2026-07-16 · 8 min read
Websites block scrapers on five signals: IP reputation, request rate, browser fingerprint, behavior, and session state. Scraping without getting blocked means scoring like a normal visitor on all five at once: a real rendering browser, reputable rotating IPs, human pacing, coherent sessions, and respect for robots.txt and rate limits. Proxies alone fix one signal out of five, which is why proxy-only setups still get caught. Here is each signal, what trips it, and what actually fixes it.
Last updated July 2026
Robot console · WR-01
Standing by Running · s Complete · rows
1 · Pick a target
3 · Fields to extract
Agent log
Crawl graph
Extracted data · rows
Want this data fresh every morning, without lifting a finger?
REASON 1
A careless bot can generate more load than a site's entire human audience. Server bills are real, and an operations team that sees one client IP request 40 pages a second is right to cut it off. A scraper that paces itself simply never becomes this problem.
REASON 2
Content theft, credential stuffing, ticket-buying bots, spam signups. The defenses you meet were mostly built for these, and a data collector inherits the suspicion. That is unfair and also unchangeable, so plan for it.
REASON 3
Retailers know their prices get watched and some invest in making that hard. This is the one category where defenses target you specifically, and where tooling quality decides whether your price monitoring feed stays alive.
| Signal | What trips it | What fixes it | What does not |
|---|---|---|---|
| IP reputation | Datacenter IP ranges, IPs already flagged for abuse, thousands of requests from one address | Reputable rotating residential IPs, spreading load across addresses | One "clean" static IP; it degrades with every run |
| Request rate | Machine-regular timing, page walks faster than any human reads, hammering at 3 a.m. in the site's timezone | Human pacing with randomized spacing, caching what you already fetched, scraping only what changed | A fixed delay of exactly N seconds; regularity is itself a signature |
| Browser fingerprint | No JavaScript execution, header sets that do not match any real browser, known automation traces in the runtime | A real rendering browser with a consistent, ordinary fingerprint | Faking a user-agent string on raw HTTP fetches; the absence of JS execution gives it away |
| Behavior | Perfectly sequential crawls, zero scrolling or mouse events, requesting only data endpoints and never assets | An agent that actually navigates: loads pages fully, scrolls, clicks through pagination like a reader | Replaying one recorded path forever; sites notice the identical choreography |
| Session state | Cookies that reset every request, token mismatches, a "user" whose session never persists | Coherent sessions: keep cookies, carry state across pages the way a browser does | Clearing everything between requests to "stay anonymous"; it does the opposite |
Notice the pattern in the fix column: everything on it is a description of a normal browser being used by a normal person. That is why WebRobot's approach is not a bag of evasion tricks; the AI web agent is a real browser being used by a patient robot, and it passes these checks by construction.
HABIT 1
It is the site telling you where bots are welcome. Honoring it costs you little data and buys you standing: courts and platforms both treat it as evidence of good faith. WebRobot honors it by default.
HABIT 2
Re-downloading 10,000 unchanged pages daily is load without information. Scheduled diff-based runs, like website change monitoring, fetch a fraction of the pages and produce a better dataset.
HABIT 3
Public pages are where the favorable case law lives. Circumventing logins or paywalls you have no right to moves you from web scraping into unauthorized access. If you have credentials and authorization, that is a different, legitimate workflow.
HABIT 4
A cease-and-desist letter changes your legal position. The right response is a conversation (often about licensing the data properly), not a proxy upgrade. Treat it as a business decision, because legally it is one.
The full legal analysis, including the hiQ v. LinkedIn history, is in is web scraping legal. One scoping note: if what you actually need is to know where your company gets mentioned across the web and social platforms, that is a different job than structured extraction, and a dedicated brand monitoring platform will do it without you scraping anything.
THE DIY STACK
A proxy subscription, a headless browser framework, fingerprint tuning, retry and backoff logic, session management, and monitoring to notice when any of it silently fails. Engineers who love this exist, and for millions-of-pages workloads it is the right investment. For a team that just needs the rows, it is a part-time job that arrives without a job posting.
THE AGENT APPROACH
WebRobot's robot browses in a real browser with managed IPs, human pacing, and coherent sessions, retries transient failures, honors robots.txt, and self-heals when the page changes. You describe the data; the five signals are the vendor's problem. That is most of what you are buying with any serious web scraping tool, and it is why "free" scrapers get blocked first.
One honest caveat to close on: nobody can promise zero blocks, because the site owns its front door. Some targets are defended enough that the correct answer is their official API or a licensed data feed; the tradeoffs are in web scraping vs API. A vendor who promises 100% access on every site is telling you something about their honesty, not their technology.
FINAL ASSEMBLY
A real browser, managed IPs, human pacing, and self-healing extraction. You describe the data; WebRobot handles the rest.