T+0S · DETECTION
The run starts and the layout check fails: the product grid the agent expected is gone, replaced by a new card design. A selector-based scraper would return zero rows here, or worse, wrong ones, and say nothing.
Field manual · Unit WR-01 · Operating procedure
Automated Web Scraping: How the AI Agent Actually Works
Automated web scraping means software collects website data for you on a schedule, with no manual copy-paste. WebRobot does it with an AI agent: you describe the data in plain English, the robot browses the site like a human, extracts clean rows and repairs itself when the site changes.
01 / FULL TEARDOWN
FIG. 1 · FIVE-PHASE RUN SEQUENCE
What happens inside one robot run
Every run of the agent moves through the same five phases. Most first robots finish
phase five in under ten minutes from a cold start.
1
PHASE 1 · PARSE
Parse the task
You write one sentence, like "Get every product with title, price, rating and stock. Follow pagination." The agent turns that sentence into two things: a field schema (title as text, price as currency, rating as number, stock as status) and a browsing plan (start at this URL, expect a list, expect a next-page control). Nothing here is a CSS selector or an XPath. The plan describes intent, which is why it survives redesigns.
input: 1 sentence + 1 url · output: field schema + browse plan
2
PHASE 2 · BROWSE
Open the site like a human
The robot launches a real browser session, not a raw HTTP fetch. It waits for the page to render, scrolls to trigger lazy loading, clicks "load more" buttons, follows pagination, signs in where you allow it and fills search forms in multi-step flows. Pacing is human-like, robots.txt is honored and per-site rate limits are enforced before any request leaves the fleet. If a person can reach the data in a browser, the agent can too.
engine: full browser render · pacing: human-like · robots.txt: honored
3
PHASE 3 · EXTRACT
Extract and validate fields
On each page the agent reads the rendered layout the way a person does: it recognizes that "$24.99" next to a product title is the price even if the markup changed last week. Every row passes a schema check (types, required fields, formats), duplicates are dropped across pages and runs, and malformed values are flagged instead of silently written. The result is a table you can trust, not a dump you have to clean.
checks: type · format · required · dedupe scope: cross-run
4
PHASE 4 · SELF-HEAL
Self-heal when the site changes
Before extracting, the agent compares the live page against its last known layout. Layout drift triggers a re-read: the agent re-maps your fields against the new design and continues the run. Across the fleet the self-heal rate is 98.6% and the average repair takes 41 seconds. You get a note in the run log, not a broken spreadsheet. The rare cases the agent cannot resolve on its own are paused and surfaced instead of shipping wrong data.
self-heal rate: 98.6% · avg repair: 41s · failure mode: pause + alert
5
PHASE 5 · DELIVER
Deliver clean data where you work
Validated rows land where your team already works: Google Sheets that refresh themselves, CSV and Excel downloads, Slack change alerts, webhooks into your own systems and a REST API that returns clean JSON. Schedules run once, daily, hourly or as real-time monitors that fire the moment something changes on the page.
outputs: sheets · csv · xlsx · slack · webhook · json api
Prefer to see it instead of read it? Try the web scraper online demo, no account needed, or browse the full web scraping tool overview.
02 / FAILURE ANALYSIS
FIG. 2 · REDESIGN RESPONSE TIMELINE
What happens when a site changes
This is where selector-based tools die and where the agent earns its keep.
Here is the exact sequence on the first run after a redesign.
T+5S · RE-READ
The agent re-reads the rendered page from scratch, the same way it did on first setup. It looks for your fields by meaning: product names, prices, ratings, stock badges, wherever they moved to.
T+41S · RE-MAP
Fields are re-mapped to the new layout and validated against your schema plus historical rows: same columns, plausible values, expected row counts. Average repair across the fleet is 41 seconds.
T+42S · RESUME + NOTE
The run resumes and finishes on schedule. Your run log gets one line: "Layout changed, re-mapped 4 fields, 0 rows lost." If confidence is low, the robot pauses and alerts you instead of guessing. Wrong data never ships silently.
03 / AGENT ACTIONS
FIG. 3 · ACTION REGISTER
Every action the agent can take on a page
The full register of browsing actions, and the plan tier that includes each.
| Action | What it does | Available on |
|---|---|---|
| read | Renders the page in a real browser and reads the visible layout | All plans |
| paginate | Follows next-page links, numbered pages and "load more" buttons | All plans |
| scroll | Triggers infinite scroll and lazy-loaded content | All plans |
| click | Opens detail pages, tabs and expandable sections | All plans |
| login | Signs in to sites you own accounts on, credentials encrypted at rest | Scale and up |
| fill-form | Completes search forms and filters before extracting results | Scale and up |
| multi-step | Chains actions across pages: search, filter, open, extract | Scale and up |
| download | Fetches linked files such as PDFs and images alongside rows | Scale and up |
| monitor | Watches a page in real time and fires on change | Autopilot and up |
| autonomous-flow | Lets the agent plan its own multi-page route to the requested data | Autopilot and up |
The action register is one part of the picture. The full list of scheduling, monitoring and delivery capabilities lives on the web scraping software features page, and the AI web scraper homepage shows the console in action.
FINAL ASSEMBLY
Watch your first robot run in ten minutes
Describe the data in one sentence. The agent handles the browsing, the extraction and the next redesign.