Webhooks
Pass a webhook URL on any run and WebRobot POSTs the result envelope when the run finishes. Payloads are signed with an HMAC-SHA256 header so you can verify origin. Retries with exponential backoff for 24 hours if your endpoint is down.
Web Scraping API That Returns Clean JSON, a Scraping API With No Proxies to Run
A web scraping API lets your code request data from any website and get structured JSON back. With WebRobot you define a robot once in plain English, then trigger it with a single POST request. No proxies, no headless browsers, no selectors to maintain, and the REST API is included on every plan from $79 a month rather than sold as a separate product.
try the agent below, then call the same robot from your code
Robot console
No account needed Running · s Complete · rows
The contract
One POST request, structured rows back
Your integration is a single call and a webhook. The robot behind it already knows the source, the fields
and the pagination, so nothing in your code has to know what the page looks like.
Request
curl -X POST https://api.webrobot.ai/v1/robots/rbt_8k2fq/run \
-H "Authorization: Bearer $WEBROBOT_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": { "url": "https://shop.example.com/laptops" },
"webhook": "https://yourapp.com/hooks/webrobot"
}'Response · 200
{
"run_id": "run_01hq3xv9",
"robot_id": "rbt_8k2fq",
"status": "succeeded",
"rows": 214,
"duration_ms": 48210,
"data": [
{ "title": "ProBook 14 G11", "price": 899.00,
"currency": "USD", "rating": 4.6, "in_stock": true },
{ "title": "ZenLine 15 OLED", "price": 1249.00,
"currency": "USD", "rating": 4.8, "in_stock": false }
]
}Endpoints
Four endpoints cover the whole lifecycle
You create the robot in the workspace with a plain-English description. From then on,
everything is API. Runs are asynchronous by default; short runs can be awaited with
?wait=true.| Endpoint | What it does |
|---|---|
| POST /v1/robots/{id}/run | Triggers a run. Accepts input overrides (start URL, search term) and an optional webhook URL. Returns a run_id immediately. |
| GET /v1/runs/{id} | Run status and metadata: queued, running, succeeded, failed, row count, duration. |
| GET /v1/runs/{id}/data | The extracted rows as JSON, paginated with cursor parameters. CSV via the Accept header. |
| GET /v1/robots/{id}/data | Latest successful dataset for a robot, ideal for scheduled robots your app polls. |
Integration
Webhooks, rate limits and SDKs
Rate limits
API requests are limited to 60 per minute on Launch, 300 per minute on Scale and 1,200 on Autopilot, with standard X-RateLimit headers. Record quotas match your plan: 10k, 100k or 1M rows per month. Limits are on your account, never surprise per-request pricing.
SDKs
Official clients for Node and Python (webrobot.mjs,
webrobot.py, one file each, no dependencies) wrap runs, polling and webhook
verification. The API is plain REST + JSON, so any language works without them.
The part you never build: the robot behind the endpoint is a full ai web agent. It handles pagination, logins and layout changes on its own by re-reading the page. Your integration code stays a single POST request while the extraction logic maintains itself. Compare that with running your own headless browser fleet, or read how the same engine powers our browser automation tool and data extraction tool.
Pricing
Web scraping API pricing and what each plan includes
The API is not an add-on and there is no per-request tariff to model. Every active plan carries full REST
access, signed webhooks and the SDKs. What changes between plans is how many records you can pull, how
fast you can call, and whether the robot may sign in to a site on your behalf.
| Plan | Monthly | Billed yearly | Records a month | API rate limit | Robots | What it unlocks |
|---|---|---|---|---|---|---|
| Launch | $79 | $39 | 10,000 | 60 req/min | 5 | Daily scheduling, webhooks, 1 seat |
| Scale | $249 | $124 | 100,000 | 300 req/min | 25 | Hourly schedules, change alerts, agent actions for logins and forms, 5 seats |
| Autopilot | $699 | $349 | 1,000,000 | 1,200 req/min | 100 | Real-time monitors, 15 seats |
| Enterprise | $1,398 | $699 | 2,000,000 | 2,400 req/min | 200 | Real-time monitors, 40 seats, SSO/SAML, audit log, DPA |
A record is one row the robot delivers, counted once whether it read a list page or followed a link to a detail page to fill the fields in. That is the difference that matters when you compare a flat plan with a credit meter: on most credit-based APIs a page you want as structured fields costs several credits, and a page that answers with a 403 usually costs one too.
Comparison
Web scraping API options and what each one bills you for
Every vendor in this market sells a different unit, which is why quoted prices never line up. The useful
question is what a fixed monthly budget delivers, and how much of the surrounding work stays with you.
| API | Billing unit | Entry paid price | Structured fields | Scheduling and delivery |
|---|---|---|---|---|
| Firecrawl | Credits, 1 per page | Hobby $19/mo, Standard $99/mo | JSON format adds 4 credits a page, so 5 in total | Your cron, your retries, your sheet writer |
| Zyte | Successful requests, priced by site difficulty | Pay as you go from $0.13 per 1,000 HTTP requests | You write the spider and the parsing | Yours, or Scrapy Cloud |
| ScrapingBee | Credits, 1 to 75 per request by proxy and rendering mode | Hobby $19/mo, Freelance $49/mo | HTML by default; extraction rules or an AI query at 5 credits more | Your cron, your storage, your sheet writer |
| ScraperAPI | Credits, 1 to 75 per request by domain, rendering and proxy tier | Hobby $49/mo for 100,000 credits, Startup $149/mo for 1,000,000 | HTML by default; JSON auto parsing on some domains | Your cron, your storage, your sheet writer |
| Oxylabs | Dollars per 1,000 successful results, by target and JavaScript rendering | Micro $49/mo ($0.50 to $1.35 per 1,000), Starter $99/mo | Parsed JSON for major targets, HTML elsewhere | Scheduler included; storage and sheet writer are yours |
| Decodo | Dollars per 1,000 requests, by proxy pool and JavaScript rendering | $19/mo ($0.50 to $1.50 per 1,000), $49/mo, $99/mo | HTML, JSON, CSV or markdown; templates for common targets | Your cron, your storage, your sheet writer |
| ZenRows | Credits per successful request: 1 plain, 5 rendered, 10 premium proxy, 25 both | Build $19/mo (45,000 credits), Launch $69/mo, Growth $199/mo | HTML, or structured fields through Extract at no extra credit | Your cron, your storage, your sheet writer |
| Browserless | Units: 30 seconds of browser, 6 per MB residential proxy, 10 per captcha | Prototyping $35/mo (20,000 units), Starter $200/mo, Scale $500/mo | None, your Puppeteer or Playwright script parses the page | Your cron, your storage, your sheet writer |
| Diffbot | Credits, 1 per extracted page | Startup $299/mo; Crawl from Plus at $899/mo | Automatic, in Diffbot's fixed schema per page type | Yours |
| Apify | Compute units, proxy traffic, per-result Actor fees | Starter $19/mo plus usage | Depends on the Actor you write or rent | Built in, if the Actor supports it |
| Bright Data | Records and proxy gigabytes | $1.50 per 1,000 records pay as you go | Included on its dataset products | Delivery to cloud storage on higher tiers |
| WebRobot | Records delivered, counted once | Launch $79/mo, 10,000 records | Included: fields are how you define the robot | Built in: schedules, Sheets, Slack, Zapier, signed webhooks |
If you are pricing one of those against us, we have written the maths out in full for Firecrawl pricing, Apify pricing, ScrapingBee pricing, ScraperAPI pricing, Oxylabs pricing, Decodo pricing, ZenRows pricing, Browserless pricing and Browse AI pricing, and the head-to-head comparisons live at Firecrawl alternative, Diffbot alternative and Zyte alternative.
FAQ
Web scraping API questions
No. Every run executes on WebRobot's managed browser fleet with rotating exits, real fingerprints
and human-like pacing, always within a site's robots.txt and rate limits. You send a POST request
and receive JSON; the infrastructure is our problem.
Nothing. Your contract is the robot's output schema, not the site's HTML. When a page changes,
the agent re-maps fields on the fly and keeps returning the same JSON
keys. Schema changes you make yourself are versioned on the robot.
It depends entirely on what the vendor meters. Credit-based APIs start around $19 a month and bill per
page, with structured extraction usually costing several credits per page. Request-based APIs quote per
1,000 successful requests, from roughly $0.13 for a simple page to over $16 for a hard one rendered in a
browser. WebRobot is flat: $79 a month for 10,000 records, $249 for 100,000, with the API included.
Most cannot, at least not unattended. A fetch endpoint returns whatever a fresh anonymous browser sees,
and anything behind a sign-in needs session handling you script and maintain. On WebRobot, agent actions
on Scale and Autopilot let the robot sign in with credentials you provide, run a search and follow
pagination on a schedule. Only scrape data you are entitled to access.
Describe the fields instead of the markup. You create the robot once by typing what you want, for
example every product with name, price, SKU and stock status from a category page and its detail pages,
then trigger it with a POST request. The output schema is your contract, so no CSS selector or XPath
ever appears in your codebase.
Yes. Launch at $79 per month includes full API access with 10k records. Scale at $249 raises that
to 100k records with 300 requests per minute, webhooks on every channel and agent actions like
logins and form fills.
Evaluating options for the whole team, not just the codebase? Start with the AI web scraper overview or the general web scraping tool checklist.
One POST request. Clean JSON back.
Create your first robot in plain English, grab an API key, and delete the scraper code you were dreading to maintain.