WEBROBOT

Web Scraping API That Returns Clean JSON, Not HTML

A web scraping API lets your code request data from any website and get structured JSON back. With WebRobot you define a robot once in plain English, then trigger it with a single POST request. No proxies, no headless browsers, no selectors to maintain.

REQUEST

curl -X POST https://api.webrobot.ai/v1/robots/rbt_8k2fq/run \
  -H "Authorization: Bearer $WEBROBOT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": { "url": "https://shop.example.com/laptops" },
    "webhook": "https://yourapp.com/hooks/webrobot"
  }'

RESPONSE · 200

{
  "run_id": "run_01hq3xv9",
  "robot_id": "rbt_8k2fq",
  "status": "succeeded",
  "rows": 214,
  "duration_ms": 48210,
  "data": [
    { "title": "ProBook 14 G11", "price": 899.00,
      "currency": "USD", "rating": 4.6, "in_stock": true },
    { "title": "ZenLine 15 OLED", "price": 1249.00,
      "currency": "USD", "rating": 4.8, "in_stock": false }
  ]
}
Run the robot
01 / ENDPOINTS FIG. 1 · API SURFACE

Four endpoints cover the whole lifecycle

You create the robot in the workspace with a plain-English description. From then on, everything is API. Runs are asynchronous by default; short runs can be awaited with ?wait=true.
EndpointWhat it does
POST /v1/robots/{id}/runTriggers a run. Accepts input overrides (start URL, search term) and an optional webhook URL. Returns a run_id immediately.
GET /v1/runs/{id}Run status and metadata: queued, running, succeeded, failed, row count, duration.
GET /v1/runs/{id}/dataThe extracted rows as JSON, paginated with cursor parameters. CSV via the Accept header.
GET /v1/robots/{id}/dataLatest successful dataset for a robot, ideal for scheduled robots your app polls.
02 / INTEGRATION FIG. 2 · DELIVERY CONTRACT

Webhooks, rate limits and SDKs

WEBHOOKS

Pass a webhook URL on any run and WebRobot POSTs the result envelope when the run finishes. Payloads are signed with an HMAC-SHA256 header so you can verify origin. Retries with exponential backoff for 24 hours if your endpoint is down.

RATE LIMITS

API requests are limited to 60 per minute on Launch, 300 per minute on Scale and 1,200 on Autopilot, with standard X-RateLimit headers. Record quotas match your plan: 10k, 100k or 1M rows per month. Limits are on your account, never surprise per-request pricing.

SDKS

Official clients for Node and Python (npm i webrobot, pip install webrobot) wrap runs, polling and webhook verification. The API is plain REST + JSON, so any language works without them.

The part you never build: the robot behind the endpoint is a full ai web agent. It handles pagination, logins and layout changes on its own, with a 98.6% self-heal rate. Your integration code stays a single POST request while the extraction logic maintains itself. Compare that with running your own headless browser fleet, or read how the same engine powers our browser automation tool and data extraction tool.

03 / FAQ FIG. 3 · DEVELOPER QUESTIONS

Web scraping API questions

No. Every run executes on WebRobot's managed browser fleet with rotating exits, real fingerprints and human-like pacing, always within a site's robots.txt and rate limits. You send a POST request and receive JSON; the infrastructure is our problem.

Nothing. Your contract is the robot's output schema, not the site's HTML. When a page changes, the agent re-maps fields on the fly (average repair 41 seconds) and keeps returning the same JSON keys. Schema changes you make yourself are versioned on the robot.

Yes. Launch at $79 per month includes full API access with 10k records. Scale at $249 raises that to 100k records with 300 requests per minute, webhooks on every channel and agent actions like logins and form fills.

Evaluating options for the whole team, not just the codebase? Start with the AI web scraper overview or the general web scraping tool checklist.

FINAL ASSEMBLY

One POST request. Clean JSON back.

Create your first robot in plain English, grab an API key, and delete the scraper code you were dreading to maintain.