EVALUATING
You want to see it run
Pick a target and watch a robot extract rows in the live web scraper demo. No account, no card. Then check the how it works walkthrough for what happens on a real crawl.
Web Scraping FAQ: Cost, Legality, Logins, and Why Scrapers Break
Web scraping is the automated extraction of data from websites into structured rows. A managed scraping tool typically costs $50 to $500 per month, collecting public data is broadly lawful in the US, a modern scraper can log in and fill forms, and the usual reason a scraper breaks is that it stored a CSS selector instead of your intent.
Below: 31 answers grouped by what buyers actually ask, from the first robot to the legal review.
Last updated July 2026
Robot console · WR-01
Standing by Running · s Complete · rows
1 · Pick a target
3 · Fields to extract
Agent log
Crawl graph
Extracted data · rows
Want this data fresh every morning, without lifting a finger?
01 / QUICK ANSWERS
FIG. 1 · ANSWER SHEET
The five questions, answered in one line each
If you only have a minute, this table is the page. Everything after it is the detail.
| Question | Short answer | In WebRobot |
|---|---|---|
| How much does it cost? | Managed tools run about $50 to $500 per month depending on volume | $79, $249 or $699 flat per month |
| Is it legal? | Public data is broadly lawful in the US. Personal data and logins raise the stakes | robots.txt and rate limits honored |
| Can it log in? | Modern agent scrapers can log in, fill forms and click through multi-step flows | Agent actions on Scale and up |
| Will it break? | Selector-based scrapers break on redesigns, usually silently | Self-heals from stored intent |
| Do I need to code? | Not for a no-code agent tool. Engineers can still use an API | Plain-English sentence, or REST |
New to the category? The web scraping tool overview explains how the pieces fit together, and the WebRobot AI web scraper home page shows a full run end to end.
02 / COST
FIG. 2 · PRICE BOARD
What the market actually charges
The most common question on this page deserves real numbers, including our competitors'.
Prices below are as of July 2026, taken from each vendor's public pricing page, and vendors change them.
| Tool and plan | Price | Pricing model | Where it wins |
|---|---|---|---|
| WebRobot Launch | $79 / mo | Flat: 5 robots, 10,000 records | Predictable bill, no credit math |
| WebRobot Scale | $249 / mo | Flat: 25 robots, 100,000 records, agent actions | Logins, hourly runs, change alerts |
| WebRobot Autopilot | $699 / mo | Flat: 100 robots, 1,000,000 records | Real-time monitors, 15 seats |
| Octoparse Standard | $69 / mo billed annually | 100 tasks, 3 concurrent cloud runs | Real free tier and 500+ prebuilt templates |
| Browse AI Professional | $87 / mo | Credits, about 10 rows per credit, no rollover | Cheapest way to start, quick setup |
| Apify Starter | $29 / mo | Usage-based compute units, many Actors bill on top | Huge actor store if you have engineers |
| Bright Data / Zyte | Typically $500+ / mo | Sales-led, proxy-centric enterprise contracts | Proxy scale and enterprise procurement |
The honest read: if you want a free tier or hundreds of prebuilt templates, Octoparse has both. If you have engineers and want to pay by the compute unit, Apify is strong. WebRobot is for the buyer who wants a flat bill and a robot that keeps working after the site changes. All plan limits are on the WebRobot pricing page.
03 / GETTING STARTED
FIG. 3 · FIRST ROBOT
Getting started with web scraping
The questions people ask in the first ten minutes, before they have run anything.
Web scraping is the automated extraction of data from websites into structured rows. A program opens pages, reads the fields you care about, and writes them to a spreadsheet, a database, or an API. It replaces copy and paste, and when it is set up properly it runs on a schedule so the data stays current.
No. With WebRobot you describe the data in plain English, for example "product title, price, and stock status from every category page". The robot browses the site, locates those fields, and returns rows. There is no CSS selector, no XPath, and no Python script to maintain. Engineers can still drive everything through the web scraping API.
Minutes for an ordinary site. Paste the URL, write one sentence describing the fields, run the robot, and check the preview rows before you schedule it. Harder jobs (a login, a multi-step search, thousands of detail pages) take longer to verify, but the build step is still a sentence rather than a script.
An AI scraping agent browses a site the way a person does instead of pattern-matching raw HTML. It clicks, logs in, fills forms, paginates, and presses load more until the dataset is complete. It stores your intent, the fields you asked for, rather than a fragile CSS path, which is why it survives a redesign. The how it works page walks through a full run.
04 / WHAT IT CAN SCRAPE
FIG. 4 · REACH
What a scraper can and cannot reach
Most valuable data is not sitting in plain HTML on a single page. It is behind a click, a form, or a login.
Yes. WebRobot performs agent actions: it types credentials into the target site login form, holds the session, and extracts the pages behind it. Agent actions are included on Scale at $249 per month and on Autopilot. Use an account you are entitled to use, and check the target site terms first. Credential handling is described on the web scraping security page.
Yes. The robot drives a real browser, so pages built with React, Vue, or infinite scroll are read after they render, not before. If a person can see the value on screen, the robot can extract it. Simple HTTP parsers usually return an empty shell on these sites.
Yes. Numbered pages, next buttons, infinite scroll, and load more controls are handled by clicking through them the way a person would, until there is nothing left to load. You get one merged dataset at the end instead of twenty partial exports you have to stitch together.
Yes, on plans with agent actions. The robot types a query, sets filters, submits the form, and extracts the result rows, then repeats for every term on your list. Multi-step flows, search then open each result then pull the detail fields, are the normal shape of a real scraping job.
Yes. Build one robot per site, describe the same fields in each, and point them at the same destination. The rows land in a single sheet with a consistent schema, which is what makes competitor price tables and multi-source lead lists usable rather than a folder of mismatched CSVs.
05 / RELIABILITY
FIG. 5 · FAILURE MODES
Why scrapers break, and what to do about it
Every scraper works on day one. The interesting question is what happens on day forty, when the target site ships a redesign.
Traditional ones do. Selector-based scrapers record where the data was, a CSS class or an XPath, so one rename or one redesign returns empty cells, usually without an error. WebRobot records what the data is and re-locates the fields after a change, which is what self-healing means here. There is a longer treatment on the web scraping tool page.
Nearly always one of four causes: the layout changed and the selectors no longer match; the content is rendered by JavaScript after the initial HTML loads; the rows sit behind pagination or a login the scraper never got past; or the site is rate limiting you and serving a block page. The first two are what kill selector-based tools.
Check the run log and the row count. Silent failure is the real risk in scraping: a broken scraper does not crash, it quietly delivers zero or half the rows. Treat any run that returns far fewer rows than usual as an incident. Scale and above can alert you when what the robot sees changes.
WebRobot paces its requests, honors robots.txt, and behaves like an ordinary browser rather than hammering a server. If a target genuinely forbids automated access, we do not build tooling to defeat its protections. That is a deliberate boundary, explained in full on the security and compliance page.
06 / PRICING AND BILLING
FIG. 6 · MONEY QUESTIONS
What web scraping costs
Flat monthly plans, no credit meters, no overage invoices. The numbers below are the whole story.
For a managed tool, budget roughly $50 to $500 per month. WebRobot is flat: Launch is $79 per month for 5 robots and 10,000 records, Scale is $249 for 25 robots, 100,000 records, hourly schedules and agent actions, Autopilot is $699 for 100 robots, 1,000,000 records and real-time monitors. Yearly billing takes 20% off. Full breakdown on WebRobot pricing.
Yes. The interactive web scraper demo runs a real robot in your browser with no account and no card: pick a target, choose the fields, watch it extract the rows. It is the closest thing to a test drive without a subscription. There is no free tier on the paid plans.
One record is one row delivered: one product, one listing, one job posting, one article. Re-checking a page that has not changed does not consume records, and neither do failed or self-repairing runs. Only rows that reach you count against the monthly allowance.
No credits and no overage billing. Robots pause when the monthly allowance is reached and you get an alert with two options: upgrade the plan or wait for the next cycle. Credit meters have a nasty property, they charge you most for the runs that worked hardest.
Yes, from the workspace, in one click, with no call and no retention form. Robots keep running to the end of the paid period and you keep every row you already exported. Monthly plans are not refunded pro rata, they simply stop renewing.
07 / DATA AND DELIVERY
FIG. 7 · OUTPUT
Where the data lands
A CSV in a downloads folder is not a workflow. Data has to arrive where the work already happens.
Structured rows: CSV, Excel, or Google Sheets for spreadsheet users, JSON over the REST API for engineers. The columns are the fields you asked for, typed and consistent across runs, not raw HTML you have to clean up before anyone can use it.
Yes. Delivery targets are CSV, Excel, Google Sheets, Slack, Zapier, webhooks, and the REST API. Sheets, CSV and API delivery are on every plan. Slack, Zapier and webhooks start on Scale. The full list is on the web scraping software features page.
Yes. Launch runs daily schedules. Scale adds hourly runs and change alerts, so you hear about it when a price, a listing, or a page you watch actually moves. Autopilot adds real-time monitors for the cases where an hour is too long to wait.
Yes. Start a run with a POST request, receive results as JSON, get a webhook when the run completes. You do not maintain a proxy pool, a browser farm, or retry logic. See the web scraping API for the request shape and delivery options.
You do. Rows extracted for your account are your data. WebRobot does not resell them, share them with other customers, or fold them into a dataset we sell. You can export them and you can delete them from the workspace whenever you want.
08 / LEGAL AND COMPLIANCE
FIG. 8 · LEGAL DESK
The questions your legal team will ask
General information, not legal advice. Talk to your own counsel about your specific sources and use case.
Collecting publicly available data is broadly lawful in the United States, and US courts have generally treated reading a public web page as different from unauthorized access to a computer. The risk lives in what you collect and what you do with it: personal data, copyrighted content, and data behind a login all raise the stakes. We wrote a longer piece on whether web scraping is legal.
Breaching terms of service is usually a contract question rather than a criminal one, but it can still get your account terminated or pull you into a civil dispute. Our position is simple: if a site forbids automated access, do not point a robot at it. You choose the targets, so you carry that call.
Yes. Robots honor robots.txt directives and apply polite rate limits by default, so a crawl looks like an ordinary visitor rather than a denial of service. We do not ship features whose purpose is to defeat a site access control, and we do not treat that boundary as negotiable.
You can, but you become the controller of that data and inherit the obligations: a lawful basis, purpose limitation, retention limits, and honoring deletion requests. Most commercial scraping (prices, catalogs, listings, public company information) touches no personal data at all, which is the cheaper and calmer path. Our posture is set out on the security and compliance page.
09 / SECURITY
FIG. 9 · DATA HANDLING
Security, credentials, and your data
The short version. The long version, including a shared-responsibility table, is on the security page.
Traffic between you and WebRobot runs over TLS, and stored data, including any credentials you hand a robot, is encrypted at rest. Credentials are used for one thing only: signing your robots in to the target site you configured them for.
Encrypted, inside your workspace, and used only for the target-site logins you set up. They are never shared with other customers and never used for anything else. Good practice on your side: give the robot a dedicated account with the least privilege the job needs. See the shared responsibility table for the full split.
Access is limited to the staff who need it to run the service or to answer a support request you opened. We do not sell, resell, or share the rows your robots extract, and we do not use your extracted data to build datasets for anyone else.
You control retention. Extracted datasets and stored credentials can be deleted from the workspace at any time, including before you cancel, and anything you already exported stays yours. If you want everything removed after you leave, ask and we will remove it.
10 / NEXT STEPS
FIG. 10 · WHERE TO GO NEXT
Question not answered here?
Three routes, depending on what you are trying to decide.
BUDGETING
You need a number for finance
Every limit, seat count and integration is listed on pricing, and every capability on the features page. Flat monthly, cancel anytime.
REVIEWING
Legal or security is asking
Send them the web scraping security and compliance page. It has the shared-responsibility table, the data-handling posture, and what we do not claim.
FINAL ASSEMBLY
Stop researching, start extracting
Describe the data once. The robot runs it today, after the redesign, and every scheduled run in between.