2026-06-18 · 9 min read · WebRobot field notes
SHORT ANSWER
To scrape data from a website, choose one of three routes: copy and paste it by hand (fine for one page), point and click the fields with a browser tool or extension (fine for a few dozen stable pages), or hand the job to code or an AI agent that browses the site for you (the only option that survives pagination, logins and redesigns). The fastest no-code path is to describe the fields you want in plain English, let the robot crawl, and export the rows to CSV or Google Sheets.
Last updated July 2026
Robot console · WR-01
Standing by Running · s Complete · rows
1 · Pick a target
3 · Fields to extract
Agent log
Crawl graph
Extracted data · rows
Want this data fresh every morning, without lifting a finger?
METHOD 1
Highlight the table, copy, paste into a sheet, clean up the formatting. It sounds beneath you, and for a single page it usually is not. Zero setup, zero tooling, zero maintenance, and you can see exactly what you got.
It falls apart the moment the answer is "and now do that for 400 more pages" or "and again next Monday". Human transcription also introduces errors nobody catches until a number looks wrong in a board deck.
METHOD 2
Browser extensions and desktop scrapers let you click the elements you want. Under the hood they record a CSS selector or XPath for each field, then replay that path across every page in a list. No code, quick to set up, and genuinely useful.
The catch is that selectors are a snapshot of today's HTML. Rename a class, wrap a price in a new span, and the extraction silently returns blanks. That is the recurring tax of this method: you become the maintainer.
METHOD 3
Python with requests and BeautifulSoup gives you total control: fetch the HTML, parse it, write rows. Add Playwright or Selenium when the page renders with JavaScript. It scales, and you own it, but you also own proxies, retries, headers and every broken selector.
An AI scraping agent is the same power without the code. You describe the fields in plain English, the agent drives a real browser, clicks through pagination, and re-locates fields after a redesign because it stored your intent rather than a CSS path.
A minimal Python scraper is about fifteen lines: request the URL, hand the HTML to BeautifulSoup, loop over the product cards, pull the title and price, write a CSV. Beginners get it working in an afternoon. What nobody tells you is that the fifteen lines are five percent of the work. The other ninety-five is pagination, rate limits, cookies, headers, JavaScript rendering, retries, and the Tuesday morning six weeks later when the site ships a redesign and your file is full of nulls.
| Method | Time to first data | Skill needed | Breaks when site changes | Cost | Scales to N pages |
|---|---|---|---|---|---|
| Copy and paste | 2 minutes | None | Never (you see it) | Free, plus your hours | No. Linear human time |
| Google Sheets IMPORTHTML | 5 minutes | Basic spreadsheet | Often. Table index shifts | Free | Barely. Simple tables only |
| Point-and-click tool | 20 to 60 minutes | Patience, some CSS helps | Yes. Selector rot is the norm | Free tiers to about $70/mo | Hundreds to thousands |
| Python + BeautifulSoup | Half a day to a week | Real Python, plus HTTP and HTML | Yes, and you are the on-call | Free code, paid proxies and time | Unlimited, if you engineer it |
| AI scraping agent | Under 10 minutes | Describe the data in English | Self-heals from stored intent | $79 to $699/mo (WebRobot) | Unlimited, scheduled |
Two honest notes. Point-and-click tools have real strengths: Octoparse ships hundreds of prebuilt templates and a free tier, and if your target is a common site somebody has already built the recipe. And if you have engineers with spare capacity, Python is free and completely yours. The calculation changes when the data has to keep arriving after everyone has moved on to the next project. See the full breakdown in our comparison of the best web scraping tools in 2026.
STEP 1
Not the homepage. You want the page where the records live as a repeating list: the category page, the search results page, the directory index. Apply the filters you care about first, then copy that URL. Filtered URLs are almost always the right starting point, because the site has already narrowed the dataset for you.
STEP 2
Be specific and name them like spreadsheet columns: "product name, price, currency, stock status, rating, review count, product URL". This list is your schema. Vague requests produce vague tables, and adding a column later means re-running the crawl, so spend two minutes getting it right.
STEP 3
Paste the URL and write the instruction the way you would brief an intern: "Go through every page of this category and get the product name, price, stock status and rating for each item." A no-code web scraper built on an agent turns that sentence into a crawl plan. There are no CSS selectors to pick.
STEP 4
Do not skip this. Look at row 1, row 5 and row 10 against the live page. The classic failures are visible instantly: prices with the currency glued on, a discount price captured instead of the list price, or a "sponsored" row that is not really part of the catalog. Fix the description, run again.
STEP 5
Now unleash it on all the pages. The agent clicks "next" or scrolls until nothing new appears, and deduplicates as it goes. Sanity-check the total: if the site claims 2,400 products and you got 240, you captured exactly one page and the pagination did not fire.
STEP 6
Export to CSV or scrape the website straight to Excel, or push to Google Sheets, Slack, a webhook or the web scraping API. Then set a schedule. A dataset you have to remember to refresh is a dataset that will be three weeks stale the day someone actually needs it.
Once the rows land in a sheet, the work shifts from collection to interrogation. Most teams pivot, filter and eyeball. If the dataset gets big enough that pivots stop being useful, you can point a tool at the file and ask questions of it in plain English instead of building a warehouse for a 40,000 row CSV.
| Symptom | What is actually happening | The fix |
|---|---|---|
| You only got page one | The crawler never followed pagination, or the "next" link is a JavaScript button rather than an anchor tag | Check whether the page number appears in the URL (?page=2). If yes, generate the URL list. If not, you need a tool that drives a real browser and clicks the button. |
| The table is empty | The content is rendered client-side by JavaScript. The raw HTML your fetcher downloaded contains an empty div and a script tag | Render the page. Playwright or Selenium in code, or any agent-based tool that runs a headless browser by default. Sometimes the site calls a JSON endpoint you can hit directly, which is faster and cleaner. |
| Rows stop after 60 | Infinite scroll. New records only load when the viewport reaches the bottom | Scroll in a loop until the row count stops increasing, with a stall timeout. Watch the network tab: the scroll usually triggers a paged API call you can request directly. |
| The data needs a login | The page is gated. Public HTML shows a teaser or a paywall | Use a tool that performs agent actions: fill the form, submit, hold the session cookie, then crawl. Only for accounts you legitimately control, and read the terms first. |
| 403s and CAPTCHAs | You look like a bot: no user agent, no cookies, twenty requests a second from one IP | Slow down first. Most blocks are rate-limit blocks in disguise. Set a real user agent, add delays, respect robots.txt. Do not build CAPTCHA-bypass machinery; if a site is that determined to stop you, that is a signal, not an obstacle. |
| It worked for six weeks, now it is blank | Selector rot. The site changed a class name or moved an element, and your recorded CSS path points at nothing | Manually repair the selectors, or use a scraper that stores intent ("the price next to each product title") instead of a DOM path, so it can re-locate the field after a redesign. |
On rate limits, the practical rule is boring but effective: one request every second or two, one concurrent session per host, and back off when you see a 429. You are not trying to be clever, you are trying to be a well-behaved visitor. Scraping that takes down a site is the fastest way to turn a technical question into a legal one, which we cover in our guide to whether web scraping is legal.
Copy and paste. Genuinely. Do not install anything, do not open a terminal, do not sign up for a tool. Ten minutes with a mouse beats two hours of setup for a job that will never repeat.
A free browser extension or a point-and-click tool. You will spend an hour on setup and you will not care that it breaks next month, because you will not run it next month.
This is where the economics flip. Anything you have to babysit becomes a permanent chore, so you want an agent that paginates, logs in, schedules itself and repairs itself. That is what a web scraping tool built on an AI agent is for.
You want JSON and a webhook, not a spreadsheet. Check whether an official API exists first, since it will usually be cleaner. Our web scraping vs API breakdown walks the decision, and pricing starts at $79 per month for the Launch plan (5 robots, 10,000 records).
FINAL ASSEMBLY
The robot handles the pagination, the logins and the redesigns. You handle what the numbers mean.