STEP 01
List the sources
Paste the homepages or section pages you want to watch. The robot follows the article links off each one, so you do not list every story by hand.
News Scraper: Extract Headlines, Article Text, and Publish Dates Into a Live Feed
A news scraper turns news sites into structured rows: headline, article text, author, publish date, section, and source URL. WebRobot does it with an AI agent. Give it the publications you care about, name the fields you want, and it follows every article link, reads the story, and delivers a clean, deduplicated feed to a spreadsheet, database, or your API on a schedule.
No CSS selectors to maintain, no per-site parser to babysit. The robot stores the intent, not a brittle path, so it keeps working after a publisher redesigns its site.
Last updated July 2026
Robot console · WR-01
Standing by Running · s Complete · rows
1 · Pick a target
3 · Fields to extract
Agent log
Crawl graph
Extracted data · rows
Want this data fresh every morning, without lifting a finger?
01 / FIELDS
FIG. 1 · WHAT COMES OFF AN ARTICLE
Every field a news article exposes, in one row
A story page holds more than the headline. Here is what the robot reads off each article and what teams
do with it.
| Field | What it is | Who uses it |
|---|---|---|
| Headline and subhead | The title and standfirst on the article | Keyword and mention scanning, and quick relevance triage |
| Article text | The full body of the story | Sentiment, entity extraction, and summarization by a language model |
| Author and byline | The reporter or wire service credited | Building a media list and mapping which reporters cover your beat |
| Publish and update date | When the story ran and when it was edited | Trending coverage over time and ordering a timeline of events |
| Section and tags | The desk or topic the article sits under | Filtering to the beat that matters and routing alerts |
| Source and URL | The publication and the canonical link | Attribution, deduplication, and linking back to the original |
| Outbound links | Companies, people, and sites referenced | Relationship mapping and following a story across outlets |
The same engine reads any structured page, so a news pull can flow into the same stack as your other sources. For the category overview, start at the web scraping tool pillar, or read the walkthrough in how to scrape news articles.
02 / THREE ROUTES
FIG. 2 · FEED, API, OR SCRAPE
RSS versus a news API versus scraping the sites
There are three ways to pull news into a system, and they cover different ground. Picking the right one
keeps you from paying for coverage you do not get.
| RSS feed | News API | Scraping the sites | |
|---|---|---|---|
| Coverage | Only sites that publish a feed | A fixed set of licensed sources | Any site you can open in a browser |
| Depth | Title and snippet, rarely full text | Depends on the plan | Every field you name, including full text |
| Fields | Fixed by the feed | Fixed by the vendor | You choose them in plain English |
| Schedule | The publisher's cadence | The vendor's cadence | Yours, down to hourly |
| Cost model | Free where offered | Per-call or tiered licensing | Flat monthly plan, no per-article fee |
| Best for | A few big outlets with clean feeds | Broad, licensed archives | Niche sources, custom fields, your cadence |
Most monitoring stacks combine them: RSS for the big outlets that offer it, a scraper for the trade journals, regulator pages, and regional sites that publish neither a feed nor an API. On what you may keep and republish, see is web scraping legal.
03 / PROCEDURE
FIG. 3 · SETUP
How to scrape news articles in four steps
The first feed takes about ten minutes to set up. Every refresh after that runs itself.
STEP 02
Name the fields
"Headline, article text, author, publish date, section, source URL." The robot opens each story and reads them. No selectors, no per-site parser.
STEP 03
Filter and dedupe
Keep only articles matching your keywords or date range, and dedupe on the canonical URL so a story syndicated to three sites does not land three times.
STEP 04
Deliver and schedule
Google Sheets, a database, Slack, Zapier, or the REST API. Schedule hourly and each run flags new matching articles so alerts reach you fast.
04 / USE CASES
FIG. 4 · WHO RUNS THIS
Where a news dataset pays for itself
Media monitoring and PR
Watch dozens of outlets for your brand, executives, and competitors. Every mention lands in one feed with the author and date, so the morning clipping report writes itself and nothing gets missed between the big wires and the trade press.
Competitive and market intelligence
Track funding rounds, launches, hires, and layoffs across an industry as they are reported. Pair it with price monitoring and you see both what competitors say and what they charge.
Regulatory and compliance watch
Regulators and agencies post updates on plain pages with no feed. The robot reads them on a schedule and flags a new rule or enforcement action the day it appears, so compliance teams are not the last to know.
Financial and investment signals
Analysts pull company and sector news into a model to catch a catalyst early. A clean feed of dated, sourced articles is the raw material an investment research workflow turns into a call on a name.
Research and dataset building
Assemble a labeled corpus of articles on a topic for analysis or model training, with the date, source, and section preserved. The structure means the dataset is usable the moment the pull finishes, not after a cleanup pass.
Content and newsletter curation
Gather the day's stories on a beat into one place, ranked by source and recency, so an editor picks from a shortlist instead of trawling a dozen homepages. Store links and short extracts, not full reposts.
Pricing is flat: plans on pricing start at $79 per month with no per-article fee, so widening your source list does not multiply the bill. For a broader extraction job, see the data extraction tool.
05 / FAQ
FIG. 5 · FIELD QUESTIONS
News scraper questions, answered
A news scraper is software that reads news websites and turns their articles into structured rows:
headline, full article text, author, publish date, section, and source URL. Instead of checking a
list of sites by hand, an agent visits each source, follows the article links, reads every story,
and delivers a clean, deduplicated feed to a spreadsheet or your database that refreshes on a
schedule.
Give a no-code agent the news sites or section pages you care about, name the fields you want
(headline, text, author, date, source), and let it open each article and read them. It deduplicates
on the URL and delivers rows to Google Sheets, a database, or your API. There is nothing to code and
nothing to fix when a publisher redesigns, because the robot stores what you asked for, not a
brittle CSS path.
Facts, headlines, publish dates, and short factual snippets are generally fair to collect, and US
courts have treated scraping public web pages as lawful. Full article text is copyrighted, so use it
for internal monitoring, indexing, and analysis rather than republishing it. Respect each site's
terms and robots directives, keep the request rate polite, and store links and short extracts
rather than mirroring whole articles you do not have the right to repost.
Yes. Point the robot at a whole list of publications, trade journals, or regulator pages and it
works through them in one scheduled run, tagging every article with its source. Media-monitoring and
PR teams use this to watch dozens of outlets for brand mentions, competitor news, or regulatory
changes without opening a single tab.
Yes. Tell the robot the terms you care about (your brand, a competitor, a product, a regulation)
and schedule it hourly. Each pass flags new articles that match and pushes them to Slack, email, or
a webhook, so a mention reaches your team in minutes instead of surfacing in a morning clipping
report a day later.
RSS only covers sites that publish a feed, and it usually gives you a title and a snippet, not the
full article or the fields you actually want. A paid news API covers a fixed set of sources on its
own schedule and pricing. A scraper reads any site you can open in a browser, extracts exactly the
fields you name, and runs on your cadence, which matters when the source you need has no feed and no
API.
WebRobot is flat-priced: Launch at $79 per month ($63 yearly) covers 5 robots and 10,000 records,
enough to monitor a focused set of outlets. Scale at $249 ($199 yearly) adds hourly schedules and
100,000 records for broad media monitoring across hundreds of sources. There is no per-article fee,
so widening your source list does not multiply the bill.
FINAL ASSEMBLY
Every source you watch, in one feed that updates itself
List the sites, name the fields, and get a fresh, deduplicated news feed on the schedule you set.