How to Monitor Competitor Prices Automatically

2026-07-01 · 7 min read

To monitor competitor prices, pick the specific competitor product URLs that map to your own SKUs, run an automated check on a schedule that matches how fast the category moves, and capture more than the number: promo price, stock status and shipping cost. Route material changes to Slack or email, and append every run to a dashboard so you are looking at a trend line, not a screenshot. The whole setup takes about ten minutes once you know which pages you care about.

Run the robot

Last updated July 2026

Robot console · WR-01

Standing by Running · s Complete · rows

1 · Pick a target

2 · Tell the robot what you want

3 · Fields to extract

Agent log

▌

Crawl graph

Extracted data · rows

Want this data fresh every morning, without lifting a finger?

01 / SCOPE FIG. 1 · TARGET SELECTION

Pick the pages and SKUs worth watching

The instinct is to track everything. The teams who get value from price monitoring track a deliberately small set, because a list you act on beats a list you scroll past.

STEP 1A

Start from your revenue, not their catalog

Take the SKUs that drive most of your margin, plus anything price-sensitive enough that a competitor undercut costs you the sale. That is usually a few dozen to a few hundred products, not the whole catalog.

STEP 1B

Map SKU to URL, once

Every one of your SKUs needs a matching competitor product URL. Do this matching by hand the first time. Fuzzy title matching produces plausible garbage, and a mismatch means you reprice against the wrong product.

STEP 1C

Watch category pages too

Product pages tell you what a competitor charges. Category and "sale" pages tell you what they are pushing. Scraping a listing page catches new products and sitewide promos that a fixed SKU list will never see.

02 / CADENCE FIG. 2 · WHAT TO TRACK, HOW OFTEN

What to capture and how often to check it

Frequency should match how fast the category actually moves, and how fast you can respond. Checking a weekly-moving B2B catalog every hour just burns your rate limit and trains you to ignore alerts.

Field	Why it matters	Suggested frequency	Alert when
List price	The baseline. Useless on its own, essential as a reference	Hourly (electronics, marketplaces), daily (most retail), weekly (B2B, industrial)	Any change beyond your noise threshold, say 2%
Promo or sale price	The price a customer actually pays. This is the one that moves conversion	Same as list price	A promo starts, deepens, or ends
Discount depth	Reveals whether discounting is occasional or a standing strategy	Daily	Depth exceeds your own floor
Stock status	An out-of-stock competitor undercutting you is not undercutting you	Hourly on contested SKUs	A rival goes out of stock, or comes back
Shipping cost and threshold	Landed price is what buyers compare. $12 shipping erases a 5% discount	Weekly, plus before any promo period	Free-shipping threshold changes
Bundles and multipacks	How competitors lift average order value without cutting unit price	Weekly	A new bundle appears on a tracked SKU
New SKUs on category pages	Catches launches before your SKU map is out of date	Daily	An unseen product URL appears

On WebRobot, hourly scheduling and change alerts start on the Scale plan at $249 per month; daily scheduling is included from Launch at $79. The full breakdown is on the pricing page, and the mechanics of the checks are on the price monitoring page.

03 / DELIVERY FIG. 3 · WHERE THE DATA LANDS

Alerting and the pricing dashboard

Two different jobs, and people constantly conflate them. Alerts are for the handful of changes somebody must react to today. The dashboard is for the pattern you only see over weeks.

Alerts: high signal, low volume

Send a Slack message only when a change crosses a threshold you have decided in advance: a tracked SKU drops below your price, a rival launches a promo on a contested product, a competitor stocks out. Everything else goes to the dashboard silently. An alert channel that fires forty times a day is a channel nobody reads. WebRobot pushes change alerts to Slack, Zapier and webhooks from the Scale plan up.

Dashboard: append, never overwrite

Every run should append a timestamped row, not replace yesterday's. Overwriting destroys the only thing that makes this data valuable: history. With a table of date, SKU, competitor, list price, promo price and stock, you can answer "do they always discount on the first Friday" without another scrape. Deliver to Google Sheets or straight into Excel, or pull it from the REST API into your warehouse.

Price is only half the picture

A price drop is the visible end of a decision that was made weeks earlier. If you also see the ads they are actually running, a promo stops being a surprise and becomes something you saw coming. The same logic applies to their landing pages, shipping banners and product launches, which is a website monitoring job rather than a pricing one.

04 / PITFALLS FIG. 4 · WHERE THIS GOES WRONG

Six ways price monitoring quietly lies to you

Every one of these produces data that looks perfectly fine in a spreadsheet and is wrong.

PITFALL 1

Geo and currency variants

The same URL serves different prices and currencies depending on where the request comes from. Pin the region you care about and record the currency in the row, or you will compare a EUR price to a USD one and never notice.

PITFALL 2

Dynamic and personalized pricing

Some retailers vary price by session, device, or time of day. If you check once at 3am you are capturing the 3am price. Sample at the hours your customers actually shop, and treat a single reading as one observation, not the truth.

PITFALL 3

A/B tests

You may be bucketed into a test variant and see a price no ordinary customer sees. Repeated checks that disagree with each other are a signal, not a bug. Keep the raw readings so you can tell a test apart from a real move.

PITFALL 4

Logged-in and member prices

Trade, wholesale and loyalty pricing often only appears after login. If your competitor's real price lives behind an account, a scraper that reads only the public page is tracking a number nobody pays. Agent actions (logging in, filling forms) are on the Scale plan.

PITFALL 5

Rate limits and blocking

Hammering one URL every sixty seconds from one IP gets you challenged, then blocked, and your data goes stale without anyone noticing. Spread checks out, keep intervals sane, and render pages like a browser instead of firing raw requests.

PITFALL 6

Silent extraction failure

The worst one. The competitor redesigns, your selector-based scraper returns empty cells, and the dashboard shows a flat line that looks like price stability. Alert on missing values, not just on changed ones. A robot that stores intent instead of CSS paths re-finds the field instead.

On the legal side: collecting publicly listed prices is normal commercial practice, but the details matter. We cover the boundaries in is web scraping legal, and if you are still choosing a platform, the 2026 scraping tool comparison puts the options side by side.

05 / FAQ FIG. 5 · FIELD QUESTIONS

Competitor price monitoring questions

Pick the competitor product URLs that matter, map each one to your own SKU, then run an automated check on a schedule that captures price, promo, stock and shipping. Route material changes to Slack or email, and append every run to a dashboard so you can see trends rather than snapshots.

Match frequency to how fast the category moves. Electronics, marketplaces and anything with dynamic pricing deserve hourly checks. Most retail categories are fine daily. Slow-moving B2B or industrial catalogs can run weekly. Checking more often than you can act on just adds noise and blocking risk.

Collecting publicly listed prices is generally legal in the US, and price comparison is normal commercial practice. Stay on public pages, respect robots.txt and rate limits, avoid personal data, and do not exchange or coordinate pricing with competitors, which is an antitrust problem, not a scraping one.

The best tool is whichever one keeps working after the competitor redesigns their product page. Selector-based scrapers break quietly and hand you stale data. WebRobot stores your intent (price, promo, stock) rather than a CSS path, checks hourly on the Scale plan, and pushes change alerts into Slack.

List price alone is misleading. Capture the promo or strikethrough price, discount depth, stock status, shipping cost and threshold, bundle contents, and any on-page badges. A competitor who is 5% cheaper but out of stock with $12 shipping is not actually beating you.

Yes. Aggressive polling from one IP gets rate-limited or challenged quickly. The fix is to behave reasonably: sensible check intervals, spread requests over time, do not hammer the same URL every minute, and use a tool that renders the page like a real browser instead of firing raw requests.