How to scrape competitor prices without breaking every week

Scraping a competitor price once is a twenty-minute job. Scraping four thousand of them daily for two years, and knowing each morning which ones are wrong, is the actual problem. The method that survives is boring: prefer the structured data a site already publishes about its own products, fall back to a versioned per-domain recipe, validate every number before it reaches the database, schedule at a rate nobody notices, and monitor for silence as carefully as for errors.

How to scrape competitor prices: the pipeline in order

A price collection system has seven stages, and each one has a distinct failure mode. Building them as separate stages matters, because when something goes wrong you need to know whether you fetched the wrong page, parsed the right page badly, or stored a number that was never plausible.

1. 1Discovery. Decide which URLs on a competitor domain are the product pages you care about.
2. 2Fetch. Retrieve the page politely, with the right headers and a retry policy that backs off.
3. 3Extract. Read a price out of the response, structured data first, per-domain recipe second.
4. 4Normalise. Turn a display string into a currency-tagged decimal with a known tax and shipping context.
5. 5Validate. Decide whether the number is plausible before it is allowed anywhere near your reporting.
6. 6Store. Append an immutable observation, never overwrite a previous one.
7. 7Monitor. Watch coverage, freshness and shape, because the expensive failures are silent.

Stage one: find the product pages, do not crawl the site

The wrong instinct is to crawl a competitor domain looking for products. It is slow, it generates enormous request volume, it picks up thousands of pages you will never use, and it is the single most likely thing to get you noticed for the wrong reason.

The right instinct is to arrive at product URLs by the shortest available route, in this order of preference.

• A product sitemap. Most platforms publish one at /sitemap.xml or a sitemap index linking to product children. It is authoritative, meant to be read by machines, and usually carries a lastmod date you can use to skip untouched pages.
• A shopping feed. If the competitor advertises through a shopping channel, their catalogue is already structured somewhere public. The cheapest and most stable source there is.
• Category pagination. Walk the category you compete in, not the whole store. One request per results page, with a hard page cap so a pagination bug cannot loop forever.
• On-site search by identifier. Searching a competitor site for a GTIN or manufacturer part number often lands directly on the page you want.

Whatever route you use, resolve to a canonical URL and store the product identity separately from the URL. URLs change constantly during replatforms and SEO work. If your record of a competitor product is a URL, every URL change looks like a discontinued product. If your record is a GTIN plus a domain, a moved URL is a rediscovery problem rather than a data loss.

Stage two and three: read schema.org Product data before you write a selector

A large share of ecommerce sites publish structured data about their own products, because it is what generates rich results in search. It is usually JSON-LD in a script tag, occasionally microdata attributes in the markup. When it is present it is the best extraction target available: it is a documented schema, it is machine-readable by design, it is maintained by the site because their own search visibility depends on it, and it survives visual redesigns that would destroy a CSS selector.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Northline 20V Cordless Drill Kit, 4.0Ah",
  "sku": "NL-DRL-2040",
  "gtin13": "0715432198765",
  "mpn": "NL2040DK",
  "brand": { "@type": "Brand", "name": "Northline" },
  "offers": {
    "@type": "Offer",
    "url": "https://www.voltbay.example/p/nl-drl-2040",
    "price": "179.00",
    "priceCurrency": "USD",
    "priceValidUntil": "2026-08-31",
    "availability": "https://schema.org/InStock",
    "itemCondition": "https://schema.org/NewCondition",
    "seller": { "@type": "Organization", "name": "Voltbay" }
  }
}
</script>

That single block gives you price, currency, availability, condition, seller and three identifiers you can match on. Read it first, always. But read it sceptically, because there are four well-known ways it lies.

• Stale price. The JSON-LD comes from a template that caches separately from the price component. The page shows $159.00 and the structured data still says $179.00. Cross-check against the visible price and quarantine when they disagree by more than a rounding cent.
• List price instead of sale price. Some templates emit the regular price in offers.price and render the promotional price only in the DOM. If the page carries a sale badge and your structured price equals MSRP, treat it as suspect.
• AggregateOffer instead of Offer. A marketplace-style page emits lowPrice and highPrice across sellers rather than one price. The lowest offer from any seller is a different metric than the retailer's own price, and mixing them produces an index that drifts without explanation.
• Multiple Product nodes. Variant-heavy pages emit an array, or a hasVariant list. Taking the first node is how a 2.0Ah kit price ends up recorded against a 4.0Ah SKU.

Our own extraction reads schema.org Product data first wherever a site publishes it, and treats it as a candidate rather than as truth. The cross-checks above are why a structured-data read still passes through the same validation gate as a selector-based read.

Falling back to a per-domain extraction recipe

Where structured data is absent or untrustworthy, you need selectors. The mistake here is writing one generic price finder for the whole internet. A regular expression for a currency symbol followed by digits will happily return the price of a cross-sell accessory, a finance instalment, a shipping threshold, a bundle add-on, a strikethrough list price, or the RRP of the wrong variant.

Write a recipe per domain instead, and anchor it on structure rather than styling.

• Anchor inside the buy-box container, not on the page as a whole, so an accessory carousel can never be the match.
• Prefer attributes that exist for machine reasons over classes that exist for visual reasons: itemprop="price", data-price, data-product-id, meta tags. Class names change with every redesign; data attributes survive because code reads them.
• Capture the strikethrough price as a separate field rather than discarding it. Knowing that $179.00 is down from $219.00 is worth more than knowing $179.00, and it is your best signal that a promotion is live.
• Record the selector version alongside every observation. When a value looks wrong six weeks later, the first question is which recipe produced it.
• Fail loudly on multiple matches. If a selector returns three prices, that is not a tie to break, it is a broken recipe.

Prices rendered by JavaScript, and how a scraper breaks on them

Here is the mechanical version of the most common breakage, because knowing the shape of it makes it recognisable in five minutes rather than five days.

A retailer moves pricing into a client-side component so they can personalise it, run promotions without a full deploy, or reflect regional inventory. The HTML that arrives at your fetcher now contains an empty container. The price is fetched afterwards by an XHR call to an internal endpoint and injected into the DOM by the browser. Your selector still matches the container. The container is empty. Your extractor returns null, or worse, it returns the placeholder text and your parser reads it as zero.

Note which failure that is. It is not an HTTP error. The fetch returned 200, the page was the right page, the selector matched. Every naive health check passes. This is precisely why error rate is a poor monitor and null rate is a good one.

There are three responses, in ascending order of cost.

If you do use a headless browser, never wait on a fixed timeout. Wait on a condition: the price node exists and its text matches a currency pattern. A fixed two-second wait works on your laptop and fails at 3am under load, and the failure is a null price rather than a crash, which means you will find out about it in a report.

Normalising currency, locale and what the shopper actually pays

A price string is not a number. Turning one into the other is where a surprising share of bad data is created, because the conversions look trivial and are not.

• Decimal and thousands separators invert by locale. 1.299,00 and 1,299.00 are the same amount, and a naive parser turns one into 1.299. Parse using the storefront's locale, not your server's.
• The currency symbol is not the currency. A dollar sign can be USD, CAD, AUD or several others. Take priceCurrency from structured data where present, otherwise lock it as a per-domain constant rather than re-detecting per page.
• Tax inclusion differs by market. US storefronts typically show a pre-tax price and add sales tax at checkout. Many European storefronts show a VAT-inclusive price. Comparing the two produces a gap that is entirely an artefact.
• Shipping changes the real price. A $179.00 item with $12.00 shipping is not competing with a $185.00 item that ships free. Record the shipping threshold, and decide once whether your index compares item prices or delivered prices.
• Promotions hide in the cart. Coupon codes, automatic cart discounts and multi-buy offers mean the advertised price is not always the paid price. Record the promotional wording alongside the number.
• Pack size is the quiet killer. A four-pack at $47.96 against a single unit at $12.99 is a unit-of-measure error, not a price advantage. Normalise to a per-unit price when pack quantity is known, and refuse to compare when it is not.

Validate before you store, not after

The most valuable component in a price scraping pipeline is the gate between extraction and storage. Without it, a bad number does not stay a bad number; it becomes a data point in a chart, a trigger for a repricing rule, and a decision someone makes on a Tuesday. Once it is stored, it is expensive to unwind.

validate(observation, history, catalog, page):

  # 1. shape
  if observation.price is null:              reject("no price extracted")
  if observation.price <= 0:                 reject("non-positive price")
  if observation.currency not in EXPECTED[observation.domain]:
      reject("unexpected currency for this storefront")

  # 2. plausibility against this product's own past
  last = history.last_accepted(observation.product_id, observation.domain)
  if last exists:
      move = abs(observation.price - last.price) / last.price
      if move > 0.40:
          quarantine("moved 40%+ since last read; needs a confirming fetch")

  # 3. plausibility against the catalogue
  if observation.price < 0.25 * catalog.msrp:  quarantine("implausibly low vs MSRP")
  if observation.price > 3.00 * catalog.msrp:  quarantine("implausibly high vs MSRP")

  # 4. the list-price-instead-of-sale-price trap
  if page.has_sale_badge and observation.price == catalog.msrp:
      quarantine("sale badge present but list price captured")

  # 5. batch-level shape, checked per domain per run
  if batch.distinct_prices == 1 and batch.size > 20:
      fail_batch("every product on this domain returned the same value")
  if batch.null_rate > 0.20:
      fail_batch("null rate above threshold; treat as an extraction outage")

  accept(observation)

Three things there are worth stating explicitly. Quarantine is not rejection: the value is held for a confirming fetch, because genuine 40 percent moves do happen on clearance. Batch-level checks catch what per-row checks cannot, since the signature of a broken recipe is uniformity rather than an outlier. And a failed batch must not overwrite yesterday's good data, which is the argument for the storage model below.

Schedule politely, and store history append-only

Politeness is not only an ethical position, it is an uptime strategy. A collector a site never notices does not get blocked, and never needs the maintenance burden that evasion brings. Cap concurrency per domain rather than globally. Spread a daily run across hours rather than firing it at midnight. Honour Retry-After. Back off exponentially on 429 and 5xx, because a retry storm is how a mild rate limit becomes a permanent block.

Choose the refresh interval from category behaviour, not from ambition. Any price move shorter than your interval is invisible to you, and your average detection delay is roughly half the interval. Slow-moving durables are fine on a daily read. Categories where marketplace repricers run all day need several checks a day to mean anything.

On storage, there is one rule: never update a price row in place. Append an observation. Each observation carries the product identity, the domain, the URL as read, the price, the currency, the stock state, the promotion wording, the extraction method and recipe version, and the timestamp of collection rather than the timestamp of processing.

Append-only storage buys you four things that overwriting destroys forever: you can chart a real history rather than a current state; you can tell a price change apart from an extraction change; you can recompute an index after fixing a recipe, because the raw observations are still there; and a bad run degrades coverage instead of corrupting the record. That last property is what makes it safe for a validation gate to fail a whole batch.

Detecting breakage, because the expensive failures are silent

A scraper that throws errors is a scraper you will fix. A scraper that returns confident, wrong, unchanging numbers is the one that costs money. Monitor for the second kind.

The real cost is maintenance, not the first fetch

This is the part that vendor pages and weekend prototypes both skip. Writing the first extractor for a competitor domain takes an afternoon. Keeping forty of them correct is a permanent job, and the work does not arrive on a schedule you control.

Every source is independently redesigning, replatforming, adding bot defences, restructuring URLs, introducing regional pricing and running promotions your parser has never seen. None will tell you. The signal is usually a number that is quietly wrong, which means the cost is not the repair, it is the interval between the break and the discovery.

• Budget engineering time per source per month, not a one-off build cost. A source is a subscription, not a purchase.
• Version every extraction recipe and record which version produced every observation, so a repair can be applied retrospectively.
• Keep a raw response sample for recently changed sources, so a diff answers the question of what moved.
• Automate regeneration where you can. When a layout changes, rediscovering the price node is a task a system can do repeatedly; ours regenerates the extraction recipe automatically and re-discovers moved products by identifier rather than by URL.
• Measure coverage and freshness as first-class metrics next to the price data itself, and put them in front of whoever consumes the numbers.

There is an honest build-versus-buy answer in that list. If you monitor a handful of stable sources and have engineering capacity, build it. If you monitor dozens across marketplaces and storefronts, collection was never the hard part; maintenance, matching and validation are, and they never finish. The trade-off is in how to choose price monitoring software, and the managed version is price scraping.