Price scraping: the extraction stack, and what breaks it

Price scraping is the automated collection of prices from public product pages. The stack runs in three layers: read the retailer's own schema.org Product structured data where it exists, fall back to a per-domain extraction recipe against server-rendered HTML, and render the page in a browser only when nothing cheaper works. Each layer down costs more and fails differently. Almost none of the work is the first scrape. It is noticing the day one stops being true.

How price scraping works, layer by layer

Every price scraper does the same job: turn a URL into a number, a currency and a stock status you would bet a margin decision on. There are three routes, and the difference is not accuracy. All three work when the page cooperates. The difference is cost per page, and behavior the week a retailer ships a redesign.

Layer 1: schema.org Product JSON-LD

Most retailers hand a machine-readable price to search engines voluntarily, because rich results depend on it. That block is the best source you can read, for one reason: the retailer maintains it. Nobody redesigns a page and forgets to update the price they publish to Google.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "18V Brushless Drill Kit, 2 x 4.0Ah",
  "sku": "NL-4412",
  "gtin13": "0812345678903",
  "offers": {
    "@type": "Offer",
    "url": "https://example.com/products/nl-4412",
    "priceCurrency": "USD",
    "price": "189.00",
    "availability": "https://schema.org/InStock",
    "itemCondition": "https://schema.org/NewCondition"
  }
}
</script>

The field you want is offers.price: a plain decimal string, no currency symbol, no thousands separator, nothing to unpick. Currency sits beside it in priceCurrency. Availability is a URL enum rather than a boolean, so InStock, OutOfStock, PreOrder and BackOrder arrive distinct instead of as a guess about a greyed-out button. And gtin13 is why structured data doubles as the matching key.

• Variants are the first trap. A product with sizes or capacities often publishes an AggregateOffer with lowPrice and highPrice instead of a single Offer. Reading lowPrice as the price reports the cheapest variant, systematically below what a shopper pays, and the error is invisible because the number is real.
• Two offers, one page. Some templates emit list price and sale price as separate Offer entries. Take the lowest currently valid one, not the first in the array.
• Markup can be stale. A cached block can carry yesterday's price while the visible page shows today's. Read structured data first, cross-check the rendered page, and flag disagreements past tolerance.
• Absence is information. A domain that stops publishing structured data has changed platform or theme, so the recipe underneath is about to be wrong too.

Layer 2: per-domain extraction recipes

When a site publishes nothing usable, you write a recipe for that domain. A recipe is not one CSS selector. It is a small versioned config: which URL shapes are product pages, which node holds the shelf price, which holds the crossed-out list price so the two are never confused, how the locale formats decimals, and what a valid product page looks like at all. Recipes are cheap to run and expensive to keep true. Your first scraper is a weekend. Owning fifty is a job.

Layer 3: rendered-page extraction

Some prices only exist after JavaScript runs. For those domains you launch a browser, let the page hydrate, wait for the price node to populate rather than for an arbitrary timeout, and read the DOM. It works, and it costs a browser process plus the whole asset graph per fetch. Rendered extraction is a per-domain promotion, decided deliberately, not a default applied everywhere.

How competitor price scraping breaks, and what a resilient system does

Price feeds rarely fail loudly. They fail by delivering numbers that are no longer true, on schedule, into a dashboard that looks healthy. A vendor who cannot describe these eight failures has not met them.

The failure nobody instruments: quiet success

Most monitoring watches for jobs that fail. The expensive failure is a job that succeeds forever with a value that stopped changing. If a competitor's price on a volatile SKU has been identical for forty days while three rivals moved twice, the explanation is rarely pricing discipline. It is a static element, a cached page or the wrong node. Staleness alerting has to fire on values that stopped moving, not only on requests that stopped returning. Almost nobody writes that query.

Validation: the gap between a number and a price

Extraction produces candidate values. Validation decides which are allowed to exist. This runs before storage, not in a report afterwards.

• Digit-shift detection. A value at roughly ten times or one tenth of the trailing median is a parse error far more often than a promotion. $18.99, $189.90 and $1,899.00 are one broken selector three ways.
• Range check against the product's own history. Compare to the trailing median at that source, not a global rule. A 40 percent overnight move on a mattress is suspicious; on a graphics card at launch it is Tuesday.
• Zero, negative and null rejection. A price of $0.00 is never a price. It is a challenge page, an empty container, or a shipping field read by mistake.
• Currency agreement. The value must carry a currency, and it must be the one that source is pinned to. A silent switch from GBP to USD stops the pipeline.
• Cross-layer agreement. Where structured data and the rendered page both yield a price, they should agree. Persistent disagreement means the recipe reads the wrong node.
• Freshness and confidence gates. Anything older than its refresh interval is labeled stale, and repricing rules refuse to act on it. Low-confidence matches never reach a pricing decision, because a correct price on the wrong product is still wrong. See the product matching guide.

Price scraping software versus building it yourself

This gets decided after someone builds a prototype in two days and concludes the whole thing is two weeks. The prototype is real. The estimate is wrong, because the prototype covered the part that never needs maintaining.

Worked example, with numbers you should replace. Suppose a fully loaded engineer-day costs $800, and the build lands at 15 engineer-days, optimistic beyond a handful of domains. That is $12,000 before a single price is collected. Suppose maintenance runs one day a month: $9,600 a year, plus proxies and compute. Against $299 a month for 2,500 products at twice-daily refresh, which is $3,588 a year, the build has to return roughly $18,000 in year one. Sometimes it does. More often nobody costed maintenance at all.

What a price scraping tool has to hand back

Collected prices are only useful if they leave the tool cleanly. Every observation carries its timestamp, source URL, currency, stock status, the marketplace seller where there is one, and the match confidence that tied it to your SKU. That is what makes an argument about a number resolvable months later. Access is by REST API, webhooks on price and stock change, scheduled CSV or SFTP export, and direct pushes into connected stores. The API and webhooks are on every plan, because gating data access behind a higher tier holds your own data hostage.

curl https://api.priceintelligence.io/v1/products/NL-1084/offers \
  -H "Authorization: Bearer $PI_API_KEY"

{
  "sku": "NL-1084",
  "gtin": "00910042852",
  "your_price": 199.00,
  "currency": "USD",
  "price_index": 101.4,
  "position": "above",
  "checked_at": "2026-07-24T06:12:04Z",
  "offers": [
    { "competitor": "Voltbay",    "price": 194.99, "in_stock": true,  "confidence": 0.99 },
    { "competitor": "Harborline", "price": 201.50, "in_stock": true,  "confidence": 0.97 },
    { "competitor": "PrimeDeck",  "price": 189.00, "in_stock": false, "confidence": 0.93 }
  ]
}

Is price scraping legal?

The short version: collecting prices a retailer publishes openly on a public product page is ordinary competitive research, and buyers have walked into rival stores with a clipboard for a century. The questions that carry real risk are narrower than the debate suggests, and they concern method and use rather than scraping as such.

Our position, plainly: we collect public listing pages at a rate sources can serve, we do not authenticate into sites on your behalf, and a source that cannot be collected reliably and politely is marked degraded rather than guessed at. For the end-to-end method, see how to scrape competitor prices.