Scrape monitoring, because the worst failure is silent

The expensive failure here is almost never a wrong price. It is a price that stopped updating while the dashboard still looked fine. A scraper that breaks loudly gets fixed the same day. One that breaks quietly leaves last Tuesday's number on screen, and by then a repricing rule has run on it for a week. Scrape monitoring makes that silence impossible: coverage and freshness measured per source, automatic repair when extraction breaks, and an alert on degradation rather than on disappearance.

How a price feed goes quiet without an error

The mechanics matter, because the abstraction hides the problem. A retailer redesigns a product page and moves the price out of the server-rendered HTML into a component a script writes after load. Nothing errors. The fetch returns HTTP 200. The parser finds an empty placeholder and returns nothing.

What happens next decides whether you lose money. A naive pipeline reads an empty result as no change and keeps the previous value. The listing shows $159.00 in every view, export and rule, the chart stays a smooth line, and the one honest signal, the age of that number, is nowhere on screen.

Why totals hide it: track eleven competitor domains, let one go dark, and overall coverage falls to about 91 percent, because 10 / 11 = 0.909. That reads like rounding. It is every price from one competitor being a week old.

Coverage and freshness, the two measures scrape monitoring reports

Coverage, worked. A store tracks 4,200 product-competitor pairs and 4,061 returned a valid price last night, so coverage was 4,061 / 4,200 = 0.967, or 96.7 percent. The 139 that did not are listed and grouped by cause.

Freshness, worked. On the Growth plan the expected interval is 12 hours, so a pair whose newest observation is 31 hours old has missed two cycles. Freshness is the share of pairs inside 1.5 times that interval, 18 hours here, reported with the oldest pair per category so nothing hides in an average.

Self-healing extraction, mechanism by mechanism

Automatic repair is not one feature. It is four mechanisms catching four different breakages, and it is worth knowing which is doing the work.

1. 1

   Structured data first, so most redesigns change nothing

   Where a retailer publishes schema.org Product markup, price, currency and availability are read from it. That is the data the site hands to search engines, so it survives cosmetic redesigns. A per-domain recipe is the fallback.

2. 2

   Recipe regeneration when a layout changes

   When a domain's parse-failure rate jumps in one cycle, the recipe is regenerated from a fresh render of sampled pages, then validated against recent known-good prices. If it reproduces them within a plausible band it is promoted. If not, nothing is written.

3. 3

   Identifier-based re-discovery when a URL moves

   The most common breakage after a redesign. The old address 301s to a category page or returns 404. Rather than drop the listing, the product is re-discovered on the retailer's own catalogue by GTIN, UPC, EAN or MPN, and the new URL attaches to the same match with a fresh confidence score. History continues on one series.

4. 4

   Plausibility validation before anything is written

   Every candidate price is checked against the listing's own history and against peer listings. A $0.00, a shipping fee, a financing instalment where the shelf price was, or a fortyfold overnight jump is quarantined. A quarantined value never reaches a chart, an index or a rule.

5. 5

   A human queue when automation fails

   Some sites cannot be repaired by pattern. When regeneration fails validation twice, the source escalates to a person and the incident stays open in your account meanwhile. Automation that admits defeat beats automation that keeps writing.

Failure modes, detection, and automatic response

Rows three and four are why matching belongs to data quality. A URL is a location, not an identity. When the identity behind a URL changes, the honest response is to drop confidence and stop using the pair, not keep charting whatever price is now there.

Why we show you our own incident log

Every vendor claims high accuracy, and none of those claims are checkable, because you cannot audit a figure handed to you about data you did not collect. What is checkable is whether a vendor tells you when something broke. So the incident log lives in your account.

• What broke. The domain and the failure mode, in the plain terms used above.
• When it started and when it was restored. Both timestamps, so you can check whether a number you used on Wednesday sat inside the window.
• How many products were affected. Scope in pairs, not a severity colour. Fourteen pairs and 1,400 pairs are different conversations.
• Whether history was backfilled. Some outages recover by re-reading pages that still show the same price. Many do not.
• Whether it is still open. Affected pairs stay out of repricing until it closes.

What we do not claim

• Not 100 percent coverage, and nothing behind a login. Some sites cannot be read reliably, and we say so and stop rather than serve degraded data quietly. No member pricing, no negotiated B2B rates: if a shopper cannot see it, we do not collect it.
• No retroactive capture. Backfill works only when a page still shows the price it showed during the outage. A moment that passed uncollected stays uncollected.
• Detection is not omniscience. A new failure mode on one domain can run a cycle before it is caught. What we commit to is that it surfaces as an incident, not as a confident wrong number. Evidence capture for MAP protection is held to the same rule.

Data quality is one of six modules, all included on every plan. See how they fit together on the product overview, or how collection itself works in price monitoring.