Product matching, and why it decides whether your price data is real

Product matching is the step where a competitor listing is judged to be the same product as one of yours. Everything downstream depends on it: a price index, a position mix, a MAP case and an automated repricing rule are all built on the assumption that the two products are genuinely the same. The work is a waterfall of four methods, ordered from most precise to most forgiving, with a confidence score attached to every pair and a human review band in the middle.

Why a wrong match is worse than no match

A missing match is visible. A wrong match is not, and it produces a number that behaves exactly like a real one. Consider a stock pot. Northline Supply sells the 8 quart at $119.00. The competitor Voltbay sells the 6 quart from the same manufacturer, same lid, same finish, and because both retailers use the manufacturer's image library, the same photograph. Voltbay lists it at $79.00.

If those two are matched, the dashboard reports a $40.00 undercut, 33.6 percent below your price because 40.00 / 119.00 = 0.336. That number pulls your price index down, sits at the top of the exception list in every weekly review, and if it feeds an automated repricing rule it will try to move a $119.00 product to around $78.00. Guardrails such as a margin floor limit the damage. They do not stop the underlying error.

How product matching works: a four-layer waterfall

Run the methods in order of precision. Each layer only sees the pairs the layer above could not settle, which keeps the expensive methods off the easy cases.

Layer 1: exact identifier joins (GTIN, UPC, EAN, ISBN, MPN)

If both sides publish the same global identifier, the match is essentially settled. GTIN is the umbrella number. UPC-A is the 12-digit form used in North America, EAN-13 is the 13-digit international form, ISBN-13 is the book form carrying a 978 or 979 prefix, and GTIN-14 identifies a case or carton pack rather than the consumer unit. MPN is the manufacturer part number, which is not globally unique and is only usable when joined with a brand.

Identifiers carry a check digit, which is worth validating before you trust one. Take the UPC body 03600029145. Sum the digits in odd positions, 0 + 6 + 0 + 2 + 1 + 5 = 14, and multiply by three to get 42. Sum the digits in even positions, 3 + 0 + 0 + 9 + 4 = 16. The total is 58. The next multiple of ten is 60, so the check digit is 2 and the full code is 036000291452. An identifier that fails this arithmetic was typed by hand or truncated in a feed, and joining on it produces confident nonsense.

Why identifiers are missing, or present and wrong

• The retailer strips them. Many storefronts publish no GTIN in the page HTML or in schema.org structured data, sometimes deliberately, to make exactly this comparison harder.
• Variant collapse. A marketplace listing carries the parent GTIN while selling eight colours, so one identifier resolves to eight sellable products at different prices.
• Placeholder values. A seller with no identifier enters a distributor code, a zero-filled string, or the same code across their whole catalog. These validate as text and mean nothing.
• Bundles under the single-unit code. A two-pack listed with the single-unit GTIN. The identifiers agree, the prices are roughly double, and the result looks like a very expensive competitor.
• No manufacturer identifier at all. Private label, made-to-order and store-exclusive products often have no GTIN outside the retailer that created them.
• MPN collisions. A part number like A1234 belongs to more than one brand, and brand strings are messy: parent company, sub-brand and the retailer's own spelling rarely agree.
• Condition and region share the code. Refurbished, open box and regional voltage variants frequently carry the same identifier as the new domestic unit.

Layer 2: title and attribute matching

When identifiers are absent, the title and the spec table are what remain. This layer has two distinct parts, and treating them as one is the classic mistake.

• Normalisation. Lowercase, strip punctuation and merchandising language (new, sale, free shipping), expand unit abbreviations so qt, quart and quarts are one token, and normalise numbers so 6-qt, 6 qt and 6qt agree.
• Attribute extraction. Pull structured values from the title and spec table: brand, model, capacity, dimensions, pack count, colour, material, voltage, condition. Compare these for equality, not similarity.
• Blocking attributes. The small set where disagreement kills a match however similar the rest looks: pack count, capacity or size, model generation, condition, region variant. Each is a difference the customer notices at the door.
• Descriptive similarity. Everything else in the title, as token overlap. A supporting signal, never a deciding one.

The failure cases in this layer are all near-misses. Model successors differ by a single character, so Gen 3 and Gen 4 look almost identical. Colour variants are textually identical apart from one word. Retailers invent their own titles, so the same product appears under three different names. And bundles read as the base product with extra words, which token overlap treats as a minor difference rather than a different purchase.

Layer 3: embeddings, and the one thing they cannot do

An embedding turns a product title, and sometimes the description, into a vector, so listings with similar meaning sit close together in vector space. This is powerful for retrieval: from a competitor catalog of a hundred thousand listings, embeddings will surface the twenty candidates worth examining for your product, including ones with completely different wording.

Use embeddings for recall, never for the final decision. Semantic similarity is not numeric equality. A 6 quart stainless stock pot and an 8 quart stainless stock pot are, in meaning, almost the same sentence, and a vector model will score them as near-identical because that is what it was built to do. The distinction that matters commercially lives in one number in the title, and that number needs a rule, not a similarity score. The correct division of labour: embeddings generate candidates, attribute rules verify them.

Layer 4: image comparison

Two techniques do different jobs. A perceptual hash detects near-identical image files, catching the common case where both retailers use the manufacturer's asset library. An image embedding detects the same physical object photographed differently, catching retailers that shoot their own photography.

Images are the most expensive automated signal, because every candidate means a download, storage and compute, and they are the least reliable alone. Lifestyle photography of a product in a kitchen tells you very little. Colour variants frequently share one hero shot supplied for the whole family, which is exactly where a strong image signal misleads most. Cropping, watermarks and composited discount badges all move a perceptual hash. Treat image similarity as a tiebreaker inside the review band, not as a first-pass method.

Confidence scoring and where to set the thresholds

A confidence score is a weighted combination of the signals above, with blocking rules that override the combination. Two mechanisms, not one, because a weighted average will happily average away a contradiction that should be fatal.

signals      identifier 0.40 | attributes 0.30 | title 0.15 | brand 0.10 | image 0.05

when a signal is unavailable, redistribute its weight across the rest

score = sum(signal_value * normalised_weight)

hard rules, applied after scoring:
  identifier exact AND brand agrees      -> floor score at 0.95
  blocking attribute contradicted        -> cap score at 0.50
  price outside plausibility band        -> cap score at 0.50, flag for review

bands:
  >= 0.92   auto-accept
  0.70-0.92 human review queue
  <  0.70   auto-reject

• Auto-accept, 0.92 and above. Straight into the price feed. Raise this threshold if matches feed automated repricing, because a wrong match then moves a real price with no human in between.
• Review, 0.70 to 0.92. A queue, not a rejection. Signals are strong but incomplete, which is where a person adds the most value per minute spent.
• Auto-reject, below 0.70. Discarded, but logged. A SKU rejecting every candidate usually signals a catalog data problem on your side rather than an absence of competitors.

A worked confidence decision

Back to the stock pot. Your product is the 8 quart at $119.00. The candidate is the Voltbay 6 quart at $79.00. Voltbay publishes no identifier on the page, so the 0.40 identifier weight is redistributed across the remaining signals, which sum to 0.60. Renormalised, attributes carry 0.500, title carries 0.250, brand carries 0.167 and image carries 0.083.

That is the entire argument for blocking rules. Four of five attributes agreed, the image was nearly identical, the brand was exact. Every soft signal said yes. One hard signal said the customer would receive a different pot, and that signal has to win, because the alternative is a $40.00 phantom undercut in your price index and your repricing queue.

The human review loop

Review is not a failure of automation. It is what makes automation safe. Design it as a queue with a budget.

The budget is smaller than people expect. Example: 2,500 products against 5 competitors is 12,500 candidate pairs. Suppose 70 percent settle on identifiers and auto-accept, 22 percent auto-reject, and 8 percent land in the review band. That is 1,000 pairs. At 20 seconds each, that is 20,000 seconds, or 5.6 hours of work, once, at onboarding. Steady state after that is only new products, re-discovered matches and pairs whose confidence has dropped, which is usually a few minutes a week.

• Show the disagreement, not the similarity. Put the contradicted attribute at the top of the screen. Presenting two attractive product cards side by side and asking whether they match is how accept-bias happens.
• Three actions only. Confirm, correct to the right listing, or reject. Anything more nuanced belongs in the scoring, not in the reviewer's head.
• Corrections must change future scoring. A correction that fixes one pair and teaches the system nothing means the queue never shrinks. On our matching, corrections feed back into how comparable pairs are scored.
• Sort by disagreement, not by score. A queue that leads with near-certain pairs trains the reviewer into a rhythm of accepting, and the genuinely wrong pair arrives mid-rhythm.
• Put a merchandiser on it, not an analyst. Someone who knows the Gen 3 was discontinued in March clears ambiguous cases in seconds. Someone reading titles cold will not.

How matches break after they are made

A match is not a permanent fact. It is a claim about two pages, and one of those pages belongs to someone who changes it without telling you. There are two categories of breakage, and only one of them is easy.

Loud breakage

The URL 404s. The page returns a server error. The product is delisted. These are easy: they raise an error, land in a coverage report and get fixed. Anything that errors will be noticed.

Silent breakage, which is the dangerous kind

• The URL redirects to a category page and returns 200. The extractor finds a price on that page, usually the cheapest item in the grid, and stores it. Nothing errors. Your competitor appears to have cut prices dramatically overnight.
• The product is replaced in place. Same URL, new model year, new price. The comparison continues silently against a product you do not stock.
• A marketplace listing is merged or split. Variants collapse into a parent, or one variant is promoted to its own listing, and the identifier you joined on now resolves to a different sellable unit.
• A single-unit page becomes a bundle page. Same URL, same title plus two words, roughly double the price. Read as a competitor price increase.
• The page moves price into a JavaScript-rendered component. The HTML still returns 200 with a full page of markup, and the price element is simply absent. A fragile extractor stores nothing, or grabs the crossed-out reference price instead of the sale price.

The defences are structural rather than clever. Key matches to identifiers rather than to URLs, so when a product moves it can be re-discovered rather than lost. Re-verify the match signature on every collection, not only at match time: if the title tokens, blocking attributes or image hash shift, drop the confidence and route the pair back to review. Validate every price for plausibility before storing it. And alert on a confidence drop, because a match whose score fell from 0.96 to 0.71 overnight is telling you something specific happened on that page. That behaviour is what the data quality module exists to provide, and the scraping mechanics behind it are described separately.

A matching checklist you can run this week

1. 1Export your catalog and count how many SKUs have a valid GTIN, validating the check digits. That number is usually lower than anyone expects, and it sets your ceiling on Layer 1.
2. 2Fix your own attribute data first. Capacity, pack count and model generation belong in structured fields, not only inside a title string.
3. 3List your blocking attributes per category. Pack count and size for consumables, generation and storage for electronics, size and colourway for apparel, fitment for auto parts.
4. 4Set the accept threshold according to what the data drives. Higher if it feeds automated repricing, lower if it feeds a report a human reads.
5. 5Sample 20 auto-accepted matches, check them by hand, record the precision, and repeat monthly.
6. 6Turn on alerting for confidence drops, not just price changes, so a page change reaches you before a report does.

Matching is the highest-weighted criterion in the software selection rubric for exactly this reason, and it is step four of the competitor price monitoring procedure. Everything else in a price program is arithmetic on top of it.