Is web scraping legal? What the question actually covers

No single law makes web scraping legal or illegal in the United States. What exists is a set of separate questions a scraping programme can trigger independently: computer-crime statutes about unauthorized access, contract claims about terms of service, copyright in what you copy, database rights outside the US, and data-protection rules if the data describes people. A method can be comfortable under one and exposed under another. The practical work is knowing which questions your method raises, and no longer raising the ones you do not need.

Is web scraping legal? Start by naming which law you mean

Most arguments about scraping go nowhere because the people arguing mean different statutes. One means computer crime, one means breach of contract, a third means copyright. Those are separate bodies of law with separate elements and separate remedies, so the useful version of the question is which of these five it raises, and which of them you can stop raising.

1. 1Access. Did the collection reach a computer, or an area of one, that was not open to me? This is the computer-crime question.
2. 2Agreement. Did I accept terms forbidding automated collection, and then do it anyway? This is the contract question.
3. 3Copying. Did I take original creative expression, or only facts? This is the copyright question.
4. 4Investment. Did I extract a substantial part of a database someone invested in assembling? This is the EU database-right question, with no direct US equivalent.
5. 5People. Does the data describe an identifiable person? This is the data-protection question, and it does not arise for a shelf price.

Price collection is unusual: it can usually be run so that only the first two are live, and both are manageable. That is not true of every scraping use case. It is true of most price scraping done for competitor monitoring.

Publicly accessible pages versus anything behind authentication

The largest single factor in how a scraping dispute gets framed is whether a person without an account could have seen the same page in a browser. A product page that search engines index, that renders fully for a logged-out visitor and that carries a price is in a different category than a page requiring a login, a membership, a dealer code or an API key.

The reason is mechanical rather than moral. Computer-crime statutes are built around a gate. If a page is served to anyone who asks, there is no gate to pass. If it is served only after credentials are checked, passing it is an event a court can point at. Creating an account also means you clicked to accept terms, and it attaches your identity to the collection.

The Computer Fraud and Abuse Act, and how Van Buren narrowed it

The Computer Fraud and Abuse Act, 18 U.S.C. section 1030, is the federal computer-crime statute. It reaches someone who accesses a protected computer without authorization, or who exceeds authorized access. It carries criminal penalties and also gives private parties a civil claim, which is why it turned up so often in scraping disputes: an operator who wanted a scraper to stop could point at a federal statute rather than only at its own contract.

For years a broad reading circulated in which breaking a site's terms of service was itself a way of exceeding authorized access. Under it, an ordinary automated fetch of a public page could be argued into a federal crime because a terms document said no robots.

What Van Buren v. United States (2021) decided

In Van Buren v. United States, decided by the Supreme Court in 2021, a police sergeant used credentials he was entitled to use to look up a license plate in a law-enforcement database for a personal and improper reason. The government argued that using authorized access for an unauthorized purpose exceeded authorized access under the CFAA.

The Court disagreed. It read the phrase to cover obtaining information from areas of a computer that are off-limits to the person, rather than obtaining information the person may reach and then misusing it. The opinion described this as a gates-up-or-down question. Purpose, motive and policy violations sit outside the statute.

That mattered for scraping, because it removed the argument that a terms breach automatically converts into a federal crime. It did not answer everything. The Court expressly declined to resolve whether the authorization line is drawn only by technical measures such as passwords, or whether contracts and policies can draw it too. That open question is where scraping disputes now live.

hiQ Labs v. LinkedIn: what it settled and what it did not

hiQ Labs collected data from LinkedIn profiles visible to the public without logging in. LinkedIn sent a cease-and-desist and blocked hiQ. hiQ sued and obtained a preliminary injunction requiring LinkedIn to stop blocking it, which the Ninth Circuit affirmed. After Van Buren the case went back for reconsideration, and the Ninth Circuit reached the same conclusion again: where a site is open to the public and no authorization is required in the first place, the CFAA concept of access without authorization likely does not apply.

This gets summarised online as a ruling that scraping public data is legal. That summary is wrong in three specific ways, and the three ways are the useful part.

• It was a preliminary-injunction posture, not a final merits judgment. The standard at that stage is likelihood of success. Courts were saying the CFAA claim probably fails, which is not a holding that it always does.
• It is Ninth Circuit law. Persuasive elsewhere, not binding, and other circuits are not obliged to agree.
• It was about the CFAA, not about contract. LinkedIn later pursued its breach-of-contract theory in the district court and prevailed on it, and the litigation ended in a settlement under which hiQ was restrained from scraping LinkedIn.

That last point is the one to take away. Winning on the CFAA and losing on contract are entirely compatible outcomes. Anyone citing hiQ as blanket permission is citing half a case.

Terms of service and contract claims are a separate risk

A terms-of-service claim is an ordinary contract claim, and it asks a simpler question: was there an agreement, and was it broken? Because the elements are simpler, this is now the more common route for an operator who wants a scraper to stop.

Whether terms bind you depends heavily on how they were presented. Terms you actively clicked to accept when creating an account are routinely enforced. Terms sitting behind a footer link, never surfaced to a visitor who only read a product page, are on much weaker ground, and courts have often been sceptical of them. There is no clean rule, and the analysis is fact-specific enough to be a real question for counsel.

Two things reliably strengthen the operator's position. Account creation converts a passive notice into an accepted agreement. A cease-and-desist removes any argument that you did not know what they wanted. Continuing unchanged after a direct, addressed demand changes the character of the conduct in every framework at once, including the access question.

Separately, hitting a site hard enough to degrade it can support older trespass-to-chattels claims, which generally turn on demonstrating actual interference with or harm to the system. That risk is self-inflicted and avoidable: a collector staying well inside the traffic a site handles routinely does not create it.

Copyright, the EU database right, and when GDPR is irrelevant

Facts are not copyrightable, expression is

Copyright protects original expression, not facts. In Feist Publications v. Rural Telephone Service Co., the Supreme Court held that an alphabetical white-pages listing lacked the minimal originality copyright requires, and rejected the idea that effort alone earns protection.

A price is a fact. So is a stock status, a shipping threshold, a GTIN, or the observation that a listing was discounted on a Thursday. A product description, a lifestyle photograph, a customer review and an editorial buying guide are expression, and they belong to whoever created them. That line maps neatly onto what a price monitoring system should keep: numbers, identifiers, states and timestamps. One nuance: an original selection or arrangement of facts can attract protection even when the individual facts do not, so reading one competitor price is not the same act as reproducing their entire curated catalogue.

The EU sui generis database right

The European Union has a right the United States does not. Under the EU Database Directive, the maker of a database who made a substantial investment in obtaining, verifying or presenting its contents can prevent extraction or reutilisation of a substantial part, independently of copyright. European case law distinguishes investment in obtaining data that already existed from investment in creating it, which narrows the right, but it applies where copyright would not. If you collect from EU storefronts, do not assume the US analysis travels.

GDPR is about people, and a price is not a person

GDPR governs personal data, meaning information relating to an identified or identifiable natural person. The price of a cordless drill is not personal data. Neither is its stock status or its currency. A programme storing only product identifiers, prices, availability and timestamps is largely outside the regulation, and saying so plainly beats the vague compliance language that usually surrounds the topic.

What brings GDPR into play is everything else on the page. Marketplace seller names are frequently individual traders. Review text, reviewer handles and Q and A threads are personal data, and once stored the record is in scope. The cheapest control is to drop those fields at extraction time. If you never keep the reviewer name, you never have to answer a question about it.

robots.txt is a norm, not a statute

The Robots Exclusion Protocol is a convention site operators use to tell automated clients which paths they would prefer left uncrawled. It was eventually written up as an IETF standard, which gave it a stable specification, but a standard is not legislation. Ignoring a Disallow line is not itself a crime in the United States, and honouring one is not a defence to anything.

It matters anyway. robots.txt is a public, dated, unambiguous statement of what the operator asked for, so if a dispute happens the first exhibit is what the file said and what your collector did. A collector that quietly respected a Disallow tells a coherent story. One that walked straight through it has made every other argument harder to run.

A practical compliance checklist for price collection

None of the following guarantees an outcome. It removes the easy arguments against you, so that any dispute has to be fought on the hard ones. Every item is cheap relative to what it prevents.

1. 1

   Stay logged out

   Collect only what a visitor with no account can see. If a source genuinely requires an account, escalate it as a business decision rather than solving it in code.

2. 2

   Do not circumvent access controls

   No CAPTCHA solving, no credential sharing, no rotating identities to defeat a block aimed at you. A block is a message. Route it to whoever owns the relationship.

3. 3

   Rate limit, and stay boring

   Cap concurrency per domain, spread checks across the day, back off on 429 and 5xx, and never retry aggressively into an error. A collector that never appears in a capacity conversation never generates a harm argument.

4. 4

   Identify your crawler honestly

   Send a descriptive user agent with a URL or contact address. Anonymity looks like evasion. Being reachable means the first contact is an email rather than a letter from a law firm.

5. 5

   Take facts, not content

   Store price, currency, availability, promotion state, identifiers and timestamps. Do not store descriptions, photographs, review text or reviewer identities. Enforce it in the extraction schema so it cannot drift by habit.

6. 6

   Respect robots.txt

   Fetch it, parse it, cache it, obey it, log the decision. When a Disallow removes a source you need, ask through the front door instead of routing around it.

7. 7

   Keep records

   Retain what you requested, when, from where, what the response was, and how the collector was configured at that moment. A dispute arriving eighteen months later is answerable only from logs you already had.

8. 8

   Have a stop procedure and use it

   Decide in advance who receives a cease-and-desist, how fast that domain is paused, and who reviews it. The worst version is a letter sitting in a shared inbox while the collector keeps running.

If you are building this yourself, the mechanics are covered in how to scrape competitor prices. If you would rather not run the crawler at all, that is what our competitor price monitoring does: logged out, rate limited, with a full collection log per source.

Questions worth taking to a lawyer, not to a forum

• Which of our sources are collected while logged in, and can any move to logged-out collection?
• Which terms of service have we actually accepted by clicking, as opposed to terms that merely exist in a footer?
• What is our exposure if a source operator sends a cease-and-desist tomorrow, and what happens in the first 24 hours?
• Do we store any personal data at all, including marketplace seller names, and is that deliberate?
• Do we collect from EU storefronts, and has anyone looked at the database right for those sources?
• Do we republish anything we collect, on any customer-facing surface, and is it facts or expression?