Legal and Ethical Boundaries of Web Scraping

Scraping a public webpage is legal. Writing a script that visits that same page many times a day does not change that basic fact, because a computer reading and recording what's already sitting in public view is doing what a human would do with a notepad and a lot of free time. What actually decides a scraping project's fate is what you collected, how you got to it, and what you did with it afterward.

The Supreme Court narrowed the reach of the Computer Fraud and Abuse Act in Van Buren v. United States in 2021, ruling that "exceeds authorized access" only applies to someone who reaches data they were never allowed to touch, not someone who uses access they already have for a purpose a company dislikes. That ruling quietly killed the idea that a terms-of-service clause alone could turn scraping into a federal crime.

That's the baseline. It's also where most of the useful analysis still has to start. "Legal by default" covers public, non-personal, non-copyrighted data collected without breaking through any technical barrier, and nothing broader than that. The instant a project touches a login wall, scoops up personal information, copies protected writing, or leans on a server hard enough to slow it down, the safe harbor disappears.

Fault line one: authentication as the boundary between public and unauthorized access

The clearest dividing line in scraping law is whether there's a lock on the door. Data sitting on a page any anonymous visitor can load falls outside the CFAA's reach. Data sitting behind a login screen, a paywall, or a barrier that had to be defeated to get past it sits on the other side of that line.

Van Buren gave this test its name: gates up or down. A public page has no gate to lift in the first place, while a login wall is a gate, and forcing it open without permission is the exact behavior the CFAA was written to punish. The Ninth Circuit applied that reasoning directly to scraping in hiQ Labs v. LinkedIn, ruling that where no authentication gate exists, the CFAA's idea of access "without authorization" simply has nothing to grab onto, comparing an open public page to a building with no door to break into. The court reaffirmed that reading in 2022, holding that scraping publicly accessible data likely doesn't violate the CFAA at all, though it stopped short of ruling separately on content locked behind a login.

A California court applied similar logic in Meta Platforms v. a scraper defendant, dismissing the CFAA claims over scraping of publicly viewable posts. Meta's breach-of-contract claims got dismissed too, which is the detail that matters here: clearing the CFAA bar doesn't clear every other bar standing behind it.

The sharpest update for 2026 involves cease-and-desist letters. The Ninth Circuit held, and reaffirmed on rehearing, that a cease-and-desist letter from LinkedIn does not, by itself, create the kind of notice needed to trigger CFAA liability for scraping public data, because the statute's "without authorization" language has nothing to attach to when no technical barrier exists. A nasty email from a company's legal department does not retroactively build a gate where none stood before.

That doesn't mean every workaround is safe. Defeating a challenge-response puzzle meant to screen out bots, rotating credentials to dodge a block, or circumventing an IP ban can get read as bypassing a technical protection measure under an entirely different law: the DMCA's Section 1201. That's the theory Reddit used against Perplexity AI in a lawsuit filed in October 2025. The practical test for any developer is almost embarrassingly simple: open a logged-out browser with no special headers and see if the target page loads. If it does, the authentication fault line is clear. If it doesn't, stop and think before writing another line of code.

Fault line two: personal data under GDPR and CCPA

Clearing the authentication test doesn't clear the data-privacy test, and treating them as the same checkbox is where a lot of projects get into trouble. GDPR and CCPA apply to personal data regardless of whether that data was sitting in public view when it was collected.

Under GDPR, scraping names, emails, profile photos, or location data counts as processing personal data, and processing requires a lawful basis to stand on. "It was public" isn't one of the options. Most scraping operations lean on "legitimate interest" as that basis, but legitimate interest isn't a magic phrase, it's a documented balancing test that has to show the business reason for collecting the data doesn't override the rights of the people the data describes, and that test has to hold up if a regulator ever looks at it closely.

Scale brings its own obligations on top of that. Large-scale scraping of personal data under GDPR can require giving data subjects a privacy notice (often close to impossible in practice), responding to access and deletion requests, running a formal Data Protection Impact Assessment, and applying data minimization so only what's actually needed gets kept. None of these are optional add-ons for careful teams. They're baseline requirements once personal data and scale appear together.

The clearest case study, even if it isn't a perfect match, is Clearview AI, fined €20 million by the French data regulator CNIL in 2022 for collecting biometric and personal data without a lawful basis. No case has found a scraper liable under GDPR specifically for the act of scraping alone, but Clearview shows how hard regulators come down when personal data collection has no legal footing under it.

CCPA and its update, CPRA, add a parallel obligation in the U.S.: scraping personal data about California residents can classify the operator as a "business" under the act, which brings opt-out rights and disclosure duties along with it. The carve-out for "publicly available" data is narrower than most teams assume it is.

The practical line is this: business data like prices, product listings, and aggregate reviews with no author attached sits largely outside privacy law. The moment a dataset can be linked back to an identifiable person, the privacy fault line switches on.

Fault line three: copyright and the limits of "publicly visible

Copyright doesn't care that scraping happened. It cares what the scraped content turns into afterward. Pulling facts out of a page for analysis sits in a completely different category than copying someone's writing and republishing it, and the rise of AI training as a use case has made courts far more exacting about where that line actually falls.

The underlying principle is old and simple: facts aren't copyrightable, but original creative expression is. Scraping a stock price, a job title, or a product's listed weight copies a fact. Scraping a full article, an editor's written summary, or a photograph copies someone's protected expression, and those two things get treated very differently in court.

The sharpest current battleground is market substitution, the fair-use question of whether a new use damages the market for the original work it drew from. That question got a lot more concrete when OpenAI's own Head of ChatGPT described the product as "largely substitutive" to publishers, a quote that carries real legal weight precisely because it speaks to the exact factor courts weigh. A company's own words about its product's effect on a market tend to matter more in court than a competitor's accusation ever could.

The scale of exposure is visible in active litigation. A coalition of nearly 400 local newspaper outlets is suing Microsoft and OpenAI in Richner Communications et al., filed June 24, 2026 in the Southern District of New York. A Hachette-led class action against Google, filed July 10, 2026, alleges that copyrighted works were used to train Gemini with copyright management information intentionally stripped out along the way.

The practical test holds up cleanly against that backdrop: scraping for internal analysis of facts carries low risk, while scraping to republish, reproduce, or train a model that ends up substituting for the original source carries high risk, no matter how publicly accessible that source was to begin with.

Fault line four: server impact as a legal question

The decisions that feel purely technical, like how many threads to run or what retry interval to set, carry legal weight too. A scraper that slows a site down or knocks it offline for other users can cross from simple data collection into conduct the law treats as actionable harm. The CFAA's damage provisions and various state computer-crime statutes reach interference with service availability directly, and older trespass-to-chattels cases have treated a server as property that can be harmed by unauthorized, excessive use, even when every byte collected was public.

Robots.txt is the clearest signal a site operator gives about what volume of traffic they're willing to tolerate. A Crawl-delay directive is the site telling a crawler how fast is too fast, and ignoring it reads as evidence of bad faith if the dispute ever ends up in front of a judge.

The regulatory floor under this got a lot firmer through 2025 and 2026. The EU AI Act, enforceable for general-purpose AI providers from August 2, 2025, requires those providers to respect machine-readable opt-outs. Cloudflare built directly on top of that requirement with a Content Signals Policy, adding a Content-Signal field to robots.txt that distinguishes search crawling from AI training crawling: a site can allow search engines in while closing the door specifically on model training, and ignoring that distinction creates a timestamped record of a violation under EU law.

Cloudflare went further on July 1, 2026, announcing that starting September 15, 2026 it will block AI training and agent crawlers by default on any page carrying ads, a policy covering roughly one-fifth of all web traffic and framed explicitly as protecting publishers whose monetized pages were built for human eyes, not bot traffic. Courts have started backing the machine-readable standard with real teeth: the Hamburg Higher Regional Court ruled in December 2025 that opt-outs written into a site's terms of use in plain language don't count, because opt-outs currently have to be machine-readable, though the court left the door open, on a technology-neutral basis, to advanced natural-language processing qualifying at some point in the future. Respecting robots.txt in the EU is now a documented legal obligation.

How the four fault lines interact in AI pipelines

The scraping projects that end up in real legal trouble are rarely clean failures on one fault line. They tend to be projects that cross several lines at once, and the exposure compounds rather than adds, because clearing one defense does nothing for the other three.

Picture a project that only touches public pages, so the authentication line is clear. It also harvests personal profiles at scale, reproduces full article text to train a model, and fires requests with no rate limiting in sight. That project faces exposure on three separate fronts at the same time, and the hiQ defense that protects it on authentication has nothing to say about the other three.

AI training pipelines are the clearest real-world version of this pattern. They tend to scrape at a scale large enough to stress servers, often pull in personal data embedded inside otherwise public content, copy substantial creative material, and sometimes go through paywalls to get it. The NYT v. OpenAI case alleges three of those four: scraping at scale, copying substantial copyrighted material, and bypassing paywalls. The personal-data allegation doesn't appear in that suit. It shows up instead in separate class actions filed against OpenAI.

A new structure is forming around all of this: Cloudflare's Pay Per Crawl model, where a bot's access to content becomes a metered, paid transaction rather than a free crawl, and that's a signal that the open-web data-collection era is shifting toward permissioned, paid access as the norm rather than the exception, resetting the baseline risk calculation for any new scraping project launched from here on.

For developers building AI pipelines that depend on live web data, the cleanest path runs through structured, authorized access, meaning search and scraping APIs built to handle authentication, rate compliance, and data formatting in a manner consistent with published terms, rather than hand-rolled crawlers trying to thread separate legal needles at once. Olostep, a managed web-data API that handles browser rendering, proxies, and anti-bot infrastructure so developers can collect public web content at scale without building that stack themselves, is one concrete example of this model. The four fault lines don't average out to a single safe number. They stack, and a pipeline only needs to be reckless on one of them to end up in a lawsuit it didn't see coming.