Ethical and Legal Boundaries of Web Scraping for AI

If you're building AI systems and scraping the web to train them, here's the honest truth: the old "public data is fair game" assumption isn't dead, but it's on life support. The law hasn't caught up to AI perfectly, but courts, regulators, and infrastructure providers have moved fast enough in the past two years that "we assumed it was fine" is no longer a defensible posture. The good news is that the rules are navigable. You just have to know which rule applies to which scraping decision. That's what this piece is actually about.

How the Public/Private Data Distinction Actually Works Under Current Law

Let's start with the foundation everyone thinks they understand and almost nobody fully does.

The basic principle is sound: publicly accessible, non-password-protected content is the legally safer zone for scraping. The landmark case here is hiQ v. LinkedIn, decided by the Ninth Circuit between 2019 and 2022. LinkedIn tried to stop hiQ from scraping public profiles. The court said no. Scraping publicly visible data didn't automatically violate the Computer Fraud and Abuse Act.

That ruling still matters. It's still good law. But "public" is not a blanket shield, and here's where people get into trouble.

Three things can turn a "public" page into a legal minefield:

• PII is present. If the page contains names, emails, or any personally identifiable information, GDPR and CCPA activate. Doesn't matter that the data was publicly posted. Public visibility is not the same as consent to commercial processing.
• You agreed to terms to get there. Clickwrap agreements (where you actually click "I agree") are contractually binding even on technically accessible content. Browsewrap terms (those footer links nobody reads) are much harder to enforce because courts require proof the user actually knew about them.
• You bypassed something to get the data. Rate limits, IP blocks, authentication tokens. Circumventing these triggers CFAA and potentially DMCA exposure. That's a different conversation entirely.

The case that illustrates all of this most cleanly is Meta v. Bright Data from 2024. Meta sued Bright Data for scraping. The result was almost comically split: Meta lost on the logged-out public scraping claim, but won on the breach-of-contract claim for logged-in scraping. Same defendant. Two different outcomes based entirely on whether Bright Data was authenticated when it scraped.

So the question is never just "is this page public?" The real question is: what conditions governed how I accessed it, and does the content include PII or protected creative expression?

What the CFAA Actually Prohibits and Where Its Limits Now Fall

The Computer Fraud and Abuse Act sounds scary. It's a federal criminal statute. People hear "criminal" and assume the worst.

Here's the reality: the Supreme Court significantly narrowed the CFAA's reach in Van Buren v. United States (2021). The court held that accessing publicly available data, even against a site's Terms of Service, does not constitute unauthorized access under the statute. ToS violations, by themselves, are more likely to generate breach-of-contract claims than CFAA claims. Those are still annoying and expensive. But they're not the same as criminal federal exposure.

Where CFAA exposure becomes real is circumvention of actual technical controls. Breaking a paywall. Spoofing authentication. Bypassing IP rate limits.

Reddit v. Anthropic, filed in June 2025, shows the theory in action. Reddit alleged that Anthropic bypassed robots.txt directives and IP rate limits. The combination of ignoring technical controls alongside commercial use without authorization is what elevates the claim toward CFAA territory. It's not just a "you violated our Terms" argument. It's "you broke through our fences" — and when you break through enough fences, sooner or later someone calls the sheriff.

Here's the practical takeaway for developers: rate-limiting your own requests to stay under a site's thresholds isn't just polite engineering. It's the behavior that keeps you on the right side of the circumvention line. Going fast is a legal risk, not just a technical one.

Where Copyright Law Draws Its Line for AI Training Data

This is the messiest area, and I'll be direct: it is not fully resolved. But the direction of travel is clear.

The central question has always been whether using scraped content to train an AI model qualifies as fair use. Fair use has four factors, and the one that matters most right now is transformative use.

In 2023, the Supreme Court decided Andy Warhol Foundation v. Goldsmith. The ruling narrowed what "transformative" means. Transformation requires a different purpose, not just a different form. That distinction matters enormously for AI training.

The emerging rule looks like this:

• An AI trained on news articles that generates news summaries = same market purpose = not transformative.
• An AI trained on medical literature to assist with clinical diagnosis = different purpose = potentially transformative.

Thomson Reuters v. Ross Intelligence, decided in February 2025, was the first final court judgment on AI training data copyright. Ross lost on market-harm grounds. The AI system competed with Westlaw's core market. That was enough.

The US Copyright Office published a 108-page report in May 2025. The short version: certain AI training uses cannot be defended as fair use. Where AI outputs are substantially similar to training inputs, the USCO sees a strong argument that model weights themselves infringe reproduction rights. That's a significant position from a significant institution.

A few more data points worth knowing:

• Chain-of-custody risk is real. You don't have to do the bad scraping yourself. Using a third-party dataset assembled through unauthorized scraping exposes you to the same claims. The LAION-5B litigation established that pattern.
• Anthropic settled a copyright class action for $1.5 billion in September 2025. That number sets a floor for how seriously courts and litigants are treating these claims.
• The DMCA anti-circumvention angle is becoming the preferred plaintiff theory because it sidesteps the fair use debate entirely. The argument: bypassing YouTube's systems, Reddit's rate limits, or CDN-layer controls equals circumventing access controls under DMCA Section 1201. Reddit v. Anthropic and Reddit v. Perplexity opened this door in 2025, and YouTube creator suits against Nvidia, Snap, Meta, and Apple followed the same template.
• One important clarification: a federal court held that robots.txt alone is not a technological access control under DMCA. Ignoring it isn't circumvention by itself.

Internationally, the picture is fragmented. The EU has a text-and-data-mining exception, but rights holders can opt out. Japan has taken the most permissive stance. India has no TDM exception at all, and the ANI Media v. OpenAI case at the Delhi High Court had its judgment reserved as of April 2026. The outcome there will affect a large and fast-growing market.

How GDPR and CCPA Constrain Scraping Even When the Data Is Publicly Visible

This one surprises people. It probably shouldn't anymore, but it still does.

GDPR applies to the processing of personal data regardless of whether that data was publicly posted. Public visibility is not consent. It never was. The regulation is explicit about this.

France's data protection authority, the CNIL, fined a company called KASPR €240,000 for collecting LinkedIn data without appropriate consent. The data was publicly visible on LinkedIn. The fine wasn't for how they got it. It was for processing it without a lawful legal basis.

Under GDPR, there are six lawful bases for processing data. For scraping pipelines, two are relevant:

• Legitimate interest. This requires passing a three-part test covering purpose, necessity, and a balancing of interests. France's CNIL now explicitly considers robots.txt compliance as a factor in legitimate interest assessments. More on that in a moment.
• Consent. Rarely workable for scraped data. You have no direct relationship with the people whose data you're scraping. You can't get consent from someone you've never spoken to.

CCPA adds a California-specific layer. California residents have rights to know what data is collected, delete it, and opt out of its sale. If your scraping pipeline ingests personal information belonging to California residents, you need a compliant data inventory and a working deletion mechanism.

Here's the timing point that catches teams off guard: you need the documented lawful basis before the data enters a training set or a RAG index. Not after you've already built the pipeline. The documentation trail needs to precede the processing.

The EU AI Act stacks on top of GDPR, not instead of it. Training processes that touch personal data still need a valid GDPR legal basis independently of whatever AI Act compliance you've achieved. They are separate frameworks with separate requirements.

Data minimization helps on both the legal and engineering side. Scraping only what's necessary for your stated purpose reduces legal exposure and storage costs at the same time. Compliance and efficiency pointing in the same direction is rare. Take advantage of it.

What the EU AI Act Now Requires from Teams That Scrape Training Data

Let's get the timeline right, because this is where teams are making the most expensive mistakes.

• August 1, 2024: EU AI Act entered into force.
• August 2, 2025: General-purpose AI model obligations became enforceable. This includes training data documentation requirements.
• July 2025: The AI Office published the mandatory training-data-summary template.
• August 2, 2026: Full conformity assessments due for high-risk AI systems. Fines up to 7% of global annual turnover.
• August 2, 2027: Models already on the market before August 2, 2025, have until this date to comply.

If your model launched before August 2, 2025, you're in the grace period. But "grace period" doesn't mean "do nothing." It means you have until 2027 to get your documentation in order, not an invitation to ignore the requirements.

What the AI Act actually requires for scraping-based training pipelines:

• A public summary of your training data. Contents, modalities, sizes, and sources. The AI Office template tells you exactly what format they expect.
• Copyright compliance documentation. You need to demonstrate you respected copyright reservations and opt-outs. Robots.txt compliance is directly relevant here.
• Proactive illegal content filtering. The obligation is proactive, not reactive. You can't wait to get caught.
• Synthetic data disclosure where applicable.

The stacking problem is the thing that's catching organizations flat-footed. Building your AI Act compliance does not satisfy GDPR. They are separate documentation trails for separate frameworks. Teams that build for one and assume the other is covered will find out the hard way.

The silver lining: teams that log their data sources, track robots.txt compliance per domain, and document their copyright opt-out checks are building the paper trail the AI Act now formally requires. Good engineering hygiene and regulatory compliance are pointing at the same practices.

What robots.txt and llms.txt Actually Signal. And What They Legally Mean.

Here's where a lot of developers have strong opinions based on incomplete information.

robots.txt is standardized as RFC 9309. It is a voluntary request, not a technical barrier. There is no code in robots.txt that prevents you from accessing anything. It signals intent. It does not enforce access restrictions at the server level.

A federal court confirmed this: robots.txt alone is not a technological access control under DMCA Section 1201. Ignoring it is not, by itself, circumvention.

So why should you care about it? Because ignoring it creates compounding risk that's harder to see but very real.

• Evidence of bad faith. When combined with other technical measures, ignoring robots.txt is evidence of lack of authorization, relevant to both CFAA and DMCA claims.
• GDPR legitimate interest. France's CNIL explicitly factors robots.txt compliance into legitimate interest assessments.
• EU TDM safe harbor. Under the EU text-and-data-mining exception, rights holders opt out via robots.txt. Ignoring it removes your safe harbor.
• Reddit v. Anthropic. Bypassing robots.txt directives was part of the unauthorized access theory in that complaint.

A May 2025 Duke University pre-publication paper found that many bots never checked robots.txt, and compliance dropped as rules became stricter. The bots most likely to ignore robots.txt were doing exactly the thing that creates legal exposure.

The user-agent split is the practical sophistication point that separates serious teams from careless ones. Major AI providers now operate separate user-agents for training crawlers versus search or answer crawlers. A site can block GPTBot, which handles training, while allowing OAI-SearchBot, which handles citations. That's meaningful opt-out granularity for publishers, and meaningful signal for developers about what access is actually permitted.

As of 2026, Cloudflare manages AI traffic across three categories for its customers, including free-tier accounts: Search, Agent, and Training crawlers. The infrastructure for fine-grained control is already deployed at scale.

llms.txt is a newer signal, with roughly 10% adoption as of recent research. Rather than blocking crawlers, it guides AI to a site's best content. It's opt-in rather than opt-out framing. Think of it as a welcome mat with directions, versus a door with a lock.

How the Permission Economy Is Reshaping What "Accessing Public Data" Means in Practice

Here's the number that explains why everything shifted: Cloudflare data showed that OpenAI's crawler scraped websites 1,700 times for every single referral it sent back. Anthropic's crawler scraped 73,000 times per referral.

Publishers looked at those numbers and concluded the relationship was not working in their favor. That calculation drove everything that followed.

Cloudflare launched Pay Per Crawl on July 1, 2025. A crawler requests content. If it doesn't present payment intent, it gets an HTTP 402 Payment Required response. Cloudflare acts as the merchant of record. More than a dozen major publishers endorsed the system at launch, including AP, The Atlantic, Time, and Condé Nast.

AWS added HTTP 402 capability to its firewall on June 15, 2026. Akamai integrated monetized crawler infrastructure in September 2025. Three major CDN and cloud providers now offer paying tollbooths at the network layer. This is not hypothetical infrastructure. It's deployed and in use.

More than one million Cloudflare customers have used its AI-blocking opt-out option. That's a lot of doors being closed.

The licensing track is running in parallel. OpenAI signed deals with AP, Axel Springer, and multiple publishers. Google has agreements with Reddit, Stack Overflow, and news publishers. Licensing deal values range from $1 million to over $250 million annually as of 2025. The market for formal data access agreements exists and is active.

Legislative pressure is building too. The AI Accountability for Publishers Act was introduced in February 2026. It would require AI companies to get permission and pay publishers before scraping. It's not law yet. But it signals where congressional intent is pointing.

The practical read for developers building pipelines today: "publicly accessible" is increasingly a temporary state. Domains are moving behind payment layers, rate-limit enforcement, and formal licensing requirements. The scraping decisions you make now need to account for what access will look like in 12 months, not just what it looks like today.

The Practical Compliance Checklist for a Production AI Scraping Pipeline

After everything above, here's what it looks like in practice. Not theory. Actual steps for a production pipeline.

Before you scrape a single byte:

• Determine whether the target data contains PII. If yes, identify your GDPR and/or CCPA lawful basis before any data is ingested.
• Check whether the domain has a clickwrap agreement. If you had to agree to terms to access the content, those terms bind you.
• Review the robots.txt file and honor disallow directives. Log your compliance check per domain. That log is your paper trail.
• Assess whether the content is protected /* blocked */creative works, journalism, fiction). If yes, document your fair use analysis with specific attention to market substitution.

During the scrape:

• Respect rate limits. Do not attempt to circumvent IP blocks, authentication tokens, or other technical controls. The moment you bypass a technical measure, you've moved from potential contract dispute to potential CFAA and DMCA exposure.
• Log your user-agent honestly. Use separate user-agents for training crawlers versus retrieval or search functions where applicable.
• Collect only what you need. Data minimization is both a GDPR requirement and good engineering.

Building the dataset:

• Document sources, sizes, modalities, and collection dates. The EU AI Act's mandatory training-data-summary template tells you exactly what format is required.
• If you're using a third-party dataset, trace its provenance. Chain-of-custody liability is real. "We didn't scrape it ourselves" is not a complete defense.
• Filter for illegal content proactively. The AI Act obligation is proactive, not reactive.
• If the dataset includes works from EU jurisdictions, verify that rights holders have not reserved their rights via robots.txt or similar signals. Ignoring a valid TDM opt-out removes your safe harbor.

Ongoing operations:

• Revisit robots.txt compliance regularly. Publishers update directives. A domain that was permissive six months ago may not be today.
• Monitor for HTTP 402 responses. These are now meaningful signals that a site has moved behind a payment layer.
• Maintain separate documentation trails for GDPR and EU AI Act compliance. They overlap but they are not the same.
• If your model is or becomes commercially significant, assess whether a formal licensing agreement is more defensible than continued scraping. In several categories of content, this is no longer a philosophical question. It's a practical risk calculation.

The goal isn't to avoid scraping. Scraping is how AI pipelines get built. The goal is to scrape in ways that are documented, defensible, and honest about what you accessed and how. That's the difference between a production-grade pipeline and a litigation waiting to happen.