Ethical and Legal Boundaries of Web Scraping for AI

A developer reads one headline that says scraping public data is legal, citing hiQ v. LinkedIn. A week later, the same developer reads about Anthropic paying out in a copyright settlement for doing something that sounds a lot like the same thing. The two facts seem to contradict each other, and that confusion is the whole problem with how people talk about scraping law. Legality here doesn't hinge on the act of scraping. It hinges on four separate factors stacked on top of each other: what you collect, how you access it, which jurisdictions apply, and what you do with the data once you have it, and each of those four shifts your exposure on its own. Scraping isn't some fringe tactic either. Every major language model in production today was built substantially on scraped web data, so this is the core supply line for the entire industry, not a workaround a few teams use. What changed is scale. AI crawlers now hit the web hard enough that the practice became politically visible in a way it simply wasn't when search engine bots were the only ones out there, and that visibility is why courts, legislators, and regulators are all moving at once. The rest of this piece walks through the four factors in order, because each one changes your risk independently of the others.

What You Collect: Data Type and Baseline Risk

Before you think about how you're scraping or where your servers sit, figure out what category the data falls into, because that alone sets a floor under your legal risk that nothing downstream can erase. Government records and content released under open licenses like Creative Commons or MIT are at the bottom of the risk scale. They're explicitly meant to be public, though commercial use can still come with license terms attached, so "open" doesn't mean "no strings". Public business data, things like product prices, product specs, or basic company information, is a notch up: it's the most defensible category almost everywhere, but a site's terms of service can still box in what you're allowed to do with it. User-generated content such as reviews, forum posts, and social media posts runs moderate to high, and the issue isn't really legal technicality so much as plain consent. That content was written for other people to read, not for a training pipeline to ingest, and that mismatch creates a structural problem no amount of clever legal reasoning fully resolves. News articles and other published editorial work carry high risk right now, since publishers are actively suing and bolting AI opt-out clauses onto their terms. Copyrighted creative work, books, music, visual art, is at the very top of the risk ladder, and the Anthropic settlement put a real number on just how expensive that category can get. Personal data, names, emails, locations, behavioral profiles, is also at the top, and GDPR, CCPA, and a growing stack of state laws apply whether or not the data was sitting out in plain view on the open web.

There's a distinction inside this hierarchy that trips up a lot of teams who assume data use doesn't affect data risk. Courts treat scraping for straightforward business intelligence, say, watching a competitor's prices, differently than scraping to train an AI model, and the reason comes down to how a judge thinks about transformation. An AI model trained on a news site's archive that can then generate news-like summaries is competing with the very source it learned from, and that weakens the argument that the use was transformative. Copyright law protects creative expression far more than it protects plain facts, so a pipeline pulling technical documentation is standing on much firmer ground than one pulling novels or song lyrics, and that distinction carries into the next section. It doesn't set the ceiling, and the next three factors determine how much higher the risk climbs from there.

How you access it: why the logged-out/logged-in line keeps winning in court

Once you know what category your data falls into, the next question courts care about most is how you got it, specifically, whether you had to get past a login wall to do it. This single variable has predicted legal outcomes more reliably than almost anything else in the scraping world, and the case that proves it is hiQ v. LinkedIn. Most developers only remember the first half of that story, and they forget that the case didn't end there. In 2022, the Ninth Circuit ruled that scraping pages anyone could view without logging in likely does not violate the federal anti-hacking statute at issue, because there was no gate to get past and no unauthorized access to speak of. That sounds like a clean win for scrapers, and for a narrow legal theory, it was. But the case didn't end there. It closed with hiQ agreeing to a $500,000 consent judgment and a permanent injunction against it: a company that won on CFAA grounds still lost the war, because contract law, copyright law, and other legal theories run on entirely separate tracks from the CFAA. That full arc is the single most important data point in this entire field: winning one legal argument does not mean winning the case.

The rule that settled out of all this is fairly clean. Scraping logged-out, public pages holds up reasonably well under the CFAA. Scraping behind a login, in violation of the terms you agreed to when you created the account, doesn't. Platforms have largely stopped bothering with CFAA claims against logged-out scraping since 2024, and they've redirected their legal energy toward contract claims, copyright claims, and DMCA claims instead. Terms of service themselves split into two flavors that matter here. A browsewrap agreement, a link to terms sitting somewhere on the page you never clicked, rarely holds up against a scraper who never actively agreed to anything. Violating a site's terms isn't a criminal matter but a civil one, and a terms violation still gets you blocked, still gets you sued, and still gets treated by judges as evidence you were acting in bad faith.

The newer wrinkle involves DMCA Section 1201. Platforms are increasingly arguing that rate limiting, CAPTCHAs, and anti-bot systems constitute "technological protection measures" under DMCA Section 1201. If that argument holds up broadly in court, getting around any anti-bot system while scraping could trigger statutory damages even when the underlying data was sitting in public view the whole time, which is a real escalation from where things stood before 2025. Proxies themselves aren't illegal. What you do with them is what matters, but using a proxy to circumvent an IP block that's functioning as an access control pushes the analysis much closer to circumvention territory.

Jurisdictions: The US Patchwork and the EU AI Act's Teeth

A lot of teams assume their legal exposure follows their server location, and that assumption is wrong in a way that can cost real money. Jurisdiction follows the people whose data you collected and the markets where your AI product actually ships, so most production pipelines are operating under several overlapping legal regimes at the same time, whether anyone on the team realizes it or not.

Start with the US, where there's no single federal privacy law and instead a growing patchwork of state statutes. Nearly every one of these state laws includes a carve-out for "publicly available" information, and nearly every team reads that carve-out as broader than it actually is. Data that's visible to the public isn't automatically exempt from these laws just because anyone could technically look at it. There's a second layer sitting on top of the privacy statutes themselves: data-broker registration. If a team sells or transfers scraped personal data, that can legally make them a registered data broker in California, Texas, Vermont, Oregon, and Connecticut. California's version of this, called DROP, carries a compliance deadline of August 1, 2026, and the penalty for missing it runs $200 per request per day. Texas filed the first-ever attorney general lawsuit under a comprehensive state privacy law in January 2025, targeting Allstate and its data subsidiary Arity. And California's AB 2013, effective January 1, 2026, requires any company building generative AI to publish documentation of its training data if the model is offered to California residents, which puts scraped training corpora directly in the regulatory spotlight.

The EU is where the teeth have gotten noticeably sharper. Article 53(1)(c) of the AI Act requires any company providing a general-purpose AI model to maintain a copyright compliance policy that recognizes and respects rights reservations made under Article 4(3) of the 2019 DSM Directive. In practice, that obligation already took effect on August 2, 2025: an AI company that scrapes a site despite its robots.txt file asking crawlers to stay out is violating that obligation. As of August 2, 2026, the AI Office's enforcement powers, including the ability to levy substantial fines, became active, and robots.txt is the machine-readable mechanism rights holders have converged on for expressing TDM opt-outs. There's an exception in EU law for text-and-data-mining done for research, but it doesn't automatically extend to commercial AI training, so teams can't lean on it by default. Every document feeding into a training set now needs a provenance record attached to it, showing where it came from, when it was collected, and whether the source's robots.txt was blocking TDM at the moment of collection. Fixing that retroactively, after the fact, costs far more than building the recordkeeping in from day one. On top of the AI Act sits ordinary GDPR exposure for personal data specifically. European regulators have already fined companies for scraping personal data that was technically visible to the public, Clearview AI took hits in the Netherlands, Italy, and the UK, and those cases settled a basic point: under European law, "public" has never meant "fair game".

Downstream Use as an Independent Liability

Collecting data cleanly doesn't protect you once the data goes into a model. Courts and regulators increasingly treat the legality of the collection and the legality of the training as separate questions, so a pipeline that did everything right on the way in can still create liability based purely on what the resulting model does.

In US copyright law, this downstream question runs through the fair use test, and that test weighs four factors, each capable of swinging a case on its own. How transformative the use is matters a lot: a model trained across huge, varied sources to generate genuinely new output has a stronger claim than one that can spit back something close to the original it learned from. Market effect matters just as much, maybe more: if the trained model ends up competing directly with the source it was trained on, a summarizer that replaces the need to read the original article, that factor weighs hard against fair use. The nature of the underlying work matters too, since factual writing gets thinner copyright protection than creative work, so a model trained on technical manuals stands on steadier ground than one trained on novels. And the amount of each work used counts against a fair use defense when a model trains on entire copyrighted works, though courts have carved out room here before. Google Books survived on the idea that copying whole books can still be fair use when the transformative purpose genuinely requires copying that much.

Legislatures are starting to write rules aimed squarely at this downstream layer rather than at collection. The IAB's draft AI Accountability for Publishers Act, proposed in February 2026, would require AI companies to honor publisher terms of service, including permission and payment, before scraping their content at all. It isn't law yet, but it signals clearly where lawmakers want this to go.

Downstream risk multiplies fastest in agentic use. Picture a single Slack message that kicks off a batch of autonomous agents, and those agents fan out to scrape competitor websites, social profiles, review platforms, and news sites all at once, with no human checking each target against its terms of service or its jurisdiction first. Collection, access method, jurisdiction, and downstream use all stack on top of each other, and agentic scraping is where that stack gets tested the fastest, which is why teams running large-scale crawls increasingly route through managed infrastructure like Olostep, a web-data API built to handle browsers, proxies, and anti-bot layers for developers so the engineering team can focus on compliance decisions rather than keeping requests alive.