Robots.txt and AI Crawler Access Control

Robots.txt started life as a handshake, not a law.

That world is gone. The crawler population hitting a typical site today is a structural mismatch between a 1994 tool and a 2026 problem, and fixing it starts with a new way of thinking about what a crawler actually is.

The one distinction that makes every robots.txt decision in 2026 legible

Every AI crawler hitting a website is doing exactly one of three jobs. It is either collecting pages to train a model, indexing pages so an AI search product can cite them in an answer, or fetching a single page right now because a live human being just asked an assistant a question. Those three jobs are three different commercial relationships, and treating them as one undifferentiated "AI bot" category is the single biggest mistake in crawler management today.

Vendors have already built their systems around this exact split. The decision isn't theoretical. Anthropic runs three distinct bots: ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for real-time fetches triggered by an actual user, each one controllable on its own. Amazon, Google, and Apple all draw the same line between training access and search or assistant access.

Most sites get this wrong by reaching for the broadest possible hammer. Roughly 71% of top news publishers currently block at least one retrieval or search bot while only intending to block training. That single statistic is the whole argument for treating this as a three-way split rather than an on/off switch: block training, allow search, and handle user-triggered fetchers as their own separate category with their own rules.

The bot-by-bot map

Once the purpose-split framework is in place, the actual configuration work gets a lot less mysterious, because the defensible default for 2026 follows one consistent pattern across almost every vendor: block the training bots, allow the search and retrieval bots, and reserve a small amount of extra attention for the handful of crawlers that don't sit neatly in either bucket.

On the training side, GPTBot is OpenAI's model-training crawler, and disallowing it signals that content shouldn't feed foundation model training while leaving ChatGPT search visibility completely untouched; OpenAI publishes its IP ranges at openai.com/gptbot.json for sites that want to verify traffic rather than trust the user-agent string alone. ClaudeBot plays the same role for Anthropic, and Anthropic's own documentation states that it honors both the standard Disallow directive and the non-standard Crawl-delay directive, and that it will not attempt to bypass CAPTCHAs on sites it crawls; current IP ranges are published at claude.com/crawling/bots.json, and the older anthropic-ai and Claude-Web user-agent tokens are deprecated, so configurations still referencing them need an update. CCBot belongs to Common Crawl, the open dataset that feeds AI training across the industry broadly rather than for one company alone, and it respects robots.txt like the others. All four of these training crawlers get the same default treatment: disallow.

On the search and retrieval side, the calculus flips. OAI-SearchBot handles ChatGPT search indexing, and blocking it means the site disappears from ChatGPT's search citations entirely, even though it's run by the same company as GPTBot. The two need separate rules rather than one blanket OpenAI policy. PerplexityBot handles search indexing for Perplexity, and the company's own recommendation is to allow it for the sake of discoverability. Both of these get the opposite default: allow.

Sitting apart from both categories are the user-triggered fetchers, bots that show up because a specific person asked a specific question right now. These need a separate decision, not the training default and not the search default.

One more number makes the economic case for treating training crawlers more aggressively than anything else in this list. Cloudflare measured Anthropic's ClaudeBot crawling roughly 70,900 pages for every single referred visitor it sent back to a site, at its peak in June 2025, compared to about a 5:1 ratio for traditional Googlebot traffic. That gap between extraction and return is orders of magnitude worse for training crawlers than for search crawlers, and it's the clearest evidence that these two categories deserve opposite defaults rather than a shared policy.

Three bots that do not fit the standard matrix and require extra handling

Three crawlers break the clean training-versus-search binary, and each one breaks it for a different reason: one serves two masters at once, one has a documented compliance record that doesn't hold up, and one publishes no policy.

Googlebot, Applebot, and BingBot are the mixed-use collision, and Cloudflare has stated that when a crawler serves more than one purpose, the most restrictive matching rule wins, naming all three of these explicitly. It was the most consequential side effect of Cloudflare's September 2026 policy change, where sites that flipped on "block training" discovered their traditional search crawlers had been blocked too, because the strictest matching rule always takes precedence. The fix is precision rather than caution: block Google-Extended, the dedicated training token, instead of Googlebot itself, and block Applebot-Extended instead of Applebot.

The thread running through all three of these exceptions is the same: robots.txt can only govern what it can actually see.

What robots.txt cannot enforce

Robots.txt is a request, not a lock on the door.

Bytespider is the clearest example of why this matters.

The trend line makes this more urgent every quarter. TollBit's State of the Bots data shows that about 30% of AI-bot scrapes did not comply with robots.txt permissions in the fourth quarter of 2025, up from roughly 3.3% a year earlier. That's not a rounding error; it's a near-tenfold jump in a single year, and the file's authority is eroding exactly as fast as the bot population is growing.

There's a cost angle here too, separate from rights and citations. AI bots use 15 to 20 times more server resources per request than Googlebot does, and ChatGPT-User wastes more than a third of its requests on pages that return nothing but 404 errors. Uncontrolled crawling isn't a free pass a server can just absorb forever; it's a direct infrastructure bill.

The enforcement stack that actually works runs in a clear order of strength. IP allowlists and blocklists come next, effective for crawlers that publish verified IP ranges, like OpenAI and Anthropic both do, but useless against anything that doesn't. Robots.txt sits at the bottom of that stack in terms of raw enforcement power, but it's also the cheapest fix available and the first one worth shipping, because it costs nothing and takes effect immediately for every crawler that actually honors it.

How the CDN layer began overriding the file

As of 15 September 2026, robots.txt stopped being the only binding constraint on crawler access for a huge share of the web, because Cloudflare rebuilt its defaults from the ground up. The old single "Block AI Bots" toggle is gone, replaced by three separate categories, Search, Training, and Agent, with Training and Agent crawlers blocked by default on any page that carries advertising, while Search remains allowed. This is a platform decision made for publishers rather than one they opted into page by page, and the scope of it is uneven: paid customers running their own explicit configuration were left alone. The blast radius of this change is invisible from a simple dashboard glance and depends entirely on what plan and settings a given site was already running.

The mixed-use collision from the previous section becomes acute right here. A site that flips on "block training" at the CDN level to keep model-training scrapers out can simultaneously block Googlebot, Applebot, and BingBot, because Cloudflare resolves a multi-purpose crawler using the most restrictive matching rule, and all three of those crawlers serve both traditional search and AI-powered surfaces at once.

Cloudflare has also started moving from blocking crawlers to charging them. The Pay Per Crawl model lets a site respond to a bot request with an HTTP 402 Payment Required status instead of a flat block, naming a price, and the crawler pays to proceed. The x402 payment standard, which the Linux Foundation began stewarding in July 2026, underlies this: a server names a price and an accepted asset, the client pays, typically in a stablecoin such as USDC that clears in under a second, and retries the request with proof of payment attached.

Without the major labs that generate most of the bot traffic actually adopting it, publishers who enable the feature will mostly see their content blocked outright rather than monetized, at least for now. It's a system to watch, not a line item to build a budget around yet.

What blocking training crawlers prevents

Roughly 70% of ChatGPT citations came from content on sites that actively block GPTBot. If blocking a training crawler doesn't stop a model from citing your content, what exactly is the block doing?

The honest answer is that a robots.txt training block only governs what happens going forward. The block is a compliance signal rather than a technical barrier, and its entire force depends on whether the crawler reading it chooses to honor it. That is why the WAF layer from two sections ago matters so much for the crawlers that don't.

llms.txt and its place alongside robots.txt

The cleanest way to separate these two files is to remember that robots.txt governs whether a crawler gets in at all, while llms.txt governs how a model finds its way around once it's already inside reading pages. They operate at completely different layers, and confusing one for a substitute for the other is the single most common mistake in this part of the stack.

llms.txt is a Markdown file designed to help AI coding agents and other LLM-powered tools navigate a documentation site efficiently. A bot blocked by robots.txt stays blocked whether or not an llms.txt file exists on the same domain, because the two files are answering entirely different questions.

The practical dividing line is simple. Ship an llms.txt file if the site publishes developer documentation, SDK references, or API guides, because the AI coding tools reading those files represent a real and growing audience worth serving well. Skip it, or at least deprioritize it, for a content site whose actual goal is citation inside AI search answers, because that outcome is controlled entirely by robots.txt and the search crawlers it governs, not by a navigation file aimed at coding agents.

The emerging verification layer: Web Bot Auth

Every rule discussed so far rests on one shaky assumption: that a crawler is telling the truth about who it is. User-agent strings are self-declared. Any scraper on the internet can simply claim to be OAI-SearchBot, and robots.txt has no mechanism to check whether that claim is true.

Web Bot Auth is the effort to fix that, and it's moving faster than the standard process usually allows. The specification was formally adopted by the IETF webbotauth working group on September 1, 2026, as draft-ietf-webbotauth-httpsig-protocol-00. It remains an Internet-Draft rather than a ratified RFC, which means the enforcement layer built around it is already shipping in places before the underlying standard has been fully locked down.

A working configuration and the audit process to keep it current

The baseline configuration for 2026 follows directly from everything above: disallow the training crawlers by name, allow the search and retrieval crawlers explicitly, treat user-triggered fetchers as their own category rather than lumping them in with automated training bots, and push enforcement for non-compliant crawlers to the WAF, where robots.txt has no authority.

A working robots.txt file for this moment looks something like this:

That file needs a WAF sitting behind it to do the work robots.txt structurally cannot. Bytespider gets IP-range blocking at the WAF rather than a robots.txt entry, since no compliance policy exists to make a text-file rule meaningful. Anyone running on Cloudflare should go check the Bot Preference Sync setting directly, since the September 2026 defaults may have applied silently, and the Training and Agent categories deserve a deliberate choice rather than whatever the platform picked on its own.

From there, the configuration needs an actual audit process, not a one-time setup. After any change to the file, confirm that the pages meant to stay discoverable to search and retrieval crawlers haven't been accidentally caught by a broader disallow rule written for something else entirely.

This whole configuration question looks different from the other side of the fence, where teams build AI pipelines that need live, current web data: monitoring competitor pages, feeding fresh content into a model, tracking changes across large sets of URLs. Those teams are reading the exact same signals described in this piece, just from the opposite direction, and a tool that delivers clean, structured content while actually respecting the access rules a site has published is the foundation that kind of pipeline needs to run reliably at scale, without anyone having to hand-manage the raw crawler stack underneath it.