Politeness Rules and Rate Limiting in Web Crawlers

Politeness on the web is a fiction everyone agreed to keep telling. Robots.txt is the top layer of that fiction: a plain-text handshake between a website and whatever bot comes knocking. Below that sits the stuff with actual teeth: server rate limits, bot detection, and now infrastructure providers deciding who gets in before a crawler even reaches the front door. Skip past the top layer and the truth shows up fast: the internet runs on most bots not bothering to be rude.

The robots.txt file lives at the root of a domain and lists which user-agents can fetch which paths. It got a formal standard in 2022 (RFC 9309), though the idea dates back to 1994, which in internet years makes it basically ancient scripture. Robots.txt is a request, and no login wall sits behind it, no network block, nothing stopping a crawler from walking past it like a "no soliciting" sign on a door with no doorbell. Disallow and Allow are the only directives fully covered by the standard, controlling which paths a bot should skip. Sitemap tells search engines where the XML map of the site lives. Crawl-Delay asks for a pause between requests, and as the next section makes clear, "asks" is doing all the work in that sentence.

Plenty of sites never bother posting the sign at all. Cloudflare's data puts the share of top 10,000 domains with a robots.txt file at around 37%, meaning more than six in ten of the biggest sites on the internet haven't even written the polite note, let alone locked the door.

How Crawl-Delay works in practice, and why its effect varies by crawler

Crawl-Delay tells a crawler to wait a set number of seconds before its next request, a simple enough idea. It never made it into the core of RFC 9309, so no crawler owes it any obedience, and most of the ones that matter most don't give it any.

Bingbot documents support for it. So do a handful of AI crawlers, including ClaudeBot and CCBot. Googlebot ignores it completely, running its own separate system for managing crawl rate instead. Want to slow Googlebot down? Use Google Search Console settings, or send back HTTP 429 and 503 responses, which Googlebot automatically backs off from.

Run the math and the limits show up fast. A 10-second delay caps a compliant crawler at a few thousand pages a day, fine for a small blog and nowhere near enough for a site with millions of URLs. Some publishers lean into that gap on purpose, setting a long Crawl-Delay for AI training bots to slow the harvest without fully blocking it, landing somewhere between open access and a locked door.

Crawl-Delay only slows the crawlers that choose to obey it. Everything else just keeps coming, at full speed, without so much as a nod toward the number in the file.

The scale of AI crawler traffic that is straining the cooperative model

Crawlers have wanted different things at different points in web history. In the 2000s, they indexed pages so humans could find them. In the 2010s, they scraped for SEO tricks and competitor intel. Now they're harvesting content to train AI models, and it's the sheer volume of that harvest that's breaking the old cooperative model.

GPTBot's raw request volume rose 147% between July 2024 and July 2025. Meta-ExternalAgent rose 843% over that same window. Those aren't rounding errors; that's a different category of traffic showing up at the door. Bytespider, ByteDance's crawler, hit more than 40.4% of all Cloudflare-protected domains as of mid-2025, the most active AI crawler measured, with GPTBot following at 35.5% and ClaudeBot at 11.2%.

The old deal is what's actually collapsing here, and the numbers are worth sitting with instead of skimming past. Search crawlers used to run on an implicit trade: crawl the page, send back a visitor. A traditional search crawler historically crawled around 14 pages for every human referral it generated, a ratio that at least resembled fair exchange. By June 2025, OpenAI's crawl-to-referral ratio sat at 1,700 to 1. Anthropic's sat at 73,000 to 1, according to Cloudflare. Call that what it is: a different transaction altogether, one where the crawler takes and almost never gives anything back in the form of traffic.

Whether AI crawlers actually respect robots.txt

Checking robots.txt and obeying robots.txt are not the same behavior, and that gap is where most of the publisher frustration in this whole story actually lives. Most people assume the two are interchangeable, but they aren't.

Plenty of AI crawlers do check the file, particularly when deciding what goes into a training set or gets cited in an answer, and the reputable ones say so publicly. Checking isn't obeying, though. Anthropic caught public criticism in 2024 over reports it ignored robots.txt rules outright. Bytespider's compliance is spotty enough that site owners who actually want it gone don't bother trusting a Disallow line to do the job; they reach for a WAF rule or an IP block instead. There's a structural reason this keeps happening: no audit trail, no signed credential, no API key proving a bot is who its user-agent string claims to be. Ignore a robots.txt directive and nothing technical stops you, with no fine, no lockout, no consequence built into the protocol itself.

Publishers noticed, and they stopped asking nicely. Between July 2025 and January 2026, the number of sites actively blocking AI crawlers ran nearly seven times higher than the number blocking traditional search crawlers like Googlebot. That reaction looks overdue rather than aggressive once you see the earlier numbers: as of July 2024, only about 2.98% of the top million sites on Cloudflare's network actively blocked AI bot traffic, even though AI bots were hitting roughly 39% of those same sites. Bots touching 39% of sites while only 3% push back is the whole story in two numbers. Robots.txt tells a crawler where it's welcome; it carries no built-in way to stop the ones that don't care, and pretending otherwise is how publishers got caught flat-footed.

How server-side rate limiting and bot detection actually enforce limits

This layer carries actual consequences, where robots.txt only carries a suggestion.

HTTP 429, "Too Many Requests," is the server telling a crawler it blew past its allotted rate. A well-built crawler responds with exponential backoff, waiting longer between each retry instead of hammering the endpoint again immediately. Some APIs hand over headers showing exactly how much quota is left and when it resets, and a crawler worth its salt reads those headers instead of guessing.

HTTP 503, "Service Unavailable," shows up when a server's overloaded. Googlebot backs off automatically when it sees one, and any crawler operating in good faith should copy that behavior rather than treat a 503 like a dare.

Rate limiting splits into two different worlds, and mixing them up is a common mistake. API-based limiting is tied to a token or account: documented, predictable, and when you hit the ceiling, the response tells you exactly why. Website-level rate limiting is usually IP-based, often undocumented, and enforced by bot-detection middleware that owes nobody an explanation. That second world is the murkier one to work against, since there's no quota page to check before getting cut off.

Good crawler design routes around that murk with batching: pull URLs in smaller groups, add delays between batches, run the whole thing through asynchronous task queues, something managed crawling APIs like Olostep handle at the infrastructure level, so several scrapers can work at once without all of them slamming the same endpoint at the same second. Bot detection checks far more than IP addresses now, cross-referencing device fingerprints, TLS fingerprints, behavioral patterns, and header validation, mostly because IP blocking alone stopped working the moment bots started rotating addresses faster than security teams could blacklist them.

How infrastructure providers have moved from passive routing to active enforcement

Cloudflare routes more than 16% of global internet traffic, so its policy decisions ripple across a huge slice of the web whether individual site owners weigh in or not. In July 2025, Cloudflare flipped a switch that mattered: it started blocking AI crawlers by default. That's the first time a major infrastructure provider treated non-human access as something a bot has to opt into, rather than something it gets automatically, and that's the plumbing of the internet deciding the old free-for-all is over.

Alongside that came Pay Per Crawl, a monetization layer Cloudflare rolled out in mid-2025 and still runs in beta. Publishers name a price; AI companies decide whether they're willing to pay it. Skip the toll and the response is a 402, "Payment Required," instead of the page. Call it an attempt to rebuild an economic exchange where one used to exist and quietly vanished, given the referral ratios mentioned earlier.

There's a sharper line now between crawler purposes, and it's the line that actually matters going forward. Retrieval crawlers, fetching content for citations or live answers, generally get let through, since they keep a site visible in AI-generated results. Training crawlers, hoovering up data to bake into a model, increasingly get blocked or metered instead. Google-Extended is the clearest working example: a separate robots.txt user-agent that lets a site opt out of AI training use without touching how Googlebot indexes the site for search. Same company, two different bots, two different sets of rules, which is either clever engineering or a tacit admission that one crawler can't be trusted to represent both jobs honestly.

The standards world is catching up too, slowly. An IETF draft on a Robots Exclusion Protocol User Agent Purpose Extension would let robots.txt itself distinguish crawler purpose instead of relying on separate user-agent strings for every use case. Another draft on vocabulary for expressing content preferences for AI training aims at the same problem from a different angle. Neither is finished business, but both are proof the people writing these standards know the current setup is patched together, not designed.

What well-behaved crawlers at scale actually look like in practice

Strip away the acronyms and the baseline is simple: read robots.txt, honor the Disallow paths, respect Crawl-Delay when the target crawler actually supports it. That's table stakes, not an achievement, and treating it as anything more is how a lot of "AI-friendly" crawler marketing gets away with saying nothing at all.

Handling errors correctly matters more than people give it credit for. A 429 means back off with increasing wait times, not a faster resend. Rate-limit headers, when a server bothers to include them, should shape the pacing before a limit even gets hit. A 503 gets treated as a signal to slow down, full stop, not an invitation to push harder.

Batching and queue design round out the practical side: pull URLs in smaller groups, space batches out, lean on task queues so several scrapers run at once without all converging on the same server at the same moment. Well-run crawlers increasingly separate themselves by purpose too, running distinct user-agents or access paths depending on whether they're retrieving content live or hoovering it up for training, because that's exactly the distinction publishers and infrastructure providers are now building their rules around.

Put the whole stack together and it runs three layers deep. Cooperative signals (robots.txt, Crawl-Delay) sit on top, and only crawlers that choose good behavior follow them, which by this point in the story should read less like a technicality and more like the whole problem. Server-side responses (429s, 503s, IP-based limits, detection middleware) sit in the middle, enforcing pace whether a bot wants to comply or not. Infrastructure-level gates (CDN blocking, pay-per-crawl) sit at the bottom, deciding who gets through the door before any other layer even comes into play.

Building all three layers from scratch for every project is a real engineering cost, and it's not one worth paying more than once. That's the actual argument for reaching toward managed data APIs instead of hand-rolling a robots.txt parser, backoff logic, and retry system for the fifth time this year. Let infrastructure built for exactly this handle the politeness delays and the structured output (clean Markdown, HTML, or JSON), and engineering time goes toward the actual product instead of toward reinventing exponential backoff on a Tuesday afternoon.

The cooperative model is still evolving, and llms.txt is the clearest sign of where it's heading. The proposed Markdown-based standard has been gaining adoption across a growing number of sites. It gives language models structured, inference-time access to a site's content instead of forcing them to scrape a page and guess at its shape. Think of it as a menu handed to the crawlers a site actually wants to feed, alongside another rulebook for keeping the rest of them out.