Polite Crawling and Robots.txt Compliance
Crawlers that ignore robots.txt are multiplying fast and harder to stop.
Robots.txt compliance and rate-limiting are the plumbing that decides whether an AI crawler keeps running or gets shut off cold. Cloudflare's 2025 numbers put total crawler traffic up 18% in a single year, with GPTBot alone growing 305%. Get the mechanics wrong at that scale and the failure isn't philosophical, it's operational: the data stops flowing, the pipeline goes quiet, and someone spends a Tuesday debugging why a 429 storm took down the ingestion job.
What robots.txt actually specifies — and where the standard stops
The Robots Exclusion Protocol goes back to 1994, when Martijn Koster wrote it up as a way for site owners to wave crawlers off certain paths. It stayed an informal gentleman's agreement for almost thirty years until Google open-sourced its own parser and the protocol got formalized as RFC 9309 in 2022.
Here's what the RFC actually pins down. Rules get grouped under User-agent tokens. Allow and Disallow directives follow a most-specific-match-wins rule, so the longest matching path beats a shorter, more general one. Matching on the product token itself is case-insensitive. And if nothing in the file matches your crawler's name, with no wildcard group either, then no rules apply to you at all.
That last point is the one worth sitting with, because most engineers get it backwards. The instinct is to assume no match means blocked by default. No match actually means wide open. Anyone building a parser who codes the safe-default assumption has built the wrong parser.
What the standard leaves out matters just as much. Crawl-delay is a widely used extension, honored by Bingbot and a handful of AI crawlers, but it was never part of the core spec, and Googlebot ignores it outright. AI-specific directives (things like flagging content as off-limits for training but fine for search) are still stuck in draft form at the IETF. Nothing there is a finished standard yet.
The real limitation sits underneath all of it: robots.txt is voluntary, a note taped to the door rather than a lock on it. Nothing about the protocol stops a crawler from reading the file and ignoring it anyway. For anyone building a crawler, that gap between what the spec says and what bots actually do is the starting point for every decision that follows.
How the AI crawler landscape changed the compliance environment
The bot population exploded. Something like 25 major AI crawlers were active by 2024-2025, up from about 12 in early 2023. Publishers noticed fast. ClaudeBot showed up on 2,382 sites in December 2023 and was on 30,000 within four months. PerplexityBot went from 157 sites in January 2024 to 31,000 by April. By 2025, crawlers from ChatGPT, Claude, Meta, and others show up in the robots.txt files of over 560,000 sites. Publishers aren't just tolerating bots anymore; they're naming names, one line at a time.
One shift worth understanding on its own: major providers now split their crawlers into two jobs. One bot trains models, another handles live search and citations. Major providers like OpenAI and Anthropic each run separate crawlers for training versus live search and citations. That split lets a publisher block the training crawler while still letting the search crawler cite them, which sounds like a small technicality until a pipeline mixes the two up and gets locked out of content it should've had access to.
Cloudflare's data shows GPTBot's share of AI crawler traffic climbing from 5% to 30% between May 2024 and May 2025, with Meta-ExternalAgent entering the mix at 19%. The bot population is bigger and more specialized, and the sites on the other end have gotten a lot better at telling one bot from another.
The compliance gap — who honors the protocol and who doesn't
Some crawlers play it straight. Googlebot, GPTBot, ClaudeBot, and PerplexityBot all document their user-agents publicly and say, on the record, that they follow standard directives.
Then there's the gray zone, and it's a real one, not a technicality. When someone asks ChatGPT or Perplexity to summarize a specific page, the provider treats that fetch as a request made by a person, not automated crawling. OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Perplexity says its PerplexityUser agent "generally ignores" robots.txt for the same reason. A site owner who configures robots.txt expecting it to cover all AI traffic has no idea this category even exists, until the server logs say otherwise.
The trend line is heading the wrong direction, and it's not subtle. TollBit's Q1 2025 data shows the share of bots ignoring robots.txt climbing from 3.3% to 12.9% in a single quarter, with 26 million AI scrapes bypassing robots.txt files in March 2025 alone.
The Perplexity case is the one worth remembering, because it turns an abstract problem into a specific, documented one. Cloudflare looked into complaints from customers whose robots.txt disallows weren't being respected. What they found: when the official Perplexity crawler identity got blocked, the requests didn't stop. They switched to a generic browser identity instead, posing as Chrome on macOS. Automated traffic dressed up as a regular human visitor, walking straight through the door it had just been shown out of. Compare that to OpenAI's ChatGPT Agent, which stopped when blocked and signed its HTTP requests using the Web Bot Auth standard instead of finding a workaround. Same industry, two very different postures, and only one survives contact with a skeptical security team.
That gap is exactly why Web Bot Auth matters. User-agent strings are self-declared, so the whole compliance system only binds crawlers that choose to identify themselves honestly. Cryptographic signing changes that: it lets a site actually verify who's knocking, instead of just trusting the label on the door. A crawler that identifies itself correctly and follows the rules is running in a neighborhood where some of its neighbors don't, and that's exactly why the neighborhood is putting up better fences.
How sites are responding — and why non-compliant crawlers get caught
Only about 14% of the top 10,000 domains have written AI-specific rules into their robots.txt as of 2024-2025, but that number is climbing fast, concentrated on exactly the sites with the most valuable content to protect. Only 17% of sites specifically block GPTBot by name, even though 89% have some robots.txt rules configured at all. That gap says most sites have a general policy but haven't gotten around to targeting AI crawlers individually. As of mid-2024, only about 2.98% of the top million sites on Cloudflare's network actively blocked AI bot requests, despite AI bots touching roughly 39% of those same properties. The space between "bots are here" and "sites are responding" is closing fast, and any crawler betting on that gap staying open is betting on the wrong trend.
How do sites catch the non-compliant ones? None of the methods are exotic. Sites use a range of detection approaches — checking request patterns, verifying crawler identities against known ranges, and flagging traffic volumes no human session would produce.
Get caught and the response ranges from an IP ban to a CAPTCHA wall to a flat 429; in some cases, sites just quietly serve misleading content instead of the real page. A common pattern now: block the training crawlers like GPTBot and anthropic-ai, while leaving general crawl access open. A pipeline that misidentifies itself, or just ignores the rules, ends up locked out of content it would otherwise be free to use. Ahrefs looked at 140 million websites and found an average block rate of 5.7% for major AI crawlers. That number understates things badly for any crawler that's already built up a bad reputation.
The legal exposure that non-compliance creates for AI pipeline operators
Robots.txt has no legal force on its own. Commentators have called it a "keep off the grass" sign, and that's about right: a request, not a lock. Circumventing an actual technical barrier is a different story, though, and that's where the Computer Fraud and Abuse Act comes into play. In the EU, robots.txt compliance is increasingly read as part of GDPR and Digital Services Act obligations, and GDPR fines run up to €20 million. That makes this a budget-line risk, not a footnote.
The litigation is already here. Reddit's October 2025 federal lawsuit against Perplexity leans on DMCA anti-circumvention provisions, unjust enrichment, and unfair competition, describing the conduct as "industrial-scale" evasion of technical controls. Encyclopedia Britannica and Merriam-Webster have a suit testing whether an AI company can build a knowledge base on extracted content without asking first. Amazon's complaint against Perplexity cites terms-of-service violations, unauthorized access under the CFAA, and harm to the customer experience.
The hiQ v. LinkedIn ruling established that scraping publicly available data doesn't automatically violate the CFAA. That settled one narrow question, not the whole field, and it doesn't mean scraping is broadly legal under every US statute. What the current wave of lawsuits is really testing is whether ignoring robots.txt on top of violating a site's terms of service changes that calculus. For engineering teams, the exposure is real and the audit trail needs to match the scale of the collection. This risk is already showing up in court dockets against the exact kind of high-volume data collection most production pipelines run every day.
The mechanics of a rate-limiting strategy that keeps a crawler running
Start with Crawl-delay. Where a site publishes one, honor it. Where it doesn't, pick a sensible default instead of hammering the server as fast as the connection allows, because "as fast as possible" is how a crawler ends up on a blocklist by Thursday.
Run two rate limits at once, not one. A global limit protects the crawler's own infrastructure. A per-domain limit protects the site on the other end. Neither substitutes for the other, and treating them as interchangeable is how a burst on one high-traffic domain quietly starves every other job in the queue. As a rule of thumb, no single domain should see more requests per second than a busy human clicking around the site would generate at peak.
Then there's back-off, and this is where most pipelines quietly fall apart. A 429 or a 503 with a Retry-After header is the site explicitly saying "slow down," and the crawler needs to read that signal, not just retry blindly. Exponential back-off with some jitter mixed in stops every paused request from firing again at the exact same instant and re-triggering the block. Treat a 5xx error as temporary, unless it keeps showing up across several retry cycles, at which point it's telling you something else is wrong entirely.
Spread requests out over time instead of firing them in a burst, and where a site publishes a preferred crawl window, use it. Send an honest User-Agent string with a contact email or URL attached, too. A lot of site operators will whitelist, or at least deprioritize instead of ban, a crawler they can actually get in touch with. The payoff for all of this: a crawler that never trips a rate limit ends up faster in aggregate than one that gets throttled or banned every few weeks and has to claw its way back from zero.
Parsing and caching robots.txt correctly at scale
Fetch robots.txt once per domain per crawl session, never once per URL. Re-fetching it on every single request isn't just slow, it's a violation of the spirit of the protocol in its own right. Cache the result and respect whatever Cache-Control or Expires header comes back; if the site doesn't specify one, default to caching for hours, not minutes.
A few matching rules from RFC 9309 trip up a surprising number of homegrown parsers. User-agent matching is case-insensitive, full stop. The most specific path wins by length, so a rule for /api/private/ beats a rule for /api/. No matching group and no wildcard group means no restrictions apply at all; don't default to blocking just because nothing matched, that's the mistake from earlier showing up again at the code level. Allow and Disallow can both apply to the same path, and whichever is more specific wins regardless of which directive it is.
Status codes matter too, and they're easy to get backwards. A 404 on /robots.txt means there are no restrictions at all, crawl away. A 401 or 403 means treat the whole site as off-limits. A 5xx means retry with back-off, since a server error is not the same thing as permission, no matter how convenient that interpretation would be.
The wildcard character in paths, like Disallow: /search?*, never made it into the original 1994 standard, but it's everywhere in practice, and RFC 9309 acknowledges it as common usage now. Any parser that can't handle it misfires constantly. Test against the ugly cases too: empty files, malformed directives, files that are enormous, Unicode showing up in paths. These aren't theoretical edge cases. They show up in production robots.txt files regularly, and a naive parser breaks on every single one.
Identifying your crawler correctly and why it changes how sites treat you
User-agent strings are self-declared, and that cuts both ways. A crawler that misidentifies itself gets none of the benefit of rules written specifically for it, and might get caught by rules meant for an entirely different bot altogether.
A compliant User-Agent string needs three things: a product token matching whatever name the crawler documents publicly, a contact URL or email so a site operator can actually reach a human, and version information where it's relevant. Major compliant crawlers like Googlebot and GPTBot publish their IP ranges and support reverse DNS lookups, so sites can confirm a request really is who it claims to be, rather than a spoof wearing the right name tag. Publishing and maintaining an equivalent IP list is the same commitment, just smaller scale, and it's the kind of unglamorous housekeeping that pays off exactly when a site operator is deciding whether to trust or block.
Web Bot Auth is where this is heading: signed HTTP requests let a site verify identity cryptographically instead of trusting a self-reported string. OpenAI's ChatGPT Agent already does this. Pipelines that adopt it early are sending a signal to every site operator watching the incoming logs, that this one identifies itself honestly. A site that can verify who's crawling it, and reach that operator directly if something goes wrong, has a lot less reason to block first and ask questions never.
