Llms.txt and Machine-Readable Site Signals

The web has always talked to machines in a separate language from the one it uses to talk to people. For decades, that language had exactly two words: robots.txt, which tells automated visitors what they may access, and sitemap.xml, which tells them what exists to be accessed. Both were built for one kind of visitor: the search crawler, a system that indexes pages and hands back a ranked list of links for a human to click. AI agents are a different animal. They don't index and return links. They read, reason, and synthesize answers directly. The questions they're asking a website are not the questions a search crawler asks. A crawler wants to know whether it can go somewhere and whether a given page exists. An AI agent wants to know what the site actually means, which parts of it matter most, and whether it can get that information in a format that doesn't burn through its context window just to find out. Neither robots.txt nor sitemap.xml was ever built to answer those questions, and that mismatch is the gap the rest of this piece is about.

What robots.txt governs

robots.txt is a permissions list, not an explanation. It tells automated clients where they can and can't go, full stop, and it has done that job since it was formally codified in RFC 9309 in September 2022, using Allow and Disallow directives tied to specific user-agent strings. Googlebot and Bingbot read it and respect it reliably, which is what it was designed for.

What's changed is who else shows up with a user-agent string. GPTBot, ClaudeBot, and PerplexityBot are now named agents that publishers can explicitly permit or block in the same file, using the same syntax that's existed for years. That makes robots.txt the single most useful lever available today. Every major AI crawler honors it, so setting it correctly is a one-file decision with outsized reach.

What robots.txt still can't do is tell an agent which content matters, offer any structured context about a site's purpose, or draw finer distinctions (allow summarization but not storage, say, or indexing but not training). It's a gate, not a guide. And the gate now has legal weight behind it: the EU's General-Purpose AI Code of Practice, published in July 2025, commits signatories to deploying crawlers that follow RFC 9309. That commitment is voluntary, so it binds only the companies that sign on. But a Hamburg court ruling from December 10, 2025 pushed the stakes further, holding that opt-outs written in plain-language terms of service don't count. Opt-outs need a machine-readable form, such as robots.txt or the TDM Reservation Protocol, though the court stopped short of naming a required list of formats. A file that used to be a polite suggestion to crawlers is becoming, in some jurisdictions, a legal instrument.

What sitemap.xml contributes

sitemap.xml answers the question robots.txt leaves open: what's actually here? It lists the URLs a site wants indexed, often with extras like last-modified dates, change frequency, and priority hints. All of that is genuinely useful for a crawler deciding what to revisit and when.

None of it helps with comprehension. A sitemap can't tell an agent which URL is the canonical product page, which is a paginated archive nobody needs, which is a login wall, and which is the marketing page that exists purely to sell something. They all sit in the list as equals. For a search engine building an index, a complete inventory of URLs is the whole job. For an AI agent trying to understand a product, a codebase, or a company, a flat list of several thousand URLs is close to useless on its own.

The cost of that gap is concrete. An agent trying to make sense of a documentation site by following a sitemap has to fetch and process a pile of noise before it finds anything worth reasoning about. That means wasted context window, added latency, and added cost, and all three get worse as the site gets bigger. Between them, robots.txt and sitemap.xml tell an agent where it can go and what's technically there. Neither tells it what's worth reading. That's the opening llms.txt was built to fill.

How llms.txt is structured

llms.txt is a comprehension file. robots.txt handles permission and sitemap.xml handles inventory, so llms.txt takes on the job neither one does: curating the content an AI system most needs in order to understand a site, in a format built for AI systems to read directly. Jeremy Howard, cofounder of Answer.AI, proposed the format in September 2024. It's a plain Markdown file sitting at a domain's root, and its structure is intentionally small.

A typical llms.txt opens with a single H1 carrying the project or site name, followed by a short blockquote, usually a sentence or two, summarizing what the site does. Below that come H2 sections grouping related links, each with a brief colon-separated description, something like API Reference: Full endpoint documentation. An optional "Optional" H2 at the bottom holds lower-priority pages that an agent working with a tight context budget can safely skip.

Markdown was a deliberate choice, not an accident. It's something a language model can parse natively without a custom schema, it's lightweight to fetch, and it's plain enough that a human maintainer can edit it without special tooling. A companion format, llms-full.txt, takes the idea a step further by concatenating an entire site's key content into one Markdown file, letting a model pull in full context with a single fetch. Neither file is a standard in the sense that the W3C or IETF would recognize. Both are community proposals, and nothing obligates a publisher to write one or a model to read one.

Who reads llms.txt today, and who does not

The single most useful thing to know about llms.txt is who's actually on the other end of it, and it isn't who most people assume. The real readers of llms.txt today are IDE-layer coding agents, not the consumer chatbots people picture when they hear "AI." Mixing those two audiences up leads to the wrong conclusion in both directions.

A coding agent figures out which library or dependency a task depends on, fetches that library's llms.txt, pulls in only the linked pages relevant to the job, and then writes code. In that loop, the file is doing real, measurable work. Consumer AI assistants are a different story. Google has confirmed that Search ignores llms.txt, and OpenAI, Anthropic, Perplexity, and xAI haven't made any public statement that their consumer products act on it.

That raises a fair question: if none of the big consumer assistants read the file, is maintaining it just theater for a standard nobody enforces? Coding agents are precisely the audience that API and SDK publishers are trying to reach, and for that audience, the file earns its keep. The tooling growing up around llms.txt backs this up. Lighthouse now checks for its presence, a sign that infrastructure teams are taking it seriously even without a single model provider confirming production use of it. Adoption tracks that same pattern: large enterprises have been slow to pick it up, while developer-documentation sites adopted it fast, which lines up exactly with where its real readers live.

Why the permission stack does not stop at llms.txt

llms.txt handles one half of the equation: what to read. A separate, fast-growing set of signals governs what an agent is allowed to do with what it reads. That distinction between reading and using is turning into a legal one, not just a technical one.

TDMRep, the Text and Data Mining Reservation Protocol, is built specifically for that second half. It defines two properties, a boolean called tdm-reservation and a URL called tdm-policy pointing to licensing terms, implemented through a file at /.well-known/tdmrep.json, through HTTP response headers, or through HTML metadata. The Hamburg court ruling gives this real teeth: it found that natural-language opt-outs weren't sufficient at the time in question (2021), and that a valid opt-out needed to be machine-readable. The court kept its standard technology-neutral, leaving the door open on whether plain-language opt-outs could count in some future context, but the direction is clear enough.

The EU's General-Purpose AI Code of Practice, published in July 2025, builds this into its compliance language directly: signatories commit not just to following the Robot Exclusion Protocol but to identifying and complying with other machine-readable protocols. That puts TDMRep on the compliance surface for anyone who signs, not just on a list of nice-to-haves. Cloudflare has gone further still. As of July 2026, Cloudflare splits crawlers into three categories (Search, Agent, and Training) and attaches a content-use signal of use=immediate, reference, or full, extending the existing Content Signals convention inside robots.txt. New domains on Cloudflare now default to blocking AI Training and Agent crawlers on any page running ads, while Search crawlers stay allowed, and a pay-per-crawl option is pushing the conversation from a binary allow-or-deny toward something closer to licensing. For anyone building a pipeline that consumes third-party web content, these aren't abstract policy debates. They determine what can legally be ingested, stored, used to train a model, or surfaced in an answer, and getting them wrong is turning into a legal exposure, not just bad manners.

The permission stack at the infrastructure level

Every time an AI agent makes a request to a web server, it runs into this entire stack at once, and the infrastructure underneath that agent decides whether the signals get read correctly, get ignored, or get missed. A well-built agent checks robots.txt first to confirm it's allowed in, reads llms.txt next if it's there to figure out what matters, and only then starts pulling content. Skipping any one of those steps makes the agent's behavior unreliable or noncompliant, whichever matters more for the job at hand.

Newer work on permission manifests for web agents is trying to formalize that exact sequence, treating discovery, parsing, and respecting these signals as one composable stack. A related protocol called terms.txt pushes the idea further into agentic commerce, where an agent might need to negotiate access terms on the fly. That's an early signal of where this is all heading as agents take on more autonomous work.

A plain cost problem compounds this. Fetching raw HTML and trying to extract meaning from it burns far more tokens than fetching clean Markdown would, and the signal layer, llms.txt and llms-full.txt especially, cuts that overhead by pointing an agent straight at content that's already structured. Pipelines that read and act on this signal stack, instead of treating every page on the internet as equivalent raw HTML to be scraped, end up cheaper to run, faster, and more in line with what publishers actually intended. The Model Context Protocol, originally built by Anthropic and open-sourced in late 2024, sits right next to this stack rather than inside it: llms.txt makes a site readable, MCP makes it interactive, and OpenAI adopted MCP in March 2025. Knowing which problem each one solves saves a lot of wasted engineering effort.

Recommendations for Developers Building AI Pipelines

The right move with this whole stack is to match each signal to the audience it actually serves and act accordingly. Setting robots.txt correctly, with named AI user-agents included, is the highest-priority item on the list for any site owner or any developer building on top of third-party content, since every major AI crawler respects it, it now carries legal weight in EU jurisdictions, and getting it wrong either exposes content that was meant to be private or blocks access that was supposed to be open.

Writing an llms.txt makes sense for any site publishing an SDK, an API, or developer documentation, because IDE agents like Cursor, Continue, and Cline genuinely read it, and the effort of putting together a well-structured file is small next to the payoff of showing up correctly inside AI-assisted coding workflows. It makes much less sense as a strategy aimed at visibility inside consumer AI assistants such as ChatGPT or Google AI Overviews, since no major provider has confirmed acting on it there, and large-scale studies so far have found no measurable lift in citations from having one.

TDMRep and the IETF's emerging AIPREF directive deserve attention now even though neither is fully settled, and publishers with training-sensitive content shouldn't wait around for another court ruling before putting machine-readable opt-outs in place. For teams building pipelines that consume web data rather than publish to agents, the stack is something to respect on the way in: reading robots.txt, treating llms.txt as a prioritization hint, and checking TDMRep before storing anything builds a pipeline on sturdier, more defensible ground than one that treats the entire web as undifferentiated HTML. For documentation-heavy sites specifically, llms-full.txt is the fastest win available, since a single concatenated Markdown file of the important content cuts out the multi-fetch overhead of chasing individual links and happens to be the format IDE agents prefer when they want full context in one pass.

The IETF's AIPREF working group is expected to move its specifications toward ratification later in 2026, at which point the permission side of this stack gets a standards-track mechanism with the same kind of staying power robots.txt has had for decades. Developers who understand how the pieces fit together now, rather than waiting for the standard to land fully formed, will be the ones who implement it cleanly when it does.