How is llms.txt different from robots.txt?
They differ in status, in purpose, in format, and in when they are read. The one thing they share is that neither can make anything obey it.
How is llms.txt different from robots.txt?
robots.txt and llms.txt do opposite jobs and sit at opposite ends of the standards spectrum. robots.txt is specified by IETF RFC 9309, a Standards Track document published in September 2022 and co-authored by a Google employee; it is a line-based file that tells crawlers what they may fetch, and it is read at crawl time. llms.txt is a markdown file proposed by one author, with no ratification and no documented consumer; it invites agents toward selected content, and the proposal says it is meant to be used on demand at inference. Calling llms.txt "robots.txt for AI" inverts the function and overstates the status.
The status gap, stated exactly
RFC 9309 is titled "Robots Exclusion Protocol". It runs on the IETF stream, it is Standards Track, it was published in September 2022, and its authors are M. Koster, G. Illyes, H. Zeller and L. Sassman.
Gary Illyes is a Google employee, so Google formally co-authored the specification behind robots.txt. Google has separately and explicitly declined to support llms.txt for Search. That asymmetry conveys the status difference better than any adjective.
llms.txt has no equivalent anywhere. No number, no stream, no working group, no editors beyond the person who wrote it. Version 2 arrived in August 2026 because its author decided it should.
The enforcement point that gets argued backwards
A common framing says robots.txt is enforced and llms.txt is not. That is wrong in a way worth correcting, because it credits robots.txt with power it does not have.
RFC 9309 states plainly that its rules are not a form of access authorization. Compliance with robots.txt is voluntary. A crawler that ignores it is violating a convention, not bypassing a control, and if you need actual enforcement you need authentication, rate limiting or blocking at the edge.
So the real distinction is not enforcement. It is that robots.txt has a ratified specification and universal voluntary adherence from the parties that matter, while llms.txt has neither ratification nor a single documented adherent.
Purpose, timing and shape
robots.txt is exclusionary. It defines what a crawler may and may not fetch, and it is consulted before the fetch happens. Its scope is the origin root and nowhere else.
llms.txt is editorial. It selects a small number of destinations and describes them, and it may point off-site, which no sitemap does. Since version 2 it can live at any subpath and govern the URLs beneath it.
The specification separates them by timing rather than by authority. Its wording is that robots.txt tells automated tools what access to a site is considered acceptable, such as for search indexing bots, while llms.txt information is used on demand, when an agent needs information about a topic while assisting a user. It adds that the expectation was inference rather than training, and that this is how the file has been used.
The format difference is not cosmetic either. robots.txt is line-based directives meant for a parser. sitemap.xml is schema-constrained XML meant for a parser. llms.txt is markdown meant for a language model, which is why it carries prose notes after each link.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Formal status | IETF RFC 9309, Standards Track | sitemaps.org protocol, not an RFC | One author's proposal, v2 Aug 2026 |
| Format | Plain text directives | XML, schema-constrained | Markdown |
| Purpose | Access control | Enumerate indexable pages | Curate an entry path for agents |
| Directionality | Excludes and permits | Exhaustive | Selective and editorial |
| Scope | Origin root only | Site plus declared alternates | Root or any subpath, most specific wins |
| Off-site links | Not applicable | No | Yes |
| When read | At crawl time, before fetching | At crawl and index time | On demand at inference |
| Enforcement | Voluntary, not access authorization | Voluntary hint | No enforcement, no committed consumer |
| Consumer commitment | Honoured by all major crawlers | Honoured by all major search engines | None documented by any platform |
Why the sitemap comparison is the more useful one
If you want an analogy for llms.txt, sitemap.xml is closer than robots.txt. Both are invitations rather than restrictions, both are hints, and neither guarantees anything about what a consumer does next.
The proposal argues that a sitemap cannot substitute for it, on three grounds: sitemaps generally will not list the LLM-readable versions of pages, they do not include URLs to external sites, and the documents they cover will in aggregate be too large to fit in a context window.
The first two are factual. The third is the proposal's own reasoning about context budgets, and it is worth labelling as such rather than as a measured result.
The status difference still holds even in the friendlier comparison. sitemap.xml is a published protocol with multi-search-engine agreement behind it. llms.txt has one author and no agreement at all.
Practical consequence for a site owner
These files do not overlap, so publishing one never substitutes for the other. If you want to allow or restrict a specific AI crawler, that decision lives in robots.txt and in each platform's own opt-out settings. llms.txt has no vocabulary for permissions and expresses none.
Getting robots.txt wrong has immediate, verifiable consequences. Getting llms.txt wrong costs you a file nothing was committed to reading. Spend your care accordingly.
A worked example of what a misconfigured robots.txt actually does, measured on a live site, is in the crawler access study linked below.
What is actually established, and how
Sorted by how strong the evidence is, not by how convenient it is.
| Claim | Basis |
|---|---|
| RFC 9309 is an IETF Standards Track document published in September 2022, titled Robots Exclusion Protocol. | Documented by the platform |
| RFC 9309 authors include Gary Illyes, a Google employee. | Documented by the platform |
| RFC 9309 states that its rules are not a form of access authorization. | Documented by the platform |
| The llms.txt proposal distinguishes itself from robots.txt by timing: crawl-time access rules versus on-demand information at inference. | Documented by the platform |
| llms.txt may link to external sites; sitemap.xml may not. | Documented by the platform |
| Since v2, an llms.txt may live at any subpath and covers the URLs beneath it. | Documented by the platform |
| sitemap.xml is a published protocol at sitemaps.org and is not an IETF RFC. | Documented by the platform |
| No platform documents any commitment to consume a third-party llms.txt, unlike robots.txt which all major crawlers honour. | We measured this |
What nobody can currently tell you
Stated because the alternative is implying a certainty that does not exist.
Whether the on-demand-at-inference model the proposal describes is how any deployed system actually behaves. The spec states an expectation about its own use; no platform has published a retrieval flow confirming or denying it.
Whether the subpath precedence rule added in v2 is implemented by anything. It is a clear rule with no known implementer.
How closely robots.txt compliance holds among the many smaller AI crawlers. Universal adherence is well established for the major ones and is not measured here for the long tail.
What people get wrong about this
-
llms.txt is the robots.txt for AI.
What is actually the case
Wrong on both function and status. One restricts access and is governed by a ratified IETF specification. The other invites attention and is governed by nobody. The proposal itself separates the two explicitly.
-
robots.txt is technically enforced, so it is the serious file.
What is actually the case
It is serious because it is ratified and universally honoured, not because it is enforced. The RFC says outright that its rules are not access authorization. Enforcement requires authentication or blocking, not a text file.
-
You can use llms.txt to keep AI companies from training on your content.
What is actually the case
The format has no directives, no allow, no disallow and no permission vocabulary of any kind. Training controls live in robots.txt tokens and in each platform's own opt-out mechanisms.
-
llms.txt replaces your sitemap.
What is actually the case
They answer different questions. A sitemap enumerates everything indexable; an llms.txt deliberately selects a few destinations and can point off-site. Search engines have committed to reading one of them.
How to check this on your own site
You should not have to take our word for any of it.
- Fetch your own robots.txt from the live origin and diff it against the file in version control. A CDN can inject managed rules that never appear in your repository.
- Read RFC 9309 section by section rather than a summary of it. It is short, and the access-authorization sentence is the one most often paraphrased away.
- Confirm your robots.txt is at the origin root. Unlike llms.txt under v2, a robots.txt at a subpath governs nothing.
- Check that your llms.txt contains no attempt at allow or disallow syntax. If it does, someone has confused the two formats.
Questions we get asked constantly
Do I need both files?
You need robots.txt if you want any crawler policy at all, because that is the file every major crawler reads. llms.txt is optional and nothing has committed to reading it.
Can llms.txt block a crawler?
No. It has no directive vocabulary. It is a list of links with notes, and a crawler that reads it gains suggestions rather than instructions.
Which file do AI crawlers actually obey?
robots.txt. OpenAI, Anthropic and Perplexity all document robots.txt tokens for their crawlers. None of the three documents reading a third-party llms.txt.
Is llms.txt closer to a sitemap than to robots.txt?
Yes, in function. Both are invitations rather than restrictions. The difference is that sitemap.xml has multi-search-engine agreement behind it and llms.txt has one author.
Where this came from
Every factual claim above traces to one of these. Each entry says what it supports and the date it was read, because platform documentation changes without notice.
-
RFC 9309: Robots Exclusion Protocol — IETF
Standards Track status, September 2022 publication, the author list including Gary Illyes, and the statement that the rules are not a form of access authorization.
Primary source · read 2026-09-03
-
The /llms.txt file, v2 (markdown source) — llmstxt.org
The proposal's own comparison to robots.txt by timing and purpose, its inference-not-training expectation, its three arguments that a sitemap cannot substitute for it, and the v2 subpath precedence rule.
Primary source · read 2026-09-03
-
Sitemaps XML format — sitemaps.org
That sitemap.xml is a published multi-search-engine protocol, XML-based and schema-constrained, and not an IETF standard.
Primary source · read 2026-09-03
-
Top ways to ensure your content performs well in AI experiences — Google Search Central
That Google documents no additional requirements to appear in AI Overviews or AI Mode, and no need to create machine-readable or AI text files. This is the basis for the statement that Google, whose employee co-authored RFC 9309, has declined to adopt llms.txt.
Primary source · read 2026-09-03
-
Overview of OpenAI crawlers — OpenAI
That OpenAI documents robots.txt tokens, user agents and IP ranges as the site-owner control surface for its crawlers.
Primary source · read 2026-09-03