What is llms.txt?
A markdown index for AI agents, proposed by one person in 2024, rewritten in August 2026, and still ratified by nobody.
What is llms.txt?
llms.txt is a proposed markdown file that gives AI agents a short, curated index of links to a site's most useful pages. Jeremy Howard of Answer.AI published the proposal on 2024-09-03, and the specification at llmstxt.org was revised to version 2 in August 2026. It is a proposal from one author at one private company, described in the document itself as an informal overview open for community input. It is not an IETF RFC, a W3C Recommendation, or a WHATWG living standard, and no major AI platform documents its crawler consuming a third-party llms.txt.
What the specification actually requires
The format is a markdown file with a fixed opening order. An H1 naming the project or site is the only required element. Everything after it is optional.
After the H1 may come a blockquote holding a short summary, then any markdown content that does not use headings, then any number of H2 sections containing lists of links. Each link may be followed by a colon and a note explaining what is behind it.
The rationale is token economy. The spec argues that HTML wraps information in navigation, ads and JavaScript, that context windows remain too small for most websites in their entirety, and that every wasted token costs time and money.
That is why the file is meant to stay short. In the spec's own phrasing, the detail lives behind the links and is fetched only when needed.
Version 2 exists, and most circulating advice predates it
The document at llmstxt.org now carries the header "The /llms.txt file, v2" and is dated August 2026. It also keeps the original 2024-09-03 byline, so a reader can easily mistake the current page for the first release. There is a separate changes page.
Three v2 revisions matter to anyone implementing it. First, a file may sit at the origin root or at any subpath, and it covers the URLs beneath its own path. Where more than one file applies, agents should use the most specific.
Second, v2 endorses serving a clean markdown twin of each page, either by appending .md or by replacing the extension. Version 1 permitted only the appended form.
Third, and this was the headline addition, v2 defines discoverability through link relations. A rel="alternate" type="text/markdown" pointer names a page's markdown version, and rel="describedby" names the llms.txt covering it. Both work as HTML link elements or as HTTP Link response headers, so a CDN can advertise them without any page being edited.
The author also considered and rejected the RFC 8615 /.well-known/ route, on the grounds that well-known URIs live only at the origin root while many authors control only a path on a shared host. That rejection was deliberate, not an oversight.
Where it sits against files that are standardised
The most common error in writing about this topic is calling llms.txt a standard. The document never uses that word about itself. It says "proposal", and its closing section states that the specification is open for community input.
There is no working group, no draft process, no editors beyond the author, and no normative versioning. Version 2 was issued by the same person based on what he learned from two years of adoption. That is a defensible way to run a proposal and it is not a standards process.
None of this makes the file fake. Plenty of load-bearing web conventions were never ratified. The accurate framing is adoption without ratification, and the adoption is heavily weighted toward publishing rather than consuming.
| File | Formal status | Governance |
|---|---|---|
| robots.txt | IETF RFC 9309, Standards Track, September 2022 | IETF, multi-vendor |
| sitemap.xml | Published protocol at sitemaps.org, not an RFC | Multi-search-engine agreement |
| llms.txt | One author's proposal, v2 dated August 2026 | One person, one company |
llms-full.txt is not in the specification, in either version
This is the finding most likely to surprise a practitioner, and it is not ambiguous. A search of the full v2 source at llmstxt.org/index.md for the string "llms-full" returns zero matches. The file has never been part of the proposal.
What version 1 did define was llms-ctx.txt and llms-ctx-full.txt, which were artifacts generated by a command line tool that expanded an llms.txt into a single context. Version 2 deleted that tooling from the proposal, and with it the special meaning of the Optional section.
So llms-full.txt is a platform convention rather than a specified artifact. It appears to originate with documentation hosts, principally Mintlify. It is genuinely widespread: docs.anthropic.com/llms-full.txt and developers.openai.com/llms-full.txt both returned HTTP 200 on 2026-09-03, and Anthropic's own llms.txt points readers at it.
It also cuts against the reasoning behind llms.txt. The proposal exists because context is scarce and tokens cost money. A single concatenated dump of an entire documentation set is the opposite of a small curated index, and the "full" naming encourages people to treat one as a longer edition of the other.
We should be direct about our own position here. This site publishes both files. The llms-full.txt we serve is a convention we adopted from platform practice, not something the specification asked for, and anyone citing the spec as authority for that file is citing a document that does not contain it.
What is actually established, and how
Sorted by how strong the evidence is, not by how convenient it is.
| Claim | Basis |
|---|---|
| The proposal was authored by Jeremy Howard of Answer.AI and first published 2024-09-03. | Documented by the platform |
| The specification at llmstxt.org is at version 2, dated August 2026, while retaining the original 2024 byline. | Documented by the platform |
| An H1 naming the project or site is the only required element of the format. | Documented by the platform |
| Version 2 allows the file at any subpath, with the most specific applicable file taking precedence. | Documented by the platform |
| Version 2 adds rel="alternate" and rel="describedby" discoverability, usable as HTTP Link headers. | Documented by the platform |
| The string llms-full does not appear anywhere in the v2 specification source. | We measured this |
| Version 2 removed the context-expansion tooling that defined llms-ctx.txt and llms-ctx-full.txt. | Documented by the platform |
| llms-full.txt is served at HTTP 200 by both Anthropic and OpenAI documentation sites. | We measured this |
| The specification is not an IETF RFC, a W3C Recommendation, or a WHATWG living standard. | Documented by the platform |
What nobody can currently tell you
Stated because the alternative is implying a certainty that does not exist.
How many sites publish an llms.txt file worldwide. The spec asserts "thousands"; Ahrefs found roughly 38,000 valid files inside its own 137,210-domain sample and warns that its customers skew technical, so its adoption figure is an upper bound. Those two numbers measure different populations and no reliable web-wide count exists.
Whether version 2 will be adopted, ignored, or superseded. It was published in August 2026 by a single author with no ratification process, and there is no mechanism that would tell you whether implementers have moved to it.
Whether the new HTTP Link header discoverability is read by anything. It is the most implementable part of v2 and we found no platform documentation acknowledging it.
What people get wrong about this
-
llms.txt is an emerging web standard.
What is actually the case
The specification calls itself a proposal and an informal overview, open for community input. There is no IETF, W3C or WHATWG process behind it and no governance beyond its author. The word "standard" comes from agency copy, not from the document.
-
The spec defines both llms.txt and llms-full.txt.
What is actually the case
It defines only llms.txt. The string llms-full appears zero times in the v2 source. Version 1 defined llms-ctx.txt and llms-ctx-full.txt as tool-generated files, and version 2 removed that tooling outright.
-
The file has to live at the root of your domain.
What is actually the case
Since version 2 it may live at any subpath and covers the URLs under that path, so /docs/llms.txt governs everything beneath /docs/. Where several files could apply, the most specific one wins.
-
A bigger llms.txt is a better llms.txt.
What is actually the case
The stated design goal is a file small enough to fit in context, with detail behind the links. Size works against the only rationale the proposal offers for its own existence.
How to check this on your own site
You should not have to take our word for any of it.
- Fetch https://llmstxt.org/index.md rather than the rendered page. The markdown source is the cleanest form of the spec and it carries the v2 header.
- Search that source for "llms-full" before believing any article that says the spec defines it.
- Read https://llmstxt.org/changes.md to see what version 2 removed, which is the part almost no commentary covers.
- Request your own /llms.txt with curl and confirm it returns 200 with a text/plain or text/markdown content type, and that the first line is an H1.
Questions we get asked constantly
Who created llms.txt?
Jeremy Howard, co-founder of Answer.AI. The proposal appeared on 2024-09-03 as a post on answer.ai and as the site llmstxt.org at the same time.
Is llms.txt an official standard?
No. It has no RFC number, no W3C status and no WHATWG status. The specification describes itself as a proposal and an informal overview that is open for community input.
What format is the file?
Markdown, not plain-text directives and not XML. That is a real difference from robots.txt and sitemap.xml, and it is deliberate: the file is meant to be read by a language model, not parsed by a rule engine.
Does llms.txt control whether AI companies train on my content?
No. It expresses no permissions at all. Training and crawling controls are expressed through robots.txt and through each platform's own opt-out mechanisms.
Where this came from
Every factual claim above traces to one of these. Each entry says what it supports and the date it was read, because platform documentation changes without notice.
-
The /llms.txt file, v2 (markdown source) — llmstxt.org
The required structure, the token-economy rationale, subpath scope, markdown twins, link-relation discoverability, the rejection of /.well-known/, the self-description as a proposal open for community input, and the absence of any mention of llms-full.txt.
Primary source · read 2026-09-03
-
Changes to the llms.txt proposal — llmstxt.org
That version 2 removed the context-expansion tooling and the special meaning of the Optional section, which had defined llms-ctx.txt and llms-ctx-full.txt.
Primary source · read 2026-09-03
-
The /llms.txt file (original proposal post) — Answer.AI
Authorship by Jeremy Howard and the original publication date of 2024-09-03.
Primary source · read 2026-09-03
-
RFC 9309: Robots Exclusion Protocol — IETF
That robots.txt is an IETF Standards Track document published in September 2022, which is the contrast used in the status table.
Primary source · read 2026-09-03
-
llms-full.txt (documentation) — Anthropic
That llms-full.txt is deployed in the wild at major vendors, returning HTTP 200 on 2026-09-03.
Primary source · read 2026-09-03
-
llms-full.txt (developer documentation) — OpenAI
A second vendor deployment of the unspecified llms-full.txt convention, returning HTTP 200 on 2026-09-03.
Primary source · read 2026-09-03
-
Sitemaps XML format — sitemaps.org
That sitemap.xml is a published multi-search-engine protocol rather than an IETF RFC, the middle row of the status table.
Primary source · read 2026-09-03