Should your website publish an llms.txt file?
Probably yes if you publish documentation. Probably immaterial otherwise. Nobody can honestly tell you it will change an outcome.
Should your website publish an llms.txt file?
Publish an llms.txt if you have documentation that coding agents might navigate, because that is the one use case with measured evidence behind it and the file costs very little to produce. For a small business marketing site, treat it as a cheap and unproven experiment rather than an investment. Google Search states it ignores the file entirely, no platform has committed to reading yours, and as of 2026-09-03 no published study connects publishing an llms.txt to any change in citation, inclusion or visibility. Anyone quoting you a result from it is describing something nobody has measured.
The honest decision, by site type
If your site is software documentation or an API reference, the case is strongest. Coding agents were the largest identified category of AI fetches in the only substantial log study available, and the specification itself says documentation is where these files are used most heavily.
If your site is a normal business site, the file is not going to hurt and is unlikely to be the thing that moves anything. Google states that maintaining one will neither harm nor help visibility or rankings in Google Search.
If somebody is charging you meaningfully for one, that is the point to push back. The file is a short markdown list of links. Its production cost should be roughly its value.
| Question | Status as of 2026-09-03 |
|---|---|
| Does Google Search use it? | Documented: no. Explicitly listed as ignorable. |
| Is it in any standards process? | No. One author, one company, no ratifying body. |
| Do coding agents fetch it? | Observed: yes, at low volume, top requesters in log data. |
| Do consumer assistants fetch it for grounding? | No evidence found in either direction beyond ~2% of a tiny sample. |
| Has any platform committed to reading it? | No. No documentation from any of the five major platforms. |
| Does publishing it change an outcome? | Unknown. No study, no experiment, no methodology published anywhere. |
The evidence gap that should govern the decision
There is fetch data. There is no outcome data. That single sentence is the most important thing on this page and it is the thing vendor content most consistently omits.
We looked for any study, log analysis, or controlled before-and-after showing that adding this file changed a site's citation rate, its inclusion in an answer, or its visibility in any AI system. We found none. Several vendor posts assert the effect; none of the ones we checked published a method.
This is not a claim that the file does nothing. It is a claim that the question has not been answered, which is a different and more useful thing to know. Treating an unanswered question as a negative result would repeat the error in the other direction.
A related trap: even the log studies measure requests, not use. The researchers behind the largest one wrote that fetching does not imply the bot acted on the contents.
If you publish one, do it the way the spec describes
Start with an H1 naming the site, which is the only element the format requires. Add a blockquote summarising what the organisation does in one or two sentences.
Then list the pages you would actually want an agent to read, grouped under H2 headings, each link followed by a colon and a short note about what it contains. Ten to thirty good links beat two hundred indiscriminate ones.
Keep it small. The whole argument for the format is that context is scarce, so a file that has grown past a screen or two has lost the plot.
For a worked example of the shape, OpenAI serves a root index at developers.openai.com/llms.txt that routes to a dozen per-product files, each of which links to .md versions of individual pages. It is close to a model implementation of what version 2 describes.
Consider the version 2 additions, which almost nobody has implemented. Serve a markdown twin of each linked page, and advertise both the twin and the covering llms.txt using rel="alternate" and rel="describedby". These can be set as HTTP Link headers at your CDN, which means they cost one configuration change rather than a template rewrite.
Review it when the site changes. A file full of URLs that now 404 is worse than no file, because the only systems reading it are following the links.
Where our own practice is imperfect
This site publishes both an llms.txt and an llms-full.txt, and we sell AI search optimization, so read what follows with that interest in mind.
The second of those files is not part of the specification. The string llms-full appears nowhere in either version of the proposal. We picked it up from platform convention, the way most publishers did, and it concatenates rather than curates, which runs against the token-scarcity argument that justifies the format in the first place.
We keep it because major documentation platforms serve one and some agents may look for it by name. That is a convention argument, not a specification argument, and it is worth saying out loud rather than filing under best practice.
Our commercial page already puts it plainly: llms.txt is a proposed convention that no major AI company has committed to consuming, and we publish it because it is cheap and it may help, not because anyone has promised to read it. Nothing in this research changes that sentence. The research narrows it: coding agents do fetch these files, and there is still no evidence that fetching them changes what any system says about you.
What to do instead if you only have time for one thing
Confirm that AI crawlers can reach your ordinary pages at all. A CDN default or a managed rule can block agents you never chose to block, and that failure is invisible from inside your repository.
Google's own guidance points the same direction. It advises prioritising effective SEO over what it calls AEO and GEO hacks, and names creating unnecessary AI text files as one of the things to skip.
Publishing this file takes twenty minutes. Making sure the pages it points at are fetchable, accurate and worth quoting is the work that has evidence behind it.
What is actually established, and how
Sorted by how strong the evidence is, not by how convenient it is.
| Claim | Basis |
|---|---|
| Google documents that maintaining an llms.txt will neither harm nor help visibility or rankings in Google Search. | Documented by the platform |
| Google documents that it is completely fine to maintain the file for other services or systems that use it. | Documented by the platform |
| The specification states that these files are used most heavily for software documentation, where coding agents follow them. | Documented by the platform |
| Coding agents were the largest identified AI category fetching llms.txt in the Ahrefs May 2026 log data. | We measured this |
| No study, experiment or before-and-after analysis links publishing llms.txt to a change in any measured outcome. | Not publicly documented |
| llms-full.txt is not defined in either version of the specification. | We measured this |
| Google advises against creating unnecessary AI text files as an optimization tactic for Search. | Documented by the platform |
| An H1 is the only element the format requires; everything else is optional. | Documented by the platform |
What nobody can currently tell you
Stated because the alternative is implying a certainty that does not exist.
Whether publishing the file produces any benefit at all. This is the largest gap in the entire subject: fetch data exists, outcome data does not, and nobody has run the experiment that would settle it.
Whether the coding-agent use case transfers to non-documentation sites. The measured use is concentrated in software docs, and there is no data on whether an agent navigating a plumber's website behaves the same way.
Whether keeping llms-full.txt helps, hurts or does nothing. It is unspecified, widely deployed, and unmeasured, including on this site.
How platform behaviour will change. There is no changelog, no deprecation policy and no announcement channel for any of this, which is why this page carries a 90-day review interval.
What people get wrong about this
-
Publishing llms.txt gets you cited by AI assistants.
What is actually the case
No published evidence supports this. Fetch logs show who requests the file; nothing connects a request to a citation. A vendor asserting a result here is asserting something that has not been measured.
-
You need an agency to build one.
What is actually the case
It is a markdown file with a heading, a summary and a list of links. If a quote treats it as a deliverable of substance, the quote is describing effort that the format does not require.
-
Publishing llms-full.txt is following the specification.
What is actually the case
The specification does not contain that filename in any version. It is a documentation-platform convention. We publish one ourselves and it is convention, not compliance.
-
Google penalises sites for publishing these files.
What is actually the case
Google documents the opposite of a penalty and the opposite of a benefit. Its position is that Search ignores the file, so it changes nothing in either direction.
How to check this on your own site
You should not have to take our word for any of it.
- Request your own file with curl and confirm a 200 response, a sane content type, and no stray HTML wrapper from your host.
- Click through every link in the file, or script it, and confirm none of them 404. Broken entries are the most common defect in published files.
- Watch your access logs for 30 days after publishing, then compare the request count against the fetch rate for a single ordinary page. Set expectations from your own numbers, not from a vendor's.
- Search your site logs for the same period before publishing. Without a before window you cannot say anything about after.
- Ask whoever recommended the file which platform documents consuming it. The honest answer today is none of them.
Questions we get asked constantly
Will publishing llms.txt hurt my SEO?
No. Google documents that maintaining these files neither harms nor helps rankings or visibility in Google Search, because Search ignores them.
How long should the file be?
Short enough to sit comfortably in a context window, which is the format's entire justification. A curated list of the pages that matter, not an export of your sitemap.
Should I publish llms-full.txt as well?
It is not in the specification, so this is a judgement call rather than compliance. We publish one on the grounds that it is cheap and some agents look for the filename, and we have no evidence that it helps.
How often should I update it?
Whenever the pages it references change or move. The only failure mode with real consequences is an agent following a link that no longer resolves.
Is it worth paying for?
Not much. It is a small markdown file. The work worth paying for is making the underlying pages accurate, reachable by crawlers, and worth quoting.
Where this came from
Every factual claim above traces to one of these. Each entry says what it supports and the date it was read, because platform documentation changes without notice.
-
Optimizing your website for generative AI features — Google Search Central
That Google Search does not use these files, that maintaining one neither harms nor helps visibility or rankings, that doing so for other services is completely fine, and the advice to skip unnecessary AI text files as a tactic.
Primary source · read 2026-09-03
developers.google.com/search/docs/fundamentals/ai-optimization-guide
-
llms.txt study: server logs across 137,210 domains — Ahrefs
That coding agents were the largest identified AI category of fetches in May 2026 log data, and the authors' caveat that a fetch does not establish that the file was read or acted upon.
Primary source · read 2026-09-03
-
The /llms.txt file, v2 (markdown source) — llmstxt.org
The required H1, the small-file rationale, the claim that documentation is the heaviest use case, the markdown-twin guidance, and the rel="alternate" and rel="describedby" discoverability mechanism.
Primary source · read 2026-09-03
-
llms.txt (developer documentation index) — OpenAI
A thorough deployed implementation: a root index routing to a dozen per-product files, each linking to markdown versions of individual pages.
Primary source · read 2026-09-03
-
llms.txt (documentation index) — Anthropic
That a major documentation platform publishes both files and points readers from the specified one to the unspecified one, which is the convention this site also follows.
Primary source · read 2026-09-03