Do AI assistants actually read llms.txt?
Coding agents fetch it. Search grounding shows no sign of it. Nobody has published an outcome study either way.
Do AI assistants actually read llms.txt?
Some things fetch llms.txt, and they are probably not the ones site owners have in mind. An Ahrefs server-log study of 137,210 domains found that 97% of llms.txt files received zero requests in May 2026, while the top individual requesting bots among the rest were Claude-Code and GPTBot. Coding agents accounted for around 10% of fetches and assistant-class bots for about 2%. Separately, as of 2026-09-03, none of Google, OpenAI, Anthropic, Perplexity or Microsoft documents any crawler or assistant of its own consuming a third-party site's llms.txt.
Three different claims that get treated as one
Almost every confident answer to this question rests on a category error. Separate these before reading anyone's evidence.
A: a company publishes an llms.txt for its own documentation. That is a decision about its own site. It tells you nothing about what its crawlers do.
B: a company's documentation describes its crawler consuming other people's llms.txt. This is the only category that matters to a site owner deciding whether to publish one.
C: a company tells site owners whether publishing one is worth their time.
The industry routinely cites A as proof of B. OpenAI, Anthropic, Perplexity and Cloudflare all serving their own llms.txt is category A, and it is the single most misread fact in this topic.
| Company | A: publishes its own | B: documents crawler consuming third-party | C: guidance to site owners |
|---|---|---|---|
| Yes, for Gemini API docs | No mention found | Yes: Search says ignore it. Chrome Lighthouse audits for it. | |
| OpenAI | Yes, extensively | No mention found | No mention found |
| Anthropic | Yes, including llms-full.txt | No mention found | No mention found |
| Perplexity | Yes | No mention found (search incomplete) | No mention found |
| Microsoft | Not established | No mention found (weakest search) | No mention found |
The best available evidence is server logs, and it exists
Ahrefs published a study on 2026-06-15 covering all 137,210 domains in its Web Analytics product that received traffic during May 2026, using bot-analytics log data for requests to /llms.txt.
The headline number, verbatim: "97% of those files received zero traffic in May 2026. Nothing fetched them at all." Of roughly 38,000 domains serving a valid file, about 1,100 got any request whatsoever.
The composition of the requests that did happen is the interesting part. About 96% came from bots, mostly non-AI ones. Named AI tools accounted for 19.5% of fetches, split roughly into coding agents at 10%, training crawlers at 5%, and assistants at 2%. SEO audit tools took 21%, unidentified bots 14%, general web crawlers including Googlebot 13%, and technology profilers such as BuiltWith 11%.
Roughly 12% of all fetches were the industry studying itself. Slackbot requested these files more often than PerplexityBot did.
Claude-Code and GPTBot were the top individual requesters. That matches what the specification claims about its own use: llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials.
The caveats the study's own authors printed, which usually get dropped
Fetching is not reading. The authors wrote that many bots may have fetched the file without ever acting on what is inside, and described their own figures as a ceiling on actual consumption.
The sample is skewed. Ahrefs Web Analytics customers are more technical and more SEO-aware than the web at large, so the study explicitly labels its 28% adoption figure an upper bound.
One more observation from the same work is easy to miss and worth keeping: AI bots did not request /llms.txt on domains that had no such file. Nothing was probing speculatively. The fetches happened where something had already advertised the file's existence.
What Google says, twice, and inconsistently
Google Search Central lists llms.txt under things you can ignore for Google Search. The guidance states that Google Search itself does not use these files, and that maintaining one will neither harm nor help visibility or rankings because Search ignores them.
The sentence that usually gets cropped from that quote is the one immediately after it: Google says it is completely fine to maintain such files for other services or systems that use them. Google is drawing the same line this page draws.
Meanwhile Chrome for Developers ships a Lighthouse audit for llms.txt under its agentic browsing checks, last updated 2026-05-05, describing the file as an emerging convention and noting that without it agents may spend more time crawling a site to understand its structure.
Those two positions are reconcilable if you separate search grounding from browser agents, but Google has not reconciled them in writing. A site owner who reads one page comes away with the opposite impression from one who reads the other. That contradiction is the finding, and pretending one page overrules the other would be inventing a resolution Google has not published.
The Lighthouse audit is also weaker than it sounds. It only flags server errors when fetching the file, and a 404 is marked Not Applicable because providing the file is optional at the moment.
What about the "no AI system uses llms.txt" quote
John Mueller of Google posted on Bluesky on 2025-06-17 that no AI system currently uses llms.txt, adding that consumer chatbots fetch pages for training and grounding but none of them fetch the llms.txt file.
Two things about that quote get lost. It is a social media post, not documentation. And it is from June 2025, and measurement has since overtaken its strongest form: the Ahrefs logs put Claude-Code and GPTBot at the top of the requester list.
His narrower point, that consumer chatbots are not fetching the file for grounding, still holds up well against the data. Quoting the universal version as current fact in 2026 does not.
What is actually established, and how
Sorted by how strong the evidence is, not by how convenient it is.
| Claim | Basis |
|---|---|
| An Ahrefs study of 137,210 domains found 97% of llms.txt files received zero requests in May 2026. | We measured this |
| Claude-Code and GPTBot were the top individual bots requesting llms.txt in that dataset. | We measured this |
| Coding agents made up roughly 10% of fetches; assistant-class bots roughly 2%. | We measured this |
| The study authors state that fetched does not mean read, and call their numbers a ceiling. | Documented by the platform |
| Google Search Central documents that Google Search does not use llms.txt files. | Documented by the platform |
| Chrome ships a Lighthouse audit for llms.txt under agentic browsing, last updated 2026-05-05. | Documented by the platform |
| No documentation from Google, OpenAI, Anthropic, Perplexity or Microsoft describes a crawler of theirs consuming a third-party llms.txt. | We measured this |
| OpenAI's crawler documentation describes robots.txt as the site-owner control surface and mentions llms.txt only as a pointer to its own docs index. | Documented by the platform |
What nobody can currently tell you
Stated because the alternative is implying a certainty that does not exist.
Whether any of these crawlers actually consumes a third-party llms.txt internally. We established only that none of them documents doing so, and absence of documentation is not evidence of non-use. No amount of further documentation reading could close this gap.
Whether the files that were fetched changed any model output. Fetch data exists; nothing published connects a fetch to an answer, a citation or an inclusion.
Microsoft's position in either direction. The search of Microsoft Learn and the Bing Webmaster blog returned nothing relevant and was the least thorough of the five, so this is not established rather than confirmed absent.
Perplexity's full crawler policy. Its technical FAQ page returned a JavaScript shell that could not be read, so its position is less well established than the others.
The claim, circulated from one practitioner's screenshots, that OpenAI polls llms.txt every 15 minutes for freshness. The raw logs were never published and the user agent was not verified against OpenAI's IP ranges.
What people get wrong about this
-
OpenAI and Anthropic support llms.txt, because they publish their own.
What is actually the case
Publishing a file for your own documentation is a publishing decision. Neither company documents its crawler reading anyone else's. Google publishes one for its Gemini API docs while simultaneously documenting that Google Search ignores llms.txt.
-
Nothing reads llms.txt.
What is actually the case
That was the common position in 2025 and the logs no longer support it. Claude-Code and GPTBot topped the requester list in the largest published study. The volume is small and coding agents dominate it, but zero is the wrong number.
-
97% of these files get zero traffic, so the file is useless.
What is actually the case
The statistic measures whether files were requested, not whether they help when something does read them. The same study documents real coding-agent use and warns that its own sample skews technical.
-
Google confirmed llms.txt helps with AI Overviews.
What is actually the case
Google documents the opposite in two separate places, naming llms.txt as something to ignore for Search including its generative features. The only Google surface encouraging the file is a Chrome developer tool audit that checks for server errors.
How to check this on your own site
You should not have to take our word for any of it.
- Grep your own access logs for requests to /llms.txt, grouped by user agent and by day. This is the only measurement that tells you about your site rather than about somebody else's sample.
- Verify any AI user agent you find by reverse DNS or against the platform's published IP ranges. A user-agent string is self-reported and trivially spoofed.
- Compare the request rate for /llms.txt against the request rate for a normal HTML page over the same window. The ratio is more informative than the raw count.
- Check whether the fetch of your llms.txt is ever followed by fetches of the URLs it links to. A fetch with no follow-through is a fetch, not a use.
- Run Lighthouse with the agentic browsing category on a page of your site to see what Chrome actually checks, which is server errors and nothing else.
Questions we get asked constantly
Does ChatGPT read my llms.txt?
OpenAI does not document ChatGPT or any of its crawlers doing so. GPTBot does appear in server logs as a frequent requester of these files, but a fetch is not evidence that the content influenced an answer.
Does Claude read my llms.txt?
Anthropic's crawler documentation does not mention it. Claude-Code was one of the two top requesters in the Ahrefs logs, which is consistent with coding agents using these files to navigate documentation.
Does Perplexity use llms.txt?
Not documented. In the Ahrefs data, Slackbot fetched these files more often than PerplexityBot did.
If a bot fetched the file, did it work?
Unknown. That distinction is the central limit of all current evidence on this subject, and the researchers who produced the best dataset say so themselves.
Where this came from
Every factual claim above traces to one of these. Each entry says what it supports and the date it was read, because platform documentation changes without notice.
-
llms.txt study: server logs across 137,210 domains — Ahrefs
Sample of 137,210 domains for May 2026; the 97% zero-traffic figure; the composition of requests by bot category; Claude-Code and GPTBot as top individual requesters; and the authors' own caveats that fetched does not mean read and that the sample skews technical.
Primary source · read 2026-09-03
-
Optimizing your website for generative AI features — Google Search Central
That llms.txt is listed under things to ignore for Google Search, that Search does not use these files, and the following note that maintaining one for other systems is completely fine. Read as the markdown version Google publishes of the same page.
Primary source · read 2026-09-03
developers.google.com/search/docs/fundamentals/ai-optimization-guide.md.txt
-
llms.txt Lighthouse audit (agentic browsing) — Chrome for Developers
That Chrome ships an llms.txt audit, that it describes the file as an emerging convention, and that it flags only server errors while treating a 404 as Not Applicable.
Primary source · read 2026-09-03
developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt
-
Overview of OpenAI crawlers — OpenAI
That the documented site-owner control surface is robots.txt, and that the single llms.txt mention points to OpenAI's own documentation index rather than describing crawler behaviour.
Primary source · read 2026-09-03
-
Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
That Anthropic documents crawler behaviour and robots.txt controls without any mention of consuming a third-party llms.txt.
Primary source · read 2026-09-03
-
PerplexityBot and crawler guidance — Perplexity
That Perplexity's bots documentation contains no crawler policy regarding third-party llms.txt files.
Primary source · read 2026-09-03
-
John Mueller: no AI system currently uses llms.txt — Bluesky (via Search Engine Roundtable)
The 2025-06-17 statement by a Google employee that no AI system used llms.txt at that time, treated here as an informal social post that later measurement has partly overtaken.
Secondary source · read 2026-09-03
-
Google's llms.txt guidance depends on which product you ask — Search Engine Journal
Secondary reporting that the Search and Chrome positions diverge, corroborating the contradiction described from the two Google documents directly.
Secondary source · read 2026-09-03
www.searchenginejournal.com/googles-llms-txt-guidance-depends-on-which-product-you-ask/575431/