What does Cloudflare's AI crawler blocking actually do?
A training opt-out expressed as a request, frequently described as something much larger, including once by us.
What does Cloudflare's AI crawler blocking actually do?
Cloudflare's managed robots.txt setting generates a robots.txt file disallowing a fixed list of AI user agents. As documented on 2026-09-03 that list was Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. Every one of those is a training crawler or a training control token. No search crawler appears on the list, and the same generated file sets a Content-signal line reading search=yes, ai-train=no, use=reference. So it is a training opt-out, not a block on AI search visibility, and Cloudflare states that robots.txt compliance is voluntary rather than enforced.
The generated block, exactly as documented
Cloudflare publishes the file it produces, so there is no need to guess at the membership. This is what the documentation showed when we read it.
Two features of that file deserve attention before the list itself. It opens with a wildcard section that allows everything and sets content signals. Then it disallows eight named agents.
The wildcard section reads: User-Agent: *, Content-signal: search=yes, ai-train=no, use=reference, Allow: /. Search is explicitly permitted. Training is explicitly refused.
| Disallowed agent | Operator | Role per operator documentation |
|---|---|---|
| GPTBot | OpenAI | Training crawler |
| ClaudeBot | Anthropic | Training crawler |
| Google-Extended | Training and grounding control token, not a crawler | |
| Applebot-Extended | Apple | Training opt-out control token, not a crawler |
| CCBot | Common Crawl | Open web archive crawler, commonly used downstream for training |
| meta-externalagent | Meta | Single token covering training and product indexing |
| Bytespider | ByteDance | Classified by Cloudflare as an AI Crawler |
| Amazonbot | Amazon | Not covered by the operator documentation reviewed here |
What is not on the list
Read the absences, because they carry the meaning. OAI-SearchBot, Claude-SearchBot and PerplexityBot are not disallowed. Neither are ChatGPT-User, Claude-User or Perplexity-User. Neither are Googlebot or bingbot.
Cross-referenced against the operator documentation, that is precisely the configuration that keeps ChatGPT search, Claude search and Perplexity visibility open while declining to contribute training data.
Whether you want that trade is a real decision with defensible answers on both sides. What it is not is an AI visibility outage.
We described this wrongly the first time
This section exists because we made the error ourselves and it is the most useful thing on the page.
While auditing our own site we found a managed block being served that nobody had committed. Our first account of it said the site was blocking AI crawlers and implied a serious hit to AI search presence.
Having actually read the operator documentation for each agent in the list, that was an overstatement. Every entry was a training crawler. The file was explicitly signalling search=yes. The correct description was a training opt-out applied by default.
The genuine finding survived the correction, and it is narrower and more useful: the robots.txt in version control was not the robots.txt being served, and the only way to see the difference was to request the file over HTTPS and read it. The full incident, including the wrong first reading, is written up as a study.
A request, not a wall
Cloudflare is unusually direct about the limits of what this feature does, and the distinction is worth quoting rather than summarising: "robots.txt compliance is voluntary. The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level. Some crawler operators may disregard your robots.txt directives (instructions like Disallow: /) and crawl your content regardless."
For enforcement, Cloudflare points to a separate product, AI Crawl Control, formerly named AI Audit. That is where allow and block rules for individual crawlers live, along with robots.txt compliance monitoring and a pay per crawl option in private beta.
The two are designed to be used together, one to state the preference and the other to enforce it. Turning on the managed file alone gives you the statement without the enforcement.
How it merges with a file you already have
If your site already returns a robots.txt with an HTTP 200, Cloudflare prepends its managed block before your existing content and serves the combination as one response. If no file exists, it creates one.
That prepend behaviour is why the mismatch is easy to miss. Your own directives are still there, further down, unchanged and looking correct. The added rules sit above them.
There is a related default that catches people who never enabled anything: domains on the Free plan with no robots.txt of their own and no managed file will be served the Content Signals Policy when a crawler requests robots.txt. Cloudflare notes that this policy defines the categories without expressing any specific preference.
The setting itself lives under Security Settings, filtered by bot traffic, described as setting your preference to block training in robots.txt. It is available on all plans.
Content Signals, and the part that is still moving
Content Signals are machine-readable directives that categorise permitted uses rather than blocking agents. Cloudflare defines search as building a search index and returning links and short excerpts, ai-input as feeding content into models for real-time answers, and ai-train as training or fine-tuning models.
The generated policy carries a legal assertion in its header, stating that restrictions expressed via Content Signals are express reservations of rights under Article 4 of the European Union copyright directive of 2019.
A fourth signal, content-use, is described as being tested, with values covering interaction without retention, indexing with excerpts and links, and full summarisation. Managed customers get the middle value. Treat that one as unstable.
One known false alarm: Cloudflare notes that Google Search Console may report a syntax not understood warning for Content Signals, and says it has observed no impact on crawl rates or SEO as a result.
Reading the violation numbers correctly
Cloudflare can report which crawlers ignored your directives, which is genuinely useful, with one caveat that changes how you read the chart.
Violations are not logged in real time. Cloudflare warns that adding a new Disallow rule causes past requests to that path to appear as violations, even though they were compliant at the time they were made.
So a spike immediately after a rule change is an artefact of how the report is computed, not evidence that an operator started misbehaving.
What is actually established, and how
Sorted by how strong the evidence is, not by how convenient it is.
| Claim | Basis |
|---|---|
| Cloudflare managed robots.txt generates and maintains a robots.txt that instructs known AI crawlers to stay away from content. | Documented by the platform |
| The documented managed block disallows Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. | Documented by the platform |
| The managed file sets a Content-signal line with search=yes, ai-train=no and use=reference, and allows all agents by default. | Documented by the platform |
| No search crawler appears in the managed disallow list, so ChatGPT search, Claude search and Perplexity visibility are untouched by it. | Inference, not documentation |
| Cloudflare states that robots.txt compliance is voluntary and directs site owners to AI Crawl Control if they want enforcement. | Documented by the platform |
| When a site already serves a robots.txt, Cloudflare prepends its managed block before the existing content. | Documented by the platform |
| A live site was found serving a 3,079 byte robots.txt containing a managed block and a Content-signal directive, against a 1,243 byte file in its repository. | We measured this |
| How many sites have this setting enabled without their operator knowing. | Not publicly documented |
What nobody can currently tell you
Stated because the alternative is implying a certainty that does not exist.
The membership of the managed disallow list is maintained by Cloudflare and can change without notice. The eight agents above are what the documentation showed on the retrieval date, and that list should be treated as volatile rather than as a stable fact.
Amazonbot is on the list but is not covered by the operator documentation reviewed for this article, so we are relying on Cloudflare classification rather than on a primary Amazon source for its role.
Cloudflare renamed AI Audit to AI Crawl Control recently enough that the rename is itself documented. Third-party writeups using the old name may describe behaviour that has since changed.
The content-use signal is explicitly described as being tested. Its values and its treatment by any crawler could change, and no operator documentation we reviewed says it is honoured.
We cannot verify from outside whether any operator actually honours Content Signals. Cloudflare defines the vocabulary. Compliance is a separate question and none of the crawler operators in this research documented reading these directives.
Our own case is a sample of one site, on one platform, on one day. It shows the default existed and what removing it did. It does not tell you how widespread the configuration is.
What people get wrong about this
-
Turning this on blocks AI crawlers from your site.
What is actually the case
It writes a robots.txt request. Cloudflare states the file does not prevent access at a technical level and points to a separate product for actual enforcement.
-
The managed list hides you from ChatGPT and Perplexity.
What is actually the case
The disallowed agents are training crawlers and training control tokens. The search agents at OpenAI, Anthropic and Perplexity are absent from the list.
-
If your repository robots.txt is correct, your served file is correct.
What is actually the case
The managed feature prepends its rules to whatever you publish, so both files can be individually correct while the served combination says something you never wrote.
-
A jump in reported violations means a crawler started ignoring you.
What is actually the case
Cloudflare warns that violations are not logged in real time, so adding a rule retroactively marks earlier compliant requests to that path as violations.
How to check this on your own site
You should not have to take our word for any of it.
- Fetch the file your visitors and crawlers actually receive: curl -sS https://example.com/robots.txt
- Look for a line reading BEGIN Cloudflare Managed content near the top. Its presence means directives were added above your own.
- Check for a Content-signal line and read its values. The managed default sets search to yes and ai-train to no, which is a training refusal, not a search block.
- List every disallowed user agent and look each one up in its operator's own documentation before deciding anything. The category matters more than the count.
- If you want the managed rules gone, turn the setting off in Cloudflare under Security Settings rather than editing your own file, since your file was never the source of those lines.
- Re-fetch and diff against your repository copy to confirm the served file and the committed file now match.
Questions we get asked constantly
Is this the same thing as AI Crawl Control?
No. Managed robots.txt writes a preference file. AI Crawl Control, previously called AI Audit, provides visibility and allow or block rules that enforce access decisions.
Does it cost anything?
Cloudflare documents managed robots.txt for AI crawlers as available on all plans.
Will it affect my Google rankings?
The only Google token in the list is Google-Extended, and Google states that token does not impact inclusion in Google Search or act as a ranking signal. Googlebot is not disallowed by the managed block.
Should a small business turn it on?
That depends on whether you want your content in training datasets, which is a values and licensing question rather than a visibility one. Decide it deliberately instead of discovering it later in a robots.txt you did not write.
Where this came from
Every factual claim above traces to one of these. Each entry says what it supports and the date it was read, because platform documentation changes without notice.
-
Managed robots.txt — Cloudflare
The exact managed disallow list, the Content-signal defaults, the prepend merge behaviour, the voluntary-compliance warning, and availability on all plans. Page last updated Aug 3, 2026.
Primary source · read 2026-09-03
developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/
-
AI Crawl Control — Cloudflare
AI Crawl Control was formerly AI Audit and provides visibility, per-crawler allow or block policies, robots.txt compliance monitoring and pay per crawl in private beta.
Primary source · read 2026-09-03
-
AI Crawl Control: track robots.txt — Cloudflare
Violations are not logged in real time, so a newly added Disallow rule retroactively flags earlier requests to that path. Page last updated Apr 23, 2026.
Primary source · read 2026-09-03
developers.cloudflare.com/ai-crawl-control/features/track-robots-txt/
-
AI Crawl Control: bots reference — Cloudflare
Bytespider and CCBot are categorised as AI Crawler, while OAI-SearchBot, Claude-SearchBot and PerplexityBot are categorised as AI Search. Page last updated Apr 23, 2026.
Primary source · read 2026-09-03
-
OpenAI Bots — OpenAI
GPTBot is a training crawler and is a separate control from the search token, which is why its presence in the managed list is a training opt-out.
Primary source · read 2026-09-03
-
Google common crawlers — Google
Google-Extended does not affect Google Search inclusion or ranking, so its presence in the managed list carries no Search consequence.
Primary source · read 2026-09-03
developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers