What is the difference between an AI search crawler and an AI training crawler?
Three categories, three different consequences when you block one, and several operators who label them explicitly.
What is the difference between an AI search crawler and an AI training crawler?
A training crawler collects pages to build datasets used to train foundation models. A search crawler builds an index that AI products query at answer time, and it is what governs whether a site can be surfaced and cited. OpenAI, Anthropic and Perplexity all publish this split in their own documentation, and Cloudflare classifies the same bots into categories it calls AI Crawler, AI Search and AI Assistant. There is a third category worth knowing: the user-triggered fetcher, which loads a page because a person asked a question in that moment. Blocking a training crawler and blocking a search crawler produce entirely different outcomes.
The three categories, defined by the operators themselves
Training. The crawler gathers content that may end up in a dataset used to train a model. Anthropic describes ClaudeBot as "collecting web content that could potentially contribute to their training". OpenAI puts GPTBot in the same role.
Search. The crawler builds an index that the product consults when someone asks a question, which is the pipeline that produces citations. Anthropic writes that Claude-SearchBot "navigates the web to improve search result quality for users".
User-triggered retrieval. Nothing is being indexed. A person asked something, and the system fetched one page to answer it. Anthropic's framing: "Claude-User supports Claude AI users."
These are not marketing distinctions. Each has its own robots.txt token at most major operators, and each has a different effect when denied.
| Operator | Training | Search | User-triggered |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | none documented | PerplexityBot | Perplexity-User |
| Google-Extended (control token, not a crawler) | Googlebot | Google-GeminiNotebook, Google-Agent | |
| Microsoft | none documented; robots meta tags instead | bingbot | none documented |
| Apple | Applebot-Extended (control token, not a crawler) | Applebot | none documented |
| Meta | meta-externalagent | meta-externalagent (same token) | meta-externalfetcher |
| Common Crawl | CCBot, indirectly (see below) | not applicable | not applicable |
Why blocking the wrong one is expensive
Deny a training token and you are declining to contribute content to a model dataset. Your pages can still be indexed, retrieved and cited by the search side of the same company.
Deny a search token and you have opted out of being surfaced in that product's answers. Perplexity states the reverse case directly: to appear in results, it recommends allowing PerplexityBot in robots.txt and permitting its published IP ranges.
Deny a user-triggered fetcher and you are declining live retrieval when a person asks about you by name. Anthropic warns that disabling Claude-User "may reduce your site's visibility for user-directed web search".
These outcomes are not interchangeable, which is why a single blanket rule aimed at "AI bots" usually costs more than the person writing it intended.
The operators who do not fit the three-way split
Perplexity documents no training crawler at all. Both of its agents carry an explicit disclaimer that they are not used to collect content for training AI foundation models. On its own documentation, Perplexity is search and retrieval only.
Microsoft has no AI training user agent. Bing's crawler article lists bingbot and a set of preview and ads crawlers, and nothing AI-specific. Training control at Microsoft is expressed through robots meta tags instead, where noindex is documented as also meaning do not use the content for training Microsoft generative AI models.
Apple runs one crawler for two purposes. Applebot powers search across Spotlight, Siri and Safari, and the same crawled data may also help train Apple foundation models. Applebot-Extended is the opt-out for the second use and does not crawl anything itself.
Meta collapses the axes entirely. Its documentation says meta-externalagent "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly". One token, two uses, no way to accept one and refuse the other.
Common Crawl is a fourth shape again. It calls itself a non-profit producing an open repository of web crawl data, and does not describe CCBot as either a training or a search crawler. The training relevance is downstream, in what other people do with the corpus.
robots.txt obedience is not uniform, and the gaps are on the user-triggered side
Automatic crawlers are consistently documented as obeying robots.txt. The user-triggered fetchers are where the commitments get soft, and each operator words it differently.
OpenAI on ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." That is not a commitment in either direction.
Perplexity is blunter about Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."
Anthropic makes no such carve-out. Its blanket statement covers all three of its bots without exempting Claude-User, which is a genuine difference between operators rather than a difference in wording.
Google keeps its own three-way split: common crawlers "always obey robots.txt rules when crawling automatically", while its special-case Google-Safety agent is documented as ignoring robots.txt outright.
| Agent | Operator | What the documentation says |
|---|---|---|
| ChatGPT-User | OpenAI | Rules may not apply, because the action is user-initiated |
| Claude-User | Anthropic | No exemption stated; covered by the blanket robots.txt commitment |
| Perplexity-User | Perplexity | Generally ignores robots.txt rules |
| meta-externalfetcher | Meta | May bypass robots.txt rules |
A third party who labels the same bots
Cloudflare assigns every bot it knows about a category, and its taxonomy lines up with the operator documentation rather than cutting across it.
GPTBot, ClaudeBot, CCBot, Bytespider and Google-CloudVertexBot are filed as AI Crawler. OAI-SearchBot, Claude-SearchBot and PerplexityBot are filed as AI Search. ChatGPT-User, Claude-User and Perplexity-User are filed as AI Assistant. Googlebot and BingBot are filed as Search Engine.
That is useful corroboration from outside the companies being described, and it is a convenient way to sanity-check a block list before you deploy it.
What is actually established, and how
Sorted by how strong the evidence is, not by how convenient it is.
| Claim | Basis |
|---|---|
| OpenAI, Anthropic and Perplexity each publish separate robots.txt tokens for automatic crawling and for user-triggered retrieval. | Documented by the platform |
| Perplexity documents no training crawler, and both of its published agents state they are not used to collect content for training foundation models. | Documented by the platform |
| Microsoft documents no AI-training user agent and expresses training control through robots meta tags such as noindex, noarchive and nocache. | Documented by the platform |
| Apple uses a single crawler, Applebot, for both search and training, with Applebot-Extended as a non-crawling control token for the training use. | Documented by the platform |
| Meta documents one agent, meta-externalagent, covering both model training and product indexing, with no separate control for each use. | Documented by the platform |
| Cloudflare categorises the same bots as AI Crawler, AI Search, AI Assistant and Search Engine, matching the split the operators publish. | Documented by the platform |
| Whether a given AI product actually restricts itself to the crawler category it documents, once the data is inside the company. | Not publicly documented |
What nobody can currently tell you
Stated because the alternative is implying a certainty that does not exist.
The Meta documentation could not be fetched directly. Every attempt returned HTTP 400, and the content was read through a rendering proxy, so the Meta quotes here sit one confidence step below the rest and the full Meta user-agent strings are deliberately not published.
No operator in this set documents reading a publisher-supplied llms.txt file. Several publish an llms.txt for their own documentation, which is the opposite direction. This is an absence of documentation, not a documented refusal.
Anthropic publishes no full user-agent strings, only the three robots.txt tokens, so there is no primary source for the literal ClaudeBot request header.
The tokens anthropic-ai and claude-web appear nowhere in Anthropic current documentation. We could find no Anthropic page that describes, deprecates or retires them, so their status is genuinely unknown rather than confirmed dead.
Common Crawl does not state that CCBot exists to feed model training. Whether a specific corpus release ends up in a specific model is decided by whoever downloads it, and is not visible from Common Crawl documentation.
None of these operators publishes a last-updated date on every relevant page. Perplexity and Bing expose no revision date at all, so their content can only be dated to our retrieval date.
What people get wrong about this
-
AI crawlers are one category, so one rule handles them.
What is actually the case
At least three categories exist with separate tokens at most major operators, and denying all of them removes search and live retrieval as well as training.
-
Every AI bot obeys robots.txt.
What is actually the case
Automatic crawlers are documented as obeying it. Perplexity-User is documented as generally ignoring it, meta-externalfetcher may bypass it, and OpenAI says rules may not apply to ChatGPT-User.
-
Perplexity trains models on the pages it crawls.
What is actually the case
Perplexity documentation attaches an explicit training disclaimer to both of its agents and names no training crawler anywhere on that page.
-
CCBot is a training crawler.
What is actually the case
Common Crawl describes CCBot as building an open repository of crawl data. It makes no training claim of its own, though third parties commonly use the corpus that way.
-
If you only wrote Googlebot rules, you only addressed Google.
What is actually the case
Apple documents that when robots instructions mention Googlebot but not Applebot, Applebot follows the Googlebot instructions.
Questions we get asked constantly
If I block only training crawlers, can AI products still cite me?
Yes, on the operators that separate the two. The search tokens, OAI-SearchBot, Claude-SearchBot and PerplexityBot, are untouched by a training-only block list.
Which category does Googlebot fall into?
Google documents Googlebot as its search crawler, and Google states that crawling preferences for it affect Search including Discover, Images, Video and News.
Is there any operator where you cannot separate training from indexing?
Meta, on its published documentation. A single agent covers both uses and no per-use control token is offered.
Does an AI assistant fetch count as a crawl?
The operators treat it as a distinct thing. It is one page requested at a person's direction rather than systematic crawling, and it carries different robots.txt commitments.
Where this came from
Every factual claim above traces to one of these. Each entry says what it supports and the date it was read, because platform documentation changes without notice.
-
OpenAI Bots — OpenAI
OpenAI runs GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot, and states robots.txt rules may not apply to the user-initiated agent.
Primary source · read 2026-09-03
-
Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
Anthropic documents three robots covering model development, web search and retrieval at a user direction, with no robots.txt exemption for the user agent.
Primary source · read 2026-09-03
-
Perplexity Crawlers — Perplexity
Perplexity documents no training crawler, disclaims training use for both agents, and states Perplexity-User generally ignores robots.txt.
Primary source · read 2026-09-03
-
Google crawlers and fetchers overview — Google
Google splits its clients into common crawlers, special-case crawlers and user-triggered fetchers, and only the common crawlers always obey robots.txt when crawling automatically.
Primary source · read 2026-09-03
developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers
-
Robots meta tags and attributes that Bing supports — Microsoft
Microsoft expresses generative AI training control through robots meta tags rather than a dedicated AI user agent.
Primary source · read 2026-09-03
www.bing.com/webmasters/help/robots-meta-tags-and-attributes-that-bing-supports-5198d240
-
About Applebot — Apple
Applebot serves both search and training, Applebot-Extended is a non-crawling training opt-out, and Applebot follows Googlebot rules when no Applebot rules exist.
Primary source · read 2026-09-03
-
Meta web crawlers — Meta
meta-externalagent covers both training and indexing under one token, and meta-externalfetcher may bypass robots.txt. Read through a rendering proxy, not a direct fetch.
Primary source · read 2026-09-03
developers.facebook.com/docs/sharing/webmasters/web-crawlers/
-
CCBot — Common Crawl
Common Crawl describes CCBot as building an open repository of crawl data and makes no training or search claim for it.
Primary source · read 2026-09-03
-
AI Crawl Control: bots reference — Cloudflare
Cloudflare categorises these bots as AI Crawler, AI Search, AI Assistant and Search Engine.
Primary source · read 2026-09-03