ClaudeBot
What ClaudeBot means
ClaudeBot is Anthropic's training crawler. Anthropic describes it as collecting web content that could contribute to training its generative AI models, and lists it alongside two other agents, Claude-SearchBot for search indexing and Claude-User for fetches a person asked for.
More precisely
Anthropic publishes robots.txt tokens rather than full user-agent strings, so `ClaudeBot`, `Claude-SearchBot` and `Claude-User` are the identifiers to write rules against. Anthropic's bots honor robots.txt and support the non-standard Crawl-delay extension, and source IPs are listed at claude.com/crawling/bots.json. Restricting ClaudeBot is documented as signaling that a site's "future materials should be excluded" from training datasets, and the rules must be repeated on every subdomain.
Why it matters
Blocking ClaudeBot is a training signal only. It does not stop Claude's search indexing or user-directed retrieval, each of which has its own token, and Anthropic warns that blocking its IP addresses instead is counterproductive because it prevents robots.txt from being read at all.
Example
To opt out of training but stay retrievable, disallow `ClaudeBot` and leave `Claude-SearchBot` and `Claude-User` alone.
Where this came from
Definitions here follow the specification or documentation that defines the term, not a paraphrase of another glossary.
-
Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
ClaudeBot is the training crawler, the three separate tokens and their purposes, Crawl-delay support, the published IP list, the per-subdomain requirement, and the warning against IP blocking. Article dated April 7, 2026.
Primary source · read 2026-09-03