Web crawler
Also called crawler, bot, spider, robot.
What Web crawler means
A web crawler is an automated client that requests pages over HTTP without a person driving each request, typically following links to find more pages. RFC 9309 puts it simply: "Crawlers are automated clients," and notes that search engines run them to recursively traverse links for indexing.
More precisely
Crawlers are distinct from user-triggered fetchers, which retrieve one URL because somebody asked for it in the moment. Operators now split their fleets along three axes, and Cloudflare classifies them the same way: AI Crawler for training, AI Search for indexing that feeds answers, and AI Assistant for user-initiated retrieval. The axes carry different robots.txt commitments, and the user-triggered agents are the ones most likely to say robots.txt may not apply.
Why it matters
A single vendor may operate three agents with three purposes against the same site. Deciding per vendor rather than per purpose is how publishers accidentally trade away answer visibility while trying to decline training.
Example
OpenAI runs GPTBot for training, OAI-SearchBot for search and ChatGPT-User for user requests, and each is controlled by its own robots.txt group.
Where this came from
Definitions here follow the specification or documentation that defines the term, not a paraphrase of another glossary.
-
RFC 9309: Robots Exclusion Protocol — IETF
The definition of crawlers as automated clients, and the description of search engines using them to recursively traverse links for indexing.
Primary source · read 2026-09-03
-
AI Crawl Control: bots reference — Cloudflare
The three-way categorisation of AI bots into AI Crawler, AI Search and AI Assistant, and which operator agents fall into each. Page last updated Apr 23, 2026.
Primary source · read 2026-09-03