robots.txt
Also called Robots Exclusion Protocol.
What robots.txt means
robots.txt is a plain-text file at the root of a site that tells automated clients which paths they may fetch. It is a real standard: RFC 9309, "Robots Exclusion Protocol," an IETF Standards Track document published in September 2022 by M. Koster, G. Illyes, H. Zeller and L. Sassman.
More precisely
The rules "MUST be accessible in a file named /robots.txt (all lowercase) in the top-level path of the service." Records are grouped by user-agent product token and carry Allow and Disallow rules. Crawlers must parse at least 500 kibibytes and should not reuse a cached copy for more than 24 hours unless the file is unreachable. Compliance is voluntary by design: the RFC states that "these rules are not a form of access authorization."
Why it matters
It is the mechanism every major AI operator names as the supported opt-out, which makes it the only publisher-side control with documented adherence. Cloudflare puts the limit bluntly: the file "does not prevent crawlers from accessing your content at a technical level."
Example
A group is a token line followed by its rules, such as `User-agent: CCBot` on one line and `Disallow: /` on the next.
Where this came from
Definitions here follow the specification or documentation that defines the term, not a paraphrase of another glossary.
-
RFC 9309: Robots Exclusion Protocol — IETF
Standards Track status, September 2022 publication, the four authors, the mandatory file name and location, the 500 kibibyte parsing minimum, the 24 hour caching guidance, and the access authorization disclaimer.
Primary source · read 2026-09-03
-
Managed robots.txt — Cloudflare
robots.txt compliance is voluntary and the file does not prevent access at a technical level. Page last updated Aug 3, 2026.
Primary source · read 2026-09-03
developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/