User agent
Also called UA, user-agent string.
What User agent means
A user agent is any client program that initiates an HTTP request, which covers browsers, crawlers, command-line tools and applications alike. The User-Agent header field, defined in RFC 9110, "contains information about the user agent originating the request."
More precisely
robots.txt rules are not matched against the whole header. RFC 9309 defines a "product token", limited to letters, underscores and hyphens, which "SHOULD be a substring of the identification string that the crawler sends." Crawlers match tokens case-insensitively, combine every matching group, and fall back to the `*` group only when no token matches. Because the header is self-reported it can be forged, so operators publish IP ranges or reverse DNS hostnames for verification.
Why it matters
Rules have to be written against the token an operator publishes. A rule aimed at a full browser-style string, or at a token nobody uses, silently matches nothing and looks identical in the file to one that works.
Example
`User-agent: GPTBot` matches the token OpenAI publishes; a line naming the whole `Mozilla/5.0 ... GPTBot/1.4 ...` string does not.
Where this came from
Definitions here follow the specification or documentation that defines the term, not a paraphrase of another glossary.
-
RFC 9110: HTTP Semantics, section 3.5 and section 10.1.5 — IETF
The term user agent refers to any client program that initiates a request, and the User-Agent header field contains information about the user agent originating the request.
Primary source · read 2026-09-03
-
RFC 9309: Robots Exclusion Protocol — IETF
The product token character set, the requirement that it should be a substring of the crawler identification string, case-insensitive matching, group combination, and the fallback to the "*" group.
Primary source · read 2026-09-03