Google-Extended
What Google-Extended means
Google-Extended is a robots.txt control token rather than a crawler. It manages whether content Google has already crawled may be used to train Gemini models and to ground answers in Gemini Apps, the Vertex AI API for Gemini, and Grounding with Google Search on Vertex AI. It does not control Google Search, and Google states plainly that it is not a ranking signal there.
More precisely
Google writes that the token "doesn't have a separate HTTP request user agent string" and that "crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." There is therefore no separate IP range or reverse DNS check for it. Its documented scope names Gemini Apps and Vertex AI. It does not name AI Overviews or AI Mode, and Google's AI features documentation points site owners at Googlebot directives for Search instead.
Why it matters
This is the token most often disallowed by publishers who believe they are opting out of AI Overviews. Google's own wording says the opposite about Search, and the AI features page hedges with "in some of Google's other systems" without listing which.
Example
A robots.txt group of `User-agent: Google-Extended` with `Disallow: /` withholds content from Gemini training and Vertex grounding while leaving Google Search crawling and ranking untouched.
Where this came from
Definitions here follow the specification or documentation that defines the term, not a paraphrase of another glossary.
-
Google's common crawlers — Google
The full documented scope of Google-Extended, the statement that it has no separate user-agent string, and the statement that it does not impact inclusion in Google Search and is not a ranking signal. Page last updated 2026-07-14 UTC.
Primary source · read 2026-09-03
developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
-
AI features and your website — Google
Google directs site owners to robots.txt directives for Googlebot as the control for Search, and describes Google-Extended as covering training and grounding "in some of Google's other systems". Page last updated 2025-12-10 UTC.
Primary source · read 2026-09-03