What is the Google-Extended user agent?
Google-Extended · Google · Training crawler
Google-Extended is not a crawler and has no user agent string of its own. It is a robots.txt product token: Googlebot still fetches your pages, and the Google-Extended rule decides whether that content may be used to train Gemini models and to ground answers in Gemini Apps and the Vertex AI API for Gemini. Google states it does not affect inclusion or ranking in Google Search.
What it does
Google-Extended collects page content to be used in model training. Controls use of your content for Gemini training and for grounding answers in Gemini Apps and Vertex AI. Does not affect Google Search or AI Overviews.
It is a robots.txt usage-control token, not a crawler: it never requests a page, so it never appears in your logs. The operator's main crawler does the fetching.
Google-Extended vs Googlebot
Googlebot is the crawler: it makes the HTTP requests, renders your pages and builds the Google Search index. Google-Extended makes no requests at all. You will never see it in your server logs. It is a separate line in robots.txt that Google reads as a usage instruction for content Googlebot has already fetched.
That is why the two are independent. Disallowing Google-Extended leaves Googlebot, your Search rankings and your appearance in Search features untouched. Disallowing Googlebot removes you from Google Search, whatever the Google-Extended line says.
AI Overviews and AI Mode are part of Google Search, so they follow Googlebot and your snippet controls (nosnippet, max-snippet), not Google-Extended.
robots.txt rules for Google-Extended
Pick one. Each group goes in the robots.txt at your site root, alongside your existing groups.
User-agent: Google-Extended
Disallow: /User-agent: Google-Extended
Allow: /User-agent: Google-Extended
Allow: /
Disallow: /members/Should you block Google-Extended?
Only if you do not want your content used to train Gemini models or to ground answers in Gemini Apps and the Vertex AI API for Gemini. Blocking it can make your pages unavailable as grounding sources there. It has no effect on Google Search or AI Overviews.
Your content stops being used for Gemini model training and for grounding in Google's generative products. Your Google Search ranking is entirely unaffected — this token does not touch Search at all.
To allow it
User-agent: Google-Extended
Allow: /To block it
User-agent: Google-Extended
Disallow: /Put the rule in your robots.txt at the site root. A group that names Google-Extended explicitly takes precedence over your User-agent: * group, so naming it is how you make your intent survive later edits to the wildcard rules.
Verifying a real Google-Extended request
Google-Extended is not a crawler with its own user agent making requests. It is a robots.txt token that controls how content already fetched by Google's standard crawlers may be used. There is no request to verify, which makes it unique on this list.
The mistake people make
Expecting to see Google-Extended in your server logs. You will not, because nothing requests pages under that name. People spend real time trying to confirm it is working by looking for traffic that does not exist.
Frequently asked questions
- What is the Google-Extended user agent?
- Google-Extended is not a crawler and has no user agent string of its own. It is a robots.txt product token: Googlebot still fetches your pages, and the Google-Extended rule decides whether that content may be used to train Gemini models and to ground answers in Gemini Apps and the Vertex AI API for Gemini. Google states it does not affect inclusion or ranking in Google Search.
- Should I block Google-Extended?
- Only if you do not want your content used to train Gemini models or to ground answers in Gemini Apps and the Vertex AI API for Gemini. Blocking it can make your pages unavailable as grounding sources there. It has no effect on Google Search or AI Overviews.
- How do I block or allow Google-Extended in robots.txt?
- Add a group to the robots.txt at your site root. To opt out: "User-agent: Google-Extended" followed by "Disallow: /". To opt in explicitly: "User-agent: Google-Extended" followed by "Allow: /". Leaving it out means the rules in your * group apply.
- Does blocking Google-Extended affect Google Search rankings?
- No. Google states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. It only governs whether your content is used for Gemini model training and for grounding in Gemini Apps and the Vertex AI API for Gemini. It does not remove you from AI Overviews, which are part of Search.
- Why can't I find Google-Extended in my server logs?
- Because it never makes a request. Google's documentation says Google-Extended does not have a separate HTTP request user agent string; your pages are fetched by Googlebot, and the Google-Extended rule controls how that content may be used.
Further reading
- The Complete Guide to robots.txt for AI Crawlers: Which Bots to Allow
GPTBot vs OAI-SearchBot, ClaudeBot vs Claude-User, Googlebot vs Google-Extended: which AI crawlers to allow, which to block, and copy-ready robots.txt templates.
- How to Create and Optimize an llms.txt File for Your Website
The llms.txt format, where the file goes, how to keep it token-efficient, working examples for Next.js and static sites, and an honest account of who uses it.
- What is Generative Engine Optimization (GEO)? The Complete 2026 Guide
How ChatGPT Search, Perplexity and Claude find, read and cite pages, why that differs from Googlebot, and what the research says actually improves AI visibility.
Test your site against 15 AI crawlers now
The free scan parses your robots.txt, evaluates every AI crawler against it, and tells you how much of a page Google-Extended would actually come away with.
Run the free scan