Google-Extended lets website owners control certain AI uses by Google without affecting classic Google Search. Google explains that Google-Extended is a product token that lets you control certain AI uses by Google in robots.txt. It is not a separate HTTP user agent and is not the same as Googlebot.Open glossary entry → is not a separate Crawling is the process by which a search engine bot fetches a URL and downloads its content. It is the first step before a page can be indexed.Open glossary entry →; crawling happens via existing Google user agents, while Google-Extended is used in The robots.txt is a publicly accessible text file in a site's root directory. It tells crawlers which URLs they may fetch and therefore controls crawling.Open glossary entry → as a product token.
The most common mistake is confusing it with Googlebot is Google's web crawler, which fetches web pages to crawl and index them for Google Search.Open glossary entry →:
User-agent: Google-Extended/Disallow: /controls only AI use,User-agent: Googlebot/Disallow: /endangers classic Google visibility.
So if you want to restrict Google AI use, do not accidentally block Googlebot. The two tokens serve different purposes and must be handled separately and deliberately.