Googlebot

GooglebotGooglebot is Google's web crawler, which fetches web pages to crawl and index them for Google Search.Open glossary entry → is the automated crawlerCrawling is the process by which a search engine bot fetches a URL and downloads its content. It is the first step before a page can be indexed.Open glossary entry → Google uses to fetch the web. It follows links, loads HTMLHyperText Markup Language (HTML) is the markup language for web pages. It defines the structure and content of a page, such as headings, paragraphs, links and images.Open glossary entry → and resources, and makes the content available for crawling, renderingJavaScript rendering is the process by which a browser or crawler executes JavaScript code to build the final, visible page. Content that is not already in the delivered HTML only appears after this step.Open glossary entry → and indexingIndexing is the step in which a search engine processes crawled content and decides whether a URL is added to the search index. Only indexed pages can appear in search results.Open glossary entry →. Google primarily uses a mobile Googlebot (mobile-first indexing).

Googlebot can be controlled via robots.txt and meta robotsThe meta robots tag is an HTML element in a page's <head> that gives search engines indexing and link-following instructions such as noindex or nofollow.Open glossary entry → directives such as noindex. It is important to distinguish it from AI product tokens: Googlebot concerns classic Google Search, while Google-Extended only controls certain AI uses. Blocking Googlebot by mistake risks losing classic Google visibility. Server logs reveal whether Googlebot receives successful responses (status code 200HTTP status 200 (OK) means a page was found and served successfully. Only 200 URLs should appear in the sitemap and serve as canonical targets.Open glossary entry →).