Indexing

IndexingIndexing is the step in which a search engine processes crawled content and decides whether a URL is added to the search index. Only indexed pages can appear in search results.Open glossary entry → is the step in which Google processes previously crawledCrawling is the process by which a search engine bot fetches a URL and downloads its content. It is the first step before a page can be indexed.Open glossary entry → content and decides whether a URL is added to the search index. Only indexed pages can appear in search results.

Crawling and indexing are two separate steps. A page may be crawled but, in some cases, should not appear in the index. This is exactly what the noindex directive controls.

To keep a page out of the index, it should remain crawlable and carry a noindex instruction. Blocking it via robots.txt is not a reliable way to prevent indexing: a blocked URL can still appear in search results if other pages link to it.

• Indexing = inclusion in the search index
• controlled via noindex
• robots.txt prevents crawling, not reliably indexing

From crawling to ranking #

Indexing sits between crawling and ranking: a crawled page is analyzed, deduplicated and — if it passes quality thresholds — stored in the index. Only indexed pages can rank. "Crawled – currently not indexed" in Search ConsoleGoogle Search Console (GSC) is a free Google tool that shows how a website performs in organic search, with data on impressions, clicks, click-through rate and average position, plus indexing and technical issues.Open glossary entry → usually signals thin, duplicated or weakly linked content rather than a technical defect.

Checking and improving indexing #

Use the URL Inspection tool for individual pages and the Page Indexing report for patterns. Improve coverage by consolidating near-duplicates, strengthening internal linksInternal linking refers to links that connect pages within the same website. They control how search engines and users find and thematically classify content.Open glossary entry → to important pages, keeping sitemaps accurate, and reserving noindex for pages that genuinely should stay out.