Indexing

IndexingIndexing is the step in which a search engine processes crawled content and decides whether a URL is added to the search index. Only indexed pages can appear in search results.Open glossary entry → is the step in which Google processes previously crawledCrawling is the process by which a search engine bot fetches a URL and downloads its content. It is the first step before a page can be indexed.Open glossary entry → content and decides whether a URL is added to the search index. Only indexed pages can appear in search results.

Crawling and indexing are two separate steps. A page may be crawled but, in some cases, should not appear in the index. This is exactly what the noindex directive controls.

To keep a page out of the index, it should remain crawlable and carry a noindex instruction. Blocking it via robots.txt is not a reliable way to prevent indexing: a blocked URL can still appear in search results if other pages link to it.

• Indexing = inclusion in the search index
• controlled via noindex
• robots.txt prevents crawling, not reliably indexing