Crawling

CrawlingCrawling is the process by which a search engine bot fetches a URL and downloads its content. It is the first step before a page can be indexed.Open glossary entry → is the process in which an automated bot (such as GooglebotGooglebot is Google's web crawler, which fetches web pages to crawl and index them for Google Search.Open glossary entry →) requests a URL and downloads its HTMLHyperText Markup Language (HTML) is the markup language for web pages. It defines the structure and content of a page, such as headings, paragraphs, links and images.Open glossary entry → along with resources like images, CSSCascading Style Sheets (CSS) is the language used to define the layout and design of web pages. CSS controls colours, fonts, spacing and arrangement, among other things.Open glossary entry → and JavaScript. Crawling is the prerequisite for content to be processed and indexedIndexing is the step in which a search engine processes crawled content and decides whether a URL is added to the search index. Only indexed pages can appear in search results.Open glossary entry → at all.

Which URLs a crawler may fetch can be controlled through the robots.txt file. Its main purpose is to manage crawler traffic and avoid overloading servers.

It is important to distinguish crawling from indexing: crawling only means fetching a page, not adding it to the search index. For search engines to read instructions such as noindex or a canonical tagThe canonical tag is an HTML link element that tells search engines the preferred (canonical) URL of a page. It helps consolidate duplicate or similar URLs onto a main version.Open glossary entry →, a page must be crawlable. If it is blocked via robots.txt, those signals cannot be detected reliably.

• Crawling = fetching a URL
• controlled via robots.txt
• prerequisite for indexing and reading robots directives