Crawl Budget

Crawl budgetCrawl budget describes how many URLs a search engine crawls on a site within a given period. It mainly becomes relevant for very large websites.Open glossary entry → describes how many URLs a search engine fetches on a website within a given period. Roughly, it results from crawlCrawling is the process by which a search engine bot fetches a URL and downloads its content. It is the first step before a page can be indexed.Open glossary entry → capacity (how much the server can handle) and crawl demand (how important and fresh the content is).

For most small and medium sites, crawl budget is not a bottleneck. It becomes relevant mainly for very large websites and shops with many URLs, for example through filter and parameter pages.

Crawl budget is used inefficiently by, among other things:

• dead linksDead links are links pointing to a URL that is no longer reachable, usually returning status code 404. Internal dead links that lead users and crawlers into a dead end are especially problematic.Open glossary entry → and 404 URLs,
• redirectA redirect forwards users and search engines from one URL to another. It can be permanent (301) or temporary (302) and transfers signals to the target URL.Open glossary entry → chains,
• large numbers of similar parameter and filter URLs,
• duplicate contentDuplicate content refers to identical or very similar content reachable under several URLs. It rarely triggers a penalty but causes a loss of control over which version becomes the main one.Open glossary entry →.

A clean structure helps: clear canonicalization signalsCanonicalization signals are all the cues search engines use to determine a page's canonical main version, including the canonical tag, internal links, the sitemap, redirects and hreflang.Open glossary entry →, a tidy XML sitemapAn XML sitemap is a machine-readable file listing a website's important URLs. It helps search engines discover content more efficiently but does not replace internal linking or guarantee indexing.Open glossary entry →, deliberate control of unimportant URLs, and cleaning up dead links. A canonical tagThe canonical tag is an HTML link element that tells search engines the preferred (canonical) URL of a page. It helps consolidate duplicate or similar URLs onto a main version.Open glossary entry → alone does not solve a crawl budget problem, but it supports clearer classification of URLs.