robots.txt

The robots.txtThe robots.txt is a publicly accessible text file in a site's root directory. It tells crawlers which URLs they may fetch and therefore controls crawling.Open glossary entry → is a text file in a website's root directory, for example at https://example.ch/robots.txt. Using rules such as Disallow, it tells search engine crawlers which URLs they may fetch. This controls crawlingCrawling is the process by which a search engine bot fetches a URL and downloads its content. It is the first step before a page can be indexed.Open glossary entry → and helps avoid overloading servers.

Importantly, robots.txt controls crawling, not indexingIndexing is the step in which a search engine processes crawled content and decides whether a URL is added to the search index. Only indexed pages can appear in search results.Open glossary entry →. A page blocked via Disallow can still appear in search results if other pages link to it. It is also not a security measure, since the file is publicly visible. Sensitive content belongs behind a login, password protection or server-side access control.

Unsupported rules such as Noindex: or crawl-delay should not be used. The robots.txt can also reference the XML sitemapAn XML sitemap is a machine-readable file listing a website's important URLs. It helps search engines discover content more efficiently but does not replace internal linking or guarantee indexing.Open glossary entry →.

• controls crawling, not indexing
• not a security or indexing safeguard
• should not broadly block CSSCascading Style Sheets (CSS) is the language used to define the layout and design of web pages. CSS controls colours, fonts, spacing and arrangement, among other things.Open glossary entry → and JavaScript