X-Robots-Tag

The X-Robots-TagThe X-Robots-Tag is a robots directive sent in the HTTP header. It allows indexing rules such as noindex to be applied to non-HTML files like PDFs or images.Open glossary entry → is a robots directive delivered not in the HTMLHyperText Markup Language (HTML) is the markup language for web pages. It defines the structure and content of a page, such as headings, paragraphs, links and images.Open glossary entry → but in the HTTPHTTP (Hypertext Transfer Protocol) is the protocol through which browsers and servers exchange web content. HTTPS is the encrypted, secure variant of this protocol.Open glossary entry → (Hypertext Transfer Protocol) header. This allows indexingIndexing is the step in which a search engine processes crawled content and decides whether a URL is added to the search index. Only indexed pages can appear in search results.Open glossary entry → rules to be set server-side.

Example:

X-Robots-Tag: noindex

The X-Robots-Tag is especially useful for files that contain no HTML and therefore cannot hold a meta robots tagThe meta robots tag is an HTML element in a page's <head> that gives search engines indexing and link-following instructions such as noindex or nofollow.Open glossary entry →, for example:

• PDFs,
• Word documents,
• images,
• download files,
• entire file types via server rules.

As with the meta robots tag, the resource must be crawlableCrawling is the process by which a search engine bot fetches a URL and downloads its content. It is the first step before a page can be indexed.Open glossary entry → for search engines to read the directive; it must not be blocked via robots.txt.