Meta Robots Tag

The meta robots tagThe meta robots tag is an HTML element in a page's <head> that gives search engines indexing and link-following instructions such as noindex or nofollow.Open glossary entry → is an HTMLHyperText Markup Language (HTML) is the markup language for web pages. It defines the structure and content of a page, such as headings, paragraphs, links and images.Open glossary entry → element in a page's <head>. It tells search engines whether a page may be indexedIndexing is the step in which a search engine processes crawled content and decides whether a URL is added to the search index. Only indexed pages can appear in search results.Open glossary entry → and whether links on the page should be followed.

Example:

<meta name="robots" content="noindex, follow">

Common values are index/noindex (allow or prevent indexing) and follow/nofollow (follow links or not). Multiple values should ideally be combined into a single tag to avoid conflicts.

The meta robots tag is suited to normal HTML pages. For search engines to read it, the page must be crawlableCrawling is the process by which a search engine bot fetches a URL and downloads its content. It is the first step before a page can be indexed.Open glossary entry →; blocking it via robots.txt prevents this. For non-HTML files such as PDFs, the X-Robots-TagThe X-Robots-Tag is a robots directive sent in the HTTP header. It allows indexing rules such as noindex to be applied to non-HTML files like PDFs or images.Open glossary entry → in the HTTPHTTP (Hypertext Transfer Protocol) is the protocol through which browsers and servers exchange web content. HTTPS is the encrypted, secure variant of this protocol.Open glossary entry → header is used instead.