noindex & robots.txt: Control Crawling and Indexing Correctly

A single noindex in the wrong place can remove important pages from Google. An overly restrictive robots.txt file can prevent Google from crawlingCrawling is the process by which a search engine bot fetches a URL and downloads its content. It is the first step before a page can be indexed.Open glossary entry → important content in the first place. And the most critical point: If a page is blocked via robots.txt, Google cannot reliably see a noindex directive placed on that page.

RankScan has two measures for this in the Website Health category:

• “Content pages excluded from indexingIndexing is the step in which a search engine processes crawled content and decides whether a URL is added to the search index. Only indexed pages can appear in search results.Open glossary entry →”: Content pages carry a noindex, in the meta robots tagThe meta robots tag is an HTML element in a page's <head> that gives search engines indexing and link-following instructions such as noindex or nofollow.Open glossary entry → or in the X-Robots-TagThe X-Robots-Tag is a robots directive sent in the HTTP header. It allows indexing rules such as noindex to be applied to non-HTML files like PDFs or images.Open glossary entry →.
• “Pages blocked for crawlers”: Pages respond with HTTPHTTP (Hypertext Transfer Protocol) is the protocol through which browsers and servers exchange web content. HTTPS is the encrypted, secure variant of this protocol.Open glossary entry → 401 or 403. The server asks for a login or denies access.

Both measures have low severity in RankScan. The reason: the block is often intended, for example on login or account pages. So first check whether a system page is affected.

Important: Neither measure checks robots.txt. What this article explains about robots.txtThe robots.txt is a publicly accessible text file in a site's root directory. It tells crawlers which URLs they may fetch and therefore controls crawling.Open glossary entry → is general expertiseExpertise is the subject-matter competence behind a piece of content. It is the second E in E-E-A-T and should be visible through author details and supporting evidence.Open glossary entry →. It helps you find causes, but it is not a RankScan measure.

All three kinds of block can happen by accident, especially after relaunches, staging releases, content management system (CMS)A content management system (CMS) is software for creating, managing and publishing web content, usually without programming knowledge. Well-known examples are WordPress or Shopify.Open glossary entry → updates or plugin changes.

In this article, you will learn how noindex, robots.txt and access blocks work, how they differ, which mistakes are especially dangerous and how to handle both RankScan measures.

Crawling vs. Indexing: The Most Important Difference #

To use noindex and robots.txt correctly, you need to distinguish between two processes.

Crawling #

During crawling, a bot fetches a URL. The bot downloads HTMLHyperText Markup Language (HTML) is the markup language for web pages. It defines the structure and content of a page, such as headings, paragraphs, links and images.Open glossary entry → (Hypertext Markup Language, the markup language for web pages), images, CSSCascading Style Sheets (CSS) is the language used to define the layout and design of web pages. CSS controls colours, fonts, spacing and arrangement, among other things.Open glossary entry → (Cascading Style Sheets, the language used for web page layout), JavaScript or other resources.

The robots.txt file can control which URLs a crawler is allowed to fetch.

Google describes robots.txt as a file that tells search engine crawlers which URLs on a site they may access. It is mainly used to manage crawler traffic and avoid unnecessary server load.
Source: Google Search Central: robots.txt Introduction and Guide

Indexing #

During indexing, Google processes crawled content and decides whether a URL should be added to the search index.

noindex controls exactly this step: the page may be crawled, but should not appear in the search index.

Google explains that noindex can be used as a meta tag or HTTP header (Hypertext Transfer Protocol, the web’s transfer protocol) to prevent indexing. Important: the page must not be blocked by robots.txt, otherwise Google cannot see the noindex instruction.
Source: Google Search Central: Block Search indexing with noindex

What Does noindex Mean? #

noindex is a robots directive that tells search engines:

This page should not appear in search results.

Typical HTML implementation in the <head>:

<meta name="robots" content="noindex, follow">

This means:

• noindex: Do not index the page.
• follow: Links on the page may be followed.

follow is usually not strictly required, but is often used to make the intention clear.

Google documents that robots meta tags and X-Robots-Tags can only be read if crawlers have access to the page.
Source: Google Search Central: Robots Meta Tags

Meta noindex vs. X-Robots-Tag #

There are two important ways to set noindex.

1. Meta Robots Tag in the HTML #

For normal HTML pages:

<meta name="robots" content="noindex">

or:

<meta name="robots" content="noindex, follow">

The tag belongs in the page’s <head>.

2. X-Robots-Tag in the HTTP Header #

For PDFs, images, files or server-side rules, noindex can be set in the HTTP header:

X-Robots-Tag: noindex

This is especially useful for:

• PDFs,
• Word documents,
• images,
• download files,
• automatically generated files,
• entire file types.

Google points out that the X-Robots-Tag is especially useful for controlling the indexing of non-HTML files.
Source: Google Search Central: Meta tags and attributes that Google supports

What Does “Content Pages Excluded from Indexing” Mean? #

The RankScan measure “Content pages excluded from indexing” means: the crawl found a noindex on content pages. RankScan reads two places for this: the meta robots tag in the HTML and the X-Robots-Tag in the HTTP header. The value none also counts; it stands for noindex, nofollow. In the X-Robots-Tag, RankScan only recognises the plain lowercase form, such as X-Robots-Tag: noindex. RankScan does not recognise a directive with a bot name such as googlebot: noindex.

Rating in RankScan: low severity · low effort · work type Development.

RankScan only counts content pages, for example the homepage, articles, products, categories and contact pages. RankScan does not count legal, system and confirmation pages. Functional pages such as gallery, booking or newsletter are left out as well. The page type assignment is an estimate. So a system page can still appear in the list. If only such an intended page remains after you mark the measure as done, the measure shows “Not fixed yet”. Ignore it then.

A noindex can be correct.

Examples of intentionally set noindex:

• internal search results,
• cart,
• checkout,
• login,
• thank-you pages,
• filter pages without search value,
• thin archive pages,
• staging or test pages,
• internal documentation,
• duplicate or low-value pages.

It becomes problematic when important pages are affected:

• homepage,
• service pages,
• product pages,
• category pages,
• guide articles,
• location pages,
• landing pages,
• pages in the sitemap,
• pages with organic rankings,
• pages with backlinks,
• pages with internal linksInternal linking refers to links that connect pages within the same website. They control how search engines and users find and thematically classify content.Open glossary entry →.

Procedure #

1. First check whether a page is a system page: login, account, password or form status. There, noindex is correct. Keep it.
2. Open the remaining pages and check whether they should appear in search results.
3. Pass these pages on to your developers or agency to remove the noindex setting. The noindex often sits in an SEOSearch Engine Optimization (SEO) covers all measures that help a website become more visible in unpaid (organic) search results.Open glossary entry → plugin, in the page template or in a server rule for the X-Robots-Tag.
4. Keep noindex only on pages that deliberately stay out of search.
5. Mark the measure as done once the change is live.

How RankScan checks it: The next full crawl checks all affected pages. RankScan then reports “Fixed” or “Not fixed yet” with the number of pages still missing.

What Does “Pages Blocked for Crawlers” Mean? #

The RankScan measure “Pages blocked for crawlers” means: the affected pages respond with HTTP 401 or 403. The server asks for a login or denies access.

Rating in RankScan: low severity · low effort · work type Development.

The two status codes mean:

• 401 Unauthorized: The server asks for a login, for example via a login form or HTTP Auth.
• 403 Forbidden: The server denies access, for example because of a permission rule or a firewall.

The measure does not check robots.txt. A page that is only blocked via robots.txt responds normally with 200. RankScan does not report it in this measure.

Google treats 401 and 403 like most 4xx codes: the crawlers signal that the content does not exist. If the address was already indexed, Google removes it from the index.
Source: Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers

Login, account and admin areas are rightly closed. A block on public content, however, keeps search engines and AI systems from reading it.

Typical causes of a block on public pages:

• The password protection of the staging environment stayed active after launch.
• An area is accidentally only open to logged-in users.
• A firewall or bot protection blocks crawlers but lets browsers through.
• File permissions on the server are set incorrectly.

Procedure #

1. Open the affected addresses and check whether they belong to login, account or admin areas.
2. Keep these areas blocked. If every block is intended, ignore the measure. An ignored measure does not reopen. So check the blocks yourself after a relaunch.
3. Pass the list of public content on to your developers or agency to remove the block. If a page opens normally in the browser, review the firewall and bot protection rules with them.
4. Mark the measure as done.

How RankScan checks it: The next full crawl checks all affected pages. RankScan then reports “Fixed” or “Not fixed yet” with the number of pages still missing.

The Critical Mistake: robots.txt Blocks noindex #

The most common misunderstanding:

User-agent: *
Disallow: /old-page/

and at the same time on the page:

<meta name="robots" content="noindex">

At first glance, this looks doubly safe. In reality, it is contradictory.

If Google is not allowed to crawl the page because of robots.txt, Google cannot see the noindex tag on that page. The URL may still appear in search results, for example if other pages link to it.

Google explicitly states: For noindex to work, the page must not be blocked by robots.txt and must be accessible to the crawler.
Source: Google Search Central: Block Search indexing with noindex

The rule is:

If a page should leave the index: keep it crawlable and set noindex.

robots.txt Is Not a Security Mechanism #

The robots.txt file is publicly accessible:

https://example.ch/robots.txt

It is not password protection and not access control.

Not suitable for:

• confidential documents,
• customer data,
• internal files,
• staging content,
• private PDFs,
• admin areas with sensitive information.

For sensitive content, you need:

• login,
• password protection,
• HTTP Auth,
• server-side access control,
• VPN (Virtual Private Network),
• IP restriction,
• secure role and permission management.

Robots.txt is based on cooperation. Reputable crawlers respect it, but it does not protect content from direct access.

noindex, nofollow, index follow: What Means What? #

These terms are often confused.

In practice, noindex, follow is often more useful than noindex, nofollow when a page should not be indexed but its internal links are still useful for crawling and orientation.

In an older blog post, Google recommends combining multiple robots values in one meta tag where possible to avoid conflicts.
Source: Google Search Central Blog: Using the robots meta tag

noindex in robots.txt? Do Not Use It #

Sometimes you see rules like:

User-agent: *
Noindex: /internal/

This is not a reliable Google rule.

In 2019, Google clarified that unsupported robots.txt rules such as noindex, nofollow and crawl-delay would no longer be supported as unofficial rules.
Source: Google Search Central Blog: A note on unsupported rules in robots.txt

If you want to prevent indexing, use:

<meta name="robots" content="noindex">

or:

X-Robots-Tag: noindex

Not:

Noindex: /path/

Using robots.txt Correctly #

This section is general expertise. RankScan does not check robots.txt in either of the two measures in this article.

The robots.txt file is useful when you want to control crawling.

Example:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/

Sitemap: https://example.ch/sitemap.xml

Typical useful applications:

• exclude technical areas from crawling,
• reduce internal search result crawling,
• control low-value parameter areas,
• reduce server load,
• manage specific bots,
• specifically allow or block crawlers for Artificial Intelligence (AI),
• reference sitemaps.

Important: Do not generally block CSS or JavaScript files if Google needs them for rendering.

A URL blocked via robots.txt can still appear in search results if Google discovers it through links. If a page should stay out of the index, set noindex and keep the URL crawlable.
Source: Google Search Central: Technical requirements

When to Use noindex, robots.txt, Canonical or Redirect? #

These tools serve different purposes.

Prioritisation: Which Cases Are Really Critical? #

Not every block has the same relevance. The following table is a general professional assessment. RankScan shows a single severity for both measures: low. RankScan does not measure the robots.txt cases in these measures.

The most important rule:

First sort out system pages. Then check whether important, internally linked or sitemap-relevant pages are affected.

Content Error or Template Error? #

Distinguish whether an issue affects individual pages or entire page types.

Individual Issue #

Examples:

• a landing page was accidentally set to noindex,
• a single article has an incorrect search engine optimization (SEO) setting,
• an old campaign page is blocked,
• a single PDF has an incorrect header.

Solution: Fix the page or file.

Template or System Issue #

Examples:

• all blog articles are noindex,
• category pages inherit a staging setting,
• robots.txt blocks an entire directory,
• an SEO plugin sets global noindex,
• shop filters are broadly blocked incorrectly,
• the head template outputs incorrect meta robots,
• CDNA content delivery network (CDN) is a network of distributed servers that delivers static content closer to the user, reducing loading times.Open glossary entry → (Content Delivery Network, a network of distributed servers for faster delivery) or WAF (Web Application Firewall) blocks relevant bots.

Solution: Fix the template, CMS configuration, plugin, deployment process or infrastructure.

System issues are usually more urgent because they affect many URLs.

Typical Causes of noindex Errors #

1. Staging Settings Were Carried Over #

During development, noindex is correct. On the live site, it is fatal.

Common causes:

• WordPress setting “Discourage search engines”,
• environment variable set incorrectly,
• SEO plugin copied from staging,
• robots rules not adjusted.

2. CMS or SEO Plugin Misconfigured #

A plugin can set entire content types to noindex:

• categories,
• tags,
• author pages,
• product archives,
• custom post types,
• search pages.

3. Template Outputs Incorrect Robots Meta Tags #

Example:

<meta name="robots" content="noindex">

is placed in the global head and therefore output on every page.

4. Canonical/noindex Conflicts #

A page is named as the main version via canonical, but is also noindex. That is contradictory.

5. JavaScript or Tag Manager Sets Meta Robots #

Meta robots should not be generated or changed client-side without control. For search engines, the directive must be reliably detectable.

Typical Causes of robots.txt Errors #

This section is general expertise as well. It helps you when Google Search ConsoleGoogle Search Console (GSC) is a free Google tool that shows how a website performs in organic search, with data on impressions, clicks, click-through rate and average position, plus indexing and technical issues.Open glossary entry → shows blocked pages.

1. Global Disallow from Staging #

User-agent: *
Disallow: /

Useful on staging. Critical in production.

2. Wrong Directory Blocked #

Disallow: /blog/

when the blog should be indexed.

3. Assets Blocked #

Disallow: /assets/
Disallow: /js/
Disallow: /css/

This can make rendering and quality evaluation harder.

4. Parameters Blocked Too Broadly #

Sometimes filter or parameter pages are blocked so broadly that important landing pages are affected.

5. Crawler-Specific Rules Set Incorrectly #

Example:

User-agent: Googlebot
Disallow: /

This directly blocks Google and is usually fatal on a live site.

noindex and robots.txt for AI Crawlers #

Robots.txt does not only control GooglebotGooglebot is Google's web crawler, which fetches web pages to crawl and index them for Google Search.Open glossary entry →; it can also affect AI crawlers.

Examples of AI user agents:

• GPTBotGPTBot is the crawler documented by OpenAI that typically fetches content for training future models. It can be specifically allowed or blocked via robots.txt.Open glossary entry →,
• OAI-SearchBotOAI-SearchBot is the crawler documented by OpenAI for ChatGPT Search. It fetches content so that public websites can appear as a source in ChatGPT summaries, snippets and answers.Open glossary entry →,
• ChatGPT-UserChatGPT-User is the user agent documented by OpenAI for user-triggered retrieval. It fetches a page when a user enters a specific URL or query in ChatGPT.Open glossary entry →,
• ClaudeBotClaudeBot is the web crawler used by Anthropic. Website owners can control its access via robots.txt and block it if needed.Open glossary entry →,
• PerplexityBotPerplexityBot is the crawler documented by Perplexity, intended to make websites visible in Perplexity search results and link them as a source. According to Perplexity, it is not used to train foundation models.Open glossary entry →,
• Google-ExtendedGoogle-Extended is a product token that lets you control certain AI uses by Google in robots.txt. It is not a separate HTTP user agent and is not the same as Googlebot.Open glossary entry →.

If you block AI crawlers broadly, this can affect AI Visibility. At the same time, it can be strategically sensible to limit training while allowing live retrieval.

This topic is closely related to the article Controlling AI Crawlers: robots.txt & llms.txtllms.txt is a voluntary Markdown file in a website's root directory that gives language models a compact overview of important content. It is a proposal or de facto format, not a binding official web standard.Open glossary entry →. That article also covers the RankScan measure “AI crawlers blocked in robots.txt”.

Important for this article:

• robots.txt controls crawling.
• noindex controls classic search indexing.
• AI crawlers can be blocked via robots.txt.
• Blocking AI crawlers is not automatically bad, but it should be intentional.
• The normal Googlebot should not be accidentally blocked if classic Google visibility should be maintained.

What to Do After a RankScan Finding #

The steps of both measures are listed above in their sections. If many pages are affected, this general procedure helps you find the cause.

Step 1: Sort Out System Pages and Group URLs #

First separate login, account, password and admin pages. There, noindex and 401 or 403 are correct.

Group the remaining addresses. Check:

• individual URLs,
• page types,
• directories,
• templates,
• sitemaps,
• product areas,
• blog areas,
• staging/test paths,
• assets.

Step 2: Define the Indexing Goal #

For each group, ask:

Should this URL appear in Google?
Should this URL be crawled?
Should this URL only be accessible to users?
Should this URL be protected?

Only then decide between noindex, robots.txt, canonical, redirectA redirect forwards users and search engines from one URL to another. It can be permanent (301) or temporary (302) and transfers signals to the target URL.Open glossary entry → or login.

Step 3: Identify Contradictions #

Watch for combinations such as:

Step 4: Fix the Cause #

Typical actions:

• remove noindex from important pages,
• lift the access block (401 or 403) on public pages,
• make robots.txt rules more precise,
• remove staging rules from production,
• clean up the sitemap,
• remove noindex pages from the sitemap,
• make CSS/JS crawlable again,
• resolve canonical/noindex conflicts,
• check SEO plugin configuration,
• correct the template,
• check WAF/CDN bot rules.

Step 5: Check in Google Search Console #

Use:

• URL Inspection Tool,
• page indexing,
• crawl stats,
• robots.txt report,
• live test,
• sitemap report.

Google Search Console shows, among other things, whether a URL is excluded by noindex or whether crawling is blocked by robots.txt.

Step 6: Verify the Fix and Mark It as Done #

After the correction, check:

• Has noindex been removed from important pages?
• Have noindex pages been removed from the sitemap?
• Is robots.txt no longer too restrictive?
• Are CSS/JS accessible?
• Are important URLs indexable?
• Do public pages respond with 200 instead of 401 or 403?
• Has Google been prompted to recheck?

Then mark the measure as done in RankScan. The next full crawl checks all affected pages. RankScan then reports “Fixed” or “Not fixed yet” with the number of pages still missing.

What a Good Indexing Check Looks For #

The following list is a general checklist for a complete review. RankScan measures only part of it, see the section below.

A complete review covers:

• pages with noindex,
• important pages with noindex,
• noindex pages in the sitemap,
• noindex on canonical URLs,
• canonical pointing to noindex target,
• robots.txt blocking important content,
• robots.txt blocking whole directories,
• robots.txt blocking CSS/JS,
• Disallow: / on the live site,
• Googlebot-specific blocks,
• AI crawler blocks,
• noindex in the meta tag,
• noindex in the X-Robots-Tag,
• contradictory robots meta tags,
• robots.txt + noindex conflicts,
• status code and indexability, for example 401 or 403 on public pages,
• mobile/desktop differences,
• template or CMS patterns.

What RankScan Measures #

• noindex on content pages: in the meta robots tag or in the X-Robots-Tag, also as none. This results in “Content pages excluded from indexing”.
• HTTP 401 or 403: on all crawled pages. This results in “Pages blocked for crawlers”.
• AI crawlers in robots.txt: This means search and retrieval crawlers that robots.txt blocks completely with Disallow: /, in a group of their own or via User-agent: *. RankScan does not report blocks of training crawlers or of single directories. This results in “AI crawlers blocked in robots.txt”. The article AI crawlers and llms.txt explains this measure.

RankScan has no measure of its own for the other points on the list. Check them yourself when needed, for example with Google Search Console. They include Googlebot blocks in robots.txt, noindex pages in the sitemap and conflicts with the canonical.

Example: Invisible Relaunch Caused by noindex #

Situation #

A website is prepared on a staging system. The following is correctly set there:

<meta name="robots" content="noindex, nofollow">

After launch, this setting remains active. In addition, the guide section stays behind the staging password protection.

RankScan reports:

“Content pages excluded from indexing”
“Pages blocked for crawlers”

Analysis #

• The service pages still carry the noindex from staging.
• The guide pages respond with 401 because the password protection is still active.
• The login page of the customer portal responds with 403. This block is intended.
• Several noindex pages are still in the sitemap.
• Google Search Console also shows: robots.txt still contains Disallow: /staging-assets/. Some JS files are blocked as a result. RankScan does not report this point.

Solution #

1. Keep the customer portal login page blocked.
2. Remove noindex from the service pages.
3. Correct the SEO plugin configuration.
4. Remove the password protection from the guide section.
5. Regenerate the sitemap.
6. Clean up robots.txt and allow CSS/JS.
7. Protect staging via HTTP Auth in future.
8. Use Google Search Console URL inspection.
9. Mark both measures as done in RankScan.

The next full crawl checks all affected pages. Afterwards, only the login page responds with 403. This block is intended. The measure then shows “Not fixed yet”, with one page. Ignore it then. An ignored measure does not reopen. So check the blocks yourself after a relaunch.

Result #

The pages are crawlable and indexable again. The issue was not a content problem, but a technical deployment error.

Common Mistakes #

Mistake 1: Using robots.txt as Indexing Protection #

Disallow prevents crawling, not reliably indexing.

Mistake 2: Combining noindex and robots.txt #

If Google is not allowed to crawl a page, Google cannot see noindex.

Mistake 3: Publishing the Live Site with Staging Rules #

Disallow: / or global noindex do not belong on the production website.

Mistake 4: Leaving noindex Pages in the Sitemap #

The sitemap should only contain URLs that are intended to be indexed.

Mistake 5: Blocking CSS and JavaScript #

Google needs to be able to render pages. Important resources should be crawlable.

Mistake 6: Using nofollow Unnecessarily #

nofollow on internal pages can unnecessarily weaken internal link signals and crawling.

Mistake 7: Using robots.txt for Sensitive Data #

Robots.txt is public. Protection requires authentication or server-side restrictions.

Mistake 8: Leaving Public Pages Behind an Access Block #

A password protection left over from staging or an overly strict firewall also locks out crawlers. If a public page responds with 401 or 403, Google cannot read it.

Checklist: Checking noindex and robots.txt #

Use this checklist:

• Have you sorted out system pages such as login, account and admin first?
• Are there important pages with noindex?
• Do public pages respond with 401 or 403?
• Are there noindex pages in the sitemap?
• Is there Disallow: / on the live site?
• Does robots.txt block important directories?
• Does robots.txt block CSS, JS or images?
• Are important pages crawlable?
• Are pages that should leave the index crawlable and noindex?
• Are there canonical/noindex conflicts?
• Are there multiple contradictory robots meta tags?
• Are X-Robots-Tags on PDFs set correctly?
• Are staging pages protected by authentication?
• Have SEO plugin and CMS settings been checked?
• Has the setup been tested in Google Search Console?
• Did you mark the measure as done in RankScan?

In addition, 404 errors help narrow down the cause cleanly and prioritize the next SEO actions.

Conclusion: Separate Crawling and Indexing Cleanly #

noindex and robots.txt are not interchangeable tools. If you confuse them, you risk removing important pages from Google or preventing Google from reading decisive signals in the first place.

The most important rule is:

robots.txt controls crawling. noindex controls indexing.

Access blocks follow a third rule: 401 and 403 belong on login, account and admin areas, not on public content.

RankScan rates the measures “Content pages excluded from indexing” and “Pages blocked for crawlers” with low severity. Many blocks are intended. An accidental block on an important page still costs visibility.

The best approach is:

1. sort out system pages such as login and account,
2. group the remaining URLs,
3. define the indexing goal,
4. check noindex, access blocks and robots.txt rules,
5. identify contradictions,
6. release important pages,
7. align sitemaps, canonicals and internal links,
8. test in Google Search Console,
9. mark the measure as done in RankScan.

The next full crawl then shows whether the unintended blocks are gone.

This turns indexing control from a risk into a controlled part of your website-health strategy.

Sources and Further Reading #

• Google Search Central: Block Search indexing with noindex
• Google Search Central: Robots Meta Tags
• Google Search Central: robots.txt Introduction and Guide
• Google Search Central: Technical requirements
• Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers
• Search Console Help: robots.txt report
• Google Search Central: Meta tags and attributes that Google supports
• Google Search Central Blog: A note on unsupported rules in robots.txt
• Google Search Central Blog: Using the robots meta tag