llms.txt & robots.txt: Control AI Crawlers Without Losing AI Visibility

Your content can be technically strong, up to date and prepared for search engine optimization (SEO)Search Engine Optimization (SEO) covers all measures that help a website become more visible in unpaid (organic) search results.Open glossary entry →, and still barely appear in ChatGPT, Perplexity or other search systems powered by artificial intelligence (AI).

The reason is often not the content, but the technical setup. AI crawlersAI crawlers are automated bots that fetch web pages for AI providers in order to find content, make it available for search functions, retrieve it for answers, or, depending on the provider, use it for model training.Open glossary entry → are blocked by robots.txt, firewalls or the bot management of a content delivery network (CDN)A content delivery network (CDN) is a network of distributed servers that delivers static content closer to the user, reducing loading times.Open glossary entry →. A CDN is a network of distributed servers that delivers pages faster. Sometimes the block is intentional. Often, however, it happens by accident.

RankScan has two measures for this topic:

• “AI crawlers blocked in robots.txtThe robots.txt is a publicly accessible text file in a site's root directory. It tells crawlers which URLs they may fetch and therefore controls crawling.Open glossary entry →”: robots.txt blocks crawlers responsible for AI search or live fetching. If RankScan finds the block in the firewall instead, the measure can be called “AI crawlers probably blocked by a firewall”.
• “No llms.txtllms.txt is a voluntary Markdown file in a website's root directory that gives language models a compact overview of important content. It is a proposal or de facto format, not a binding official web standard.Open glossary entry →”: There is no file at /llms.txt. This is an observation, not an error. The measure is a hint with no action needed.

The weighting is clear: Blocked search and fetch crawlers are a critical technical problem. A missing llms.txt is not.

The best solution is therefore neither “allow everything” nor “block everything”. Decide deliberately: Which bots may crawlCrawling is the process by which a search engine bot fetches a URL and downloads its content. It is the first step before a page can be indexed.Open glossary entry → for training? Which bots do you need for AI search and live fetching?

Why AI Crawlers Are a Website Health Topic #

AI search systems use web content in different ways: for training, search, summaries, source links or fetching on request. For companies, this creates a new technical dependency.

If a search or fetch crawler cannot reach your website, this can have several consequences:

• Content is not considered, or is considered less, in AI search systems.
• Your brand is not mentioned in answers.
• Competitors appear as sources even though your content would be stronger.
• Monitoring and optimization become harder because it is unclear where the block occurs.

The problem is often invisible. A page can be indexed in Google and still be unreachable for specific AI crawlers.

Typical causes include:

• restrictive robots.txt,
• Web Application Firewall,
• Cloudflare or CDN bot management,
• rate limiting,
• blocked user agents or IP ranges,
• staging rules that accidentally stayed live,
• security plugins with broad bot blocks.

What Does “AI crawlers blocked in robots.txt” Mean? #

The RankScan measure “AI crawlers blocked in robots.txt” means: your robots.txt excludes exactly the AI crawlers that decide whether your website can show up in AI answers. These are search-index crawlers and live-fetch agents.

Rating in RankScan: critical severity · low effort · work type Development.

What RankScan measures:

• RankScan reads robots.txt in every crawl and compares it with the crawler list further below.
• A group counts if it names the AI crawler and contains Disallow: /. If the same group contains Allow: /, the block does not apply.
• If User-agent: * blocks the whole website with Disallow: /, every AI crawler without a group of its own counts as blocked. A crawler with its own group follows only that group.
• Blocking single directories such as Disallow: /internal/ does not trigger the measure.
• If robots.txt only blocks training crawlers such as GPTBot or ClaudeBot, no measure is created. That is a deliberate decision and costs no visibility.

Selective access separates the training question from discoverability. Your content stays citable without automatically feeding model training.

Steps #

1. Have your developers or agency adjust the robots.txt file of your website.
2. Have every search and fetch bot allowed: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBotPerplexityBot is the crawler documented by Perplexity, intended to make websites visible in Perplexity search results and link them as a source. According to Perplexity, it is not used to train foundation models.Open glossary entry →, Perplexity-User, Applebot, meta-webindexer, meta-externalfetcher, DuckAssistBot, YouBot and MistralAI-User. The table below shows what each of these bots is for. RankScan counts each of these bots.
3. You can keep blocking training crawlers.
4. Mark the measure as done once the new file is live.

How RankScan checks it: The next crawl reads robots.txt in full, even if it does not reach every page. RankScan then reports “Fixed” or “Not fixed yet”.

Variant: “AI crawlers probably blocked by a firewall” #

RankScan is rolling out this check gradually. It does not yet appear in every account.

robots.txt can be clean and AI crawlers still cannot get through. In that case, a firewall or the bot management of a CDN is usually blocking them.

This is how RankScan detects the case:

• RankScan fetches the home page once as its own crawler. This fetch serves as a control.
• Then RankScan fetches the same page as OAI-SearchBot, PerplexityBot and Claude-SearchBot. RankScan does not fetch as agents that robots.txt blocks. If robots.txt blocks the agent for the whole website, the robots.txt measure applies. This includes a block via User-agent: *.
• If the control gets the page but an AI crawler does not, RankScan reports a probable block. RankScan reads from the response headers which service blocks, for example Cloudflare.
• If an AI assistant has cited a page of your website in the last 30 days, its crawler does not count as blocked. CDNs often recognize real bots by their IP address and only let those through.
• If the control fails or the server responds with an error, RankScan measures nothing in that run.

Rating in RankScan: If all three crawlers are blocked, the severity is critical. With one or two, it is high. Low effort · work type Development.

The block only hits AI crawlers. Visitors and Google see the website as usual. That is why it rarely gets noticed.

1. Have your developers or agency check the bot settings of your CDN or firewall.
2. Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Claude-User.
3. You can keep blocking training crawlers such as GPTBot and ClaudeBot.
4. Mark the measure as done once the setting is saved.

How RankScan checks it: In the next run, RankScan fetches the home page as an AI crawler again.

What Are AI Crawlers? #

AI crawlers are automated bots that fetch web pages. AI providers use them to find content, make it available for search features, fetch it for answers or use it for model training.

OpenAI distinguishes several user agents in its own documentation, including GPTBot, OAI-SearchBot and ChatGPT-User. They serve different purposes and can be controlled separately via robots.txt.
Source: OpenAI Platform: Overview of OpenAI Crawlers

For website owners, this distinction is essential. Anyone who blocks all OpenAI bots across the board does not only block training, but also the search and fetch functions that matter for visibility in ChatGPT.

Training, Search Index and Fetching: The Critical Difference #

Not every AI bot does the same thing. For a sensible robots.txt strategy, distinguish three purposes.

1. Training #

In training, content flows into future AI models. Crawlers such as GPTBot or ClaudeBot serve this purpose.

OpenAI writes: blocking GPTBot indicates that the content should not be used to train generative AI foundation models.
Source: OpenAI Platform: Overview of OpenAI Crawlers

User-agent: GPTBot
Disallow: /

This can make sense for legal, strategic or editorial reasons. A training block costs no visibility in AI answers. Google-Extended is the exception: blocking it also excludes grounding in Gemini Apps and Vertex AI. In RankScan, this block does not lower the score.

2. Search Index #

Search-index crawlers build the index an AI product answers from. They include OAI-SearchBot from OpenAI, Claude-SearchBot from Anthropic and PerplexityBot from Perplexity. Blocking them removes you as a source.

OpenAI writes: sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers.
Source: OpenAI Platform: Overview of OpenAI Crawlers

Anthropic writes: blocking Claude-SearchBot prevents Claude from indexing your content for search.
Source: Anthropic Support: Does Anthropic crawl data from the web?

Perplexity describes PerplexityBot as a crawler that surfaces and links websites in Perplexity search results. According to Perplexity, this bot is not used to train foundation models.
Source: Perplexity Docs: Perplexity Crawlers

Not every assistant has its own search index crawler. Microsoft Copilot draws on the Bing index, which Bingbot builds. By common account the same holds for ChatGPT's web search, but this is not settled. So allow Bingbot. What Bing knows about your website is shown in Bing Webmaster Tools.

3. Fetching on Request #

Fetch bots retrieve a page live while an AI assistant answers a user. They include ChatGPT-User, Claude-User and Perplexity-User.

For example, a user enters your URL in ChatGPT and expects a summary. Whether a block in robots.txt prevents the fetch depends on the provider. According to Anthropic, blocking Claude-User prevents Claude from retrieving your content in response to a user query.
Source: Anthropic Support: Does Anthropic crawl data from the web?

OpenAI and Perplexity point out that their fetch bots do not follow robots.txt rules in every case. The reason: a user triggered the fetch.
Sources: OpenAI Platform: Overview of OpenAI Crawlers, Perplexity Docs: Perplexity Crawlers

robots.txt: What It Can Do and What It Cannot Do #

robots.txt is located in the root directory of your website:

https://example.com/robots.txt

It tells crawlers which areas they may access and which they may not. Google describes robots.txt as a tool for managing crawler traffic, not as a way to reliably keep web pages out of Google or off the web.
Source: Google Search Central: robots.txt Introduction and Guide

A simple example:

User-agent: *
Disallow: /internal/

Sitemap: https://example.com/sitemap.xml

This means: cooperating crawlers should not access /internal/. The sitemap also shows where important indexable URLs can be found. How robots.txt and noindexnoindex is a robots directive telling search engines that a page should not appear in search results. The page must remain crawlable for it to work.Open glossary entry → work together is explained in the article on noindex and robots.txt.

Important:

• robots.txt is based on cooperation.
• Reputable crawlers usually respect it.
• Not all bots follow it reliably.
• Confidential content must not be protected only through robots.txt.
• Only one group applies to each crawler: the one with the most specific matching user agent. It ignores all other groups, including User-agent: *.
  Source: Google Search Central: How Google interprets the robots.txt specification

A 2025 study examined robots.txt compliance among different scrapers. It concluded that not all bot categories reliably follow robots.txt. So do not treat robots.txt as your only protection mechanism.
Source: arXiv: Scrapers selectively respect robots.txt directives

The Most Important AI Crawlers and User Agents #

The table follows the list RankScan uses to read your robots.txt. It assigns each user agent to a purpose. Only blocking search-index or fetch bots triggers the measure “AI crawlers blocked in robots.txt”.

For fetch bots, robots.txt is not a reliable block. A user triggers the fetch, and some providers do not apply robots.txt then. If you want to prevent access reliably, block it on the server, for example with a firewall or access protection. You control inclusion in ChatGPT search with OAI-SearchBot, not with ChatGPT-User.

Anthropic no longer uses anthropic-ai and Claude-Web. They still appear in many robots.txt files, so RankScan keeps recognizing them.

The regular Googlebot is not on the list. It is not an AI crawler in the sense of this measure. If you block it, you lose classic Google Search.

Googlebot vs. Google-Extended: The Most Common Mistake #

A particularly important point is the distinction between Googlebot and Google-Extended.

• Googlebot is relevant for classic Google Search.
• Google-Extended is a control token for certain AI uses by Google.

Google explains that Google-Extended is not a separate user agent. Crawling is done with existing Google user agents; Google-Extended is used in robots.txt as a product token. According to Google, Google-Extended controls whether content is used to train future Gemini models. It also controls grounding in Gemini Apps and in “Grounding with Google Search” on Vertex AI. Grounding means: the model receives content from the Google Search index for a request. According to Google, Google-Extended affects neither inclusion in Google Search nor ranking.
Source: Google Crawling Infrastructure: Google-Extended

Example:

User-agent: Google-Extended
Disallow: /

This rule is different from:

User-agent: Googlebot
Disallow: /

The second rule puts your classic Google visibility at risk. Therefore:

If you want to restrict Google AI use, do not accidentally block Googlebot.

Google AI Overviews and AI Mode are part of Google Search. According to Google, Google-Extended does not control them. For these features, the Search controls apply: robots.txt for Googlebot, nosnippet, data-nosnippet, max-snippet and noindex.
Source: Google Search Central: AI features and your website

GPTBot robots.txt: How to Control OpenAI Correctly #

Treat the OpenAI crawlers separately. Many companies want to restrict training but keep visibility in AI search:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

This is often more sensible than blocking all OpenAI crawlers across the board. The same pattern applies to Anthropic with ClaudeBot, Claude-SearchBot and Claude-User.

Three Strategies for robots.txt and AI Crawlers #

There is no universally correct setting. The right strategy depends on how openly your content may be used.

Strategy A: Maximum AI Visibility #

This strategy suits websites that want to be as open as possible for AI systems. Without a block, the general group applies to all bots:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Suitable for:

• guide portals,
• SaaS websites,
• public documentation,
• companies with a strong thought leadership focus,
• brands that want maximum discoverability.

Risk: Depending on the provider, content may also flow into training.

Strategy B: Block Training, Allow AI Search #

This strategy suits companies that do not want to release content for model training but want to stay visible in AI answers. Note: Blocking Google-Extended also excludes grounding in Gemini Apps and Vertex AI, not only training. If you want to stay visible in Gemini Apps, leave Google-Extended open. In RankScan, this block does not lower the score.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /

Sitemap: https://example.com/sitemap.xml

Suitable for:

• B2B companies,
• publishers with a differentiated data strategy,
• brands with high content value.

Risk: The exact separation between training, search and fetching depends on the provider and can change.

Strategy C: Protect Sensitive Areas #

This strategy suits websites where public content should remain visible, but individual areas must not be crawled.

User-agent: *
Disallow: /internal/
Disallow: /staging/
Disallow: /downloads/confidential/

Sitemap: https://example.com/sitemap.xml

Do not give an AI crawler its own group with Allow: / here. It would then ignore the general group and fetch the blocked areas. If you do need a separate group, repeat the Disallow lines in it.

Important: Confidential content must not be protected only through robots.txt. If content is truly private, you need login protection, server-side rules or WAF configurations.

What Is an llms.txt? #

llms.txt is a Markdown file in the root directory of a website:

https://example.com/llms.txt

It is meant to give language models a compact overview of important content. Jeremy Howard of Answer.AI proposed the idea in 2024. The file is meant to help large language models (LLMs) capture the relevant content of a website faster.
Sources: llms.txt: The /llms.txt file, Answer.AI: /llms.txt proposal

The realistic assessment: llms.txt is a voluntary proposal, not a web standard. No AI system is known to read the file. It does not change your visibility in search engines or AI answers.

What Does “No llms.txt” Mean? #

The RankScan measure “No llms.txt” means: there is no file at /llms.txt. This is an observation, not an error.

Rating in RankScan: low severity · low effort · work type Development. The measure is a hint with no action needed. It does not count toward your open measures.

Steps #

1. Check whether a tool or partner explicitly requires the llms.txt file.
2. If not, ignore the measure.
3. Otherwise have your developers or agency create the file at /llms.txt, following the requirements of the tool or partner.
4. Mark the measure as done.

How RankScan checks it: The next crawl checks the file, even if it does not reach every page.

If you do create the file: it is public. Do not list internal addresses or confidential information in it.

What an AI Crawler Check Generally Looks At #

The following list is a general checklist. It goes beyond what RankScan measures.

• Is /robots.txt reachable?
• Are there blanket blocks for all bots?
• Are search-index or fetch bots explicitly blocked?
• Is Googlebot accidentally affected by AI rules?
• Is Google-Extended used correctly?
• Are training, search index and fetching controlled separately?
• Is the sitemapAn XML sitemap is a machine-readable file listing a website's important URLs. It helps search engines discover content more efficiently but does not replace internal linking or guarantee indexing.Open glossary entry → referenced in robots.txt?
• Do AI crawlers receive 401, 403, 429 or 5xx in the server logs?
• Does a firewall block relevant bots despite a permissive robots.txt?

RankScan measures two of these points. First, RankScan reads robots.txt and reports site-wide blocks of search-index and fetch bots. Second, in the firewall variant, RankScan fetches the home page as an AI crawler. RankScan does not check server logs, rate limits or subpages.

How to Check Whether AI Crawlers Are Blocked #

1. Open robots.txt Directly #

Open this in your browser:

https://your-domain.com/robots.txt

Look for rules such as:

User-agent: OAI-SearchBot
Disallow: /

or:

User-agent: *
Disallow: /

Such rules block AI crawlers completely. The second rule blocks all bots that have no group of their own.

2. Check Server Logs #

Server logs show which bots visit your website and which status codes they receive.

Important status codes:

If OAI-SearchBot, PerplexityBot or other search and fetch bots repeatedly receive 403, that is a clear warning sign.

3. Check Firewall, CDN and Bot Management #

Many blocks do not originate in robots.txt, but at the infrastructure level.

Typical systems:

• Cloudflare,
• Akamai,
• Fastly,
• WordPress security plugins,
• bot management rules,
• rate limiting,
• blocks by country or by Autonomous System Number (ASN), the identifier of a network range.

Cloudflare offers a managed robots.txt. By default, it only blocks training crawlers such as GPTBot, ClaudeBot and Google-Extended. This block does not trigger a measure in RankScan. Note: Blocking Google-Extended also excludes grounding in Gemini Apps and Vertex AI. If you also set the search policy there to “Block”, the block hits search bots. RankScan can then report the measure.
Source: Cloudflare Docs: Managed robots.txt

Common Mistakes With robots.txt and AI Crawlers #

Mistake 1: Blocking All Bots #

User-agent: *
Disallow: /

This rule makes sense for staging websites. On a live website, it prevents classic SEO and AI visibility. RankScan reports it as “AI crawlers blocked in robots.txt”.

Mistake 2: Blocking Googlebot Instead of Google-Extended #

Blocking Googlebot puts classic Google Search at risk. To control Gemini training and grounding in Gemini Apps and Vertex AI, use Google-Extended. Google-Extended does not control AI Overviews and AI Mode.

Mistake 3: Treating Training and Search the Same #

Many teams block all bots of a provider together. For visibility in ChatGPT, Claude and Perplexity, however, you need the search-index and fetch bots: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot.

Mistake 4: Treating robots.txt as a Security System #

robots.txt does not protect confidential content. If you want to protect sensitive data, you need server-side access control.

Mistake 5: Selling llms.txt as a Lever #

No AI system is known to read llms.txt. The file brings neither better rankings nor more mentions in AI answers. Spend the time on access for search and fetch bots instead.

Mistake 6: Forgetting the Firewall #

robots.txt can look correct while a WAF still blocks access. Without server logs or a fetch as an AI crawler, the problem often stays invisible.

Example: When the Firewall Prevents AI Visibility #

A B2B company ranks well in Google, but is rarely mentioned as a source in ChatGPT and Perplexity.

robots.txt looks clean:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Still, the server logs show:

OAI-SearchBot    403 Forbidden
PerplexityBot    403 Forbidden

The cause is not robots.txt, but the Web Application Firewall. It automatically blocks unknown bots. In RankScan, this case can appear as “AI crawlers probably blocked by a firewall”.

The solution:

1. Have your developers or agency check the bot settings of the firewall.
2. Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Claude-User.
3. You can keep blocking training crawlers such as GPTBot and ClaudeBot.
4. Mark the measure as done once the setting is saved.

The example shows: AI visibility is not only content optimization. It starts with technical accessibility.

Checklist: Control AI Crawlers Deliberately #

Check your website:

• Is /robots.txt reachable?
• Is the sitemap linked in robots.txt?
• Is there a blanket block via User-agent: *?
• Is the regular Googlebot blocked?
• Is Google-Extended used deliberately?
• Are training, search index and fetching controlled separately?
• Are OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed?
• Are ChatGPT-User and Claude-User allowed?
• Are there WAF, CDN or security rules that block AI bots?
• Do search and fetch bots receive status code 200HTTP status 200 (OK) means a page was found and served successfully. Only 200 URLs should appear in the sitemap and serve as canonical targets.Open glossary entry → in the server logs?

In addition, the AI Readiness Score and JavaScript SEO help narrow down the cause cleanly and prioritize the next SEO actions.

Conclusion: Control AI Crawlers Deliberately Instead of Blocking Them by Accident #

AI visibility does not come from good content alone. It requires AI systems to reach your content technically.

The most important measure is therefore the technical check:

• Can search-index and fetch bots access your website?
• Are they blocked by robots.txt?
• Are they blocked by a firewall, CDN or bot management?
• Are training, search index and fetching cleanly separated?

robots.txt is the access control.
Server logs and WAF rules show what really happens.
An llms.txt can wait until a tool or partner asks for it.

For many companies, the best strategy is differentiated control:

• deliberately allow or block training,
• enable search index and fetching,
• protect sensitive areas technically,
• check server logs and WAF rules regularly.

This keeps your website from becoming invisible to AI systems by accident.

Sources and Further Reading #

• OpenAI Platform: Overview of OpenAI Crawlers
• Google Search Central: robots.txt Introduction and Guide
• Google Search Central: How Google interprets the robots.txt specification
• Google Crawling Infrastructure: Google-Extended
• Anthropic Support: Does Anthropic crawl data from the web?
• Perplexity Docs: Perplexity Crawlers
• Cloudflare Docs: Managed robots.txt
• llms.txt: The /llms.txt file
• Answer.AI: /llms.txt proposal
• arXiv: Scrapers selectively respect robots.txt directives