ClaudeBot

ClaudeBot is the crawlerCrawling is the process by which a search engine bot fetches a URL and downloads its content. It is the first step before a page can be indexed.Open glossary entry → of Anthropic, the provider of the Claude family of AI assistants. Anthropic describes for website owners the option to allow or block ClaudeBotClaudeBot is the web crawler used by Anthropic. Website owners can control its access via robots.txt and block it if needed.Open glossary entry →'s access via robots.txtThe robots.txt is a publicly accessible text file in a site's root directory. It tells crawlers which URLs they may fetch and therefore controls crawling.Open glossary entry →.

Example to block it:

• User-agent: ClaudeBot
• Disallow: /

As with other AI crawlersAI crawlers are automated bots that fetch web pages for AI providers in order to find content, make it available for search functions, retrieve it for answers, or, depending on the provider, use it for model training.Open glossary entry →, the decision is a matter of data strategy. If you do not want to release content for certain AI uses, you can restrict ClaudeBot. If you aim for the broadest possible AI VisibilityAI Visibility describes whether and how visible a brand, website or person is in AI-generated answers. It measures mention, position, description and source linking across multiple platforms and prompts.Open glossary entry →, you allow it. robots.txt is cooperation-based and not a security mechanism; sensitive content needs login, server or WAF protection.