GPTBot is one of several OpenAI user agents. OpenAI explicitly distinguishes it from OAI-SearchBot (ChatGPT Search) and ChatGPT-User (user-triggered retrieval). This distinction matters because the bots serve different purposes.

GPTBot is the crawler documented by OpenAI that typically fetches content for training future models. It can be specifically allowed or blocked via robots.txt.Open glossary entry → is typically associated with model training. If you do not want content used for training, you can block it:

  • User-agent: GPTBot
  • Disallow: /

It is important not to confuse GPTBot with the search and retrieval bots. Blocking all OpenAI bots indiscriminately may also block the functions relevant for ChatGPT visibility. A common strategy is to block GPTBot while allowing OAI-SearchBot is the crawler documented by OpenAI for ChatGPT Search. It fetches content so that public websites can appear as a source in ChatGPT summaries, snippets and answers.Open glossary entry → and ChatGPT-User is the user agent documented by OpenAI for user-triggered retrieval. It fetches a page when a user enters a specific URL or query in ChatGPT.Open glossary entry →.

GPTBot and the other OpenAI crawlers #

GPTBot collects content for training future OpenAI models. It is a separate decision from OAI-SearchBot (the Indexing is the step in which a search engine processes crawled content and decides whether a URL is added to the search index. Only indexed pages can appear in search results.Open glossary entry → behind ChatGPT search) and ChatGPT-User (live fetches during conversations). Blocking GPTBot only opts you out of training — it does not remove you from ChatGPT's search results or live browsing.

Deciding whether to allow it #

Publishers with paid content often block GPTBot; marketing sites usually benefit from being represented accurately in future models and leave it allowed. Whatever the choice, make it explicitly in The robots.txt is a publicly accessible text file in a site's root directory. It tells crawlers which URLs they may fetch and therefore controls crawling.Open glossary entry → for each bot, and review the decision as the AI landscape changes.