GPTBot

DefinitionGPTBot is OpenAI's web crawler for training its generative AI foundation models. Blocking it in robots.txt signals that your content should not be used for training.

GPTBot is not the crawler that puts pages into ChatGPT search. That is OAI-SearchBot, a separate user agent (OpenAI docs (opens in a new tab)). So you can disallow GPTBot and still allow OAI-SearchBot if you want to be citable in ChatGPT search without contributing to training.

Like most crawlers, GPTBot fetches HTML without running JavaScript, so it never loads a JavaScript tag such as AI Rank Lab's AI bot tracker. Use server logs to see its visits.

Learn it step by step

• AI crawlers vs AI-referred visitors (AEO and GEO fundamentals)
• What the tracker counts, and what it doesn't (AI bot tracker)

Related terms

• AI crawler: An AI crawler is a bot run by an AI company that fetches web pages, for model training, for an AI search index, or because a user asked the assistant to read a page.
• robots.txt: robots.txt is a text file at the root of your site that tells crawlers which paths they may or may not fetch, rule by rule for each user agent.
• LLM (large language model): A large language model is an AI model trained on very large amounts of text to predict and generate language. GPT, Claude and Gemini are LLM families.