AI crawler

DefinitionAn AI crawler is a bot run by an AI company that fetches web pages, for model training, for an AI search index, or because a user asked the assistant to read a page.

Companies now split crawlers by purpose. OpenAI has GPTBot (training), OAI-SearchBot (ChatGPT search) and ChatGPT-User (user-triggered visits). Anthropic has ClaudeBot, Claude-SearchBot and Claude-User. Perplexity has PerplexityBot and Perplexity-User. Google uses Googlebot for Search and a robots.txt token, Google-Extended, for Gemini training. You can allow search crawlers and block training ones.

Most crawlers don't run JavaScript, so JavaScript analytics tools and tags don't see them; check your server or CDN logs. See AI crawlers vs AI-referred visitors.

Learn it step by step

• AI crawlers vs AI-referred visitors (AEO and GEO fundamentals)
• What the tracker counts, and what it doesn't (AI bot tracker)

Related terms

• GPTBot: GPTBot is OpenAI's web crawler for training its generative AI foundation models. Blocking it in robots.txt signals that your content should not be used for training.
• robots.txt: robots.txt is a text file at the root of your site that tells crawlers which paths they may or may not fetch, rule by rule for each user agent.
• AI-referred traffic: AI-referred traffic is people who arrive on your site by clicking a link in an AI answer, from assistants such as ChatGPT, Gemini, Claude, Perplexity or Copilot.
• Tracking tag: The tracking tag is the small JavaScript snippet you add to your site so AI Rank Lab can record AI-referred visitors and AI agents that load your pages in a browser.