Skip to content

Log inBook a demo Start free

AI crawlers vs AI-referred visitors

Why bots that read your site and people sent by AI answers are measured differently.

Lesson 4 of 8
Beginner
7 min read
3:17 video
Checked against the product 2026-11-03

Listen to this lesson · 3:16

In shortAI crawlers are bots that fetch your pages: some collect training data (GPTBot, ClaudeBot), some build search indexes (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and some fetch a page because a user asked (ChatGPT-User, Claude-User). AI-referred visitors are people who click a link in an AI answer. Crawlers show up in server logs, visitors show up in analytics. Mixing them up leads to wrong conclusions.

Two kinds of AI traffic

When people say "AI traffic" they mean one of two very different things. Crawlers are automated programs run by AI companies. They request your pages, read the HTML and leave. AI-referred visitors are humans. They asked ChatGPT, Gemini, Claude, Perplexity or Copilot a question, saw your page cited and clicked through.

The two need different tools to see, answer different questions, and you control them in different ways. Crawlers tell you whether AI systems can read your site. Visitors tell you whether AI answers are sending you people.

The crawlers you should know

Sources: OpenAI, Anthropic, Perplexity and Google crawler documentation, checked November 2026.
User agent	Company	What it's for	Effect of blocking it
GPTBot	OpenAI	Training generative AI foundation models	Signals your content should not be used for training
OAI-SearchBot	OpenAI	Surfacing websites in ChatGPT's search features	Your pages stop appearing as ChatGPT search sources
ChatGPT-User	OpenAI	Visits a page because a user asked ChatGPT to	OpenAI says robots.txt rules may not apply, as the user started it
ClaudeBot	Anthropic	Collecting content that may be used for training	Excludes future content from training
Claude-SearchBot	Anthropic	Improving Claude's search results	Affects search indexing, not training
Claude-User	Anthropic	Fetching a page for a Claude user's question	Claude can't retrieve your page for users
PerplexityBot	Perplexity	Surfacing and linking websites in Perplexity search; not used for foundation models	Your pages stop being surfaced in Perplexity results
Perplexity-User	Perplexity	Fetching a page when a user asks	Perplexity says it generally ignores robots.txt
Google-Extended	Google	A robots.txt token (no crawler of its own) for Gemini training and some grounding	No effect on Google Search, AI Overviews inclusion or ranking

Read the official pages before you change anything: OpenAI (opens in a new tab), Anthropic (opens in a new tab), Perplexity (opens in a new tab) and Google (opens in a new tab). Google's AI Overviews and AI Mode use ordinary Googlebot crawling. The controls Google documents for them are the Search ones: Googlebot rules in robots.txt, and nosnippet, data-nosnippet, max-snippet or noindex (Google Search Central (opens in a new tab)).

Training, search and user fetches are separate choices

Each company now splits its crawlers by purpose, so you can make three separate decisions. A common setup is to block training and allow search, because search crawlers are what make you citable:

User-agent: GPTBot then Disallow: / (no OpenAI training)
User-agent: ClaudeBot then Disallow: / (no Anthropic training)
User-agent: Google-Extended then Disallow: / (no Gemini training; Search unaffected)
Leave OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot allowed.

Check your CDN and firewall too

robots.txt is a request, not a lock, and the opposite problem is common: a security setting at your host or CDN quietly blocks AI search crawlers that robots.txt allows. If you never appear as a source anywhere, check the firewall rules as well as robots.txt. OpenAI says changes can take about 24 hours to reach its search systems.

How each one is measured

Crawlers are best seen in your server or CDN access logs, filtered by user agent. Most crawlers fetch raw HTML and don't run JavaScript, so JavaScript analytics tools such as GA4 never see them.

AI-referred visitors run your pages in a real browser, so analytics see them. They arrive with a referrer such as chatgpt.com, perplexity.ai, gemini.google.com or claude.ai, and links from ChatGPT often carry utm_source=chatgpt.com. Some clicks from apps arrive with no referrer at all and land in Direct traffic, so referral numbers are a floor, not a total. See Track AI referral traffic in GA4.

What the volumes look like

Across 3,906 AI-referred page views recorded by the AI Rank Lab tag on 31 sites (July 2025 to November 2026; page views, not unique users), ChatGPT sent 54% of AI referrals, Gemini 19%, Claude 13%, Perplexity 12% and Copilot 2%. At the median site, AI referrals were about 0.2% of all events the tag recorded. Small, but these are visitors who arrive having already read a recommendation.

How AI Rank Lab helps

The AI bot tracker (AI search → Monitoring and reports → AI bot tracker) reads data from a JavaScript tag you install on your site. That makes it a tool for the visitor side: it shows AI-referred visits, which pages they land on and which assistant sent them, plus AI agents that load pages in a real browser. Crawlers that never execute JavaScript, such as GPTBot, ClaudeBot and PerplexityBot, never load the tag, so they are not in its numbers. Use your server logs for those.

The tag is billed and pauses at zero

The tracker charges per block of tracked requests (see the cost table), and it stops collecting when your credit balance reaches zero. Recharging restarts it, but the gap is not back-filled.

Check yourself

1. You block GPTBot in robots.txt. What happens to your pages in ChatGPT search?

2. Why doesn't a JavaScript analytics tag show visits from most AI crawlers?

3. You want to opt out of Gemini training but stay in Google AI Overviews. What do you do?

0 of 3 answered.

Progress is saved in this browser.

Transcript

In this lesson, we'll separate two things people often lump together as AI traffic: AI crawlers and AI-referred visitors. Crawlers are programs run by AI companies. They request your pages, read the HTML, and leave. Visitors are people. They asked an AI assistant a question, saw your page cited, and clicked. Crawlers tell you whether AI systems can read your site. Visitors tell you whether AI answers send you people. Different questions, different tools. Let's meet the crawlers. Each company now splits them by purpose. OpenAI has GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for pages a user asks ChatGPT to open. OpenAI says robots.txt rules may not apply to that last one, because a person started it. Anthropic has ClaudeBot for training, Claude-SearchBot for search, and Claude-User for fetches on a user's behalf. Perplexity has PerplexityBot for its search results, which it says isn't used to train foundation models, and Perplexity-User, which it says generally ignores robots.txt. Google has Google-Extended. That's not a crawler, it's a robots.txt token. It controls whether your content is used for Gemini training and some grounding. It doesn't affect Google Search or AI Overviews. Because they're split, you can make separate choices. A common setup is to block the training crawlers and allow the search crawlers, because search crawlers are what make you citable. One warning. Sometimes a firewall or CDN setting blocks AI search crawlers that your robots.txt allows. If you never show up as a source anywhere, check both. Now, measurement. Crawlers are best seen in your server or CDN logs. Most of them don't run JavaScript, so tools like GA4 never see them. Visitors do run JavaScript, so analytics see them. They arrive from chatgpt.com, perplexity.ai, gemini.google.com or claude.ai. Links from ChatGPT often carry utm source chatgpt dot com. Some app clicks arrive with no referrer and land in Direct, so treat referral numbers as a floor. How big is this traffic? Across about thirty-nine hundred AI-referred page views recorded by the AI Rank Lab tag on thirty-one sites, ChatGPT sent just over half, then Gemini, Claude and Perplexity. For the median site it was about 0.2 percent of everything the tag recorded. Small, but these people arrive already recommended. AI Rank Lab's AI bot tracker uses a JavaScript tag, so it's built for the visitor side. It shows AI-referred visits, landing pages and which assistant sent them. Crawlers that never run JavaScript don't load the tag, so use your server logs for those. The tracker is billed per block of requests and pauses if your credits run out. In the next lesson, we'll look honestly at structured data and llms.txt.

Put the lesson to work

Paste your site and get a free SEO, AEO and GEO audit with the fixes, then come back to the next lesson.

Start free Book a demo