AI Crawlers in 2026: Every User Agent and Which to Allow

AI crawlers are automated bots run by AI companies that fetch web pages for one of three jobs: collecting training data for models (GPTBot, ClaudeBot), building the search index that AI answers cite (OAI-SearchBot, Claude-SearchBot, PerplexityBot), or fetching a page live because a user asked an assistant about it (ChatGPT-User, Claude-User, Perplexity-User). Blocking a training crawler keeps your content out of future models; blocking a search crawler or user fetcher removes you from AI answers and citations. That difference is the whole decision.

This page is a reference you can act on. Every user agent below was checked against the operator's own documentation on October 18, 2026, with a link to that page. You also get a decision matrix by business goal, copy-ready robots.txt groups, the verification method each operator supports (with the date stamped inside each published IP list), and an honest look at why JavaScript analytics cannot show you most crawler traffic.

What are AI crawlers, and what are the three classes?

An AI crawler is any automated client that an AI company uses to request your pages. Operators now split their bots by purpose, and the purpose decides what blocking costs you. Cloudflare's network data shows how lopsided the mix is: over the 12 months to mid-2025, 80% of AI crawling was for training, 18% for search and 2% for user actions, with training rising to 82% in the most recent six months of that window.

Training crawlers

Training crawlers collect content that may be used to train or fine-tune foundation models. GPTBot, ClaudeBot, Meta-ExternalAgent, Amazonbot, MistralAI-Training and CCBot (Common Crawl's open dataset, which many model builders use) belong here. Blocking them is a signal that your future content should stay out of training sets. It does not remove content already collected, and it does not affect whether AI search can cite you.

Search and index crawlers

Search crawlers build the retrieval index that AI answers are grounded in and that citations link back to. OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Amzn-SearchBot, MistralAI-Index and DuckAssistBot are AI-specific. Googlebot, Bingbot and Applebot are classic search crawlers whose index now also feeds AI features: Google AI Overviews and AI Mode, Microsoft Copilot, and Siri answers. Blocking a search crawler is the expensive mistake, because it removes you from the answers your buyers read. If you want the mechanics of how one of these indexes is used, see how ChatGPT search works.

User-triggered fetchers and agents

User-triggered fetchers visit a single URL because a person asked an assistant to read it, summarize it or act on it. ChatGPT-User, Claude-User, Perplexity-User, Google-Agent, Meta-ExternalFetcher, Amzn-User and MistralAI-User belong here, plus signed browser agents such as ChatGPT's Cloud browser. Several operators state that robots.txt may not apply to these, because a human initiated the request.

Control tokens that are not crawlers

Two names on every list never fetch anything themselves. Google-Extended is a robots.txt product token that Google reads while crawling with its normal user agents; it controls whether content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Applebot-Extended does the same for Apple's foundation models. You will not find either string in your access logs, which is why "I blocked Google-Extended but still see Google traffic" is expected behavior.

The AI crawler list: every major user agent, verified October 18, 2026

The table lists the token you put on a User-agent: line, who runs it, its class, what the operator's own documentation says about robots.txt, how you can verify it, and the official page we checked. "Per doc" means the robots.txt column paraphrases the operator; where a doc is silent, we say so rather than guess.

Two things the table settles. First, Anthropic's current page lists exactly three bots: ClaudeBot, Claude-SearchBot and Claude-User. Older lists still carry "anthropic-ai" and "Claude-Web", which do not appear in Anthropic's documentation as of October 2026. Second, OpenAI's version numbers move: on the day we checked, OpenAI's example strings showed GPTBot/1.4 and OAI-SearchBot/1.4, so match on the token, never the full string.

Bots without official documentation

Some bots you will see in logs have no operator page we could find on October 18, 2026. ByteDance's Bytespider is the best known; xAI's Grok-related agents are another. Third-party directories disagree on their behavior, so we have left them out of the verified table. Cloudflare's managed robots.txt does include a Bytespider disallow, and Cloudflare reported Bytespider's share of AI crawling falling from 14.1% to 2.4% between mid-2024 and mid-2025. If one of these bots matters to you, enforce your policy with a WAF rule rather than relying on robots.txt.

Do AI crawlers respect robots.txt?

Documented training and search crawlers say they do; user-triggered fetchers often say they do not; and robots.txt is never access control. The standard itself, RFC 9309, states that "these rules are not a form of access authorization." It also lets crawlers cache your file for up to 24 hours, which is why every operator quotes a delay before changes apply.

Compliance claims are not the same as observed behavior. In August 2025 Cloudflare published a test in which brand-new domains that disallowed everything were still described in detail by Perplexity. Cloudflare said it observed a declared Perplexity-User agent making 20-25 million daily requests and an undeclared Chrome-like agent making 3-6 million, then de-listed Perplexity as a verified bot. That is one vendor's finding about one operator, but it shows why a user agent string alone proves nothing.

What does blocking each class of AI crawler cost you?

Blocking training crawlers costs you little in visibility today; blocking search crawlers or user fetchers costs you citations immediately. The table pairs each class with the cost the operators themselves describe.

The scale of the exchange matters too. Cloudflare measured how many pages each AI platform crawls per visitor it refers back: in July 2025 Anthropic sat at about 38,000 crawls per referred visitor and Perplexity at 194. Those ratios are the economic argument publishers make for blocking training crawlers while keeping search crawlers open.

Which AI crawlers should you allow or block? A decision matrix by business goal

Start from what you sell. If being recommended by an AI assistant can win you a customer, allow every search crawler and user fetcher, and treat training as a separate, smaller decision.

How do you block AI training crawlers but allow AI search in robots.txt?

Give each training token its own group with Disallow: / and leave search crawlers covered by your normal rules. A crawler obeys only the most specific group that names it, so a training block never spills over to search bots.

Snippet 1: block training, keep AI search and user fetchers

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

RFC 9309 allows several User-agent lines to share one group, and matching is case-insensitive. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot fall through to the * group and stay allowed.

Snippet 2: be explicitly open to AI search

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Meta-WebIndexer
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
User-agent: DuckAssistBot
Allow: /
Disallow: /cart/
Disallow: /account/

Naming search bots explicitly protects you from a future blanket rule written by someone else, such as a plugin that adds User-agent: * with Disallow: /. Keep private paths disallowed inside the group, because a named group replaces the * group for that bot rather than adding to it.

Snippet 3: three gotchas that break real files

• Applebot inherits Googlebot rules. Apple says that if your file doesn't mention Applebot but does mention Googlebot, Applebot follows the Googlebot group. A Googlebot-specific block can silently remove you from Siri and Spotlight.
• Amzn-SearchBot inherits other search bots' rules when it isn't named, per Amazon. Name it if you want a different outcome.
• Subdomains need their own file. Anthropic and Amazon both say they read robots.txt per host, so docs.example.com needs its own copy.

You can build and check these groups with AI Rank Lab's free robots.txt generator, and our earlier robots.txt templates for AI crawlers go deeper on full-access and selective-access setups.

How do you verify that an AI crawler is real and not spoofed?

Never trust the user agent alone: anyone can send "GPTBot" in a header, and Common Crawl warns on its own page that crawlers falsely identify themselves as CCBot. Operators support three verification methods, and which one you use depends on the operator.

1. Reverse DNS, then forward DNS. Run host on the IP from your log, check that the hostname ends in the operator's domain, then run host on that hostname and confirm it returns the same IP. Google documents this for googlebot.com, google.com and googleusercontent.com in its verification guide; Apple uses *.applebot.apple.com; Common Crawl uses *.crawl.commoncrawl.org (IPv4 only).
2. Match the IP against the operator's published list. OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, DuckDuckGo, Mistral and Amazon publish IP ranges. This is the only method for OpenAI, Anthropic and Perplexity, which do not document a reverse DNS pattern.
3. Check a cryptographic signature (Web Bot Auth). OpenAI's ChatGPT Cloud browser signs its requests under RFC 9421 with a Signature-Agent header of "https://chatgpt.com". Google is testing the same IETF draft for some Google-Agent requests, signed as https://agent.bot.goog, and its Web Bot Auth page says to keep IP checks as a fallback because not every request is signed.

# Reverse then forward lookup (Google example)
host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

# IP-list check for OpenAI's search crawler
curl -s https://openai.com/searchbot.json | python3 -c "
import json, sys, ipaddress
ip = ipaddress.ip_address(sys.argv[1])
nets = [p.get('ipv4Prefix') or p.get('ipv6Prefix') for p in json.load(sys.stdin)['prefixes']]
print(any(ip in ipaddress.ip_network(n) for n in nets))
" 203.0.113.7

How fresh are the published IP lists?

We downloaded every published AI crawler IP file on October 18, 2026 and read the creationTime field each operator writes into it. An old date may simply mean the ranges have not changed, but it tells you how much to trust a hard-coded allowlist, and it shows which lists cannot account for traffic the operator never declared.

Two practical lessons. Google's and OpenAI's ChatGPT-User lists changed within the last two weeks, so a firewall allowlist you copied in the summer is probably already wrong; fetch these files daily or weekly instead. And Anthropic says that blocking its IPs "may not work correctly or persistently guarantee an opt-out" because it stops Anthropic from reading your robots.txt, so use robots.txt to opt out and IPs only to verify.

How can you see AI crawler activity: server logs or JavaScript tags?

Use server or CDN logs for crawlers and JavaScript analytics for human visitors who arrive from AI assistants. A JS tag runs only when something executes the page's JavaScript, and most AI crawlers do not. Vercel's analysis of its network found that none of the major AI crawlers rendered JavaScript; ChatGPT's and Claude's crawlers fetched JS files (11.50% and 23.84% of their requests) but did not execute them. The exceptions were Gemini, which uses Googlebot's rendering, and Applebot. That study is from December 2024, so retest on your own site, but it matches what we see.

Google Analytics makes the gap wider by design. GA4 automatically excludes traffic from known bots and spiders, and you cannot switch that off or see how much was removed. GA4 is the right tool for AI referral traffic, which we cover in how to track AI referral traffic in GA4, and the wrong tool for crawler counts.

We see the same pattern in AI Rank Lab's own tracking data. Our JavaScript tag reliably records people who click through from ChatGPT, Gemini, Perplexity and Claude, but it sees only a small, unrepresentative slice of crawler traffic, which is why we do not publish crawler counts from it.

A five-minute log check

1. Download a day of raw access logs from your host or CDN (Nginx and Apache both log the user agent in the default combined format).
2. Count requests per AI token: grep -oiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Meta-WebIndexer|meta-externalagent|Amazonbot|Amzn-SearchBot|Applebot|bingbot|Googlebot|CCBot|DuckAssistBot|MistralAI-[A-Za-z]+" access.log | sort | uniq -c | sort -rn
3. For the top tokens, check status codes. A wall of 403s or 429s usually means a WAF or CDN rule is blocking a bot your robots.txt allows.
4. Verify a sample of IPs with the methods above before you trust any count.
5. Check which URLs search bots request. If OAI-SearchBot or Claude-SearchBot only fetch your homepage, look at internal linking and your sitemap.

Tools that show AI crawler activity

Disclosure: AI Rank Lab is our product. We list it first because this is our site, and we've kept the same criteria for every tool below.

• AI Rank Lab - AI bot tracking records AI assistant referrals and the AI bot visits its tag can see, per page, next to your AI visibility and citation tracking. Pair it with logs for full crawler counts, for the reasons above.
• Cloudflare AI Crawl Control - if your site is proxied through Cloudflare, this is the most complete crawler view available, because it sees every request at the edge, including bots that never run JavaScript. It also shows robots.txt compliance per crawler.
• Log file analyzers such as Screaming Frog's Log File Analyser or GoAccess - work on raw logs from any host and are strong for one-off audits.
• Raw logs plus the grep above - free, exact, and the baseline every other tool should match.

For a deeper monitoring routine built around the three biggest bots, see our GPTBot, ClaudeBot and PerplexityBot monitoring guide.

How do Cloudflare and other CDNs control AI bots?

CDNs enforce at the network edge what robots.txt can only request. Cloudflare's AI Crawl Control, available on all plans, shows which AI services hit your site, lets you allow or block each crawler, tracks robots.txt violations, and offers pay per crawl in private beta.

Cloudflare's AI bot policies now use the same three-way split as this article. Its Block AI Bots settings offer separate Search, Agent and Training presets, each with "block on all pages", "block on pages with ads" or "allow". The docs say that from September 15, 2026, new domains get updated defaults: Training and Agent bots blocked on pages that show ads, Search allowed, and mixed-purpose crawlers that combine search and training blocked by any training block. If you put a site on Cloudflare after that date and wonder why ChatGPT-User or Claude-User gets 403s on your blog, check this setting first.

Cloudflare can also manage robots.txt for you. With Bot Preference Sync on, its managed robots.txt prepends disallow groups for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, plus a Content Signals line (search=yes, ai-train=no), ahead of your existing rules. Content Signals are a Cloudflare-proposed vocabulary (search, ai-input, ai-train) that states how content may be used after it is fetched; whether a given operator honors them is up to that operator.

Other providers support the same pattern. Perplexity's docs include WAF guidance for Cloudflare and AWS WAF using its IP lists, and OpenAI's help center shows how to allowlist its signed Cloud browser in Akamai (bot name "ChatGPT Agent") and Cloudflare (detection ID 129220581). Whatever your CDN, the order of operations is the same: decide per class, express it in robots.txt, enforce it at the edge only where you need a hard block, and recheck your logs after any CDN change.

Google and Bing: the two special cases

Does blocking Google-Extended affect Google Search or AI Overviews?

No. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Its AI features documentation is explicit that robots.txt rules for Googlebot are the control for Search, including AI Overviews and AI Mode, and that you limit what appears using nosnippet, data-nosnippet, max-snippet or noindex. Google-Extended covers Gemini model training and grounding in Gemini Apps and Vertex AI, which are separate from Search.

How do you control whether Microsoft Copilot uses your content?

Microsoft has no separate AI crawler token; Bingbot does the crawling, and you control AI use with page-level robots meta tags. The most recent official guidance we found is the Bing team's 2023 announcement, written when Copilot was still called Bing Chat: content tagged noarchive is not included or linked in Bing's chat answers and is not used to train Microsoft's foundation models; content tagged nocache may appear with URL, title and snippet only; content with neither tag can be used fully. Both tags leave you in normal Bing results. Blocking Bingbot in robots.txt would also remove you from Bing search, so use the meta tags instead.

Limitations and how we checked

On October 18, 2026 we read each operator's crawler page listed above, downloaded every published IP list, and recorded the creationTime inside each file. We could not render Microsoft's JavaScript-only crawler help page, so the Bingbot row relies on Microsoft's Bing Webmaster blog, its Verify Bingbot tool and its published bingbot.json. Robots.txt columns paraphrase what each operator says, not what we independently measured; Cloudflare's Perplexity findings are the main public evidence that the two can differ. Operators rename and add bots often (Google has replaced its Google-NotebookLM fetcher token with Google-GeminiNotebook, for example), so treat any list, including this one, as accurate on its check date.

If you want to see whether your crawler policy is actually earning citations, AI Rank Lab tracks your mentions and cited pages across ChatGPT, Claude, Gemini, Perplexity, Google AI Overviews and AI Mode, next to the AI referral traffic those answers send you.