How Claude Web Search Works: Tools, Bots and Citations

Claude web search is a tool that Claude decides to call when a question needs current or changing information. Claude writes a search query, Anthropic's servers run it against a web index and return titles, URLs and page dates, Claude can then fetch full pages, and the answer it writes cites the sources it used. Whether your site can appear in that answer is decided mostly by two things you control: whether Anthropic's search and user-fetch bots are allowed in your robots.txt, and whether your page ranks in the index Claude searches.

This guide explains that pipeline using Anthropic's own documentation, separates what Anthropic has confirmed from what researchers have reverse-engineered, lists the three Anthropic user agents and what blocking each one does, and compares how search works in claude.ai, the Claude API and Claude Code. It also includes what our own tracking shows about how stable Claude's citations are and how much traffic Claude sends. All documentation was checked on October 24, 2026.

How to read this article: confirmed, reported and our data

Discussion of Claude search mixes official facts with guesses, so every claim below falls into one of three groups:

• Confirmed - stated in Anthropic's API documentation, Claude Code documentation, help center or blog.
• Reported - third-party testing, reverse engineering or journalism. Often credible, never official.
• Our data - aggregated, anonymized measurements from AI Rank Lab, always shown with method and sample size.

Does Claude search the web?

Yes. Claude can search the web on every Claude plan, in claude.ai, the desktop and mobile apps, the Claude API and Claude Code. Anthropic launched web search on March 20, 2025 for paid US users and updated the same post on May 27, 2025 to say web search "is now available globally on all Claude plans."

How you switch it on depends on the surface. According to Anthropic's help article on enabling and using web search, you turn it on per chat from the "+" menu in the chat box. On Team and Enterprise plans, an Owner or Primary Owner first has to enable it for the organization under Organization settings, then Capabilities. The same article says users on the newer Claude experience have no toggle at all, because "Claude searches the web when it helps." It also notes that Claude may use an approximate location inferred from your IP address to localize results.

Should you turn web search on or off in Claude?

Leave it on when your question depends on anything that changes: prices, product versions, news, current rankings or a specific company's details. Turn it off when you want an answer only from the conversation and the files you uploaded, or when you are on the free plan and want to save usage, since fetched pages enter the context window. Even with search on, Claude will not search for every message.

When does Claude search the web instead of answering from training data?

Claude searches when a request depends on information that is current, changing or outside its training data, and answers directly when the request draws on stable knowledge. Anthropic spells this out in its web search tool documentation:

For site owners, the important row is the third one. A buyer asking "which CRM is best for a 10-person agency" is asking about specific products whose details change, so that kind of prompt usually triggers a search. A question like "what is a CRM" usually does not, and the answer then comes from training data, which you cannot influence quickly.

The trigger is steerable. Developers can tell Claude in a system prompt to search more or less readily, and cap searches with the max_uses parameter. Anthropic's docs say simple factual questions typically use 1 to 3 searches, while comparative or multi-entity research can use 10 or more.

How does Claude web search work, step by step?

Claude web search runs as a server tool: Anthropic executes the search, not your application and not the model itself. Based on Anthropic's API documentation for web search and web fetch, one search-backed answer goes through these steps:

1. Decide. Claude reads the prompt and decides whether it needs current information.
2. Write a query. Claude writes its own search query, which can differ from what the user typed. Scrunch's launch-week review noted that claude.ai shows the search terms it used.
3. Search. Anthropic's API runs the query and returns results. Each result carries a url, a title, a page_age field described as "when the site was last updated," and encrypted page content.
4. Filter (newer models). With the web_search_20260209 tool version and later, on Claude 4.6 and newer models, Claude can write and run code that filters search results before they reach its context window. Anthropic calls this dynamic filtering.
5. Repeat. Claude can search again with refined queries in the same turn.
6. Fetch (optional). When it needs the full text of a specific page, Claude can call the web fetch tool on a URL that already appeared in the conversation, for example in a search result.
7. Answer and cite. Claude writes the response, and each claim drawn from the web carries a citation to the source page.

The fan-out in steps 2 and 5 matters for content strategy. Claude does not search the user's exact words once; it can run several narrower queries, so a page that cleanly answers one specific sub-question can be found even when it would never rank for the broad prompt.

What search engine does Claude use?

Anthropic has not publicly named the index behind Claude's text search, but the evidence points strongly to Brave Search. This is the "reported" category, so here is what each piece of evidence actually shows:

• Subprocessor list (March 2025). TechCrunch reported that Anthropic added "Brave Search" to its subprocessor list the week web search launched. TechCrunch asked Anthropic to comment and the article shows no response.
• Matching results. Developer Simon Willison found that the ten results Claude cited for one query matched Brave's results for the same query, and that Claude's search function schema was titled "BraveSearchParams."
• Overlap test. Profound's March 2025 analysis found that 13 of 15 Claude citations (86.7%) matched Brave's top non-sponsored results. The sample was three queries, so treat it as an early signal, not a measurement.
• A second provider (unverified by us). A July 2026 Xponent21 review of Anthropic's subprocessor list reports that Brave was still listed and that turbopuffer, a vector and full-text search database, was added under web search in May 2026. Its role is not public.

Two things are confirmed. Anthropic's help center says image results in Claude are powered by Bing. And Anthropic's Claude Code tools reference says the WebSearch tool runs against "Anthropic's web search backend" and that "the search backend is not configurable."

We tried to load Anthropic's subprocessor page on October 24, 2026 to confirm the current entries, but it renders client-side and returned no list to our fetch, so we could not re-verify it ourselves. The practical takeaway does not depend on it: check whether your key pages appear on Brave Search for the questions you care about, because if the reports are right, a page missing from Brave's index has little chance of showing up in Claude's text results.

What is the difference between ClaudeBot, Claude-User and Claude-SearchBot?

Anthropic documents three user agents, and each one does a different job. The table below is based on Anthropic's support article on how Anthropic's bots crawl the web (dated April 7, 2026, checked October 24, 2026) and the Claude Code tools reference.

Anthropic says its bots honor robots.txt and support the non-standard Crawl-delay directive. If your goal is "no training, but stay citable," this is the configuration that matches Anthropic's descriptions:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Add the rules to the robots.txt of every subdomain you want covered, because each subdomain has its own file. Two further details from Anthropic's page are worth knowing:

• Do not block by IP. Anthropic warns that blocking its IP addresses "may not work correctly or reliably," because it stops the bots reading your robots.txt.
• Verify by IP instead. Anthropic publishes its crawler IP ranges at claude.com/crawling/bots.json. When we downloaded it on October 24, 2026, the file was dated October 20, 2026 and listed 14 IPv4 prefixes. PPC Land reported 23 prefixes in August 2026, so the list changes. Fetch it automatically rather than copying ranges into a firewall rule once.

A frequent cause of accidental blocking is not robots.txt at all but a bot-protection rule in a CDN or firewall that challenges unfamiliar user agents. A JavaScript challenge page is unreadable to Claude-User, so the page is effectively blocked even though robots.txt allows it. For every AI crawler and its controls, see our full list of AI crawler user agents.

How is web search different in claude.ai, the Claude API and Claude Code?

The same models sit behind all three surfaces, but the tools, defaults and limits differ, which is why their answers and citations differ. This comparison is built from Anthropic's API docs, help center and Claude Code documentation, checked October 24, 2026.

Does Claude Code have a web search tool?

Yes. Claude Code ships with a WebSearch tool that calls Anthropic's server-side search and a WebFetch tool that reads pages. The Claude Code tools reference documents several behaviors that matter to anyone who publishes developer content:

• WebSearch returns titles and URLs and "doesn't fetch the result pages," so Claude Code reads your page only if it chooses to fetch it.
• WebFetch converts HTML to Markdown and then runs the user's extraction prompt against it in a separate model call. Anthropic calls this "lossy by design": Claude sees an extract of your page, not the page.
• WebFetch sends an Accept header that prefers Markdown over HTML, so a server that supports content negotiation can return Markdown directly.
• Responses are cached for 15 minutes, a cross-host redirect is not followed automatically, and before fetching, Claude Code sends the hostname to Anthropic for a safety blocklist check, per its data usage documentation.
• Amazon Bedrock does not expose the server-side web search tool, so Claude Code users on Bedrock do not get WebSearch.

Why do Claude and Claude Code behave like different answer engines?

Because they use the web differently. Profound's August 2026 study, Claude and Claude Code are distinct answer engines, ran 1,724 prompts between July 13 and 23, 2026 and analyzed 24,135 responses with web search enabled on both. Its findings:

• Claude searched in over 93% of responses; Claude Code in 13%.
• For the same prompt, the two products shared about 1 in 5 brands.
• About three quarters of Claude Code's agent visits went to documentation, informational and pricing pages, compared with 5% for Claude. About 60% of Claude's visits went to robots.txt files, sitemaps and home pages.

Search Engine Journal's write-up notes the caveats: Profound sells AI visibility tracking, page types were labeled by an AI model, and the visit data covers one month on domains Profound monitors. The direction is still useful. If developers are your buyers, your documentation and pricing pages are what Claude Code reads, and most Claude Code answers come from training data rather than live search.

How does Claude choose and show citations?

Claude cites the specific search result or fetched document that supports each claim. In the API, citations are always enabled for web search, and each citation includes the page URL, its title and up to 150 characters of the cited text. In claude.ai, the help center says responses include direct citations, source links and relevant quotes where appropriate. For web fetch in the API, citations are optional and off by default.

What Anthropic has not published is a ranking formula: why one result in a set gets cited and another does not. What the mechanism does tell us:

• A page can only be cited if it was retrieved. If it is not in the search results for one of Claude's queries, or was not fetched, it cannot be cited.
• Citations attach to passages. The API's cited_text field points at a short span of text. A page with a clear, self-contained sentence that answers the question gives Claude something to cite; a page that buries the answer across several paragraphs gives it less.
• Freshness is visible to the model. Every search result includes page_age, so Claude can tell an old page from a new one. Whether it prefers newer pages is not documented.
• Search rank appears to carry through. The Brave overlap tests above suggest Claude mostly cites from the top of the results it gets, with limited reordering. That is reported, not confirmed.

How stable are Claude's citations from run to run?

Not very stable, though more stable than most engines we track. We measured this across 227 successful runs of 11 Claude prompt-engine pairs tracked in AI Rank Lab between February 13 and October 22, 2026, counting only pairs with at least five runs (median tracking span about 25 days). The comparison with other engines uses the same method:

Overlap is the Jaccard similarity of the cited domains in two consecutive runs: 1 means identical, 0 means nothing in common. Three findings stand out for Claude:

• Claude kept the largest stable core of the engines we track: a median 10% of the domains it cited for a prompt appeared in every run. That still means about nine in ten cited domains rotated.
• Across the tracking window, Claude cited a median 2.71 times as many distinct domains as it did in a single run. One check shows less than half the picture.
• Mentions were sticky in one direction: 6 of 11 Claude pairs mentioned the tracked brand in every run, none never mentioned it, and 5 flipped. A brand Claude already knows tends to stay, but a single "not mentioned" result is not proof that Claude ignores you.

The sample is small and comes from a limited set of brands, so read it as directional. Our AI answer stability study has the full method and how many runs you need before trusting a change.

How much referral traffic does Claude send to websites?

A meaningful but minority share. Across the sites running the AI Rank Lab tracking tag between July 27, 2025 and October 24, 2026, we recorded 3,764 page views whose referrer was an AI assistant. Claude (referrer claude.ai) accounted for 459 of them, or 12.2%, spread across 12 sites. ChatGPT accounted for 55.8%, Gemini 17.9% and Perplexity 12.5%.

Two limits apply. These are page-view events, not unique visitors, and they count only clicks that arrive with a claude.ai referrer, so visits from Claude Code, API-built apps or clicks that strip the referrer are not in this number. To set this up in your own analytics, follow our guide to tracking AI referral traffic in GA4.

How do you get your website cited by Claude?

Make your pages reachable by the bots Claude uses for search, make sure they rank in the index it searches, and write passages that answer one question cleanly. In order of impact:

1. Allow Claude-SearchBot and Claude-User in robots.txt on every subdomain. Blocking ClaudeBot is your choice and does not affect search.
2. Check your firewall and CDN. Make sure bot-protection rules do not challenge Anthropic's user agents. Verify real Anthropic traffic against bots.json instead of blocking unknown IPs.
3. Put the content in the HTML. Anthropic's web fetch tool does not render JavaScript. If your answers, prices or specs load client-side, Claude may fetch an empty shell.
4. Check the index Claude searches. Search your target questions on Brave Search. If you are not there, Claude's reported text index probably does not have you either, and classic SEO work (crawlability, internal links, earning links) is the fix.
5. Answer in passages. Lead each section with a direct, self-contained answer to one sub-question under a heading that names it. That gives Claude a span it can cite in its 150-character citation window.
6. Keep key facts in visible text. Claude Code's WebFetch converts pages to Markdown and extracts from that; facts that exist only in structured data or images are easy to lose.
7. Keep pages current and dated. Claude searches mostly for changing information, and every result carries a page age. Update pricing, versions and statistics, and show the date.
8. For developer products, invest in docs and pricing pages. Per Profound, these are the pages Claude Code's agent visits most.
9. Measure repeatedly, not once. Given the variance above, judge changes on several runs over weeks.

AI Rank Lab automates parts of this list: the AEO audit checks crawler access, rendering and answer structure across a site, and brand visibility tracking handles step 9. For broader tactics across two engines, see our Claude and Gemini playbook.

Which robots.txt setup fits your goal?

What can't you control in Claude web search?

Several parts of the pipeline are outside any site owner's reach:

• Whether a search happens. Users can turn search off, developers can cap it, and Claude skips it for stable-knowledge questions.
• The queries. Claude writes its own, and they vary between runs.
• Ranking and citation choice. Anthropic publishes no ranking factors and offers no submission or webmaster tool for Claude.
• The surface. A developer's app can restrict Claude to allowed_domains, and Claude Code answers most questions without searching at all.
• Run-to-run variance. As our data shows, the same prompt cites different domains from one week to the next.

This mirrors ChatGPT, with different plumbing. For the OpenAI side, see how ChatGPT search works.

How do you monitor which pages Claude cites?

Manually: pick 10 to 20 questions your buyers ask, run them in Claude with search on, and record whether your brand is mentioned and which URLs are cited. Repeat weekly, because one run is not representative. Then check your server logs for Claude-User and Claude-SearchBot requests to confirm the bots can reach you. Tools that automate the prompt runs:

Disclosure: AI Rank Lab is our product. We list it first because this is our site, and we've kept the same criteria for every tool below.

• AI Rank Lab - runs the prompts you choose on Claude, ChatGPT, Gemini and Perplexity, stores every run, records mention, position and cited URLs, and connects the gaps to an audit, Action Plan and content publishing. Its MCP connector also lets you pull this data from inside Claude. See brand visibility tracking.
• Profound - enterprise-focused, and publishes the largest public research on Claude and Claude Code behavior, including the study cited above.
• Peec AI - prompt tracking across engines with strong public research on retrieval and reranking.
• Otterly AI - frequent public GEO experiments and multi-engine monitoring.

Methodology and limitations

On October 24, 2026 we read Anthropic's API documentation for the web search and web fetch tools, the Claude Code tools reference and data usage pages, the Claude help center articles on web search and on Anthropic's crawlers, and Anthropic's launch post, and downloaded Anthropic's bots.json IP list. We also analyzed 16 ranking articles and reports on Claude search. Claims about Brave Search and turbopuffer are labeled reported because Anthropic has not confirmed them in its documentation, and we could not load the current subprocessor list ourselves. Profound, Xponent21 and several other sources sell AI visibility services, as we do. Our first-party numbers come from a small number of prompts and sites, are aggregated and anonymized, and describe our users' tracking, not the web as a whole.

If you want to see which of your pages Claude actually cites for the questions that matter to you, and how that changes week to week, start tracking with AI Rank Lab.