When Claude answers a question with web search turned on, it doesn't just summarize what it already knows. It runs a live search, pulls back a handful of pages, and drops inline citations next to the specific claims those pages supported. If your page is one of the ones it pulled, your brand shows up as a clickable source inside someone's answer. If it isn't, you don't exist in that conversation at all, no matter how well you rank on Google.

Getting into that citation set isn't mysterious, but it also isn't the same game as classic SEO. Claude's retrieval pipeline, its crawlers, and its citation behavior all have specific, documented mechanics. Here's what actually determines whether Claude cites you, verified against Anthropic's own documentation rather than guesswork.

How Claude's web search actually works

Claude's web search is a tool, not a background process that runs on every message. According to Anthropic's own documentation, Claude decides whether a request needs live information: recent events, current prices, anything about an organization or product that might have changed since training. Settled facts don't trigger a search. Neither does a math problem or a request to brainstorm. When it does search, the flow is: Claude writes a query, the API runs it and returns results, and Claude can repeat that loop several times in a single turn before writing its final answer.

That repeat-the-loop detail matters more than it looks. For a simple factual question, Anthropic's own guidance says one to three searches is typical. For a comparison question ("which of these three tools handles X better"), the documentation notes ten or more searches isn't unusual in a single turn. That means a comparison page you publish isn't competing against one other page for one citation slot; it's one candidate among many Claude pulls in while it builds up the full picture across several queries. Publishing a direct comparison page, rather than hoping Claude stitches your single-product page together with someone else's, gives you a better shot at being the source for the comparison itself rather than just one data point feeding into it.

The citations in that final answer aren't decorative. Every web_search_result_location carries the source URL, the page title, and up to 150 characters of the exact text Claude pulled from the page. That cited text is capped deliberately: Anthropic's documentation describes it as "cited text," not "the paragraph," which tells you something about what gets lifted. A single clean sentence that directly states a fact travels through this pipeline intact. A claim that only makes sense after three paragraphs of throat-clearing gets paraphrased, mangled, or skipped.

Each search result also carries a page_age field showing when the page was last updated. Anthropic doesn't publish the exact weighting it gives that signal, but its presence in every result tells you freshness is part of what the model sees when it's deciding which source to lean on for a time-sensitive claim.

On the API side, web search costs $10 per 1,000 searches on top of standard token usage, and developers can cap usage with max_uses or restrict it to specific domains with allowed_domains. None of that changes what makes content citable, but it matters if you're trying to reproduce Claude's citation behavior programmatically to audit your own visibility, which is covered below.

The index sitting behind Claude's search

Claude doesn't maintain its own general-purpose web index the way Google does. In March 2025, TechCrunch reported that Anthropic appeared to be using Brave Search to power Claude's web search feature, based on Anthropic quietly adding Brave Search to its subprocessor list and a parameter named BraveSearchParams turning up inside Claude's search function. Anthropic has never confirmed this directly in its product documentation, so treat it as well-sourced reporting rather than an official architecture diagram.

If it's accurate, the practical implication is straightforward: Claude's candidate pool for any given query is shaped by what Brave's independent index already ranks well. You don't optimize for "Claude" as some separate destination. You optimize for being retrievable and well-ranked in a general web index, then make sure the page itself is structured so an LLM can lift a clean answer out of it once it's in the candidate set. That's two different problems, and most sites only work on the second one.

The three crawlers Anthropic actually runs

This is the part people get wrong most often, usually by treating "Claude's crawler" as one thing. Anthropic's support documentation names three separate bots, each with its own job and its own robots.txt token:

Bot

What it does

robots.txt token

Blocking it means

ClaudeBot

Crawls the web to collect training data for future models

`ClaudeBot`

No effect on today's citations; it's about model training, not retrieval

Claude-User

Fetches a specific page in real time when a Claude user's question requires it

`Claude-User`

Claude can't retrieve your content when answering a user's live query

Claude-SearchBot

Indexes content to improve the quality and accuracy of Claude's search results

`Claude-SearchBot`

Reduced visibility and accuracy in Claude's user-facing search

The detail worth sitting with: Anthropic says all three of these bots respect robots.txt, including Claude-User, the one triggered by an actual person's question. That's a deliberate design choice, and it's not universal. OpenAI and Perplexity both draw a sharper line around their own user-triggered fetchers (ChatGPT-User and Perplexity-User), and Perplexity says explicitly that Perplexity-User "generally ignores robots.txt rules" because a human asked for that specific page. Anthropic applies the same rule to all three of its bots. If you block Claude-User, you're not just opting out of bulk indexing, you're opting out of Claude being able to answer a direct question about your page at all, even when a real user asked for it by name.

Anthropic also publishes its crawler IP ranges at claude.com/crawling/bots.json, which is worth pulling into your WAF allowlist if you're running aggressive bot-blocking rules and don't want to accidentally catch Anthropic's fetchers in a net meant for scrapers.

One more wrinkle worth flagging for anyone running a Cloudflare bot-fight-mode setup or a CDN with a default "block known bots" posture: those products frequently ship with AI crawlers pre-blocked at the edge, above and separate from your robots.txt. A perfectly permissive robots.txt file doesn't help if your CDN is returning a 403 to Claude-User before the request ever reaches your server. If you've set robots.txt correctly and still aren't seeing any Anthropic user-agent hits in your logs, check the CDN or WAF layer next, not your robots.txt syntax.

What makes a page citable in the first place

Assuming your robots.txt isn't blocking anything, citability comes down to how easy your content is to lift cleanly.

State the answer first, then support it. If someone asks Claude "what's the minimum age to open a Roth IRA," the page that gets cited is usually the one with a direct sentence like "there's no minimum age" sitting right under a matching heading, not the one that spends two paragraphs on historical context before getting there. Models extract passages near the heading that matches the query's intent. Bury the answer and you've made it harder to extract, even if the information is technically on the page.

Use headings that match real questions, not vague section labels. "Eligibility requirements" tells a model less than "Who can open a Roth IRA." The closer your H2s and H3s read like the actual query a user typed, the easier it is for Claude's retrieval step to match your passage to the question.

Make individual facts self-contained. A statistic, a definition, a step in a process: each one should make sense pulled out of its paragraph, because that's exactly what happens when it gets cited. "This increased by 12% that year" is useless once separated from the three sentences of setup above it. "Enrollment increased 12% in 2025" survives extraction intact.

Corroboration helps more than uniqueness for factual claims. If three independent, credible sources state the same number or the same rule, a model is more confident treating it as citable fact. Being the only source saying something unusual, for a plain factual claim, makes you less likely to get picked up, not more; the model has no way to confirm you're right and nothing else to triangulate against. Original research, proprietary data, and genuine first-hand reporting are the exception here: that's exactly the kind of content that's worth being the sole source for, because there's nothing to corroborate against and the model has nowhere else to pull it from.

Keep pages genuinely current when the subject matter is time-sensitive. Given that page_age rides along with every search result, a page that hasn't been touched since a rule changed is working against itself on anything regulatory, pricing-related, or tied to a specific year.

Structured data earns its keep here too, even though it's not the headline factor. Article, FAQ, and HowTo schema don't force a citation on their own, but they give Claude's retrieval step an unambiguous, machine-readable version of the same claim you've already made in prose. Think of schema as a second, cleaner copy of your answer sitting next to the first one, not a replacement for writing the answer clearly in the first place. A page with great prose and no schema will usually out-cite a page with perfect schema wrapped around a vague paragraph.

Author and publisher signals matter more than most GEO advice admits. A byline with a real name, a published date, and a visible "last updated" line aren't decorative; they're part of how a model (and the human eventually reading its answer) judges whether a claim is worth repeating. An unsigned, undated page asserting a specific number reads as less trustworthy than the same number on a page with a named author and a recent update date, even when the number itself is identical. This is the same instinct a careful reader has, and there's no reason to assume a model trained to reason about text would skip it.

robots.txt decisions you actually need to make

Most sites don't need a complicated policy here. A reasonable default for a publisher who wants citations:

User-agent: ClaudeBot
Crawl-delay: 1

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

Leave Claude-User and Claude-SearchBot open if you want Claude's search feature to be able to retrieve and cite you at all. Blocking either one takes you out of consideration entirely for that part of the pipeline; there's no partial credit.

ClaudeBot is the one decision that's genuinely about values, not visibility. It collects data for training future models, and blocking it has no bearing on whether today's Claude cites your existing pages in search. If you're fine with your content potentially informing a future model's training data, leave it open. If you'd rather opt out of that while still being eligible for citations, block ClaudeBot specifically and leave the other two alone. Crawl-delay is a reasonable middle ground if your concern is server load rather than principle.

How to check whether Claude already mentions your brand

There's no Search Console equivalent for this yet, but you have four real options.

Ask directly, in a fresh, logged-out-feeling conversation (or an incognito window, so cached context from earlier chats doesn't skew the answer), with web search enabled. Try your brand name alongside a handful of the actual questions your customers ask, not just "tell me about [brand]." A generic brand query will often pull from training data alone with no search triggered; a specific, current question is more likely to force a live search where citations show up.

If you're a developer, hit the Messages API directly with the web search tool enabled and log the web_search_tool_result and citation blocks for a batch of your target queries. This is the only way to check systematically and repeatably rather than one conversation at a time, and it's the same mechanism third-party tracking tools are built on top of.

Check your server logs for ClaudeBot, Claude-User, and Claude-SearchBot user-agent hits on your own pages. A spike in Claude-User requests to a specific URL tells you Claude is actively fetching that page in response to real user questions, even before you've confirmed a citation. It's a leading indicator.

Dedicated AI visibility trackers like Otterly, Profound, and Scrunch AI run a bank of prompts against Claude, ChatGPT, Perplexity, and Gemini on a schedule and report back where your brand shows up and where a competitor shows up instead. Worth it if you need this tracked continuously across a team rather than checked by hand once a quarter.

Few AI visibility tools track Claude, and coverage often sits on higher plans or paid add-ons. Our roundup of Claude rank trackers shows which ones do and what it costs.

Blocking ClaudeBot doesn't have to mean disappearing from Claude's answers. Our guide on whether to block AI crawlers explains the difference between training and search bots.

What doesn't work

Keyword-stuffing your headings doesn't move the needle. Claude's retrieval is matching meaning, not strings, so "Best Roth IRA Providers 2026 Top Rated Guide" in an H2 doesn't out-rank a competitor who just wrote "Which Roth IRA Provider Is Best" and answered the question cleanly underneath it.

Cloaking, or serving different content to bots than to human visitors, is a bad trade even where it technically works, because it's the kind of thing that gets your domain treated as untrustworthy once discovered, and corroboration-dependent citation models punish untrustworthy sources hard.

Paying for a service that guarantees Claude citations is not something anyone can actually deliver. Anthropic doesn't sell placement, and there's no pay-to-play layer in the pipeline described above. Anyone selling "guaranteed AI citations" is selling the same content and structure fundamentals covered here, repackaged.

Blocking ClaudeBot out of a vague sense that it'll help your AI visibility is backwards; as covered above, it has no effect on citation eligibility either way, so there's no visibility upside to blocking it and no visibility downside to leaving it open. Block it or don't based on your actual stance on training data, not based on a theory about search rankings it doesn't participate in.

Does blocking ClaudeBot stop Claude from citing my site?

No. ClaudeBot collects training data for future models. Citation eligibility runs through Claude-User and Claude-SearchBot, which are separate bots with separate robots.txt tokens.

Does Claude use Google's search index?

Not based on current evidence. Reporting points to Brave Search powering Claude's web search tool, which is a different index than Google's, built from different crawl data and ranking signals.

Can I pay Anthropic or a third party to guarantee my site gets cited?

No legitimate route exists for that. Improving your odds means improving retrievability (not blocking the relevant bots) and improving extractability (clear, self-contained, well-corroborated answers near matching headings).

How often does Claude's search index update?

Anthropic hasn't published a refresh cadence for the underlying index. The page_age field on search results suggests freshness is evaluated per-result at query time rather than on a fixed crawl schedule you can plan around.

Is this different from showing up in ChatGPT and Google AI Overviews?

The underlying content principles overlap heavily (clear structure, direct answers, real corroboration), but the specific crawlers, the index each one draws from, and the robots.txt rules each applies are all different. A page blocked from one engine isn't necessarily blocked from another, so check each one's crawler policy separately rather than assuming one robots.txt decision covers all of them.

If you do one thing after reading this, make it the server log check. Pull 30 days of access logs, grep for Claude-User and Claude-SearchBot, and see which of your URLs Anthropic's bots are actually requesting right now. That tells you, today, whether you're even in the game before you spend time optimizing content for a model that was never going to see it. For the broader context on where this fits alongside other engines, the AI search category covers how Perplexity and Gemini handle the same problem differently, and the technical SEO checklist covers the crawlability fundamentals that sit underneath all of it.