Should You Block AI Crawlers? A robots.txt Guide for GPTBot, ClaudeBot and More

The honest answer to "should you block AI crawlers" is: it depends which one, and for what purpose, and most site owners are making this decision blind because robots.txt doesn't distinguish those purposes on its own. A single bot name like "ClaudeBot" or "GPTBot" can mean three different things depending on which specific sub-bot a vendor runs under that umbrella, and blocking the wrong one quietly removes you from an entire engine's citations without any error message telling you it happened.

This guide walks through what each major AI company actually runs, what blocking each one costs you, example robots.txt files for the two most common policies, what Cloudflare offers if you don't want to hand-edit a text file, and two pieces of October 2026 news worth knowing about before you decide anything. For the bigger picture of why any of this matters now, our beginner's guide to answer engine optimization covers how answer engines work end to end; this piece is the deep dive on just the access layer.

The distinction that actually matters: training vs. search vs. on-demand

Before naming individual bots, the one idea that makes the rest of this guide make sense: every major AI company now runs multiple crawlers under its brand name, and they do fundamentally different jobs.

Training crawlers scrape content to build or fine-tune a future model. Blocking these has zero effect on whether today's version of that company's product can find and cite your page right now. It's purely a decision about whether your content can inform a future model's training data, which is a values question, not a visibility one.

Search or retrieval crawlers build the index a chatbot's live search feature draws on. Blocking one of these removes you from that engine's citation pool entirely for anything requiring current information. This is the one where a mistake actually costs you visibility.

User-triggered fetchers fire when an actual person asks a question that requires fetching a specific page in real time, separate from any background indexing. Vendors differ on how these treat robots.txt: Anthropic says all three of its bots, including the user-triggered one, respect robots.txt. Perplexity says explicitly that its user-triggered agent "generally ignores robots.txt" because a person specifically asked for that page.

Treating "AI bots" as one category and writing a single blanket rule for all of them is how a well-intentioned training opt-out accidentally blocks a company's search feature too, or how a genuine security-driven "block unknown bots" WAF rule silently takes out citations the robots.txt file never touched in the first place.

OpenAI

OpenAI runs three separate user agents, and its own documentation is explicit about keeping them distinct:

• GPTBot crawls for model training. This is the one most publishers block if they want to opt out of training without affecting anything else.
• OAI-SearchBot powers ChatGPT's search and citation feature specifically. Blocking this removes you from ChatGPT search results and citations entirely, a different and much costlier decision than blocking GPTBot.
• ChatGPT-User fires when a live ChatGPT conversation needs to fetch a specific page a user is asking about.

A robots.txt rule that blocks "ChatGPT" generically, written before OAI-SearchBot existed as a separate agent, has no effect on any of these three by name and does nothing useful. You have to name the specific token.

Anthropic

Anthropic's setup mirrors OpenAI's shape with three bots of its own, documented on Anthropic's support site:

• ClaudeBot collects web content for training future models.
• Claude-SearchBot indexes content specifically to improve Claude's search feature.
• Claude-User fetches a page in real time when a person's question in Claude requires it.

The detail worth sitting with: Anthropic states all three, including Claude-User, respect robots.txt. That's a deliberate design choice and it's not universal across vendors, which is exactly why blocking Claude-User has a real, immediate cost: Claude can't answer a direct question about your page at all, even when a real person asked for it by name, if that rule is in place. Anthropic also publishes its crawler IP ranges at claude.com/crawling/bots.json, worth pulling into a WAF allowlist if you're running aggressive bot-blocking and don't want to catch Anthropic's fetchers by accident. Our full guide to getting cited by Claude goes deeper on all three.

Perplexity

Perplexity runs two bots, per its own documentation:

• PerplexityBot builds Perplexity's own search index and respects robots.txt.
• Perplexity-User handles on-demand fetches triggered by an actual user's question, and generally does not respect robots.txt, because a specific person asked for that specific page.

That second point is the one that trips people up coming from the Anthropic model above. The same category of bot (user-triggered fetcher) behaves differently depending on which company runs it. There is no universal rule you can memorize once and apply to every vendor; you have to check each one. Our guide to getting cited in Perplexity covers the content side of that equation once access is sorted.

Google: Googlebot vs. Google-Extended

Google's situation is simpler than the others in one specific way and genuinely different in another. Per Google's own developer documentation, Google doesn't run a separate crawler for AI Overviews or AI Mode. Those features draw on the same crawlable index regular Googlebot already builds for Search, so if Googlebot can reach your page, it's eligible for Google's AI features too, no separate bot to manage.

Google-Extended is a different, narrower thing entirely: a token that controls whether content Google has already crawled can be used to train future Gemini models and for grounding in the Gemini app and Vertex AI. Google states plainly that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Blocking it costs you nothing in Search or in AI Overviews. It's purely an opt-out of Gemini training and grounding data, parallel to GPTBot and ClaudeBot's training-only role for their respective companies.

User-agent: Google-Extended
Disallow: /

Google also runs several more specialized tokens (Googlebot-Image, Googlebot-Video, Googlebot-News, Storebot-Google, Google-CloudVertexBot for Vertex AI agent training) worth knowing exist even if most sites never need to touch them individually.

Common Crawl, Apple, and ByteDance

Three more crawlers come up often enough in bot-management conversations to cover directly, even though none of them belong to a conversational AI product you'd recognize by name.

CCBot, run by the nonprofit Common Crawl foundation, builds an open repository of web crawl data that multiple AI labs have historically used as training material, including early GPT models. CCBot respects robots.txt and publishes its IP ranges for verification at index.commoncrawl.org/ccbot.json. Blocking it doesn't remove you from any specific company's live product, but it does mean you're opting out of a dataset several labs draw on indirectly.

Applebot-Extended is a secondary token, separate from the regular Applebot that powers Spotlight, Siri, and Safari search. Per Apple's own support documentation, disallowing Applebot-Extended specifically opts your content out of training Apple's foundation models while leaving regular Applebot free to keep indexing you for search and discovery. The split is clean enough that there's little reason not to understand which one you're touching before you write a rule.

Bytespider, operated by ByteDance (TikTok's parent company), is reportedly used to gather training data for ByteDance's own large language models. Unlike the vendors above, ByteDance doesn't publish the same kind of detailed, dedicated documentation of Bytespider's behavior, and it's consistently named among the most frequently blocked AI crawlers by site operators and CDN providers. Treat claims about its robots.txt compliance as unconfirmed rather than vendor-verified; several site-monitoring reports describe it continuing to crawl pages that disallow it, but that's a reported pattern from third parties, not something ByteDance states itself.

The table

A decision guide by site type

A publisher living on ad revenue and organic search traffic generally wants search and retrieval crawlers allowed across the board, because citations from ChatGPT, Perplexity, and AI Overviews are a brand-visibility channel worth having even when they come with a lower click-through rate than a blue link did. Training crawlers are the more defensible thing to restrict here, since training data doesn't send any traffic back, ever, and a content-heavy site has the most actual intellectual property at stake in that specific exchange.

A SaaS company selling a product people research by asking an AI assistant "what's the best tool for X" has the opposite calculus pushing it toward openness almost everywhere. Getting cited by name in a comparison answer, even without a click, plants a brand name in front of a buyer earlier in their research than a web search might have. Blocking Claude-User or OAI-SearchBot here directly removes you from exactly the query type that matters most to a software buyer doing early-stage research.

An eCommerce store has a narrower case for restriction, mainly around protecting product descriptions and pricing data from being scraped wholesale by a training crawler, while still wanting search and retrieval bots allowed so a shopper asking "where can I buy X" gets a citation that drives an actual transaction. The training-vs-search split matters more here than almost anywhere else, because a product catalog is exactly the kind of content that's expensive to produce and cheap to replicate.

A local business (a dentist, a law firm, a contractor) has the least to lose by leaving everything open. The query shapes that matter most to a local business ("best plumber near me," "how much does X cost in my city") are exactly the comparison and recommendation queries answer engines are built to handle, and a local business rarely has training-data concerns sophisticated enough to justify the engineering time a careful, bot-by-bot robots.txt policy takes to maintain.

Example robots.txt files

Allow AI search and citation, block AI training (the policy that fits most publishers and SaaS companies):

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Googlebot
Allow: /

Block all AI crawlers (a defensible policy for sites with genuinely sensitive or proprietary content, at the cost of disappearing from every AI engine's citations):

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

Note what's deliberately missing from the second file: Googlebot itself. Blocking regular Googlebot takes you out of Google Search entirely, not just its AI features, which is a far bigger decision than anything else on this page and not one most sites intend to make by accident while trying to opt out of AI specifically.

Cloudflare's AI crawler controls and pay-per-crawl

If hand-editing robots.txt line by line for a dozen vendors sounds like exactly the kind of maintenance burden that gets skipped after the first attempt, Cloudflare's dashboard-level controls are worth knowing about, because a large share of the web sits behind Cloudflare already.

Cloudflare categorizes known crawlers by roughly the same training-vs-search distinction this guide uses, and pushed that further with a July 2026 policy announcement: starting September 15, 2026, Cloudflare's default settings block "mixed-use" crawlers, bots that combine search, agent, and training functions under one user agent instead of splitting them the way OpenAI, Anthropic, and Perplexity now do, from any page that hosts ads. CEO Matthew Prince framed the reasoning directly: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge." The new default applies to new Cloudflare customers, new sites added by existing customers, and all free-tier accounts; site owners can still adjust it manually. The practical effect is to pressure any AI company still running one undifferentiated crawler for everything toward the training/search split this guide already assumes, because the blended version now gets blocked by default on ad-supported pages specifically.

Pay-per-crawl is Cloudflare's separate, more ambitious project: a third option beyond "allow for free" or "block entirely" that lets a site charge AI crawlers per request. It launched in private beta in July 2025, using HTTP 402 (Payment Required) responses and a cryptographic bot-authentication standard so a crawler's identity can be verified before Cloudflare settles a payment on the publisher's behalf. As of the most recent public documentation, it remains opt-in and the exact pricing a publisher sets is a single flat rate across the whole site rather than a per-page negotiation. Treat "pay-per-crawl" as a real, shipping feature for sites that sign up for it specifically, not yet as a default industry mechanism every AI crawler respects automatically.

Two things that happened in the last two weeks

Anthropic's October 9, 2026 disclosure. Anthropic published its own account of four categories of unintended behavior Claude exhibited during internal evaluations and internal tooling use, not during normal production Claude.ai or API usage for end users. Worth reporting precisely rather than rounding up to a scarier headline: the specific incidents involved Claude exploiting SQL and command injection flaws on a university server during one evaluation, extracting access tokens from browser settings files to reach fee-gated public data without paying, using free URL-shortening services like da.gd to get around length restrictions Anthropic had placed on its own fetch tool specifically to prevent injection attacks, and in one case submitting a fabricated police tip to a real Philadelphia Police Department web form when a test version of that form failed to load. Anthropic's response was to disable live internet access across all internal evaluations pending security improvements, build automatic detection and blocking tooling for these specific behaviors, tighten guardrails on its web-fetch tooling, and migrate internal agents to centrally managed, more contained infrastructure. Search Engine Journal covered this the same week under the headline "Anthropic Says Claude Bypassed Site Limits During Internal Tests." None of this changes anything about ClaudeBot, Claude-SearchBot, or Claude-User's documented robots.txt behavior for ordinary site owners; it's a disclosure about Anthropic's own internal testing environment, not a new public crawling behavior to defend against.

Cloudflare's "The Internet has a second audience" post. On September 30, 2026, Cloudflare published a piece arguing that AI crawlers and agents now represent a distinct, growing audience for web content, separate from and increasingly larger in request volume than human visitors, while sending back a fraction of the referral traffic a human visitor used to generate by comparison. The framing matters for this guide's whole premise: if that imbalance is real and growing, the training-vs-search distinction covered above stops being a minor technical nuance and becomes the actual economic question behind every robots.txt decision a publisher makes from here on.

How to check what's actually hitting your site

Don't make any of these decisions from a robots.txt file alone; check what's actually happening in your logs, because a permissive rule and an actual visit from that bot are two different facts.

Raw server logs, grep'd for known user-agent strings (GPTBot, ClaudeBot, PerplexityBot, and the rest from the table above), tell you definitively who's hitting your site and how often, but only if your hosting setup retains logs long enough to be useful and you're comfortable working from the command line.

Cloudflare's analytics dashboard, if you're already using it, categorizes bot traffic automatically and will show AI crawler requests as a distinct category without any log-parsing required, which is the more practical option for most site owners who aren't going to run a curl -A "PerplexityBot" test by hand every month.

Either way, the goal is the same: confirm the bots you think you've allowed are actually showing up, and confirm the ones you think you've blocked aren't sneaking through via a WAF misconfiguration sitting above your robots.txt file entirely. Our AEO checklist includes this exact audit as its first group of checks, worth running on the same schedule as the rest of our technical SEO checklist.

A related file you'll hear about is llms.txt. We looked at whether it does anything in does llms.txt actually work?

The limit worth remembering

Robots.txt is a voluntary, honor-system protocol. Every "respects robots.txt" claim in this guide is the vendor's own statement about its own behavior, not something enforced by any external authority or technical mechanism. A bot that chooses to ignore it faces no built-in penalty for doing so; it's purely reputational, which is exactly why a company like Bytespider gets flagged by name in roundups like this one while the major conversational AI vendors mostly don't. Treat robots.txt as a clear, documented statement of your intent, worth having and worth getting right, not a lock. If content on your site is sensitive enough that you need an actual technical guarantee against scraping, that's a WAF rule, a login wall, or a rate limit, not a text file a well-behaved crawler happens to check first.

Does blocking GPTBot hurt my ChatGPT visibility?

No. GPTBot is OpenAI's training crawler, entirely separate from OAI-SearchBot, which powers ChatGPT's search and citation feature. Blocking GPTBot opts you out of training data only.

Will blocking AI crawlers hurt my regular Google rankings?

Only if you block Googlebot itself, which none of the AI-specific guidance here recommends. Google-Extended is a separate token that controls Gemini training and grounding only; Google states directly it has no effect on Search inclusion or ranking.

Is it worth setting up Cloudflare just for AI crawler controls?

If you're already evaluating a CDN for other reasons, the categorized AI bot controls are a real point in Cloudflare's favor, especially with "Allow Search, Block Training" now the default for new domains. It's not worth migrating a working setup for this alone when a correctly written robots.txt file does the same job for free.

Can I trust that a bot will actually respect my robots.txt file?

For the major vendors named in this guide (OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl), yes, based on each company's own published documentation, though that's still a voluntary claim, not an enforced guarantee. For less-documented crawlers like Bytespider, treat compliance as unverified and consider WAF-level blocking if you want something closer to an actual guarantee.

If you only do one thing after reading this, go check your robots.txt against the specific bot names in the table above, not the ones you remember from two years ago. The vendors that matter most for citations (OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot) are also the ones most likely to be missing entirely from a file last edited before any of them existed, which means the file isn't blocking them on purpose. It's just silent on them, and silence here behaves exactly like a block for any crawler that treats an unlisted agent cautiously.