Should you block AI crawlers? GPTBot, ClaudeBot, and the training-vs-citation choice

”AI” is not one door: training bots vs citation bots

The headline that drives this question — “AI is scraping everyone’s content, shouldn’t I just shut the door?” — hides the fact that every serious AI company runs not one bot but a family of them, separated on purpose (Inspeccia, 2026). Training bots crawl in bulk to build model weights, run continuously with no user trigger, and return no attribution; search bots fetch pages in real time when someone asks a question, and they cite the source with a link back (Pixelmojo, 2026). Those are different jobs, and they answer to different rules in your robots.txt.

The 2026 line-up is worth knowing by name. OpenAI runs GPTBot (training), OAI-SearchBot (the index behind ChatGPT search and citation) and ChatGPT-User (an on-demand fetch when someone asks ChatGPT to open a link); Anthropic mirrors this with ClaudeBot (training), Claude-SearchBot (search) and Claude-User (on-demand) (Rankry, 2026). Perplexity runs PerplexityBot (search) and Perplexity-User (live fetch), and Google-Extended is not a crawler at all but a robots.txt token controlling whether your content trains Gemini (Rankry, 2026). One caution: Anthropic’s old strings anthropic-ai and Claude-Web are deprecated, so a rule still using them blocks nothing (Mersel, 2026).

Does blocking AI crawlers hurt my Google ranking?

No, and this is the fear that keeps people from making a clean decision. Blocking GPTBot has no effect on Google, because GPTBot and Googlebot are entirely separate systems (Mersel, 2026). The confusing one is Google-Extended, and Google has explicitly confirmed that blocking it opts your content out of Gemini and Vertex AI training only, never your Search inclusion or ranking (Pixelmojo, 2026). The only bot that removes you from Google Search when blocked is Googlebot itself.

That separation is what makes a training block safe to consider on its merits. Blocking a training crawler is a genuine choice about your intellectual property — some publishers object to uncompensated training use on principle — and you can make it without paying an SEO tax (Rankry, 2026). The mistake is assuming the block is invisible: even if you disallow a training bot today, your content may already sit in model weights from earlier crawls (Pixelmojo, 2026).

The middle ground: block training, allow citation

For a business that wants to be found, the optimal setup in 2026 is to block the training crawlers while allowing the search ones, so your content appears in AI answers with attribution and referral traffic but is not fed into models you don’t control (SEOScore, 2026). This is worth stating plainly because a lot of sites get it half-right and pay for it: 79% of top news sites block training bots, but 71% also block the retrieval bots — cutting their own citations in the same file (Pixelmojo, 2026). The reason to keep citation bots open is that the channel is real: AI-referred traffic has been measured converting around 4.4x better than standard organic search (Mersel, 2026).

There is now a cleaner way to express intent than block rules alone. In late 2025 Cloudflare introduced a Content Signals Policy for robots.txt that states how content may be used after it is fetched, with three signals: search for indexing and showing links, ai-input for real-time AI answers such as retrieval and grounding, and ai-train for training or fine-tuning models (Rankry, 2026). It lets you say “read me, cite me, but don’t train on me” as a preference rather than a blunt disallow. One timing nuance is worth knowing before you edit anything: OpenAI updated its crawler architecture in late 2025 so that its search and training bots share information, and blocking GPTBot now only affects future training runs — content already ingested stays in the model. So blocking training is a forward-looking choice about your next content, not a way to claw back what’s already been learned (Rank and Convert, 2026).

robots.txt is a request, not a wall

Before treating robots.txt as protection, understand what it is: a voluntary request, formalised as RFC 9309, enforced by good-faith compliance rather than by statute (DataImpulse, 2026). The major documented crawlers — Googlebot, GPTBot, ClaudeBot, PerplexityBot — generally honour it, and you can confirm that in your logs, but three caveats matter. Compliance is opt-in, and some crawlers such as Bytespider and Perplexity’s stealth fetchers have been documented ignoring it (Anagram, 2026). User-triggered real-time fetches are a gray zone: when a person asks an assistant to read a specific URL, providers often treat that as user-directed access rather than crawling, so your crawl rules may not apply (DataImpulse, 2026). And a user-agent string can be faked, so a real identity check means a reverse DNS lookup, not trust in the header (Anagram, 2026).

The practical consequence: if you truly need to stop a bot rather than ask it to leave, enforce it at the edge with a firewall or WAF rule, because a CDN rule overrides robots.txt anyway (Rankry, 2026). This cuts both ways — research on Cloudflare’s network found many sites accidentally blocking major AI crawlers at the CDN while their robots.txt said “allow,” so the two layers have to agree (DataImpulse, 2026).

The trap nobody mentions: an open door to a blank room

Here is the part that makes the whole robots.txt debate moot for a large share of sites. Research from Vercel and MERJ found that roughly 69% of AI crawlers cannot execute JavaScript, so if your site renders its content client-side, those bots receive a near-empty page regardless of your robots.txt settings (Mersel, 2026). You can allow every citation bot in existence and still be invisible in AI answers, because there was nothing in the HTML they fetched to read.

This is why access and readability are two different problems, and why we treat the second as the real one. Opening your robots.txt is step zero; making sure your most important content exists in the static HTML a bot receives is the step that actually earns citations (Anagram, 2026). It is the same argument we make in our guide on JavaScript and web performance: the safest content is the content that renders without a script running at all.

robots.txt vs llms.txt: access versus navigation

These two files are often confused, and they do opposite things. robots.txt governs access — which crawlers may fetch which paths; llms.txt governs navigation — it guides AI systems toward your most valuable content once they’re allowed in (DataImpulse, 2026). One is a gate, the other is a signpost, and a site aiming for AI visibility wants both set correctly. Adoption of llms.txt still sits around 10% of domains, which makes it a low-effort, low-risk signal that few competitors have bothered with (Mersel, 2026). We cover what it does, and honestly what it doesn’t yet do, in our guide on what llms.txt is.

How to verify it actually worked

Editing the file is the easy part; confirming it took effect is where sites get a false sense of safety (Rankry, 2026). The proof is your server access logs: look for each bot by name and check the response codes, where allowed bots should return 200 and disallowed ones should stop appearing entirely (Rankry, 2026). A directive you never verified is a directive you’re only hoping works.

Check the CDN layer first, because it can block or challenge bots before a request ever reaches your robots.txt (Rankry, 2026). This matters more since Cloudflare began blocking AI crawlers by default on new domains from July 2025, which means a brand-new site can be invisible to AI without a single line in its own robots.txt (Inspeccia, 2026). If your logs and your intent disagree, the edge is usually where the answer is.

What we do, and what we’d tell you to do

Our own stance is the visibility-maximising one: allow the bots that can cite you, keep an edge rule ready for the ones that misbehave, and — the part most guides skip — build the site as static HTML so the open door leads somewhere a crawler can actually read. On a client-rendered site, permission is theatre; on a static one, permission is an invitation the bot can accept. There is now a third setting beyond allow and block — charging — which our guide on whether you should charge AI crawlers weighs, along with why it fits publishers far more than most businesses. For a site that wants to be an answer, the file settings and the architecture have to point the same way, which is the whole argument of our pillar on whether your website is readable by AI.

If we were advising you on the file itself: allow OAI-SearchBot, Claude-SearchBot and PerplexityBot so you stay eligible for citations; decide training (GPTBot, ClaudeBot, Google-Extended) on your own IP terms, knowing it costs nothing on Google; block the pure bulk scrapers that return nothing; and verify all of it in your logs. Then spend the rest of your effort where it compounds — on the content and the structure an AI reads once it’s through the door.