We scored 137 UK B2B sites for AI crawler access. The headline finding is not how many block AI — it is that almost every blocker blocks the harmless class.
AI user agents split into two classes, and conflating them produces audit findings that are either alarmist or wrong. Training crawlers affect whether a model learns your brand over time. Retrieval agents fetch your page while the model answers a live question. Blocking the first is a defensible policy choice. Blocking the second means a model that wants to cite you cannot open the page.
GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider, cohere-ai, Amazonbot.
These collect content that may inform future training. Blocking them slows how well a model knows your brand exists in the abstract. It does not stop a model citing you today.
OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-Web.
These fetch a page during the answer. A block here is the expensive one: the assistant has decided your page is relevant to a live question and is refused at the door.
Blocking of some kind was uncommon, and where it existed it was almost entirely the training class. More striking was the uniformity: several blocked sites were serving byte-identical robots.txt files — a content-signals header declaring search=yes, ai-train=no, use=reference, followed by Disallow: / for the training set and nothing else.
That is Cloudflare's default AI policy. It is not a misconfiguration, nobody fumbled it, and it does exactly what its owners want: permit search and live reference, refuse training. Telling one of those companies they have “blocked ChatGPT” is both false and instantly checkable — the fastest way to lose a prospect who knows their own stack.
Genuine retrieval blocks were rare. Across the first hundred sites checked we found one: Disallow: / against ChatGPT-User and PerplexityBot alongside the training set. Live retrieval closed, almost certainly without anyone intending it. A ten-minute fix worth a great deal of citation.
A scanner that greps robots.txt for known AI user agents and reports a count produces a finding that is technically accurate and commercially useless. It cannot distinguish the company that made a deliberate content-licensing decision from the company that accidentally closed its own front door, and it treats both as the same alarm.
The effect on outbound is worse than useless. A generic “your site blocks AI crawlers” email reaching a CTO who configured that policy on purpose reads as an automated scan, because it is one.
Never report that AI crawlers are blocked without naming which agent, stating whether it is training or retrieval, and saying what that specifically costs. Concretely:
PerplexityBot is disallowed”, not “AI crawlers are blocked”.Open yourdomain.com/robots.txt and search for each retrieval agent by name: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-Web. A Disallow: / against any of them, or a global User-agent: * block with no explicit allow, is the finding worth acting on today. Everything else is a policy conversation, not an emergency.
If you would rather have it checked properly — alongside extractability, entity resolution and who is being cited for your category questions instead of you — send us your domain. The check is free and comes back without a call.
A training crawler collects pages to inform future model training. A retrieval agent fetches a page while the model is composing a live answer. Blocking training affects whether a model learns you exist over time; blocking retrieval means the model cannot open your page even when it wants to cite you.
The main ones are OAI-SearchBot and ChatGPT-User for OpenAI, PerplexityBot and Perplexity-User for Perplexity, and Claude-Web for Anthropic. These are separate from training crawlers such as GPTBot, ClaudeBot, Google-Extended, CCBot and Applebot-Extended.
It is a trade-off, not a mistake. Blocking GPTBot prevents your content being used in training while leaving live retrieval and citation available. Many organisations block it deliberately as content policy, and Cloudflare's default AI setting does exactly this.
Read your robots.txt and check each named agent individually rather than relying on a wildcard summary. Look specifically for Disallow rules against OAI-SearchBot, ChatGPT-User and PerplexityBot — those are the ones that cost live citations.