AI crawlers explained

Almost every argument about AI crawlers collapses the two classes into one, which is why the advice is contradictory. “Block the AI bots” and “you need to be in AI answers” are both reasonable positions, and they are not in conflict once you separate the agents by what they actually do.

What is the difference between a training crawler and a retrieval agent?

A training crawler collects pages that may be used in building or updating a model. The effect, if any, is diffuse and slow: over time a model may become more familiar with your brand as a concept. Blocking one is a legitimate editorial and commercial decision, and plenty of publishers have made it.

A retrieval agent fetches your page in the seconds after a user asks a question, so the assistant can read it and quote it. There is no lag and no ambiguity about the effect. If the agent is refused, the assistant cannot cite you in that answer — not because it chose not to, but because it could not open the page.

This is why a single “block AI bots” rule is so damaging. The instinct behind it is about training. The collateral damage is entirely to retrieval.

Which AI crawlers should I know by name?

Two entries in that table catch people out. Google-Extended is not a search crawler — it governs whether your content is used for Gemini training and grounding, and disallowing it has no effect on whether Googlebot crawls you or whether you can appear in AI Overviews. And Applebot-Extended is the opt-out, not the crawler: blocking it does not remove you from Apple’s search features.

Should I block AI crawlers?

There is a defensible answer for each class, and they are different answers.

For most B2B companies, blocking a training crawler protects nothing of value and blocking a retrieval agent costs you the answers. If you take one rule away from this page: allow retrieval, decide about training on the merits.

How do I write the robots.txt rules?

Directives are per-agent and evaluated by the most specific matching group, so a blanket rule and a named rule can coexist. A configuration that allows retrieval while declining training looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

If you have no reason to decline training either, the simplest correct file is a permissive one — and a permissive file has the advantage that it cannot be wrong about an agent that did not exist when it was written.

Why does my robots.txt say allow but the agent still gets blocked?

Because most blocks are not in the file. They are managed bot rules at the CDN, and they are the single most common cause of an unexplained absence.

Content delivery networks added AI-bot categories to their bot-management products, and some are switched on by default or enabled during a routine security review by somebody who was not thinking about search. The rule lives in a dashboard, not in your repository, so no code review will ever surface it.

Check, in this order: your robots.txt, your CDN bot-management settings, any WAF rules matching on user agent, and any origin-level rules in nginx or Apache. The conflict is almost always between the first and the second.

How do I test what an agent actually receives?

Make the request as the agent and read the status code. Nothing else settles it:

for ua in "OAI-SearchBot/1.0" "ChatGPT-User/1.0" "PerplexityBot/1.0" "GPTBot/1.0"; do
  printf "%-22s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://yourdomain.com/
done

A 200 is a pass. A 403 is a refusal. A 429 is a throttle, which behaves like a refusal from the agent’s point of view. And do the same test against a page deeper in the site, not just the homepage — rules are often scoped to paths.

Then check that the response actually contains your content:

curl -s -A "OAI-SearchBot/1.0" https://yourdomain.com/key-page | grep -c "a sentence from the page"

A 200 containing an empty JavaScript shell is a pass on access and a failure on everything downstream.

What does a blocked agent actually cost?

It is worth being precise, because the cost is not evenly spread across your funnel.

A blocked retrieval agent costs you the answers where a buyer is comparing options — the category questions, the “best X for Y” questions, the ones asked before anyone knows your name. Those are the answers you most want to be in, because being named there is the entire mechanism by which a stranger discovers you inside an assistant.

What it does not cost you is the branded question. Someone who already knows your company and asks about you by name will usually get an answer assembled from whatever the model already holds and whatever third-party sources it can reach. That answer may be stale, or wrong, and you have no ability to correct it while the agent cannot open your site — which is its own argument for allowing retrieval even if you never expect discovery traffic from it.

There is a second-order cost that is easy to miss. When an assistant cannot fetch your page but can fetch a competitor’s comparison article that describes you, the description of your company that reaches the buyer is the one your competitor wrote. Allowing retrieval is partly about being cited, and partly about being the most authoritative source on the subject of yourself.

What about llms.txt?

A proposed file listing your key pages in a form convenient for language models. Its status is genuinely contested: Google has said publicly that it is not used for its AI features, while other vendors recommend it and some agents do request it.

The sensible position is proportionate: it costs almost nothing to publish, it is read by some agents, and it is not a substitute for anything. Ship it, and do not let it displace crawler access or schema work, which are the things with demonstrable effects. The full argument, with what Google actually said →

How do I find out which AI crawlers are already visiting?

Your server or CDN logs already know. Filter requests by user agent for the names in the table above and you get a factual picture of which agents reach you, how often, which pages they request, and — crucially — which status codes they receive.

1. Export a week of access logs, or open your CDN’s analytics filtered by user agent.
2. Search for each agent name: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot, ClaudeBot.
3. For each, note the status codes. A retrieval agent receiving 403s is the finding.
4. Note which pages they request. It tells you what the assistants are interested in.
5. Be aware that user agents can be spoofed. The major operators publish IP ranges for verification if you need certainty.

An agent that never appears at all is also information: either it has not discovered you, or something upstream is refusing it before it reaches your logs — which points back at the CDN.

What changes when new AI crawlers appear?

They will. New assistants launch, operators split one agent into two, names change. Two habits keep you safe without constant attention.

• Prefer a permissive default. A file that allows everything and disallows only named training crawlers you have deliberately chosen to refuse cannot accidentally block an agent that did not exist when you wrote it. A file that disallows everything and allows a whitelist will silently exclude every new retrieval agent.
• Review quarterly. Check the operators’ published crawler documentation, check your logs for unfamiliar AI user agents, and check your CDN has not added a new default category.

What should I check this week?

1. Read your robots.txt and find the named retrieval agents. If they are not there, note it.
2. Run the loop above against your homepage and two deep pages.
3. Open your CDN’s bot-management settings and look for an AI or “AI scrapers” category. Note what it is set to.
4. Fetch one key page as an agent and grep for a sentence you expect to see.
5. Fix anything returning a non-200 to a retrieval agent. That is the whole of gate one.

Further reading

• How to check whether AI can see your website
• Training versus retrieval, with what we found across 137 UK B2B sites
• The practical AEO guide — the five gates in working order
• The AEO checklist — 24 checks, ordered
• Get the free AI visibility check