The requests to run, the trap that hides the real answer, and what to do with the result.
Request your own pages while identifying as each named AI agent and compare the status codes to what a browser gets. If a named agent receives a 403 while your browser receives a 200, the model cannot reach you — whatever robots.txt says.
Loading your site in a browser tells you nothing about this. You are not the user agent that matters, and the block is usually applied by user agent.
Open yourdomain.com/robots.txt and look for these names specifically. They are permissioned separately from Googlebot and from each other.
| Agent | Class | Blocking it means |
|---|---|---|
OAI-SearchBot | Retrieval | ChatGPT search cannot fetch you live — expensive |
ChatGPT-User | Retrieval | A user asking ChatGPT to open your page gets nothing |
PerplexityBot | Indexing | Perplexity has nothing of yours to cite |
GPTBot | Training | Long-term familiarity only — a defensible choice |
ClaudeBot | Training | As above |
Google-Extended | Training | Does not affect Google Search or AI Overviews access |
A blanket User-agent: * / Disallow: is fine. What causes problems is a well-meant block added during an “AI scraping” conversation that swept the retrieval agents in with the training crawlers.
This is the step that finds the real problem. Run it from a terminal, once per agent:
curl -s -o /dev/null -w "%{http_code}\n" \
-A "OAI-SearchBot/1.0; +https://openai.com/searchbot" \
https://yourdomain.com/your-key-pageRepeat with ChatGPT-User, PerplexityBot and GPTBot. Compare each result against a normal browser request. A 200 in the browser and a 403 for a named agent is the answer, and it is common.
The trap: most blocks of this kind are not in robots.txt at all. They are managed bot rules at the CDN, often enabled by default or switched on by someone protecting the site from scraping. The file says allow, the edge says deny, and nothing in your own repository reveals it. Check the bot-management settings in Cloudflare, Fastly, Vercel or whatever sits in front of the origin.
Fetch the raw HTML and search it for the sentence you want quoted:
curl -s https://yourdomain.com/your-key-page | grep -i "the claim you want cited"If it is not in the response, assume it is not read. Content that only exists after hydration is content a model may never see.
Ask it to open a specific URL on your site and quote a line from it. If it reports that it cannot access the page, that is a retrieval failure in plain sight. If it opens the page but paraphrases without naming you, that is an entity problem rather than an access problem — a different gate, and a slower fix.
That you are mechanically capable of being cited. Not that you will be. Access is the first of five gates; extraction, entity, corroboration and measurement all sit behind it. The full sequence →
No. The block is applied by user agent, and your browser is not the user agent in question. You have to make the request while identifying as each named AI agent and compare the status codes.
Not necessarily. The most common cause of a blocked AI agent is a managed bot rule at the CDN, which overrides the file and does not appear anywhere in your own code. Check bot-management settings at the edge as well.
The retrieval agents — OAI-SearchBot, ChatGPT-User and PerplexityBot — because they fetch your page while a model is answering a live question. Blocking training crawlers like GPTBot or ClaudeBot affects long-term familiarity and is a far less damaging choice.
No. Google-Extended governs whether your content is used for Gemini model training and grounding, and it does not control Google Search crawling or your eligibility to appear in AI Overviews, which follow standard Googlebot access.
Then access is not your problem. The next gates are extraction — whether there is a short, self-contained statement worth quoting — and entity, whether the model can identify you as a nameable source.