Free tool · 7 crawlers · policy vs reality

Can ChatGPT actually read your site?

robots.txt is a promise; the server response is the truth. One run probes your site with each AI crawler's real user-agent and puts both verdicts side by side — the WAF and CDN blocks robots.txt never mentions light up red.

The seven we probe

These are the seven with the highest visibility impact. The full registry — 16 bots, who runs them, what blocking each one costs — lives at the AI bots registry.

GPTBot TRAINING

OOpenAI — model training + retrieval corpus

live probe + robots.txt

OAI-SearchBot SEARCH INDEX

OOpenAI — the ChatGPT search index

live probe + robots.txt

ChatGPT-User LIVE FETCH

OOpenAI — live fetch when a user asks

live probe + robots.txt

ClaudeBot TRAINING

Anthropic — crawling for Claude

live probe + robots.txt

PerplexityBot SEARCH INDEX

Perplexity — answer-engine index

live probe + robots.txt

CCBot TRAINING

CCCommon Crawl — feeds many models

live probe + robots.txt

Google-Extended ROBOTS TOKEN

Google — robots token for Gemini training (not a crawler)

robots.txt only — no crawler exists

How to read a result

allowed + HTTP 200

The bot can fetch and index you. This is the goal for retrieval bots — OAI-SearchBot, PerplexityBot — even if you deliberately block training bots.

allowed + HTTP 403 / 5xx

The classic silent killer: robots.txt welcomes the bot, the WAF turns it away. robots.txt never shows this — only a real probe does. Even we got bitten by this class of bug.

blocked

An explicit robots.txt disallow. Fine when intentional — e.g. blocking Google-Extended opts you out of Gemini training without touching your Search rankings.

Google-Extended

Has no crawler of its own — it's a robots token honored by Googlebot's fetches, so there is no live probe for it. Policy-only row.