Every AI bot that knocks on your server — and what blocking it costs
The most expensive robots.txt mistake of 2026 is one line: User-agent: * Disallow: /
treats a training crawler and a search-index crawler the same — and only one of them is safe
to turn away. This registry groups the bots by what they actually do. Test your own rules
live with the Bot Access Matrix.
Training crawlers
They collect text to train models. Blocking them does not remove you from any AI search surface — this is the one group that's safe to block on principle.
Trains OpenAI's base models; also feeds retrieval corpora.
Safe to block — visibility in ChatGPT answers is unaffected.
Crawls the web for Claude's training.
Safe to block for visibility purposes.
Builds the open Common Crawl corpus that many labs train on.
Safe to block, but you exit a corpus dozens of models learn from.
Robots token: opts your content out of Apple Intelligence training (Applebot itself keeps indexing).
Training opt-out only — Spotlight/Safari visibility unaffected.
Training collection for YandexGPT.
Safe to block; Yandex Search itself uses other bots.
Search-index crawlers
They build the indexes AI search answers retrieve from. Block one of these and you disappear from that engine's answers — the most expensive robots.txt mistake of 2026.
Builds the index behind ChatGPT Search.
Blocking = you vanish from ChatGPT Search results.
Builds the index behind Claude's web search.
Blocking = absent from Claude Search answers.
Indexes for Perplexity's answer engine.
Blocking = no Perplexity citations.
The Bing index — which also feeds Bing Copilot and other AI surfaces.
Blocking costs Bing, Copilot and every product built on the Bing index.
On-demand fetchers
They fetch a specific page live, when a user asks the assistant about it. Blocking them means the assistant can't read your page even when the user pastes your URL.
Live fetch when a ChatGPT user asks about a page.
Blocking = ChatGPT can't cite the exact page a user asked about.
Live fetch on user request inside Claude.
Same trade-off as ChatGPT-User.
On-demand retrieval for a user's query.
Blocks page-level citations on request.
Live fetch for Le Chat and Mistral products.
Blocks Le Chat from reading your pages.
Robots tokens & special cases
Not everything in robots.txt is a crawler. Tokens control someone else's fetches; and one bot declares compliance it doesn't keep.
A control token, not a fetcher: opts your content out of Gemini training. Googlebot does the actual fetching either way.
Blocking opts out of Gemini training without touching Search rankings.
Indexes for Spotlight, Safari suggestions and Apple Intelligence retrieval — distinct from the -Extended token.
Blocking removes you from Apple's search surfaces.
ByteDance's crawler. Declares robots.txt compliance; bypasses have been documented repeatedly.
robots.txt alone may not stop it — enforce at the WAF/CDN by user-agent or IP range if you need it gone.
Go deeper
Bot names and IP ranges rotate — before enforcing anything, verify the exact string against the operator's live documentation. Registry roles follow the training / search / on-demand split; last reviewed July 2026.