AI Crawler List: Every Bot That Reads Your Site
At least ten AI crawlers fetch public pages in 2026. The main ones are GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, and PerplexityBot from Perplexity. CCBot and cohere-ai are here too. Each carries its own robots.txt token. The user-agent string is easy to fake, so verify a real request by its IP.
This AI crawler list does two jobs. It gives you the whole set in one table. It routes you to the per-bot card when you need exact directives. For the deepest single card, see the ClaudeBot reference.
Which AI crawlers visit your site
Google's AI Overview for "block ai crawlers," captured July 9, 2026, ships one sample robots.txt block. It names six tokens: GPTBot, anthropic-ai, Claude-Web, cohere-ai, CCBot, and Google-Extended. That is the floor, not the full roster. No marketer registry lists them all. The set is scattered across operator docs. This page assembles it.
One operator often runs several crawlers. OpenAI's bots page documents GPTBot for bulk crawling and OAI-SearchBot for answer-time search. Each setting is independent. Anthropic runs ClaudeBot plus two legacy tokens. The consequence is blunt. Block GPTBot, and ClaudeBot still reads you.
The live user-agent table
This AI crawler list runs to ten crawlers and ten tokens. One honest caveat: the user-agent string proves nothing. Anyone can send it. So match on the token. Then confirm the request by IP. IP verification is the only proof that holds. Check the source IP against the operator's published ranges. Or let Cloudflare bot management flag the verified bots. Strings and IP ranges rotate. Verify an exact string against the operator's live docs.
| Crawler | Operator | robots.txt token | How to verify a real request |
|---|---|---|---|
| GPTBot | OpenAI |
| Match OpenAI's published IP ranges. |
| OAI-SearchBot | OpenAI |
| Match OpenAI's published IP ranges. |
| ChatGPT-User | OpenAI | See OpenAI's live docs | Match OpenAI's IP ranges. |
| ClaudeBot | Anthropic |
| Use Anthropic's ranges or Cloudflare. |
| anthropic-ai | Anthropic |
| Legacy token. Match Anthropic's ranges. |
| Claude-Web | Anthropic |
| Legacy token. Match Anthropic's ranges. |
| PerplexityBot | Perplexity |
| Match Perplexity's published IP ranges. |
| Google-Extended | |
| A control token, not a fetcher. |
| CCBot | Common Crawl |
| Match Common Crawl's IP ranges. |
| cohere-ai | Cohere |
| Verify against Cohere's docs. |
One row needs a footnote. Google-Extended is not a crawler. It is a control token. It decides whether pages Googlebot already fetched may feed Gemini and Vertex AI. Blocking it stops the AI use, not the fetch. For the full ClaudeBot card, read the ClaudeBot reference.
Block or allow? The orientation
A review of a few thousand US and UK sites found 27% blocking at least one major LLM crawler. Most of it was an accident. The block lived at the CDN, WAF, or edge, not in robots.txt (r/aeo, February 18, 2026). So the first question is not whether to block. It is whether you already are.
Block-or-allow decision path. A review of thousands of sites found 27% already block a major LLM crawler, mostly by accident (r/aeo, Feb 2026).
Blocking is a legitimate choice, and this page will not talk you out of it. The exchange is one-sided. AI crawlers take content and generate answers from it. They send little or no referral traffic back. A WordPress origin still pays PHP, plugin, and database load on every request (r/Wordpress, April 28, 2026). That is a fair case for publisher control. The load can spike hard. One owner logged Meta's crawler making 7.9 million requests and burning 900-plus GB in 30 days (r/webdev, April 6, 2026). That spike is why owners start looking. And the directives are only a request. The AI Overview notes that non-compliant bots "may ignore" robots.txt.
Two facts sharpen the call. Training-time crawling and answer-time retrieval are different levers. "Sites that block AI training crawlers mostly ignore the answer time bots" (Hacker News, July 7, 2026). So check which lever your rule pulls. There is also a middle path: pay-per-crawl. Cloudflare offers a model for it. One Caddy plugin charges crawlers real USDC per request (Hacker News, February 27, 2026). It keeps the door open at a price.
The verdict, no hedge. Are AI answers a channel for you? Allow the crawlers. Then audit that you are not blocking them by accident. That is the 27% trap. Is your content the product, and the trade not worth it? Block on purpose, or charge for access. Either way, verify each request by IP. Do not let a stale robots.txt decide for you. When you allow, use llms.txt as the explicit invitation.
Every crawler we track
Every crawler on this AI crawler list gets its own reference card. You get the exact token, the IP check, the robots.txt directives, and the observed crawl rate. The ClaudeBot card is live now. GPTBot, OAI-SearchBot, PerplexityBot, and a how-to-block guide are next.
Every crawler we track
- ClaudeBot: What It Is, How to Verify It, What Blocking It Costs — ClaudeBot is Anthropic's web crawler.
No comments yet