AI Access Watchdog
Two plain-text files decide whether AI engines can see you at all: robots.txt and
llms.txt. Both change silently — a WAF preset, a plugin update, a migration.
The watchdog checks them daily and emails you only when something changed.
No dashboard to remember, no weekly digest. Silence means you're fine.
What triggers an alert
Any tracked AI crawler flips between allowed and blocked. A blanket
Disallow: / from a WAF or plugin is the most common way sites go invisible
without anyone noticing.
The file appears or disappears. SPAs and migrations love to quietly replace it with an HTML 404 — we classify that as gone, same as our validator does.
Your files become unreachable to a datacenter probe. If we can't fetch them, AI crawlers likely can't either — this failure mode once cost this very site 90 points. Post-mortem.
A new AI crawler joins our tracked registry — you get one note saying whether your current robots.txt allows or blocks it. The registry update itself becomes your alert.
Currently tracked crawlers
- GPTBot O OpenAI
- OAI-SearchBot O OpenAI
- ChatGPT-User O OpenAI
- ClaudeBot Anthropic
- PerplexityBot Perplexity
- CCBot CC Common Crawl
- Google-Extended Google
Robots verdicts follow the same rule as the Bot Access Matrix:
a bot counts as blocked only on a blanket Disallow: / with no
Allow: / override. WAF-level blocks don't show in robots.txt — check those live in
the matrix.
Frequently asked questions
robots.txt and llms.txt and compare them with the stored baseline. For robots.txt we evaluate the verdict for each tracked AI crawler (blanket Disallow: / with no override counts as blocked); for llms.txt we track whether a real file is being served at all.