Explicit rules win
Allow and Disallow directives are interpreted directly.
Robots.txt Analyzer
Check if AI crawlers (ChatGPT, Claude, Perplexity, etc.) can access and cite your content.
Methodology
Deterministic robots.txt checks, no guesswork.
This analyzer fetches robots.txt, checks explicit crawler directives, and reports the result in plain language. It does not infer permissions from marketing copy or meta tags.
Sitemap discovery is verified
The report only marks a sitemap present when the file exposes one.
Current bot names matter
The guidance tracks the crawler names and defaults the tool checks today.
That makes the result easier to trust when you are deciding whether AI crawlers can reach the pages that matter.