How to Check Whether AI Crawlers Can Reach Your Site

Robots.txt rules, the documented bot families from Google, OpenAI, Anthropic and Perplexity, and what structured data is documented to change

*Image: Blocked robots, according to website's robots.txt as of June 2026, sample size 1000 by ChippyTimeCom, licensed under CC BY-SA 4.0.*

GeoAnalyzer is a GEO/AEO audit tool that scores a site for AI recommendation readiness, per its homepage. Technical Accessibility is one of the five categories it scores, described on its About page as the question of whether AI engines can access and understand your content without barriers. The decision that category bears on is whether to change robots.txt rules for named AI user agents, and whether to treat schema markup as a lever for AI citations. Per GeoAnalyzer's pricing page, checked October 15, 2026, the free scan returns an overall 0-100 score, a tier classification of Bronze (0-59), Silver (60-79) or Gold (80-100), and the top hesitation AI might have about recommending the site; the full report is a one-time $19 payment. GeoAnalyzer's FAQ states the free preliminary score takes 30-60 seconds.

What robots.txt does and does not control

Google's introduction to robots.txt describes the file as telling search engine crawlers which URLs they can access, mainly to avoid overloading a site with requests, and states that it is "not a mechanism for keeping a web page out of Google."

The same robots.txt introduction lists three limits for anyone treating the file as a gate. The instructions cannot enforce crawler behavior, since obeying them is up to the crawler; Googlebot and other reputable crawlers follow them, and other crawlers might not. Different crawlers interpret syntax differently. A page disallowed in robots.txt can still be indexed if other sites link to it.

Google's specification page states that Google supports four fields, user-agent, allow, disallow and sitemap, and that fields such as crawl-delay are not supported. The page states that Google's crawlers select the group with the most specific user agent that matches and ignore the other groups, and that the order of groups in the file is irrelevant. Anthropic's support article on crawling and blocking states that its bots support the non-standard Crawl-delay extension, so one operator honors a directive that another ignores.

The crawler families, operator by operator

Google's overview of its crawlers and fetchers sorts them into three categories. Common crawlers such as Googlebot "always respect robots.txt rules for automatic crawls." Special-case crawlers serve specific products under an agreement between the crawled site and the Google product, which is how AdsBot ignores the global * user agent with the ad publisher's permission. User-triggered fetchers are parts of tools and product functions where a person triggers the fetch.

Google's page on its common crawlers describes Google-Extended as a standalone product token with no separate HTTP request user agent string, used in a control capacity over whether content Google crawls from a site may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI, and for grounding in those systems. The same page states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal there.

OpenAI's crawler documentation lists three user agents whose settings are independent of each other. GPTBot crawls content that may be used to train OpenAI's generative AI foundation models, and a site disallows it to indicate its content should not be used for that training. OAI-SearchBot surfaces websites in ChatGPT's search features, and OpenAI states that sites opted out of it will not be shown in ChatGPT search answers and can still appear as navigational links. ChatGPT-User handles user-initiated actions in ChatGPT and Custom GPTs, and OpenAI states that "robots.txt rules may not apply" to it. The same crawler documentation states it can take about 24 hours from a robots.txt update for OpenAI's systems to adjust.

Anthropic's support article, dated April 7, 2026, lists ClaudeBot for collecting web content that could contribute to model training, Claude-SearchBot for the relevance and accuracy of search responses, and Claude-User for user-initiated requests, which it says allows site owners to control which sites can be accessed. It states that Anthropic's bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt," that opting out requires a robots.txt entry in each subdomain, and that blocking the crawler's IP addresses "may not work correctly or persistently guarantee an opt-out."

Perplexity's crawler documentation lists PerplexityBot, described as designed to surface and link websites in search results on Perplexity and as not used to crawl content for AI foundation models, and Perplexity-User, described as supporting user actions and behaving as follows: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." The same documentation states that each setting works independently and that changes can take up to 24 hours to reflect.

The same category of bot carries different documented treatment across operators. Anthropic documents Claude-User as controllable through robots.txt, and OpenAI and Perplexity each document a user-initiated fetcher that can operate outside those rules.

Reading your own robots.txt

Token names change between operators and over time, so confirm the current ones in each operator's documentation on the day you edit and match the spelling exactly as documented.

Verification happens in server logs. Each operator publishes the IP ranges its crawlers use: OpenAI's crawler documentation links openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json; Anthropic's support article points to claude.com/crawling/bots.json; Perplexity's crawler documentation links perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json. Comparing log entries against those ranges establishes whether a request came from the declared crawler. OpenAI's crawler documentation notes that its robots.txt fetches may carry an additional robots.txt marker in the user agent, which makes them easier to identify in logs.

Google-Extended has no separate HTTP request user agent string, per Google's page on its common crawlers, so log searches for that name return nothing and the robots.txt token carries the control.

OpenAI's crawler documentation and Perplexity's crawler documentation each state that robots.txt changes can take about 24 hours to reflect, so a log check run immediately after an edit can show stale behavior. Perplexity's crawler documentation also includes WAF configuration guidance for Cloudflare and AWS that allows its bots by user agent plus the published IP ranges.

GeoAnalyzer's free Robots.txt Analyzer takes a website URL and reads the file to report whether AI crawlers can reach the content through robots.txt.

What structured data is documented to do

Google's introduction to structured data describes structured data as a standardized format for providing explicit clues about the meaning of a page and classifying its content, and states that adding it can enable rich results in Search. The same page cites Google's own case studies for rich results, including Rotten Tomatoes at a 25% higher click-through rate and Nestlé at 82% higher click-through rate.

Guidelines on the same page tell publishers not to create blank or empty pages just to hold structured data, to mark up only information visible to users, to use JSON-LD, to validate with the Rich Results Test, and to measure an effect with a before-and-after test on a few pages.

Google's documentation on AI features in Search states that a page must be indexed, eligible to be shown in Google Search with a snippet, and meet Search's technical requirements to be eligible as a supporting link in AI Overviews or AI Mode, and that there are no additional technical requirements. That documentation states that no new machine-readable files, AI text files or markup are needed, that no special schema.org structured data is required to appear in those features, that robots.txt directives for Googlebot are the control for managing how a site is crawled for Search, and that nosnippet, data-nosnippet, max-snippet or noindex limit what Search shows from a page.

None of the operator pages inspected for this article ties structured data to citation by an AI assistant. GeoAnalyzer's free schema checker is described on its page as analyzing structured data, Open Graph tags, E-E-A-T signals and AI-readiness, and the FAQ states that some suggestions, such as adding schema markup or updating technical settings, may require a developer or access to a content management system.

Where the documented control ends

Google's documentation on AI features in Search states that meeting all requirements, best practices and policies does not mean Google will crawl, index or serve a page, and that indexing and serving are not guaranteed.

For measurement, Google's documentation on AI features in Search states that pages appearing in AI features such as AI Overviews and AI Mode are included in overall search traffic in Search Console and reported in the Performance report within the "Web" search type. Server logs compared against the operators' published IP ranges show whether the named crawlers are reaching the site, which answers the access question independently of whether an assistant cites a page.

Which tokens to allow depends on which documented behavior you want to change: training use, covered by GPTBot and ClaudeBot; search surfacing, covered by OAI-SearchBot, Claude-SearchBot and PerplexityBot; or user-initiated fetches, whose documented treatment differs by operator. Confirm the current token names in each operator's documentation on the day you edit, and check the logs again about 24 hours after the change.

Disclosure: this article is published on GeoAnalyzer's own blog, geo-analyzer.com/blog/weekly, and GeoAnalyzer is the subject of this article.