Lesson 3.1 · 9/3212 min9 sources
AI Crawlers Explained: GPTBot vs OAI-SearchBot vs ClaudeBot and More
AI crawlers explained: what GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended do, and how to block training without losing citations.
You'll be able to
- ✓Name the main AI crawlers and user-agent tokens for OpenAI, Anthropic, Perplexity, Google, Apple and Microsoft
- ✓Separate training crawlers from search-index bots and user-triggered fetchers
- ✓Decide which bots to allow if you want citations but not training use
- ✓Check whether your CDN is blocking AI crawlers without your knowledge
The short answer
AI companies run separate bots for separate jobs. Training crawlers (GPTBot, ClaudeBot) collect content for model training. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index that answers cite. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) load a page when someone asks. You can block training and still allow search, so you stay citable.
Inside this lesson
- Why one "AI bot" setting is the wrong mental model
- The crawler table
- Microsoft is different
- The strategic decision
- Hands-on exercise + knowledge check
Part of the full course
Unlock every lesson, tool and exercise
One payment. Lifetime access. $59 $25