Lesson 3.2 · 10/3214 minInteractive tool5 sources
How to Configure robots.txt for AI Search (With Examples)
Configure robots.txt for AI search with three copy-ready templates, avoid the mistakes that silently block AI crawlers, and test your rules properly.
You'll be able to
- ✓Write a robots.txt that matches your AI policy using a tested template
- ✓Explain how crawlers pick which group of rules to follow
- ✓Spot the five mistakes that make sites invisible to AI search
- ✓Test your rules with a live fetch and server logs
The short answer
Configure robots.txt for AI search by giving each AI crawler a named User-agent group. Allow search bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot if you want citations. Disallow training bots like GPTBot, ClaudeBot and Google-Extended only if you object to training use. Then confirm with a live fetch and your logs that no CDN rule overrides it.
Inside this lesson
- How crawlers read your file
- Template 1: allow all
- Template 2: allow search, block training
- Template 3: block all AI
- Common mistakes
- How to test
- Hands-on exercise + knowledge check
Part of the full course
Unlock every lesson, tool and exercise
One payment. Lifetime access. $59 $25