The main crawlers
| Company | Training | Search / live answers |
|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot (search index), ChatGPT-User (fetches when a user's chat needs a page) |
| Anthropic | ClaudeBot | Claude-SearchBot, Claude-User |
| Perplexity | (not used for training, per Perplexity) | PerplexityBot (index), Perplexity-User (on request) |
| Google-Extended (a robots.txt control, not a separate bot: it tells Google whether content crawled by Googlebot may be used for Gemini training) | Googlebot (AI Overviews and AI Mode use Google Search's index) |
Names and behaviour change; check each company's crawler documentation before you edit robots.txt.
Training vs search
Blocking a training crawler keeps your pages out of future model training. Blocking a search crawler removes your pages from that engine's live answers, which usually costs visibility. Many brands therefore block or allow training by preference but keep search crawlers allowed.
Example robots.txt
# Allow AI search and live-answer crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Opt out of training (optional, your choice)
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /How ClerAEO measures it
The free scan checks crawler access as one of its AI-readiness checks, so you can see whether AI crawlers can reach your site. To check one site's robots.txt now, use the free AI crawler checker.