How to let GPTBot, ClaudeBot and PerplexityBot read your site
A practical guide to the AI crawler user-agents, how to allow or block each one in robots.txt, and how to verify they can actually reach your content.
If you want AI answers to mention and link your site, the crawlers behind them need permission to fetch it. Each major assistant ships a named user-agent, and robots.txt is where you welcome or refuse them.
The bots to know
- GPTBot — OpenAI's crawler for training and browsing.
- ClaudeBot — Anthropic's crawler.
- PerplexityBot — Perplexity's answer-engine crawler.
Allowing them in robots.txt
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://yoursite.com/sitemap.xmlBlock the same way with Disallow: / under a bot's user-agent if you would rather it stayed out. Be deliberate: blocking a crawler means that assistant can't cite you.
Verify it actually works
A rule that looks right can still fail — a stray Disallow higher up, a framework serving robots.txt as HTML, a missing sitemap link. Run your URL through the AI Crawlability checker to see, bot by bot, what is allowed and what is silently blocked, then let the skill fix the gaps in place.