AI Crawler & robots.txt Access Checker
Paste your robots.txt and see exactly which AI crawlers you are letting in or shutting out, from GPTBot and ClaudeBot to OAI-SearchBot, PerplexityBot, Google-Extended and Bingbot. Most sites block the wrong ones: it tells you which bots feed your AI-search citations versus which only train models, so you stop hiding from ChatGPT and Perplexity by accident.
| AI crawler | What it does | Type | Status |
|---|
Blocking AI bots is not one decision, it is two
Every AI company now runs more than one crawler, and they do completely different jobs. Lumping them together is how sites accidentally make themselves invisible in AI answers while thinking they only opted out of training.
- AI search & citation bots (
OAI-SearchBot,ChatGPT-User,PerplexityBot,Claude-User) fetch and index pages so the assistant can surface and link to you. Blocking these costs you visibility and clicks. Keep them allowed. - AI training bots (
GPTBot,Google-Extended,CCBot,Applebot-Extended) collect pages to train future models. Blocking them is a content-rights choice and has no effect on traffic either way. - Search bots that also feed AI (
Googlebot,Bingbot) power classic search and AI Overviews / Copilot. Never block these, or you disappear from search entirely. - A bot with no rule of its own falls back to your
User-agent: *block, so a broadDisallow: /there can silently block every AI assistant.
Want to check whether a single page is actually citable once the bots can reach it? Run it through our free AI Search Visibility Checker, or generate a manifest with the llms.txt generator.
Letting the crawlers in is step one. Knowing what people actually search is step two.
Opening your robots.txt to AI assistants only matters if the pages answer real demand. Before you write or rework a page, confirm people search the term and that a newer site can realistically rank and get cited for it. Semrush gives you live search volume, keyword difficulty, and the questions your competitors already win.
Get live keyword data free in SemrushSemrush offers a free trial with full keyword volume and difficulty data. Affiliate link: if you subscribe we may earn a commission, at no extra cost to you. We only recommend tools we would use ourselves. Disclosure.
AI crawler access, questions answered
Is this AI crawler checker free?
Yes. It runs entirely in your browser with no signup and no limit. Nothing you paste is uploaded or stored, so you can safely test a robots.txt before you publish it.
What is the difference between an AI training bot and an AI search bot?
Training crawlers like GPTBot, Google-Extended and CCBot collect pages to train future AI models. Blocking them only opts your content out of model training and has no effect on traffic. AI search and citation bots like OAI-SearchBot, PerplexityBot and ChatGPT-User fetch and index pages so AI assistants can surface and link to you. Blocking those costs you visibility in AI answers, which is real traffic.
Should I block GPTBot?
That is a content-rights choice, not an SEO one. Blocking GPTBot stops OpenAI using your pages for model training but does not remove you from ChatGPT Search, which uses a separate crawler called OAI-SearchBot. If you want to appear in AI answers, keep the search and citation bots allowed even if you block the training ones.
How does the checker decide if a bot is blocked?
It parses your robots.txt the way a crawler does: it finds the most specific User-agent group that matches each bot (falling back to User-agent: *), then evaluates whether the site root is reachable using the longest-match Allow and Disallow rules. A Disallow: / in the matching group counts as blocked. It checks root access, which is the signal that matters for whether a bot can crawl your site at all.