AI Crawler robots.txt Generator
Build a robots.txt that treats AI crawlers correctly instead of lumping them together. Pick a goal, flip each bot on or off, and copy a clean file. The toggles separate AI-search citation bots, which earn you traffic, from model-training bots, which do not, so you stop hiding from ChatGPT and Perplexity by accident.
Blocking AI bots is not one decision, it is two
Every AI company now runs more than one crawler, and they do completely different jobs. Treating them as a single switch is how sites make themselves invisible in AI answers while thinking they only opted out of training.
- AI search & citation bots (
OAI-SearchBot,ChatGPT-User,PerplexityBot,Claude-User) fetch and index pages so the assistant can surface and link to you. Blocking these costs real visibility and clicks. Keep them allowed. - AI training bots (
GPTBot,Google-Extended,CCBot,Applebot-Extended) collect pages to train future models. Blocking them is a content-rights choice with no effect on traffic. - Core search bots (
Googlebot,Bingbot) power classic search and feed AI Overviews / Copilot. Never block these, or you disappear from search entirely. - A crawler with no rule of its own falls back to
User-agent: *, so this generator writes an explicit group for each bot you block and leaves the rest allowed by default.
Already have a robots.txt? Audit it with our free AI Crawler Access Checker, then check whether a page is actually citable with the AI Search Visibility Checker.
Opening your robots.txt only pays off if the pages answer real demand.
Allowing AI assistants to crawl you is wasted if nobody searches the topic, or if a new site cannot realistically rank and get cited for it. Before you write or rework a page, confirm the demand. Semrush gives you live search volume, keyword difficulty, and the exact questions competitors already win.
Get live keyword data free in SemrushSemrush offers a free trial with full keyword volume and difficulty data. Affiliate link: if you subscribe we may earn a commission, at no extra cost to you. We only recommend tools we would use ourselves. Disclosure.
robots.txt for AI crawlers, questions answered
Is this robots.txt generator free?
Yes. It runs entirely in your browser with no signup and no limit. Nothing you choose is uploaded or stored. Pick a preset, toggle the bots you want, and copy or download the file.
Will blocking GPTBot remove me from ChatGPT?
No. GPTBot only collects pages to train future OpenAI models. ChatGPT Search uses a separate crawler called OAI-SearchBot, and live link fetches use ChatGPT-User. To stay citable in ChatGPT while opting out of training, block GPTBot but keep OAI-SearchBot and ChatGPT-User allowed. The "Allow citations, block training" preset does exactly that.
Where do I put the generated robots.txt?
Save it as a plain text file named robots.txt and upload it to the root of your domain so it is reachable at yourdomain.com/robots.txt. It must be at the root, not in a subfolder. Changes take effect the next time each crawler visits.
Should I block AI crawlers at all?
Blocking AI search and citation bots costs you visibility in AI answers, which is real traffic, so most sites should keep those allowed. Blocking AI training bots is a content-rights choice with no traffic effect either way. Never block Googlebot or Bingbot, or you disappear from search entirely.