AI Grade Tool AI Grade Tool
Free AI Robots.txt Checker

Is your site blocking
AI crawlers?

Your robots.txt file controls which AI crawlers can read your site. AI Grade Tool's Robots.txt Checker tests your domain against 17 known AI bots — including GPTBot, ClaudeBot, and PerplexityBot — and shows exactly which ones you allow or block.

Paste a URL. Get a per-crawler verdict for every major AI search engine — OpenAI, Anthropic, Google, and Perplexity — instantly.

No robots.txt on your domain? We treat that as fully open — no restrictions.

What the Robots.txt Checker evaluates

AI Search Crawlers

GPTBot, ClaudeBot, PerplexityBot, Google-Extended — the bots that power AI search answers. Blocking these removes your content from AI responses.

Training Crawlers

CCBot, Meta-ExternalAgent, Bytespider — used for model training. Blocking these is often a deliberate IP or privacy choice, not a visibility issue.

Correct precedence rules

Specific bot rules always override the wildcard User-agent: * group. We apply the same logic bots use — no false positives from wildcard blocks.

Clear verdict

Get a headline count of how many AI search crawlers you're blocking and specific guidance on which ones matter most for AI visibility.

Frequently Asked Questions

Which AI crawlers does the Robots.txt Checker test? +

AI Grade Tool's Robots.txt Checker tests 17 known AI crawlers split into two groups: 10 AI search crawlers (including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended) that power real-time AI answers, and 7 training crawlers (including CCBot and Meta-ExternalAgent) used primarily for model training.

Does blocking GPTBot hurt my Google rankings? +

No — GPTBot is OpenAI's crawler, not Google's. Blocking GPTBot prevents OpenAI from indexing your content for ChatGPT and ChatGPT Search, but has zero effect on Google rankings or Google AI Overviews, which use different crawlers.

How do I allow AI crawlers in my robots.txt? +

Add a specific User-agent rule for each crawler you want to allow, followed by 'Allow: /'. If you have a blanket 'User-agent: * Disallow: /' rule, add specific per-bot Allow rules above it — specific rules take precedence over wildcards in robots.txt.

What happens if I have no robots.txt file? +

If your domain has no robots.txt, AI Grade Tool treats it as fully open — all 17 crawlers are allowed access to all pages by default. This is the most permissive state and means no AI search crawler is blocked.

What is the difference between AI search crawlers and training crawlers? +

AI search crawlers like GPTBot and PerplexityBot retrieve content in real time to power AI-generated answers and citations. Training crawlers like CCBot collect content to train AI models and typically don't affect real-time search visibility — blocking them is often a deliberate IP or privacy choice, not a visibility issue.

Want a full AI citation readiness score?

Get your full GEO score with AI Grade Tool →