How to Improve Your AI Crawl Access
If AI crawlers can't reach your site, nothing else matters — AI simply has no information to work with.
Checking Your Current Access
The Crawler Unblock Wizard checks your live robots.txt for these bots: GPTBot, ClaudeBot, PerplexityBot, Google-Extended. If any are blocked with a Disallow: / rule, you'll see it immediately.
The Fix
Run the Crawler Unblock Wizard
It scans your robots.txt and tells you exactly which bots are blocked.
Add the Suggested Lines
Copy the generated Allow: / rules into your robots.txt for each blocked bot.
Add an llms.txt File
Use the LLM.txt Generator to create a clean summary file AI can read directly — this is quickly becoming a standard alongside robots.txt.
Verify
Visit yoursite.com/robots.txt and yoursite.com/llm.txt to confirm both are live.
Most sites don't intentionally block AI crawlers — it usually happens by accident through a security plugin or a copy-pasted robots.txt template. It's worth checking even if you think you're fine.
Related Guides
What is LLM.txt and How to Set It Up
The emerging standard for telling AI assistants exactly what your business does.
The 6 Health Categories Explained
Every way AI sees your business, broken into 6 measurable categories.
What is Agent Readiness? A Plain English Guide
Why what AI says about your business now matters as much as where you rank on Google.