Is your site blocking AI crawlers?
AI engines can't cite what they can't read — and many CDN and firewall presets block their crawlers by default. Check your robots.txt against the 12 documented AI user-agents and see whether you publish an llms.txt. Free, no login, read straight from your site's own rules.
Questions, answered
Why does blocking AI crawlers hurt my visibility?
AI answer engines (ChatGPT search, Perplexity, Google AI Overviews) can only cite sites their crawlers can read. A Disallow rule aimed at them removes you from consideration silently — you never see the traffic you didn't get. Some CDN and firewall presets block these bots by default, so many site owners are blocked without ever choosing to be.
Does blocking GPTBot remove me from ChatGPT answers?
Not directly. GPTBot gathers training data; ChatGPT search citations come via OAI-SearchBot and live fetches by ChatGPT-User. Similarly, blocking Google-Extended opts you out of Gemini training but does NOT remove you from Google AI Overviews — those use regular Googlebot. This checker labels each crawler so you can tell the difference.
What is llms.txt?
A plain-text file at /llms.txt that tells AI agents what your site is and where the important pages are — like robots.txt's welcoming cousin. It's an emerging convention: publishing one costs nothing and makes agent visits more productive.
How is this check done? Do you estimate anything?
No estimates. We fetch your live /robots.txt and /llms.txt and evaluate the rules exactly as published, per the robots exclusion standard. If we can't reach your site, we say so instead of guessing — the same no-fabricated-metrics rule as our AI-visibility grader.