Does Your Site Block AI Crawlers? The Free robots.txt + llms.txt Check

Blocked AI crawlers are an invisible failure: your site looks fine in a browser, ranks in classic Google, and yet never appears in ChatGPT or Perplexity answers because the bots that feed those answers are turned away at the door. It happens by accident more often than by choice — a security plugin's bot preset, a years-old robots.txt "block everything weird" rule, or a CDN default. Cloudflare, notably, announced in July 2025 that it blocks AI crawlers by default for new sites, so a site can be blocked without its owner ever choosing anything.

The mistake that costs citations: blocking the wrong bot

The AI user-agents are not interchangeable, and the most expensive robots.txt mistakes come from treating them as one thing. Each operator runs separate crawlers for training (feeding future models) and for answering (fetching or indexing pages so today's assistant can cite them). Blocking a training bot is a legitimate policy choice that costs you nothing in visibility. Blocking an answer-side bot silently removes you from citations.

• ✓Blocking GPTBot does NOT remove you from ChatGPT answers. GPTBot is OpenAI's training crawler. ChatGPT's search citations come via OAI-SearchBot (index) and ChatGPT-User (live fetches) — per OpenAI's own bot documentation. Many sites blocked GPTBot on principle and accidentally kept the other two open, which is fine; blocking all three is what makes you invisible.
• ✓Blocking Google-Extended does NOT remove you from AI Overviews. Google-Extended is only the Gemini-training opt-out. AI Overviews and classic search both ride on Googlebot — which means blocking Googlebot removes you from both at once, per Google's crawler documentation.
• ✓Anthropic splits the same way: ClaudeBot is the training crawler; Claude-User fetches pages live when a Claude user asks about your site.
• ✓Perplexity uses both patterns: PerplexityBot builds its answer index and Perplexity-User fetches on demand — both affect whether Perplexity can cite you.
• ✓Bingbot feeds Bing search and Copilot answers from one index.

How to check your site (free, ~10 seconds)

You can do this by hand: fetch yourdomain.com/robots.txt, find every User-agent group, and work out which of the 12 documented AI tokens each Allow/Disallow rule catches, remembering that rules for `*` apply to any bot without its own group. Or paste your domain into Answerlord's free crawler check: it fetches your live robots.txt, evaluates the published rules for each documented AI crawler exactly as the robots exclusion standard reads them, and returns a per-crawler verdict — open, limited, or blocked — split into the ones that affect citations and the ones that only affect training. No estimates: if your site can't be reached, it says so rather than guessing. It also checks a second file most sites haven't heard of yet:

llms.txt: robots.txt's welcoming cousin

Where robots.txt tells crawlers what they may not read, llms.txt (a plain-text file at /llms.txt) tells AI agents what your site *is* and where the important pages are — a proposed convention that's cheap to adopt: one Markdown file with your site's purpose and key links. Publishing one costs nothing, can't hurt, and makes an AI agent's visit more productive; the crawler check detects whether yours exists and whether it returns real content rather than an HTML error page. If you don't have one, our answer-first content guide covers the same principle at page level: state what you are, up front, in plain text.

Access is step zero, not the whole game

An open robots.txt makes citations *possible*, not *likely*. Once you've confirmed the crawlers can read you, the question becomes whether AI engines actually name you when buyers ask — and that you can also check free with a full AI-visibility grade: five real buyer-intent questions run live across ChatGPT, Perplexity, and Google AI Overviews, a 0–100 score, and the competitors being cited instead of you. Access check first, visibility grade second, then the fixes.

Verification: crawler names, purposes, and the training-vs-citation split are taken from each operator's published bot documentation (OpenAI, Google, Anthropic, Perplexity, Microsoft) as reflected in Answerlord's open crawler list; Cloudflare's default blocking of AI crawlers for new sites is from Cloudflare's July 2025 announcement. The crawler check reads a site's own published robots.txt/llms.txt — it never guesses, and unreachable sites are reported as unreachable.