Fix your own site so AI can read and place you.

Each engine sends its own bots, and each can be allowed or turned away separately. The ones that decide whether you can appear in answers today are the search and user fetchers. The training crawlers decide what future models may learn.

The bot most people block is not the one that decides whether they show up. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the one that surfaces sites in ChatGPT search; sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers. Anthropic separates ClaudeBot (training) from Claude-SearchBot and Claude-User, and Perplexity says PerplexityBot is not used for training at all. Google's crawler documentation says Google-Extended does not affect inclusion in Google Search, so blocking it does nothing to AI Overviews (the full crawler map).

Two failures are common, and neither shows up when you look at your site in a browser.

• Blocked at the edge. A CDN's bot-blocking setting can return an error to AI crawlers even when robots.txt allows them; Cloudflare announced on July 1, 2025 that it would block AI crawlers by default unless they pay. A robots.txt checker, like our free AI crawler checker, will still report every bot as allowed, because the block happens at the server.
• Empty until JavaScript runs. If your text only appears after scripts execute, many bots see a blank page, or a handful of words where a person sees thousands.

One more for Google: to be eligible as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to show a snippet, so a nosnippet or tight max-snippet rule removes it from the AI answer too.

Check it yourself in five minutes

Fetch a key page as each bot and look at the status code and the words you get back:

curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://yoursite.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "ClaudeBot" https://yoursite.com/
curl -s -A "OAI-SearchBot" https://yoursite.com/ | sed 's/<[^>]*>//g' | wc -w

A 403 or 503 means something is blocking that bot. A word count far below what you see on the page means your text depends on JavaScript. Fix the block at the CDN, and serve your main content in the HTML.

No. llms.txt is a proposed markdown file listing your key pages for language models. Google's guide to its generative AI features, last updated July 10, 2026, says publishing one will neither harm nor help your visibility in Google Search, because Search does not use it, and none of the engines we track has said it uses llms.txt for ranking.

Our honest finding: we fetched /llms.txt from the 300 domains AI engines cited most in our samples. 129 published one, and those 129 sites were cited 6,343 times. One citation pointed at a file, an llms-full.txt; none pointed at an llms.txt. Of the ten most cited domains, eight had no file and two blocked our check (the llms.txt study). Ahrefs found the same from the other end: 97% of the files in its May 2026 sample received no requests at all.

It is harmless to publish, and agents that read documentation do use it. If you publish one anyway, write it with the llms.txt generator and check the live copy with the llms.txt validator.