Field notes4 min read

Robots.txt allowed AI crawlers on 21 of the 32 sites whose servers turned them away

A robots.txt checker reads the file and stops. The server, the CDN and the firewall decide whether a crawler actually gets the page, and on our sample they turned crawlers away almost three times as often as robots.txt did.

We checked 199 sites in robots.txt and at the server

How do I check which AI crawlers my site blocks? Most checkers, our free one included, read robots.txt and stop there. The standard behind the file, RFC 9309, is blunt about its limits: "These rules are not a form of access authorization." A rule asks a crawler to stay out. Whether a request gets the page is decided by your server, your CDN and your firewall.

Proofsource audits the site of every brand it tracks. We took each site's latest audit between October 5 and November 20, 2026: 199 sites, with our own and test domains left out. Each audit reads robots.txt for every AI crawler, then asks for the homepage once per crawler, sending that crawler's published User-Agent. It records whether the server served the page or answered 401, 403, 429 or a bot challenge.

Servers refused GPTBot on 32 sites; robots.txt blocked it on 12

The server turned away a request identifying as GPTBot on 32 of the 199 sites: 16.1% (11.6% to 21.8%). robots.txt disallowed GPTBot on 12 (6.0%).

On 21 of those 32 sites, robots.txt allowed every AI crawler we read it for. A check of the file alone would have told all 21 they were open to AI.

It went the other way once. One site disallowed GPTBot in robots.txt, and its server handed the page over anyway. The file is a request in both directions.

Most server blocks hit training crawlers and spared search

On 20 sites the server refused GPTBot but served OAI-SearchBot and ChatGPT-User, and 15 of those refused ClaudeBot as well. That split looks chosen. OpenAI's crawler documentation treats its crawlers as separate settings: GPTBot collects training data, OAI-SearchBot finds pages to show in ChatGPT search. Refusing the first doesn't take you out of the second.

13 sites refused the crawlers that put pages into answers

For being cited, the crawlers that matter are the search and user agents. 13 sites (6.5%) refused at least one of OAI-SearchBot, ChatGPT-User or PerplexityBot. robots.txt disallowed them on only one of the 13.

10 sites refused every AI crawler we sent. 7 of those 10 answered with a Cloudflare bot challenge rather than a plain error page. OpenAI says a site that opts out of OAI-SearchBot won't be shown in ChatGPT search answers, and it asks sites that want to appear to allow its published IP ranges as well as its name in robots.txt.

Some of these blocks were probably never written by hand

Cloudflare added a one-click option to block AI crawlers in September 2024 and says more than a million customers turned it on. Since July 1, 2025 it has asked every new domain at sign-up whether to allow AI crawlers at all. A setting chosen once in a dashboard, or by whoever set up the account, won't show up in robots.txt. If a crawler is refused and nobody on your team remembers deciding that, look there first.

Check both in ten minutes

  • Read robots.txt for the answer crawlers first: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot. Then decide separately about the training crawlers, GPTBot and ClaudeBot.
  • Ask for your homepage with curl, setting the User-Agent to each crawler's name. A 403, a 429 or a challenge page means the server refused it, whatever robots.txt says.
  • Open your CDN's bot and AI crawler settings and read what each crawler is set to. On Cloudflare, check both the AI crawler controls and any rule that blocks AI bots.
  • If your firewall only lets verified bots through, make sure the AI companies' published IP ranges are on its allow list, then look for 403s to those crawlers in your logs.

How Proofsource checks it

Every Proofsource site audit reads robots.txt for each AI crawler and asks for your homepage as each one. When robots.txt allows a crawler and the server turns it away, the first scan reports that on its own line, with the setting to change.

Where this could be wrong

Our requests came from our servers, not from OpenAI's, Anthropic's or Perplexity's. A firewall that checks where a request comes from may refuse ours and still serve the real crawler. That applies most to the bot challenges. Treat a refusal as a reason to open your CDN settings, not as proof that ChatGPT can't read you.

One request per crawler, to the homepage only, at one moment. 5 of the refusals were 429 rate limits, which can pass. Other pages can carry other rules.

199 sites of brands tracked in Proofsource, which is not a random sample of the web. Earlier audits sent five crawlers and later ones eight, so the counts for Claude-SearchBot, Claude-User and Perplexity-User cover 189 sites. We didn't see the sites' logs, so we can't say how real crawler visits were answered.

Where the outside numbers come from

  1. RFC 9309: Robots Exclusion Protocol · IETF
  2. Overview of OpenAI Crawlers · OpenAI
  3. Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large · Cloudflare
  4. Check which AI crawlers your robots.txt allows · Proofsource
All notes

Questions

Does robots.txt block AI crawlers?

It asks them to stay out, and the major AI companies say they follow it, but it doesn't stop a request. In our audit of 199 sites, one site disallowed GPTBot in robots.txt and its server served the page anyway.

How do I know if Cloudflare is blocking ChatGPT?

Request your homepage with the User-Agent set to OAI-SearchBot and look for a 403 or a challenge page, then read your Cloudflare AI crawler settings. In our sample, 7 of the 10 sites that refused every AI crawler did it with a Cloudflare bot challenge.

If I block GPTBot, will I disappear from ChatGPT?

Not from ChatGPT search. OpenAI documents GPTBot (training) and OAI-SearchBot (search) as separate settings. On 20 of the 199 sites we checked, the server refused GPTBot and still served OAI-SearchBot.

Your own numbers beat our best post.

The shortlist in your category is being written right now, whether anyone reads this page or not. A free trial, 25 questions on four engines today, tells you if your name is in it.

100 answers. 25 Prompts/Question. Top AI engines. Free.ChatGPTPerplexityClaudeGoogle AI OverviewsGoogle AI ModeGeminiMicrosoft CopilotGrokDeepSeekMeta AI