We checked 199 sites in robots.txt and at the server
How do I check which AI crawlers my site blocks? Most checkers, our free one included, read robots.txt and stop there. The standard behind the file, RFC 9309, is blunt about its limits: "These rules are not a form of access authorization." A rule asks a crawler to stay out. Whether a request gets the page is decided by your server, your CDN and your firewall.
Proofsource audits the site of every brand it tracks. We took each site's latest audit between October 5 and November 20, 2026: 199 sites, with our own and test domains left out. Each audit reads robots.txt for every AI crawler, then asks for the homepage once per crawler, sending that crawler's published User-Agent. It records whether the server served the page or answered 401, 403, 429 or a bot challenge.
Servers refused GPTBot on 32 sites; robots.txt blocked it on 12
The server turned away a request identifying as GPTBot on 32 of the 199 sites: 16.1% (11.6% to 21.8%). robots.txt disallowed GPTBot on 12 (6.0%).
On 21 of those 32 sites, robots.txt allowed every AI crawler we read it for. A check of the file alone would have told all 21 they were open to AI.
It went the other way once. One site disallowed GPTBot in robots.txt, and its server handed the page over anyway. The file is a request in both directions.
Most server blocks hit training crawlers and spared search
On 20 sites the server refused GPTBot but served OAI-SearchBot and ChatGPT-User, and 15 of those refused ClaudeBot as well. That split looks chosen. OpenAI's crawler documentation treats its crawlers as separate settings: GPTBot collects training data, OAI-SearchBot finds pages to show in ChatGPT search. Refusing the first doesn't take you out of the second.
13 sites refused the crawlers that put pages into answers
For being cited, the crawlers that matter are the search and user agents. 13 sites (6.5%) refused at least one of OAI-SearchBot, ChatGPT-User or PerplexityBot. robots.txt disallowed them on only one of the 13.
10 sites refused every AI crawler we sent. 7 of those 10 answered with a Cloudflare bot challenge rather than a plain error page. OpenAI says a site that opts out of OAI-SearchBot won't be shown in ChatGPT search answers, and it asks sites that want to appear to allow its published IP ranges as well as its name in robots.txt.
Some of these blocks were probably never written by hand
Cloudflare added a one-click option to block AI crawlers in September 2024 and says more than a million customers turned it on. Since July 1, 2025 it has asked every new domain at sign-up whether to allow AI crawlers at all. A setting chosen once in a dashboard, or by whoever set up the account, won't show up in robots.txt. If a crawler is refused and nobody on your team remembers deciding that, look there first.
Check both in ten minutes
- Read robots.txt for the answer crawlers first: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot. Then decide separately about the training crawlers, GPTBot and ClaudeBot.
- Ask for your homepage with curl, setting the User-Agent to each crawler's name. A 403, a 429 or a challenge page means the server refused it, whatever robots.txt says.
- Open your CDN's bot and AI crawler settings and read what each crawler is set to. On Cloudflare, check both the AI crawler controls and any rule that blocks AI bots.
- If your firewall only lets verified bots through, make sure the AI companies' published IP ranges are on its allow list, then look for 403s to those crawlers in your logs.
How Proofsource checks it
Every Proofsource site audit reads robots.txt for each AI crawler and asks for your homepage as each one. When robots.txt allows a crawler and the server turns it away, the first scan reports that on its own line, with the setting to change.
Where this could be wrong
Our requests came from our servers, not from OpenAI's, Anthropic's or Perplexity's. A firewall that checks where a request comes from may refuse ours and still serve the real crawler. That applies most to the bot challenges. Treat a refusal as a reason to open your CDN settings, not as proof that ChatGPT can't read you.
One request per crawler, to the homepage only, at one moment. 5 of the refusals were 429 rate limits, which can pass. Other pages can carry other rules.
199 sites of brands tracked in Proofsource, which is not a random sample of the web. Earlier audits sent five crawlers and later ones eight, so the counts for Claude-SearchBot, Claude-User and Perplexity-User cover 189 sites. We didn't see the sites' logs, so we can't say how real crawler visits were answered.