Field notes4 min read

53% of the requests calling themselves GPTBot, ClaudeBot or PerplexityBot on six websites came from somewhere else

A bot's name is something the visitor types. The address is the proof. On half the sites we looked at, most of the AI crawler traffic was a costume, and on the other half almost none of it was.

We checked 64,222 AI crawler requests against the makers' own lists

Are the AI crawler visits in your logs real? A crawler announces itself with a user agent, a line of text such as "GPTBot/1.1". Anyone can send that line. OpenAI, Anthropic and Perplexity each publish the IP addresses their bots use, and a request is theirs only if the name and the address agree.

Six websites have connected Cloudflare to Proofsource, none of them ours. For every request that named an AI crawler, we checked the address against the list its maker publishes. We took October 12 to November 13, 2026, 33 days, and five bots: GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot from Anthropic and PerplexityBot. The counts are estimates from Cloudflare's sampled request data.

52.9% of the requests failed, and GPTBot failed most among crawlers

Of 64,222 requests naming one of the five bots, 33,964 came from an address its maker doesn't list: 52.9%. By bot, the share that failed was 83.0% for ChatGPT-User, 72.6% for GPTBot, 61.0% for ClaudeBot, 34.4% for OAI-SearchBot and 34.1% for PerplexityBot.

Googlebot, checked the same way against Google's lists, failed in 20.7% of requests on the same sites. AI crawler names were faked more often than Google's, at least here.

Three sites were mostly fakes and three were almost clean

The average hides the real story. On three sites, 69%, 84% and 96% of the AI crawler requests failed the check. On the other three, 4%, 5% and 7% did. On the worst site, ClaudeBot made 251 verified requests and 7,144 that failed.

So a single benchmark for "how much AI bot traffic is fake" doesn't mean much. Your own site could sit at either end, and only a check of your own traffic tells you which.

200 of 253 failing addresses also claimed another company's bot

Cloudflare gives us up to three sample addresses per bot, site and day. Leaving out our own servers, 253 sampled addresses failed while claiming one of the five bots. 200 of them also claimed a bot from a different company on another request, and 168 claimed bots from three or more companies. One address used 12 different bot names from nine companies.

No real crawler does that. GPTBot comes from OpenAI's list and Googlebot from Google's. An address that is both is someone trying names until one gets through, most often from rented cloud servers.

The lists miss some real traffic, so check every list

The check isn't perfect in the other direction. Claude-User, the agent Claude uses to read a page a user asked about, failed in 97.5% of requests on these sites. About half of the sampled failures came from 160.79.104.0/21, a range Anthropic publishes as the source of its outbound web fetch and web search calls on its API IP page, which isn't its crawler list.

Those visits are very likely real, and a check against the crawler list alone marks them fake. That's why we left Claude-User out of the 52.9%. If you verify by hand, check every range a company publishes for its fetchers, not only the crawler file.

What we'd do

  • Count AI crawler visits by verified address, not by name. A dashboard counting names can be more than half costume.
  • Use each maker's list: openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, claude.com/crawling/bots.json plus Anthropic's API ranges, and perplexity.ai/perplexitybot.json.
  • If your firewall lets bots in by name, match the address too. Perplexity's own documentation shows a Cloudflare rule pairing each user agent with its IP ranges.
  • Don't block a bot by name because of what impostors did. The fakes aren't the bot, and blocking the real OAI-SearchBot or PerplexityBot keeps you out of their search results.

How Proofsource shows it

When a site connects Cloudflare, Proofsource checks each AI crawler request's address against the list its maker publishes and shows verified and failed requests separately for every bot. Bots whose makers publish no list are shown as unverifiable rather than counted as real or fake. Our check has the Claude-User gap described above, and we are adding Anthropic's API ranges to it.

Where this could be wrong

Six sites is a small sample, and they connected Cloudflare to a visibility tool, so they may not look like the average site. The spread between them matters more than the 52.9%.

A list can be out of date or incomplete, as Claude-User shows. Some failures for other bots could be real traffic from addresses not yet listed. The cross-claiming addresses make that unlikely for most of them.

Our own site check sends a few requests under these bot names. At most 300 could fall in this window; removing all of them still leaves 52.7%.

Requests aren't independent, and one busy scraper can move a site's number. One outside account, a single new site over two weeks, found 6 of 33 claimed AI assistant visits real. That's a different site and method, and points the same way.

Where the outside numbers come from

  1. Overview of OpenAI Crawlers · OpenAI
  2. Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic
  3. IP addresses · Anthropic
  4. Perplexity Crawlers · Perplexity
  5. Verify Requests from Google Crawlers and Fetchers · Google
  6. 81.8% Of My 'AI Assistant' Traffic Was Fake. The Googlebot Number Was Worse · Search Engine Journal
All notes

Questions

Are the AI crawler visits in my logs real?

Check the address, not the name. On six websites from October 12 to November 13, 2026, 52.9% of requests naming an OpenAI, Anthropic or Perplexity bot came from addresses those companies don't publish. The share ranged from 4% on one site to 96% on another.

How do I verify that a GPTBot request really came from OpenAI?

Match the request's IP address against the list OpenAI publishes at openai.com/gptbot.json. OpenAI publishes separate lists for OAI-SearchBot and ChatGPT-User, Anthropic and Perplexity publish their own, and Google documents reverse DNS and IP lists for its crawlers.

Should I block AI crawlers that fail the check?

Block the failing addresses if you like, but not the bot name. Blocking OAI-SearchBot or PerplexityBot by name also keeps out the real crawler, which those companies say keeps your pages out of their search answers.

Your own numbers beat our best post.

The shortlist in your category is being written right now, whether anyone reads this page or not. A free trial, 25 questions on four engines today, tells you if your name is in it.

100 answers. 25 Prompts/Question. Top AI engines. Free.ChatGPTPerplexityClaudeGoogle AI OverviewsGoogle AI ModeGeminiMicrosoft CopilotGrokDeepSeekMeta AI