Find the CDN and firewall rules that block AI crawlers
A CDN AI crawler block check finds out whether your CDN, web firewall or robots.txt turns away GPTBot, ClaudeBot, PerplexityBot and other AI crawlers. Proofsource detects which CDN serves your site, reads its bot and firewall rules, reads robots.txt, and fetches your pages live to see what a crawler receives. Site owners and technical SEO teams get each block named, with the rule or setting that lifts it.
| Crawler | robots.txt |  Cloudflare rules | Live fetch |
|---|---|---|---|
| GPTBot | Allowed | Blocked by bot rule | 403 |
| OAI-SearchBot | Allowed | Allowed | 200 |
| ChatGPT-User | Allowed | Allowed | 200 |
| ClaudeBot | Allowed | Challenged | 403 |
| PerplexityBot | Disallowed | Allowed | 200 |
- Turn off the AI crawler block in your Cloudflare bot settings, or add a skip rule for verified AI crawlers
- Remove the Disallow for PerplexityBot in robots.txt if you want Perplexity answers to cite you
CDN AI crawler block check, in short
The facts, ready to quote.
- Detects
- Which CDN or host serves your site
- Rules read
- Cloudflare rules today; Vercel, Fastly, Akamai, CloudFront, Netlify and Google Cloud are coming soon
- Also checks
- robots.txt rules for each AI crawler, and a live fetch of any URL you give it
- Fix
- Names the blocking rule and how to lift it, then confirms the fix with a live fetch
- Protected sites
- An optional allow header and three published addresses let Proofsource read pages behind a firewall
- Crawlers
- GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and other AI crawlers
What is a CDN AI crawler block?
robots.txt is a request. The server decides who actually gets in. Between an AI crawler and your page sit a CDN, a web application firewall and often a bot management product. Many of these offer settings that block AI crawlers by default, with one switch, or as a side effect of a broad bot rule. When that happens, robots.txt still says GPTBot is allowed, your pages still load fine in a browser, and nobody on the team notices that ChatGPT, Claude and Perplexity are getting a 403 or a challenge page.
A CDN AI crawler block check looks in the places where those blocks live. It identifies which CDN or host serves the site, reads that provider's firewall, bot and access rules for anything that matches AI crawler user agents or addresses, reads robots.txt for each crawler, and then requests the page the way a crawler would to see the status code and body that come back. Each source of a block is shown on its own, because the fix is different for each: a robots.txt line, a CDN bot setting, or a firewall rule.
Proofsource runs these checks for your site and turns each block into a fix you can act on. Cloudflare rule checks are live. Checks for Vercel, Fastly, Akamai, CloudFront, Netlify and Google Cloud rules are rolling out, as is an AWS WAF allow rule Proofsource can add for you. If your site sits behind protection, you can turn on site access so Proofsource's own page checks get through, using an allow header and three published addresses you can allow-list.
How it works
Find out whether your CDN, firewall or robots.txt is turning away GPTBot, ClaudeBot or PerplexityBot, and get the rule that fixes it.
Detect your CDN
Proofsource works out which CDN or host serves your site, so it knows which rules to read.
Read the edge rules
Firewall, bot and access rules on that provider are read for anything that matches AI crawlers.
Read robots.txt
Each AI crawler gets an allowed or disallowed verdict from your robots.txt, and not determined when the file cannot be read.
Fetch the page live
A live fetch of any URL you supply shows the status code and content that come back, which catches blocks the rules alone do not show.
Get the fix
Each block is named with the rule or setting to change. On AWS WAF, an allow rule can be added for you as that integration rolls out.
See it in the product
Screens from the Proofsource app, drawn with sample data for an example brand.
| Crawler | Status | What came back |
|---|---|---|
| GPTBot | 403 | Blocked by a Cloudflare rule |
| OAI-SearchBot | 200 | Page returned |
| ClaudeBot | 403 | Blocked by a Cloudflare rule |
| PerplexityBot | 200 | Page returned |
| Googlebot | 200 | Page returned |
Allow-list these addresses or the header so our page reader gets your real page.
Every capability, live today
6 shipped capabilities behind CDN AI crawler block check, each in the product now.
CDN detection and Cloudflare check
Detects which CDN or host serves your site, then checks whether your Cloudflare rules block AI crawlers and names the rule that does.
robots.txt AI crawler check
Reads your robots.txt and tells you whether each AI crawler is allowed.
Bot access audit
Runs the bot accessibility audit for your brand and saves the result, so access is tracked over time.
Live crawlability check
Fetches any URL you supply right now and returns what came back, so you can test a fix the moment it ships.
Site access header
When you turn site access on, page checks send your brand's allow header, so a protected page answers with its content instead of a 403.
Published page reader addresses
Proofsource's page reader sends from three published, stable addresses you can allow-list on your firewall.
Who it is for
Explain a sudden drop in AI citations
Check whether a new CDN bot setting started turning AI crawlers away, and which rule to change.
Allow AI search without opening the gates
See exactly which rule blocks which crawler, so you can let search crawlers through and keep the rest of your bot protection.
Check every property at once
Main site, docs and help center often sit on different CDNs. See the result for each in one table.
Rule out access first
Before a client spends on content, confirm AI crawlers can actually reach their pages.
Why it matters
Find silent blocks
CDN blocks do not show in a browser or in robots.txt. Reading the rules and fetching live does.
Know which layer to change
robots.txt, CDN setting and firewall rule are shown separately, so the right team makes the right change.
Test fixes immediately
A live fetch confirms the block is gone the moment the rule changes.
Keep your protection
Fixes target AI crawlers specifically, so the rest of your bot defense stays in place.
Google AI OverviewsGoogle AI Mode
CDN AI crawler block check: Proofsource and other tools
Several tools check whether AI crawlers can reach a site. This table lists what each one's public pages describe.
| Capability | Proofsource | Otterly.AI | Scrunch | Semrush | AthenaHQ | Rankscale | Peec AI |
|---|---|---|---|---|---|---|---|
| robots.txt check for AI crawlers | YesAllowed or disallowed for each AI crawler, not determined when unreadable | YesCrawlability checker checks AI bot access; works on competitor sites tooSource | RelatedDeep audits flag access controls, JavaScript rendering issues, content gapsSource | YesSite Audit flags blocked AI bots including ChatGPT-User, OAI-SearchBot, PerplexityBotSource | YesDetects pages AI crawlers cannot accessSource | YesPage audit checks AI bot crawlabilitySource | YesSitemap, llms.txt and robots.txt crawlability historySource |
| CDN and firewall rules read | YesCloudflare rules live; Vercel, Fastly, Akamai, CloudFront, Netlify and Google Cloud rolling out | Not listed | Not listed | Not listed | Not listed | Not listed | Not listed |
| Crawl failures detected | YesLive fetch of any URL you supply, with status code and content | RelatedFree GEO tools include crawler simulationSource | YesDetects AI bot crawl failures, with fixesSource | Not listed | Not listed | Not listed | Not listed |
| Bot access tracked over time | YesBot access audit saved so access is tracked over time | RelatedChecks AI bot access; works on competitor sites tooSource | YesMonitors crawl health for AI botsSource | YesFlags blocked AI bots including ChatGPT-User, OAI-SearchBot, PerplexityBot, Claude-User, Claude-SearchBotSource | RelatedDetects pages AI crawlers cannot access; not a full auditSource | RelatedAI readiness score; 90+ checks; fixes; 10 to 200 audits per month by planSource | Not listed |
Where each of them shines
Otterly.AI's crawlability checker also works on competitor sites, which makes quick comparisons easy. Compare in full
Scrunch combines crawl error detection with crawl health monitoring and step-by-step fixes. Compare in full
Semrush's Site Audit names specific AI bots that are blocked, with an AI-readiness audit on its free plan. Compare in full
AthenaHQ builds crawlability into its brand integrity view, pointing out pages AI crawlers cannot access.
Rankscale folds AI bot crawlability into a page audit with more than 90 checks and an AI readiness score.
Sitemap, llms.txt and robots.txt crawlability checks with a history. Compare in full
Questions about CDN AI crawler block check
How do I know if Cloudflare is blocking AI crawlers?
Look in two places: Cloudflare's bot settings and firewall rules for anything matching AI crawlers, and the response a crawler actually gets. Proofsource reads your Cloudflare rules and fetches your pages live, so you see both the rule and its effect.
My robots.txt allows GPTBot. Why is it still blocked?
robots.txt is a request that well-behaved crawlers follow, not an access control. A CDN bot setting or firewall rule can still answer GPTBot with a 403 or a challenge. That block happens at the server, so robots.txt still reads as allowed.
Which CDNs and hosts does it check?
Cloudflare rule checks are available today. Checks for Vercel, Fastly, Akamai, CloudFront, Netlify and Google Cloud rules are coming soon: join early access and we tell you when they open. On any host, the robots.txt check and the live fetch work.
Can Proofsource fix the block for me?
On every provider today, Proofsource names the rule or setting to change and confirms the fix with a live fetch. An allow rule for AI crawlers that Proofsource adds for you on AWS WAF is coming soon: join early access and we turn it on for your workspace.
Does it work for ChatGPT, Claude, Perplexity and Google AI Overviews?
Yes. It checks the crawlers those engines use, including GPTBot, OAI-SearchBot and ChatGPT-User for ChatGPT, ClaudeBot for Claude and PerplexityBot for Perplexity. Google AI Overviews draw on Google's search index, so a Googlebot block matters there.
My site is behind a firewall. How does Proofsource read my pages?
Turn on site access. Proofsource's page checks then send your brand's allow header, and its page reader sends from three published, stable addresses you can allow-list.
Should I allow every AI crawler?
That is your choice. Many brands allow search crawlers and user fetches so they can be cited in answers, and decide separately about training crawlers. The check shows what is blocked today so the choice is deliberate.
How is this different from a robots.txt checker?
A robots.txt checker reports what your file asks for. This check also reads your CDN and firewall rules and fetches your pages live, so it finds blocks that a robots.txt checker would report as allowed.
Which AI engines does Proofsource track?
Proofsource tracks up to 10 AI engines, depending on your plan: ChatGPT, Perplexity, Claude, Google AI Overviews, Google AI Mode, Gemini, Microsoft Copilot, Grok, DeepSeek and Meta AI. The free scan and free trial run four of them: ChatGPT, Perplexity, Claude and Google AI Overviews.
Keep exploring
Try it on your own brand.
Proofsource is live: scan your brand and see CDN AI crawler block check at work with your own answers, competitors and sources.
Google AI OverviewsGoogle AI Mode