Check that AI crawlers can reach your site and read what is on it.

Crawler Access in Proofsource checks the two things that decide whether AI can learn from your site: whether GPTBot, ClaudeBot, PerplexityBot and the other AI crawlers are allowed in and actually served, and whether the page they receive contains your words or an empty JavaScript shell.

100 answers. 25 Prompts/Question. Top AI engines. Free. No card · no sales call
The Proofsource Websites screen, Site checks tab, for tesla.com: checks passed, warnings, failed and not determined, and a score per technical area including AI search access.
Site checks for tesla.com: every check, grouped by area, with AI search access among them.

A verdict for every AI crawler in your robots.txt.

Proofsource reads your robots.txt and gives a verdict for ten AI crawlers. Knowing which crawler does what matters: OAI-SearchBot and PerplexityBot fetch pages for live answers, while GPTBot and Google-Extended govern training data, so you can block training without disappearing from AI search.

  • GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and Bytespider
  • Allowed or disallowed, and not determined when the file cannot be read, never a guess
  • Google-Extended is a robots.txt token, not a separate crawler, and is checked as one

robots.txt is a request. The server decides who gets in.

A site can allow GPTBot in robots.txt and still answer it with a 403 from a CDN bot firewall. Proofsource requests your homepage as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot and records what your server actually returned. A robots.txt checker alone would report every one of them as allowed.

  • Served, turned away or unclear, per crawler
  • The status code each crawler received
  • Catches CDN and firewall blocks that robots.txt checks miss
What to fix first for tesla.com: three P1 failures, including the check that the server lets AI crawlers in.
The server check failed for tesla.com, so it is listed as a P1 fix.

See how much of your page a bot can actually read.

Many AI crawlers read the raw HTML and do not run JavaScript. Proofsource compares the words a browser renders with the words in the raw HTML. A client-rendered page can show thousands of words in a browser and a few dozen to a bot. If your content only appears after JavaScript runs, most AI crawlers never see it.

  • Rendered words against raw-HTML words, per page
  • Flags pages with no body text, no H1 or no crawlable links
  • Re-check after you ship server rendering or prerendering
What AI crawlers see for tesla.com: AI crawlers see 0% of what a browser shows on the homepage, with 1 page checked and 1 page missing content.
The share of a browser's words that an AI crawler's plain fetch also gets, per page.

If AI cannot read you, it cannot recommend you.

Find silent blocks

CDN bot settings can turn AI crawlers away without anyone on your team noticing. The server check shows it.

Make your words visible

A page that renders only in JavaScript is close to blank for most AI crawlers. The rendering check measures how close.

Block training, keep search

A verdict per crawler shows exactly which AI systems you let in, so blocking training is a choice, not an accident.

Questions buyers ask

How do I check whether GPTBot can access my website?

Check two things. First, robots.txt: it should not disallow GPTBot. Second, the server: request a page with GPTBot's user agent and confirm you get a 200, not a 403 from a CDN or firewall. Proofsource runs both checks and also confirms the page contains readable text.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content that may be used to train OpenAI models. OAI-SearchBot fetches pages for ChatGPT search results, and ChatGPT-User fetches a page when a user's conversation needs it. Blocking GPTBot does not remove you from ChatGPT search, but blocking OAI-SearchBot does.

What is Google-Extended?

Google-Extended is a robots.txt token, not a separate crawler. Disallowing it tells Google not to use your content for Gemini and Vertex AI model training. It does not affect Google Search.

Can AI crawlers read JavaScript websites?

Many cannot. Several AI crawlers read only the raw HTML a server returns and do not run JavaScript, so a client-rendered page can look almost empty to them. Server-side rendering or prerendering puts the text in the HTML where every crawler can read it.

Why would my site block AI crawlers without my knowing?

CDN and security providers offer bot-blocking settings, and some block AI training crawlers by default or with one switch. The block happens at the server, so robots.txt still says the crawler is allowed. Only a request made as the crawler shows it.

Can I try Proofsource for free?

Yes. 100 answers. 25 Prompts/Question. Top AI engines. Free. 25 buyer questions on ChatGPT, Perplexity, Claude and Google AI Overviews, today. No card. Then scanning pauses and your answers stay in your account.

How much does Proofsource cost?

Proofsource sells credits: monthly credits start from $9 a month for 500 credits. Each checked AI answer is 1 credit, and keeping a brand up to date uses daily (1,000 credits a brand a month) or weekly (350 credits). Unlimited team members on every plan. Need more one month? Add credits once.

More of the platform

Ask Proof

An assistant that answers questions about your AI visibility from your own data, and shows the queries behind every number.Explore Ask Proof

AI Visibility

Mention rate, share of voice and average position on each AI engine, with a 95% Wilson interval on every number.Explore AI Visibility

Prompts

The buyer questions you track, grouped by topic, tagged branded or unbranded, ranked by search demand, with the searches each engine runs.Explore Prompts

See every feature

Make sure AI can read you.

See whether AI names your brand, who it names instead, and what to fix. 100 answers. 25 Prompts/Question. Top AI engines. Free. No card · no sales call.

100 answers. 25 Prompts/Question. Top AI engines. Free.ChatGPTPerplexityClaudeGoogle AI OverviewsGoogle AI ModeGeminiMicrosoft CopilotGrokDeepSeekMeta AI