Free AI crawler checker: who your robots.txt lets in
The Proofsource AI crawler checker is a free robots.txt checker for site owners, SEOs and developers. Enter your domain or paste your robots.txt and it reads the rules for 16 crawlers, the AI bots from OpenAI, Anthropic and Perplexity plus Googlebot, Bingbot and others, and shows whether each is allowed, blocked or not mentioned, with the line that decides it.
| Crawler | Owner | Verdict | Line |
|---|---|---|---|
| GPTBot | OpenAI | Blocked | Line 4: Disallow: / |
| OAI-SearchBot | OpenAI | Allowed | Line 9: User-agent: * |
| ClaudeBot | Anthropic | Blocked | Line 6: Disallow: / |
| Claude-SearchBot | Anthropic | Allowed | Not mentioned, follows * |
| PerplexityBot | Perplexity | Allowed | Not mentioned, follows * |
| Googlebot | Allowed | Line 9: User-agent: * |
Free AI crawler checker, in short
The facts, ready to quote.
- Crawlers checked
- 16 AI and search crawlers
- Includes
- GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot
- Verdicts
- Allowed, blocked or not mentioned, per crawler
- Evidence
- The robots.txt line that decides each verdict
- Price
- Free, no sign-up
- Data kept
- Nothing is stored, and no AI model is involved
What is an AI crawler checker?
AI companies send crawlers to read websites. Some collect pages for training, such as GPTBot and ClaudeBot. Others fetch pages to answer a question right now, such as OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot. Your robots.txt file tells each of them what it may read. An AI crawler checker reads that file the way the crawlers do and tells you who is let in.
The difference matters. Blocking a training crawler keeps your pages out of future training sets, but does not remove you from ChatGPT or Claude answers. Blocking a search crawler does: an engine that cannot fetch your page cannot quote it. Many sites block the wrong bot by accident, through a wildcard rule or an old template.
The Proofsource checker reads your robots.txt for 16 crawlers, applies the same group and wildcard rules a crawler uses, and shows a verdict for each with the exact line behind it. It is a plain rule reader: nothing you paste is stored, and no AI model is involved.
How it works
A free robots.txt checker that says which of 16 AI and search crawlers may read your site, with the line that decides each one.
Enter a domain or paste the file
Type your domain to fetch the live robots.txt, or paste a draft to test it before you publish.
Rules are read like a crawler reads them
Each crawler's own group is found, and when it has none, the wildcard group applies.
One verdict per crawler
Allowed, blocked or not mentioned, for 16 AI and search crawlers, with the line that decides each.
Fix what you did not mean
Change the rule that blocks a search crawler you want, then run the check again.
See it in the product
Screens from the Proofsource app, drawn with sample data for an example brand.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: * Allow: /
Every capability, live today
3 shipped capabilities behind free AI crawler checker, each in the product now.
AI crawler checker
Reads a robots.txt and says which AI crawlers it lets in, crawler by crawler.
Free site tools
The free AI crawler checker, llms.txt validator and llms.txt generator, run on your live site with their working shown.
Live fetch of your files
The tools fetch your live robots.txt, llms.txt and sitemap from your domain, so you check what crawlers really see.
Who it is for
Test a robots.txt change
Paste a draft and confirm search crawlers stay allowed before the change ships.
Audit AI bot access in a minute
See every AI crawler's verdict on one screen, with the line to change for each.
Find out why AI never quotes you
Rule out the simplest cause first: a robots.txt rule that blocks the engines' search crawlers.
Check a client's site on a call
Enter the client's domain and show the result live, no sign-up needed.
Why it matters
Shows the deciding line
Every verdict points at the exact rule, so you know what to edit.
Training and search bots apart
See the difference between keeping out of training and disappearing from answers.
Nothing stored
A plain rule read. No AI model, no account, no saved data.
Instant and free
No sign-up. Results show in seconds.
Google AI OverviewsGoogle AI Mode
Free AI crawler checker: Proofsource and other tools
Checking AI crawler access is a common feature among AI visibility tools. Here is what each tool describes on its own site.
| Capability | Proofsource | Otterly.AI | Semrush | Rankscale | AthenaHQ | Ahrefs | Peec AI |
|---|---|---|---|---|---|---|---|
| Checks AI crawler access | Yes16 AI and search crawlers, verdict and deciding line for each | YesCrawlability checker: checks AI bot access; works on competitor sites tooSource | YesSite Audit flags blocked AI bots including ChatGPT-User, OAI-SearchBot, PerplexityBot, Claude-UserSource | YesPage audit checks AI bot crawlabilitySource | YesDetects pages AI crawlers cannot accessSource | RelatedBot Analytics shows real-time AI crawler visits and top pagesSource | YesSitemap, llms.txt and robots.txt crawlability historySource |
| Free with no sign-up | YesOpen to anyone, no account | RelatedFree GEO tools including crawler simulationSource | RelatedAI-readiness audit of 100 pages included on free planSource | Not listed | Not listed | RelatedBot Analytics free in betaSource | Not listed |
| Test a pasted draft | YesPaste a robots.txt before you publish it | Not listed | Not listed | Not listed | Not listed | Not listed | Not listed |
| Fetches your live files | YesLive robots.txt, llms.txt and sitemap from your domain | Not listed | Not listed | Not listed | Not listed | Not listed | Not listed |
| llms.txt check and generator | YesFree llms.txt validator and generator | Not listed | RelatedAI Search Health widget; checks llms.txt, content length, last-modified headers, robots.txtSource | Not listed | Not listed | Not listed | Not listed |
| Training and search bots told apart | YesSearch bots such as OAI-SearchBot shown apart from training bots such as GPTBot | Not listed | RelatedFlags blocked AI bots including ChatGPT-User, OAI-SearchBot, PerplexityBot, Claude-User, Claude-SearchBotSource | Not listed | Not listed | Not listed | Not listed |
Where each of them shines
A crawlability checker that also works on competitor sites. Compare in full
AI crawler checks built into a full site audit, with a free tier. Compare in full
Page audits with over 90 checks and an AI readiness score.
Crawlability findings tied to the pages AI cannot reach.
Real-time AI crawler visit analytics from Bot Analytics. Compare in full
A crawlability history across sitemap, llms.txt and robots.txt. Compare in full
Questions about free AI crawler checker
Which AI crawlers does the checker read?
Sixteen: GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI; ClaudeBot, Claude-SearchBot, Claude-User and anthropic-ai from Anthropic; PerplexityBot and Perplexity-User; Google-Extended and Googlebot; Bingbot; Applebot-Extended; CCBot; Meta-ExternalAgent; and Bytespider.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects pages for training. ChatGPT search fetches pages with OAI-SearchBot and ChatGPT-User. Blocking those does remove you from ChatGPT's answers.
Should I block training crawlers?
That is a business decision. Blocking GPTBot or ClaudeBot keeps your pages out of future training sets, but it does not stop ChatGPT or Claude citing you through their search crawlers.
Why does a crawler say not mentioned?
Your robots.txt has no group for it, so it follows the wildcard (*) group. If there is no wildcard group either, nothing restricts it.
Does robots.txt stop every bot?
No. It is a request that the major AI companies say they honor. A firewall rule is what blocks a crawler that ignores it.
Is anything I paste stored?
No. The check is a plain rule read. Nothing is stored and no AI model is involved.
My robots.txt allows everything. Why does AI still not quote my pages?
Crawlable is step one. Your server may still block AI fetchers, or your pages may render empty without JavaScript. Proofsource checks both on your live site, then shows whether 10 AI engines such as ChatGPT, Perplexity, Gemini and Copilot actually name you.
How is this different from Otterly's or Semrush's crawler checks?
Both are solid checks inside larger tools. The Proofsource checker is free with no sign-up, lets you test a pasted draft, and shows the exact line behind each of its 16 verdicts.
Which AI engines does Proofsource track?
Proofsource tracks up to 10 AI engines, depending on your plan: ChatGPT, Perplexity, Claude, Google AI Overviews, Google AI Mode, Gemini, Microsoft Copilot, Grok, DeepSeek and Meta AI. The free scan and free trial run four of them: ChatGPT, Perplexity, Claude and Google AI Overviews.
Keep exploring
Try it on your own brand.
Proofsource is live: scan your brand and see free AI crawler checker at work with your own answers, competitors and sources.
Google AI OverviewsGoogle AI Mode