Get a demo

Why can't AI crawlers read my page?

Either robots.txt blocks AI crawlers from the page, or your server refuses them. Either way, AI cannot read the page or cite it.

What it means

  • robots.txt blocks it: a rule in your robots.txt names an AI crawler, or all crawlers, and the page's path.
  • The server blocks them: your server or firewall answers AI crawlers with a refusal instead of the page.

How Proofsource fixes it

  1. Shows which crawler is blocked from which page, and by which rule.
  2. Drafts the change: the robots.txt line to change, or the allow rule for your host.
  3. After your Yes, reads the page again as each AI crawler and shows that it now gets the page.

Try it on your site

Try it on your site: a free check reads your site the way AI does and lists what Fix it would fix first.

Common questions

Will this let every bot in?

No. The change names the AI crawlers you choose. Other rules in your robots.txt stay as they are.

Read next

More answers: How often Proofsource scans AI answers · Which AI engines and models Proofsource asks · How Proofsource picks the questions it asks · What a Proofsource scan returns · Export Proofsource data, or reach it over the API and MCP · Integrations, alerts and shared reports · How to Set Up the Proofsource MCP Server (Step by Step) · Start a fix from your code: the fixes API and webhook

The shortlist is being written right now.

See whether AI names your brand, who it names instead, and what to fix.

200 answers. 25 Prompts/Question. Top AI engines. 2 days. Free.ChatGPTPerplexityClaudeGoogle AI OverviewsGoogle AI ModeGeminiMicrosoft CopilotGrokDeepSeekMeta AI

We use cookies to see how the site is used and to measure our ads. How we use cookies