An AI-ready website checklist, running on our own site
An AI-ready website gives AI crawlers and answer engines complete HTML, clear structured data and files that describe the site. This checklist covers llms.txt, JSON-LD schema, robots.txt, sitemaps, prerendering and IndexNow, exactly as proofsource.co runs them today. Free checkers fetch your robots.txt, llms.txt and sitemap.xml, and Proofsource customers can tell Bing and other engines the moment a URL changes.
| File | Found | Note |
|---|---|---|
| robots.txt | Yes | GPTBot, ClaudeBot and PerplexityBot allowed |
| llms.txt | No | No AI summary file at the site root |
| sitemap.xml | Yes | Sitemap index with 3 child sitemaps |
AI-ready website checklist, in short
The facts, ready to quote.
- Files for machines
- robots.txt, llms.txt, llms-full.txt and sitemap.xml, generated from one page table
- AI brief
- Structured facts, audience, questions and a do-not-claim list for AI engines
- Structured data
- Organization, WebSite, SoftwareApplication, BreadcrumbList, Article, FAQPage and offers
- Rendering
- Every route prerendered to complete HTML at build time
- Freshness
- Visible updated date matches schema dateModified and sitemap lastmod
- Change pings
- IndexNow tells Bing, Yandex and others when a URL changes
- Free checkers
- Fetch any site's robots.txt, llms.txt and sitemap.xml
- Headers
- Content security, framing and referrer headers, with cache lifetimes set for assets and crawler files
What is an AI-ready website?
An AI-ready website is one that AI crawlers and answer engines can fetch, read and describe correctly without guessing. Many AI crawlers read the raw HTML a server returns and do not run JavaScript, so the text must be in the first response. They follow robots.txt rules and sitemaps to decide what to fetch, and they use structured data to understand what an organization, product, article or FAQ is. Newer conventions such as llms.txt give language models a short, curated map of the site in plain text.
Readiness is a set of small, checkable habits rather than one trick: complete HTML on every route, a robots.txt that allows the crawlers you want, sitemaps with accurate last-modified dates, JSON-LD that matches what is visible on the page, a dated update line, breadcrumbs, internal links between related pages, sensible cache and security headers, and a fast way to tell search engines when something changed. None of it guarantees a citation, but missing pieces quietly keep good pages out of answers.
Proofsource builds proofsource.co this way, and this page lists each piece as it runs on our own site. You can check your own robots.txt, llms.txt and sitemap.xml with our free tools, and Proofsource customers can submit changed URLs through IndexNow so Bing, Yandex and other participating engines hear about updates right away.
How it works
The llms.txt, schema, robots.txt, sitemap, prerender and IndexNow checklist we run on proofsource.co, with free checkers for your own site.
Keep one page table
Each route has its title, description, canonical, share card and noindex flag in one table. Every machine file is generated from it.
Ship complete HTML
Every route is prerendered to full HTML at build time, so crawlers that skip JavaScript still read the whole page.
Describe the page in schema
JSON-LD for the organization, site, product, breadcrumbs, articles, FAQs and plan offers, built from what is visible on the page.
Publish the machine files
robots.txt, sitemap.xml with lastmod per locale, llms.txt and llms-full.txt are generated on every build.
Announce changes
After a deploy, changed URLs are sent through IndexNow so participating engines can recrawl them quickly.
See it in the product
Screens from the Proofsource app, drawn with sample data for an example brand.
| URL | Change | Sent | Result |
|---|---|---|---|
| driffle.ai/faq | New FAQ answers | Nov 1, 14:02 | Accepted |
| driffle.ai/compare/otter | Updated table | Nov 1, 14:02 | Accepted |
| driffle.ai/pricing | New plan text | Oct 30, 09:41 | Accepted |
IndexNow tells Bing, Yandex and other participating engines that a URL changed.
Every capability, live today
12 shipped capabilities behind AI-ready website checklist, each in the product now.
Free robots.txt, llms.txt and sitemap fetch
The public free tools safely fetch any site's robots.txt, llms.txt and sitemap.xml so you can check what crawlers find.
IndexNow submission
Proofsource tells Bing, Yandex and other participating engines when one of your URLs changes, through the IndexNow protocol. On proofsource.co, every deploy does the same for the URLs that changed.
AI brief
proofsource.co publishes structured facts, audience, common questions and a do-not-claim list written for AI engines.
Breadcrumbs and FAQ schema
Visible breadcrumbs on every page, matched by BreadcrumbList structured data. FAQPage JSON-LD is built only from questions and answers shown on the page.
JSON-LD structured data
Organization, WebSite, SoftwareApplication, BreadcrumbList, Article, FAQPage and plan offers, described in JSON-LD per page. Official social profiles are listed as sameAs in the Organization schema, which helps engines tie the brand together.
Freshness signals
A visible updated date that matches the schema dateModified and the sitemap lastmod, so freshness signals agree. sitemap.xml lists every page per language.
llms.txt and llms-full.txt
A generated plain-text summary of the site for language models, plus a full-text version.
Per-route SEO table and robots.txt
Title, description, canonical, share card and noindex for every route, kept in one table. robots.txt is generated from the same table, so crawler rules never drift from the site.
Related pages links
Each page links to related pages, so crawlers and readers find the next relevant page.
Security and cache headers
Content security, framing and referrer headers, with cache lifetimes set for assets and for crawler files.
Site verification files
Verification files for search engines sit at the site root.
Static prerender
Every route is rendered to complete HTML at build time, so crawlers that do not run JavaScript read the full page.
Who it is for
A checklist with a working example
Copy the pattern: one page table that drives HTML, schema, robots.txt, sitemap and llms.txt, so nothing drifts.
Check the basics in a minute
Run the free checkers on your site and see whether robots.txt, llms.txt and sitemap.xml are in place.
Get new pages noticed fast
Submit changed URLs through IndexNow so participating engines hear about updates the day you ship.
Hand clients a clear standard
Use the checklist as the technical baseline for every client site before content work starts.
Why it matters
Readable by every crawler
Prerendered HTML means crawlers that skip JavaScript still get your words.
Signals that agree
Dates, schema and sitemaps come from the same source, so engines never see conflicting facts.
Faster discovery
IndexNow tells participating engines about a change instead of waiting for a recrawl.
Proven on a live site
Each item runs on proofsource.co today, so it is a working pattern, not a theory.
Google AI OverviewsGoogle AI Mode
AI-ready website checklist: Proofsource and other tools
Several AI visibility and SEO tools check the files and signals AI crawlers rely on. This table lists what each one's public pages describe.
| Capability | Proofsource | Semrush | Peec AI | Otterly.AI | Writesonic | Profound | Scrunch |
|---|---|---|---|---|---|---|---|
| Checks robots.txt, llms.txt and sitemaps | YesFree fetch of robots.txt, llms.txt and sitemap.xml for any site | YesAI Search Health widget; checks llms.txt, content length, last-modified headers, robots.txtSource | YesSitemap, llms.txt and robots.txt crawlability historySource | RelatedCrawlability checker for AI bot access; works on competitor sites tooSource | RelatedSite audits, 100 to 2,500 pages per auditSource | Not listed | RelatedDeep audits flag access controls, JavaScript rendering issues, content gapsSource |
| Tell engines a URL changed | YesIndexNow submission to Bing, Yandex and other participating engines | Not listed | Not listed | Not listed | Not listed | RelatedSubmit new content directly to AI crawlers for faster discoverySource | Not listed |
| Free public checkers | YesFree llms.txt validator, llms.txt generator and AI crawler checker | YesAI-readiness audit of 100 pages included on free planSource | Not listed | YesFree GEO tools: prompt research, fan-out, content check, crawler simulationSource | Not listed | Not listed | Not listed |
| Pages readable without JavaScript | YesEvery route rendered to complete HTML at build time | Not listed | Not listed | Not listed | Not listed | YesServes fully rendered pages to AI bots on CloudFront, Cloudflare or VercelSource | YesAXP pre-renders pages so AI bots can read themSource |
| Structured data checked | YesOrganization, WebSite, Article and FAQ JSON-LD on every page, FAQ built only from visible answers | Not listed | Not listed | YesStructured data and AI-readiness review with fixesSource | Not listed | Not listed | Not listed |
| Freshness signals agree | YesVisible updated date matches schema dateModified and sitemap lastmod | RelatedChecks last-modified headers and content lengthSource | Not listed | Not listed | Not listed | Not listed | Not listed |
Where each of them shines
Semrush checks llms.txt, robots.txt and last-modified headers in its AI Search Health widget and includes a free AI-readiness audit. Compare in full
Peec AI keeps a history of sitemap, llms.txt and robots.txt crawlability, so changes over time are visible. Compare in full
Otterly.AI offers a wide set of free GEO tools, including a crawler simulation and a crawlability checker that works on any site. Compare in full
Writesonic includes site audits on its plans, covering from 100 to 2,500 pages per audit.
Profound lets teams submit new content directly to AI crawlers for faster discovery. Compare in full
Scrunch pairs deep site audits with an Agent Experience Platform that serves AI agents a machine-readable version of each page. Compare in full
Questions about AI-ready website checklist
What makes a website AI-ready?
Complete HTML on every page, a robots.txt that allows the AI crawlers you want, sitemaps with accurate dates, JSON-LD structured data that matches the visible page, clear internal links and a way to announce changes. Together they let AI crawlers fetch and understand your pages without guessing.
What is llms.txt and do I need one?
llms.txt is a plain-text file at your site root that gives language models a short, curated map of your site. It is a young convention and no engine promises to use it, but it is cheap to add. proofsource.co publishes both llms.txt and a full-text llms-full.txt.
Which schema types matter for AI search?
Start with Organization (with sameAs links to your official profiles), WebSite, BreadcrumbList, and the type that matches each page, such as Article, FAQPage, Product or SoftwareApplication with offers. Build it from what is visible on the page, never from hidden text.
Why prerender pages for AI crawlers?
Many AI crawlers read only the raw HTML and do not run JavaScript. A client-rendered page can look almost empty to them. Prerendering puts the full text in the first response.
What is IndexNow?
IndexNow is an open protocol that lets a site tell participating search engines, including Bing and Yandex, that a URL was added or changed. Proofsource customers can submit changed URLs through it, and proofsource.co pings it after every deploy.
Are the checkers free?
Yes. The llms.txt validator, llms.txt generator and AI crawler checker on proofsource.co are free and fetch the public files of any site.
Does this help with ChatGPT, Perplexity, Gemini, Copilot and other AI engines?
These are the foundations all four rely on when they fetch pages. They do not guarantee a citation, but missing pieces can keep good pages out of answers. Proofsource measures whether each engine actually cites you.
How is this different from Semrush or Peec AI site checks?
Semrush and Peec AI both check llms.txt, robots.txt and sitemaps, and Peec keeps a history of them. This page adds a working reference: every item as it runs on proofsource.co, free checkers for your own files, and IndexNow submission for customers.
Keep exploring
Try it on your own brand.
Proofsource is live: scan your brand and see AI-ready website checklist at work with your own answers, competitors and sources.
Google AI OverviewsGoogle AI Mode