Get a demo
Playbook4 min read

Engines cited 129 sites that publish llms.txt 6,343 times. Once, it was the file

An llms.txt file is a reading list for agents. Google says Search ignores it, almost nothing fetches it, and the sites engines cite most mostly don't have one.

By ProofsourcePublished

llms.txt is a reading list for agents, not a ranking file

Jeremy Howard proposed the file in September 2024 and revised it in August 2026. It is a markdown file at the root of a site. The only required part is an H1 with the site's name; after that come an optional short summary in a blockquote and lists of links to the pages an agent should read, ideally as clean markdown.

The proposal is plain about who it serves. The file is meant to be read on demand, when an agent needs information while helping a user, and it is used most heavily for software documentation, where coding agents follow it to API references. OpenAI, Anthropic and Google each publish one for their developer docs. Nothing in it controls crawling, and nothing in it asks an engine to cite you.

Google says Search ignores the file, and 97% of files got no requests in Ahrefs' May sample

Google's guide to its generative AI features, last updated 10 July 2026, puts llms.txt first in a list of things you can ignore. Publishing one, it says, will neither harm nor help your visibility in Google Search, because Search doesn't use it.

Ahrefs checked the server logs of 137,210 domains for May 2026. 28% published a valid file, which Ahrefs calls an upper bound because its customers skew technical. 97% of those files received no requests at all that month. Of the requests that did arrive, the bots that fetch pages to answer live questions, such as OAI-SearchBot and PerplexityBot, sent 1.1%. AI agents, Anthropic's coding agent among them, and training crawlers fetched far more. And no AI bot requested an llms.txt that didn't exist, so a missing file isn't a missed knock.

Of 6,343 citations to sites with a file, one pointed at it

Ahrefs measured who fetches the file. We wanted the other end: what the engines cite. Our samples from 23 August to 5 October 2026 hold 41,597 citations in 4,228 answers from ChatGPT, Perplexity, Claude and Google AI Overviews, across 18 accounts. On 8 October we fetched /llms.txt from the 300 domains cited most often.

129 returned a real file. 137 had none, or served an HTML page in its place. 34 blocked us or errored. The 129 sites with a file were cited 6,343 times. One of those citations pointed at the file: an llms-full.txt, cited by Perplexity on 2 October. No answer cited an llms.txt.

The most cited sites are mostly without one. Of the top ten, eight have no file and two blocked our check. Of the top 25, three have one.

14 of the 129 files came from one template, written for shopping agents

Fourteen files, all from online stores, followed one template headed "Agent Instructions". Each says the store is built on Shopify and tells shopping agents to install a skill so they can buy directly. Ahrefs found the same drift: Wix generates the file for its sites, and other site builders now check for one.

That matters for two reasons. Many owners never chose what their file says. And the file speaks for you to any agent that reads it, so it deserves the review you'd give a page on your site.

Publish it if it's cheap, then spend the real hour elsewhere

The file costs little and does no harm, so there's no reason to delete one you have. Just don't count it as AI visibility work.

  • If you have developer docs or an API, publish one. Coding agents are the readers the data actually shows.
  • Keep it to plain links and one line descriptions, with nothing that reads as an instruction. Put it in version control and read whatever your platform generated.
  • Check that it is a real markdown file and not your homepage served at that path. Seven of our top 100 domains returned HTML with a 200 status.
  • Spend the rest of the time on the pages engines quote for your category: your own product and comparison pages, and the third party pages already cited next to you.

Where this could be wrong

A citation is not a read. An engine could fetch the file, follow its links and cite the page it pointed to, and our data would not show that. We measured what reached the answer, not what the engine looked at.

We fetched the files on 8 October, after the sampling window closed. Some may be newer than the citations they sit beside.

The 300 domains come from what our 18 accounts track: a mix of retail, consumer apps and business software, with many Indian brands. Another set of categories could look different.

Ahrefs' figures cover one month and a customer base that skews toward SEO. Fetch volumes may grow as agents take on more browsing.

Where the outside numbers come from

  1. The /llms.txt file, v2 · Jeremy Howard, llmstxt.org (3 September 2024, modified 10 August 2026)
  2. Optimizing your website for generative AI features on Google Search · Google Search Central (last updated 10 July 2026)
  3. We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read · Ahrefs, Louise Linehan (15 June 2026)
  4. llms.txt validator · Proofsource (free tool)
All notes

Questions

Does ChatGPT or Perplexity read llms.txt?

Neither documents using it to choose sources. In Ahrefs' May 2026 log study, OAI-SearchBot, PerplexityBot and similar retrieval bots sent 1.1% of the requests that reached llms.txt files. In our samples, one of 6,343 citations to sites with a file pointed at the file.

Does Google use llms.txt for AI Overviews?

No. Google's guide to its generative AI features says Search doesn't use llms.txt or other AI text files, and that publishing one will neither help nor harm your visibility.

Should I delete my llms.txt?

No. It costs little and coding agents do read some of them. Make sure it is a real markdown file, holds only links and descriptions, and says what you want said. Our free llms.txt validator checks the format.

Related notes

Field notes4 min read

When an AI summary appears, clicks fall from 15% to 8%

Pew watched 900 people search for a month. When an AI summary showed up, people clicked a result 8% of the time instead of 15%, and clicked a source inside the summary 1% of the time. Here is what that does to a brand, and where the number could be wrong.

Method4 min read

Our tracker missed Google's AI Overview on 96% of searches. One setting fixed it

From 22 August to 7 September our sampler recorded no Google AI Overview for 193 of 201 searches. Google had shown one almost every time. The Overviews load after the page, and our data vendor only returns them if you ask. On the same 36 questions after the fix, 3 of 487 searches came back empty. Here's what that means for any number a tracker gives you about Google.

Playbook4 min read

Seven universal sites took 5% of citations in 15 categories

Ask which sites to get mentioned on and you'll get the same list: Reddit, Wikipedia, YouTube, LinkedIn. Those lists are real, and they're built from every topic people ask about. We counted the citations behind buyer questions in 15 categories instead. The universal sites barely registered. Almost everything cited belonged to one category alone.

Your own numbers beat our best post.

The shortlist in your category is being written right now, whether anyone reads this page or not. A free trial, 25 questions on four engines today and tomorrow, tells you if your name is in it.

200 answers. 25 Prompts/Question. Top AI engines. 2 days. Free.ChatGPTPerplexityClaudeGoogle AI OverviewsGoogle AI ModeGeminiMicrosoft CopilotGrokDeepSeekMeta AI

We use cookies to see how the site is used. How we use cookies