What is robots.txt?
robots.txt is a plain-text file at the root of a website that tells crawlers which paths they may fetch, under the Robots Exclusion Protocol (RFC 9309). robots.txt is the oldest control a site has over crawlers, and in the AI era it carries more decisions than ever: training, search indexing and user-triggered fetches each have their own bot, and a rule for one says nothing about the others.
How does robots.txt work?
It works as a set of groups: each names one or more user-agent tokens and lists Allow and Disallow paths. The rules are standardized as the Robots Exclusion Protocol in RFC 9309 (Koster, Illyes, Zeller and Sassman, September 2022), which sets out the details crawlers follow (RFC 9309, read Oct 15, 2026):
- The file is named robots.txt, in lower case, at the top level of the host, served as UTF-8 plain text.
- When rules conflict, the most specific match wins: the one with the most octets.
- Crawlers must parse at least 500 kibibytes of the file, and should not use a cached copy for more than 24 hours unless the file is unreachable.
- A 4xx response lets a crawler fetch anything; a 5xx response means it must assume everything is disallowed.
- The rules are not a form of access authorization: they are a request, and a listed path is publicly visible.
How do you write robots.txt rules for AI crawlers?
Name each bot by its token and decide each job separately. The common position is to stay findable in AI search while keeping content out of training: allow the search and user-fetch agents, disallow the training crawlers you object to.
Each host has its own file, so docs.example.com needs its own rules; Anthropic asks site owners to add its rules to every subdomain they want covered (Anthropic, April 7, 2026).
# Findable in AI search and user fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
# Kept out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /Which robots.txt mistakes hide you from AI answers?
The costly ones are usually accidental:
- Blocking the search agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot) while meaning to block training.
- A blanket rule under User-agent: * that also catches AI search crawlers.
- A CDN or security setting that blocks AI bots before your robots.txt is ever read.
- Rules on www but not on the docs or help subdomain engines most want to cite.
- Snippet controls on key pages: Google says a page must be eligible to show with a snippet to appear in AI Overviews or AI Mode (Google Search Central).
What does robots.txt decide in a real example?
It decides whether your pages can compete for citations at all. In the public Tesla example (Oct 4, 2026), Perplexity gave 375 citations and Claude 351 across 25 buyer questions; tesla.com got 3 and 1 of them.
A site whose robots.txt blocks those engines' search agents gives up even that. Allowing them does not win citations by itself; it puts your pages in the running.
How do you check your robots.txt for AI crawlers?
Fetch the live file from the public internet, for every host, and search it for each token by name. The free AI crawler checker reads your live robots.txt and reports which AI crawlers it allows or blocks, and Crawler access in Proofsource checks it with the CDN layer and rendered text.
Common questions about robots.txt
Does robots.txt stop AI from using my content?
Only for crawlers that honor it, and only from the time they read it. It is a request, not a lock, and user-triggered fetchers such as Perplexity-User document that they generally ignore it.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search is decided by OAI-SearchBot, and OpenAI says each setting is independent.
How long do robots.txt changes take to apply?
RFC 9309 says crawlers should not rely on a cached copy for more than 24 hours, and OpenAI says its search systems take about 24 hours to reflect a change.
Is llms.txt a replacement for robots.txt?
No. robots.txt sets crawler rules. llms.txt is a proposed reading list for language models and has no say over access.
Related terms
- GPTBot
GPTBot is OpenAI's web crawler for collecting content that may be used to train OpenAI's generative AI foundation models.
- OAI-SearchBot
OAI-SearchBot is OpenAI's crawler for surfacing websites in ChatGPT's search features, and sites that opt out of it are not shown in ChatGPT search answers.
- ClaudeBot
ClaudeBot is Anthropic's web crawler that collects web content that could contribute to training Anthropic's Claude models.
- llms.txt
llms.txt is a proposed markdown file at the root of a website that tells language models what the site is and lists the pages most worth reading.
Worked example: Tesla, public AI answers, Oct 4, 2026: Proofsource's first scan of Tesla, 25 buyer questions and 5 brand questions, one answer per question on each of four engines. A small sample from one brand, shown as an example, not a benchmark.
See where AI leaves you out, and what to fix.
See which buyer questions name your brand, who gets named instead, and what to fix. 200 answers. 25 Prompts/Question. Top AI engines. 2 days. Free. No card.
Google AI OverviewsGoogle AI Mode
