What is ClaudeBot?
ClaudeBot is Anthropic's web crawler that collects web content that could contribute to training Anthropic's Claude models. Anthropic runs three bots with separate jobs: ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for fetching a page when someone asks Claude a question. Each is named on its own in robots.txt, so a rule for ClaudeBot says nothing about the other two.
What is ClaudeBot's user agent?
ClaudeBot identifies itself with the product token ClaudeBot, and that token is what a robots.txt group must name. Anthropic's help article names its three bots (ClaudeBot, Claude-User and Claude-SearchBot) but does not print a full user-agent string, so match on the token in your logs rather than on a copied string (Anthropic's crawler article, read Oct 15, 2026).
A name in a log line can be faked. To confirm a request really came from Anthropic, check its source IP against the list Anthropic publishes at claude.com/crawling/bots.json; Anthropic says a crawler with a source IP on that list is coming from Anthropic.
Some robots.txt files still name anthropic-ai or Claude-Web, older tokens. Anthropic's current article does not mention either, so a rule that names only those older tokens is not documented to cover ClaudeBot.
What does ClaudeBot collect, and why?
ClaudeBot collects public web content that could contribute to training Anthropic's generative AI models, which Anthropic says helps the models' usefulness and safety. Disallowing it signals that your site's future content should be excluded from Anthropic's training datasets.
Anthropic also sets out how its bots behave:
- They honor robots.txt "do not crawl" directives.
- They do not try to get past CAPTCHAs or other anti-circumvention measures.
- They support the non-standard Crawl-delay rule, so you can slow ClaudeBot down without blocking it.
- A rule applies to one host, so a site with subdomains needs the rule in each subdomain's robots.txt.
How is ClaudeBot different from Claude-User and Claude-SearchBot?
ClaudeBot is about training; Claude-SearchBot and Claude-User are about being found and read when people ask Claude something. Anthropic warns that turning off either of the last two may reduce your site's visibility in Claude's search answers.
| Bot | Job | If you disallow it |
|---|---|---|
| ClaudeBot | Collects web content that could contribute to model training | Signals your future content should be left out of training |
| Claude-SearchBot | Reads the web to improve the quality and relevance of Claude's search results | Stops indexing for search, which may reduce your visibility and accuracy in Claude's search answers |
| Claude-User | Fetches a page when a person asks Claude a question that needs it | Claude cannot fetch your pages for those questions, which may reduce your visibility in user-directed search |
How do you block or allow ClaudeBot in robots.txt?
Add a group naming ClaudeBot to the robots.txt file at the root of each host. Anthropic's own example blocks ClaudeBot from the whole site with a single Disallow: / line under User-agent: ClaudeBot.
The common choice is to keep content out of training while staying findable in Claude's answers: block ClaudeBot, allow the other two. Crawl-delay is Anthropic's documented way to slow ClaudeBot instead of blocking it. Do not rely on blocking Anthropic's IP addresses instead; Anthropic says that may not work reliably, because it gets in the way of its bots reading your robots.txt. Then confirm the live file with the free AI crawler checker.
# Keep content out of Anthropic's training
User-agent: ClaudeBot
Disallow: /
# Stay findable in Claude's answers
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
# Or, instead of blocking ClaudeBot, slow it down:
# User-agent: ClaudeBot
# Crawl-delay: 1What does this mean for being cited by Claude?
Being cited by Claude depends on Claude-SearchBot and Claude-User reaching your pages; ClaudeBot is a separate decision about training. Blocking all three gives up both.
Claude's citations are spread thin even for a famous brand. In the public Tesla example (Oct 4, 2026), Claude named Tesla in 11 of 25 answers to buyer questions and gave 351 citations across those answers, only one of them to tesla.com. Its most cited site was en.wikipedia.org, 16 times. That is the citation space a search-crawler rule decides whether you can compete for, and it is measured by asking the questions, not by reading robots.txt. See how Claude chooses the brands it names.
Common questions about ClaudeBot
Does blocking ClaudeBot remove my site from Claude?
Not by Anthropic's description. ClaudeBot is the training crawler. Search indexing is Claude-SearchBot and page fetches for a user's question are Claude-User, and Anthropic says blocking those two is what may reduce your visibility in Claude's search answers.
Does ClaudeBot respect robots.txt?
Anthropic says its bots honor robots.txt directives, do not bypass CAPTCHAs, and support Crawl-delay. Put the rule in the robots.txt of every subdomain you want covered.
Should I block ClaudeBot by IP address?
Anthropic advises against relying on it: blocking its IP addresses may not work correctly or persistently, and it stops the bots reading your robots.txt. Use robots.txt for the rule, and Anthropic's published IP list only to confirm a request is genuine.
Is ClaudeBot the same as anthropic-ai?
No. anthropic-ai and Claude-Web are older tokens that some robots.txt files still carry, and Anthropic's current article does not mention them. It names only ClaudeBot, Claude-SearchBot and Claude-User, so write your rules for those three.
Related terms
- GPTBot
GPTBot is OpenAI's web crawler for collecting content that may be used to train OpenAI's generative AI foundation models.
- OAI-SearchBot
OAI-SearchBot is OpenAI's crawler for surfacing websites in ChatGPT's search features, and sites that opt out of it are not shown in ChatGPT search answers.
- PerplexityBot
PerplexityBot is Perplexity's crawler that indexes web pages so they can be surfaced and linked in Perplexity's search results.
Worked example: Tesla, public AI answers, Oct 4, 2026: Proofsource's first scan of Tesla, 25 buyer questions and 5 brand questions, one answer per question on each of four engines. A small sample from one brand, shown as an example, not a benchmark.
See where AI leaves you out, and what to fix.
See which buyer questions name your brand, who gets named instead, and what to fix. 200 answers. 25 Prompts/Question. Top AI engines. 2 days. Free. No card.
Google AI OverviewsGoogle AI Mode
