What is Claude-SearchBot?
Claude-SearchBot is Anthropic's crawler that reads the web to improve the quality and relevance of Claude's search results. Claude-SearchBot is the crawler behind Claude's search: it reads the web ahead of time so Claude has good results to draw on. Blocking it is a search decision, not a training one.
What does Claude-SearchBot do?
Anthropic says Claude-SearchBot navigates the web to improve search result quality for users, analyzing online content to improve the relevance and accuracy of search responses. Disabling it prevents Anthropic's system from indexing your content for search optimization, which Anthropic says may reduce your site's visibility and accuracy in user search results (Anthropic, April 7, 2026).
How does Claude-SearchBot behave?
Anthropic documents the same rules for all three of its bots:
- They honor robots.txt directives.
- They do not try to bypass CAPTCHAs or other anti-circumvention measures.
- They support the non-standard Crawl-delay rule, to slow a bot without blocking it.
- A rule applies per host, so each subdomain needs its own.
- Blocking by IP address may not work reliably, because it can stop the bots reading robots.txt.
How do you allow Claude-SearchBot while blocking training?
Give each bot its own group. This is the common choice for brands that want to be cited by Claude without contributing to training data. Confirm the live file with the free AI crawler checker.
# Stay findable in Claude's search
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
# Keep content out of Anthropic's training
User-agent: ClaudeBot
Disallow: /What is at stake, in a real example?
In the public Tesla example (Oct 4, 2026), Claude cited 351 pages across 25 buyer questions, more than ChatGPT and Google AI Overviews combined, and named Tesla in 11 answers.
A long source list is a large pool of citation slots, and Claude-SearchBot decides whether your pages are indexed for it. tesla.com got one of those 351; most went to reference and third-party sites.
How do you see Claude-SearchBot on your site?
Proofsource counts AI crawler requests from your Cloudflare zone or access logs every day, verified against Anthropic's published IP list, and shows which crawled pages end up cited in Claude's answers. See AI Traffic and Crawler access.
Common questions about Claude-SearchBot
Is Claude-SearchBot used for training?
Anthropic describes it as indexing content to improve search results. Training is ClaudeBot's job, and each bot has its own robots.txt rule.
What happens if I block Claude-SearchBot?
Anthropic says your content stops being indexed for search optimization, which may reduce your visibility and accuracy in Claude's search answers.
Can I slow Claude-SearchBot down instead of blocking it?
Yes. Anthropic supports the non-standard Crawl-delay rule for its bots.
How do I know a visit is really Claude-SearchBot?
Check the source IP against the list Anthropic publishes at claude.com/crawling/bots.json; Anthropic says an IP on that list is coming from Anthropic.
Related terms
- ClaudeBot
ClaudeBot is Anthropic's web crawler that collects web content that could contribute to training Anthropic's Claude models.
- Claude-User
Claude-User is the agent Anthropic uses to fetch a web page when a person asks Claude a question that needs it.
- robots.txt
robots.txt is a plain-text file at the root of a website that tells crawlers which paths they may fetch, under the Robots Exclusion Protocol (RFC 9309).
- PerplexityBot
PerplexityBot is Perplexity's crawler that indexes web pages so they can be surfaced and linked in Perplexity's search results.
Worked example: Tesla, public AI answers, Oct 4, 2026: Proofsource's first scan of Tesla, 25 buyer questions and 5 brand questions, one answer per question on each of four engines. A small sample from one brand, shown as an example, not a benchmark.
See where AI leaves you out, and what to fix.
See which buyer questions name your brand, who gets named instead, and what to fix. 200 answers. 25 Prompts/Question. Top AI engines. 2 days. Free. No card.
Google AI OverviewsGoogle AI Mode
