What is CCBot?
CCBot is the web crawler of Common Crawl, a nonprofit that publishes a free, open copy of the web that researchers and AI developers use as training data. CCBot is not an AI company's crawler, but its archive feeds many AI models. Common Crawl publishes its copy of the web for free, so blocking CCBot is a decision about open datasets that researchers and AI developers train on.
What is Common Crawl, and what does CCBot do?
Common Crawl describes itself as a 501(c)(3) non-profit dedicated to providing a copy of the internet to researchers, companies and individuals at no cost; CCBot is the crawler that collects it (Common Crawl FAQ, read Oct 15, 2026).
Its data reaches AI models through the developers who use it. OpenAI's GPT-3 paper, for example, lists a filtered version of Common Crawl as the largest part of that model's training mix (Language Models are Few-Shot Learners, May 28, 2020).
How does CCBot behave?
Common Crawl documents its crawler's rules:
- It checks robots.txt and fetches a page only if crawling it is allowed.
- It obeys Crawl-delay; Crawl-delay: 2 limits it to one request every 2 seconds.
- It honors nofollow on links for link discovery.
- It rechecks robots.txt periodically, so a new block takes effect on later visits.
- It runs on dedicated IP ranges with reverse DNS, published as JSON for verification.
How do you block or verify CCBot?
Add a group for CCBot to robots.txt. Its user-agent string is CCBot/2.0 (https://commoncrawl.org/faq/), and Common Crawl gives a reverse DNS hostname under crawl.commoncrawl.org for verifying a request (CCBot).
# Keep content out of Common Crawl's open archive
User-agent: CCBot
Disallow: /What does blocking CCBot change, in a real example?
It changes future open datasets, not today's answers. In the public Tesla example (Oct 4, 2026), Claude answered "What is Tesla?" and "Is Tesla legit?" from what the model already knew, with no sources cited: that memory came from training data collected long before.
On buyer questions the same engines cited 455 different websites, read at answer time by their own search crawlers and fetchers. A CCBot rule touches neither of those live paths; the search and user agents do.
How do you check your CCBot rule?
Read your live robots.txt on every host for a CCBot group, and decide it alongside the AI training crawlers. The free AI crawler checker reads your live robots.txt, and Crawler access in Proofsource tracks it over time.
Common questions about CCBot
Does blocking CCBot remove my pages from AI answers?
No. AI answers that search use the engines' own crawlers and fetchers. CCBot only affects Common Crawl's future archive.
Does blocking CCBot remove my content from existing datasets?
Common Crawl's documentation describes how CCBot stops crawling a blocked site. It does not say past archives are changed, so assume what was collected before stays.
Is CCBot an AI company's crawler?
No. Common Crawl is a non-profit that publishes its archive for anyone. AI developers are among the many users of that data.
Does CCBot respect Crawl-delay?
Yes. Common Crawl says it obeys the Crawl-delay parameter in robots.txt.
Related terms
- GPTBot
GPTBot is OpenAI's web crawler for collecting content that may be used to train OpenAI's generative AI foundation models.
- ClaudeBot
ClaudeBot is Anthropic's web crawler that collects web content that could contribute to training Anthropic's Claude models.
- Applebot-Extended
Applebot-Extended is a robots.txt token that lets a site opt out of Apple using its content to train Apple's foundation models, and it does not crawl anything itself.
- robots.txt
robots.txt is a plain-text file at the root of a website that tells crawlers which paths they may fetch, under the Robots Exclusion Protocol (RFC 9309).
Worked example: Tesla, public AI answers, Oct 4, 2026: Proofsource's first scan of Tesla, 25 buyer questions and 5 brand questions, one answer per question on each of four engines. A small sample from one brand, shown as an example, not a benchmark.
See where AI leaves you out, and what to fix.
See which buyer questions name your brand, who gets named instead, and what to fix. 200 answers. 25 Prompts/Question. Top AI engines. 2 days. Free. No card.
Google AI OverviewsGoogle AI Mode
