See which AI crawlers visit, from your own logs
AI crawler log analytics counts the real visits GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and other AI crawlers make to your site, read from your CDN and server logs rather than a JavaScript tag. Proofsource connects to Cloudflare or to CloudFront logs in your S3 bucket, then shows visits per bot, per page and per day. Built for SEO, web and growth teams.
| Bot | Type | Requests | Change |
|---|---|---|---|
| GPTBot | Training crawler | 1,240 | +18% |
| OAI-SearchBot | Search crawler | 912 | +41% |
| ClaudeBot | Training crawler | 610 | +6% |
| PerplexityBot | Search crawler | 402 | +12% |
| ChatGPT-User | On-demand fetch | 388 | +33% |
| Claude-SearchBot | Search crawler | 214 | New |
AI crawler log analytics, in short
The facts, ready to quote.
- Source
- Your CDN and server logs, not a JavaScript tag
- Cloudflare
- Daily AI-bot request counts per zone, refreshed on a schedule
- CloudFront
- Log files read from your S3 bucket through a read-only role you grant
- More routes
- Cloudflare Logpush, hosting log drains such as Vercel and a WordPress plugin are in early access: join and we turn them on for your workspace
- Breakdown
- Visits per bot, per page and per day
- Access
- Read-only; you can disconnect a source at any time
AI crawler log analytics, explained
AI companies send crawlers to read the web. Some collect pages for model training, such as GPTBot and ClaudeBot. Others fetch pages to answer a live question, such as OAI-SearchBot, ChatGPT-User and PerplexityBot. These bots do not run JavaScript analytics tags, so tools like Google Analytics never see them. The only reliable record of their visits is the request log kept by your CDN or web server.
AI crawler log analysis reads those logs, picks out the requests made by AI bots, and counts them by bot, page and day. It answers practical questions: are AI crawlers reaching the site at all, which pages do they read most, did visits drop after a firewall change, and did a new page get crawled after it was published. Crawling is the step before citation, so a page no AI crawler visits is unlikely to be cited.
Proofsource connects to the logs you already have. On Cloudflare it reads daily AI-bot request counts for your zones and refreshes them on a schedule. For CloudFront it reads new log files from your S3 bucket through a read-only role you grant. Cloudflare Logpush, hosting log drains such as Vercel and a WordPress plugin are in early access as further routes: join and we turn them on for your workspace. Every route ends in the same view of AI crawler visits per bot, page and day.
How it works
Counts real visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers using your CDN and server logs.
Connect a log source
Pick Cloudflare or CloudFront logs in S3. Proofsource detects which zones and projects you have.
Grant read-only access
Cloudflare uses a read-only connection; for S3 you grant a read-only role that Proofsource assumes to list and read new log files.
Pick out AI crawlers
Requests from GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and other AI user agents are counted; human and search engine traffic is left out.
Refresh on a schedule
Counts are refreshed automatically, so the chart keeps up without anyone exporting logs.
Read visits per bot and page
See daily visits per AI crawler, the pages each one reads most, and changes after you ship a fix or change a firewall rule.
See it in the product
Screens from the Proofsource app, drawn with sample data for an example brand.
| Page | AI requests | Top bot | Last visit |
|---|---|---|---|
| / | 820 | GPTBot | Today 09:12 |
| /pricing | 544 | OAI-SearchBot | Today 08:40 |
| /compare/otter | 298 | ChatGPT-User | Today 07:55 |
| /blog/fireflies-alternatives | 176 | PerplexityBot | Yesterday |
| /integrations/zoom | 0 | None | Never |
| Source | State | Last read |
|---|---|---|
 Cloudflare zone driffle.ai | Connected | Today 06:00 |
| Cloudflare zone docs.driffle.ai | Connected | Today 06:02 |
| CloudFront logs, S3 bucket driffle-cdn-logs | Connected, read-only role | Today 06:05 |
Every capability, live today
9 shipped capabilities behind AI crawler log analytics, each in the product now.
Cloudflare crawler analytics
Counts real AI-bot requests to your site per day from your Cloudflare data.
Zones and Pages projects
Lists your Cloudflare zones and Pages projects so you choose which ones to read traffic from.
Scheduled crawler refresh
Refreshes your Cloudflare crawler traffic on a schedule, so counts stay current without manual exports.
Log intake endpoint
A public endpoint that Cloudflare Logpush and other CDN log drains post crawler logs to, counted as soon as they arrive.
CloudFront logs from S3
Reads your new CloudFront log files from your own S3 bucket through a read-only role and counts AI crawler hits.
Read-only access role
The role you grant only lets Proofsource read your crawler logs, nothing else in your account.
Crawler analytics connections
Connect, inspect and remove a crawler analytics connection, and choose the zone it reads.
Crawler log sources
Manage each log source, map its fields and send logs in, so different CDNs end up in one view.
Hits per AI crawler
Counts requests per AI crawler, such as GPTBot, ClaudeBot and PerplexityBot, across every connected source.
Who it is for
Confirm AI crawlers reach key pages
Check that GPTBot, OAI-SearchBot and PerplexityBot actually request your pricing, comparison and product pages, and spot the ones they never visit.
Catch a firewall change that blocked bots
A sudden drop in AI crawler visits after a rule change shows up the next day, before it shows up in AI answers.
Decide which AI bots to allow
See which training and search crawlers visit, and how often, before deciding which ones to let in.
Add crawl data to client reports
Connect a client's Cloudflare or S3 logs and show AI crawler activity beside citations and visibility.
Why it matters
See traffic analytics tags miss
AI crawlers do not run JavaScript tags. Logs are the only place their visits are recorded.
Many CDNs, one view
Cloudflare zones and CloudFront logs in S3 feed the same per-bot, per-page chart.
Read-only by design
Proofsource reads logs and counts; it never changes your CDN or server settings from this view.
An early signal
Crawling comes before citation, so crawl changes show up days before answers change.
Google AI OverviewsGoogle AI Mode
AI crawler log analytics: Proofsource and other tools
AI crawler analytics from server logs is offered by several tools. This table lists the log routes each one's public pages describe.
| Capability | Proofsource | Profound | Peec AI | Otterly.AI | Scrunch | Ahrefs | Writesonic |
|---|---|---|---|---|---|---|---|
| AI crawler visits from logs | YesVisits per AI bot, page and day from CDN and server logs | YesWhich AI bots access content, how often, real-time bot monitoringSource | YesBot visit insights via CDN and log providersSource | YesWhich AI crawlers and agents visit your site, from server logsSource | YesReal-time feed of AI bot crawls and traffic trendsSource | YesBot Analytics shows real-time AI crawler visits and top pagesSource | YesAI Bot (Traffic) Analytics documentedSource |
| Cloudflare | YesDaily AI-bot counts per zone on a schedule; Logpush in early access | YesLog forwarding from Cloudflare among other CDNsSource | YesCloudflare among supported log providersSource | YesUpload, webhook, Cloudflare or WordPress pluginSource | Not listed | Not listed | Not listed |
| CloudFront logs in S3 | YesReads new log files from your bucket through a read-only role | YesCDN logs from Cloudflare, Fastly, CloudFront, Akamai sent to Agent AnalyticsSource | YesCloudFront among supported log providersSource | Not listed | Not listed | Not listed | Not listed |
| Hosting log drains | YesDrain URL for hosts such as Vercel, in early access | YesForwarding from AWS, Akamai, Cloudflare, Fastly, GCP, Netlify, VercelSource | Not listed | RelatedWebhook log connectionSource | Not listed | Not listed | Not listed |
| WordPress plugin | YesSends AI crawler visits every five minutes, in early access | Not listed | Not listed | YesWordPress plugin log connectionSource | Not listed | Not listed | Not listed |
| Visits per bot and per page | YesVisits per bot, per page and per day | YesAccess and frequency per AI botSource | YesPage-level AI bot crawling, citations and trafficSource | YesCrawler and agent visits, per botSource | RelatedShows which pages get AI traffic and citationsSource | YesTop pages and visits per AI crawler; free in betaSource | RelatedBot traffic analytics in the API docsSource |
| Counts that refresh themselves | YesCloudflare counts refresh on a schedule, with no manual step | YesReal-time bot monitoringSource | Not listed | Not listed | YesTraffic trends update as bots crawlSource | YesReal-time visits; free in betaSource | Not listed |
| Read-only log access you can revoke | YesA read-only role, and any source can be disconnected at any time | Not listed | Not listed | Not listed | Not listed | Not listed | Not listed |
Where each of them shines
Profound's Agent Analytics covers a very wide range of log sources, verifies genuine bots and shows which pages AI references most. Compare in full
Peec AI ties crawler visits to its My website view, with page-level crawling, citations and traffic together. Compare in full
Otterly.AI offers flexible log connections, from file upload and webhook to Cloudflare and a WordPress plugin. Compare in full
Scrunch pairs a real-time bot feed with crawl error detection and crawl health monitoring. Compare in full
Ahrefs offers Bot Analytics free in beta, next to its first-party Web Analytics. Compare in full
Writesonic documents AI bot traffic analytics next to its content and GEO agents.
Questions about AI crawler log analytics
Why can't Google Analytics show AI crawler visits?
Google Analytics counts visits by running a JavaScript tag in the browser. AI crawlers such as GPTBot and ClaudeBot request the page but do not run that script, so they never appear. Your CDN and server logs record every request, which is why Proofsource reads those.
Which AI crawlers does Proofsource count?
It counts requests from AI crawlers and on-demand fetchers by user agent, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot and PerplexityBot. Spoofed bots can be separated out with AI bot verification.
Which log sources can I connect?
Cloudflare, and CloudFront logs stored in your S3 bucket, are live today. Cloudflare Logpush, hosting log drains such as Vercel and a WordPress plugin are in early access: join and we turn them on for your workspace.
What access does Proofsource need to my S3 bucket?
You grant a read-only role that lets Proofsource list and read the crawler log files in the bucket you name. It cannot write, delete or read anything else in your account, and you can remove the role at any time.
How often are the numbers updated?
Cloudflare counts are refreshed on a schedule, and S3 logs are read as new files arrive.
Does crawling mean my page will be cited?
No, but it comes first. An AI search crawler has to fetch a page before an answer can cite it. Pages no AI crawler visits are worth checking for blocks, missing links or a missing sitemap entry.
Does it work with ChatGPT, Perplexity, Claude and Google AI Overviews?
Crawler analytics counts the bots behind ChatGPT, Perplexity and Claude directly. Google uses its regular Googlebot for AI Overviews, so those visits show in search crawl data rather than as a separate AI bot.
How is this different from Profound's Agent Analytics?
Profound's Agent Analytics is a strong, wide log-based product. Proofsource puts crawler visits in the same place as the answers, citations and fixes for your buyer questions, so a crawl change can be read next to its effect on what the engines say.
Keep exploring
Try it on your own brand.
Proofsource is live: scan your brand and see AI crawler log analytics at work with your own answers, competitors and sources.
Google AI OverviewsGoogle AI Mode