AI answer drift tracking, so you know when a change is real
AI engines give different answers to the same question from one run to the next. Proofsource asks questions several times per engine to measure that drift, runs control questions to set a noise floor, and puts a 95 percent confidence range and a one-line method under every number. Marketers and agencies can tell a real gain from normal answer noise before they report it.
| Question | ChatGPT | Perplexity | Claude |
|---|---|---|---|
| best ai meeting notes app | 4 of 5 | 2 of 5 | 5 of 5 |
| otter alternatives | 1 of 5 | 3 of 5 | 2 of 5 |
| meeting recorder with crm sync | 5 of 5 | 4 of 5 | 5 of 5 |
Answers moved most on Perplexity this week.
Answer drift and confidence ranges, in short
The facts, ready to quote.
- Repeat runs
- Each question asked several times per engine to measure drift
- Stability set
- Fixed question and engine pairs re-sampled to check consistency
- Noise floor
- Control questions measure normal noise; smaller changes are not called wins
- Confidence range
- Visibility = mentioned over answered, with a 95 percent Wilson interval
- Answered rule
- Empty or refused runs never count in a score's denominator
- Method line
- Questions, engines, reading method, runs and dates under every number and report
AI answer drift, explained
AI answer drift is how much an AI engine's answer to the same question changes between runs. Ask ChatGPT "best AI meeting notes app" ten times and you may see a brand named seven times, not ten. Drift comes from sampling randomness, live web results and model updates. It is normal, but it means a single run, or a small sample, can make your visibility look higher or lower than it really is.
That is why AI visibility numbers need error bars. A move from 30 percent to 34 percent can be noise. A move from 30 to 45 percent on a large sample probably is not. Without a measure of drift and a confidence range, teams celebrate random swings and panic over dips that mean nothing.
Proofsource asks questions several times per engine to measure drift, keeps a fixed stability set it re-samples, and runs negative control questions to set a noise floor. Visibility is mentioned answers over answered runs, counted by one answered rule, with a 95 percent Wilson interval. Every number and report carries a method line naming the questions, engines, how each answer was read, the runs and the dates.
How it works
Repeat runs measure how much AI answers move, a noise floor stops you over-reading changes, and every score carries a 95 percent range.
Ask more than once
Each question runs several times per engine, not once.
Run control questions
Negative controls measure normal answer noise and set a floor.
Count only real answers
One rule decides whether a run answered; empty or refused runs are left out.
Compute the range
Visibility gets a 95 percent Wilson interval from mentioned over answered.
Print the method
A method line under each number says what was asked, where, how and when.
See it in the product
Screens from the Proofsource app, drawn with sample data for an example brand.
Buyer questions. 412 answered runs from ChatGPT, Perplexity, Claude and Google AI Overviews, Oct 20 to 26.
| Question | Engine | Agreement |
|---|---|---|
| best ai meeting notes app | ChatGPT | 92% |
| fireflies vs fathom | Perplexity | 71% |
| ai notetaker for zoom | Claude | 88% |
Every capability, live today
9 shipped capabilities behind answer drift and confidence ranges, each in the product now.
Repeat runs and answer drift
Each question is asked several times per engine, and Proofsource measures how far the answers moved.
Stability question set
Enroll fixed question and engine pairs that are re-sampled to check how consistent each engine is.
Negative controls and noise floor
Control questions measure normal answer noise, so changes smaller than the noise floor are not called wins.
Visibility with a 95 percent range
Visibility is mentioned answers over answered runs, shown with a 95 percent Wilson confidence range.
Answered rule
One rule decides whether a finished run actually answered, and it is used in every score's denominator.
Method line under numbers
One line under each number explains how it was measured.
Method line on reports
Every report and pitch prints the questions, engines, how each answer was read, the runs and the dates.
Methodology and 95% interval
Read how answers are measured and a plain explainer of the 95 percent interval behind every number.
Daily measurement trust checks
Stability and noise floor are computed daily so you can tell real movement from random variation.
Who it is for
Report only real wins
Show clients changes that clear the noise floor, with the range beside them.
Read trends with error bars
Use confidence ranges to decide whether a dip needs action.
Defend the number upstairs
Every figure carries its method, so finance and leadership can check it.
Why it matters
No false wins
The noise floor stops random swings being reported as progress.
Honest uncertainty
A 95 percent range shows how sure each number is.
Clean denominators
Empty or refused runs never dilute a score.
Checkable method
Every number says what was asked, where, how and when.
Google AI OverviewsGoogle AI Mode
Answer drift and confidence ranges: Proofsource and other tools
Every AI visibility tool reports a visibility number. Here is what each publicly says about the number and how it is checked.
| Capability | Proofsource | Profound | Semrush | SE Ranking | Peec AI |
|---|---|---|---|---|---|
| Visibility score | YesMentioned over answered, with a 95 percent range | YesHow often brand appears in AI answers, by platformSource | Yes0 to 100 AI Visibility Score for a domainSource | YesVisibility compared across each AI engineSource | YesVisibility compared to prior 7 daysSource |
| Repeat runs, confidence range and noise floor | YesRepeat runs measure drift; 95 percent Wilson interval and negative controls | Not listed | Not listed | Not listed | Not listed |
| Number traceable to answers | YesMethod line plus saved answers behind each number | Not listed | Not listed | YesAny number opens the unedited AI answer behind itSource | Not listed |
| Trends over time | YesDaily stability and noise-floor checks separate real movement from random variance | RelatedDrag a date range or ask AI Marketer why a metric movedSource | RelatedSentiment trends over timeSource | YesChange versus previous period, sorted by biggest moversSource | YesDate filters such as last 7 days, trend chartsSource |
| Results split by engine | YesVisibility and drift measured per engine | YesHow often brand appears in AI answers, compared by platformSource | YesShows which AI models mention the brand mostSource | YesEngine-by-engine view of visibilitySource | Not listed |
| Method shown under each number | YesQuestions, engines, reading method, runs and dates printed under every number | Not listed | Not listed | RelatedReal answers captured through browser interface as a user sees them, not API samplingSource | Not listed |
Where each of them shines
Visibility, share of voice and sentiment compared by platform at enterprise scale. Compare in full
A simple 0 to 100 AI Visibility Score with monthly audience trends. Compare in full
Drill-down from any number to the unedited answer, refreshed daily.
Week-over-week comparisons of visibility, sentiment and share of voice. Compare in full
Questions about answer drift and confidence ranges
Why do AI answers change between runs?
Engines sample their wording, search live web results and update models often. The same question can name a brand in some runs and not others, which is called answer drift.
How does Proofsource measure answer drift?
It asks questions several times per engine and measures how far the answers moved, and it re-samples a fixed stability set of question and engine pairs.
What is a noise floor?
Proofsource runs control questions to measure how much answers move when nothing has changed. A change smaller than that floor is not called a win.
What is the 95 percent range on my score?
It is a Wilson confidence interval around your visibility, which is mentioned answers over answered runs. It shows the range your true visibility very likely sits in.
What counts as an answered run?
One rule decides it. Empty or refused runs are left out of every score's denominator so they never drag a number down.
What is the method line?
A short line under each number, and on every report and pitch, naming the questions, engines, how each answer was read, the runs and the dates.
Does this apply to every AI engine Proofsource tracks?
Yes. Drift, ranges and method lines apply to every engine your plan covers, up to 10 AI engines.
How is this different from a visibility score in other tools?
Profound, Semrush, SE Ranking and Peec AI all report useful visibility numbers. Proofsource adds repeat runs, a noise floor and a 95 percent range so you can tell real change from drift.
Keep exploring
Try it on your own brand.
Proofsource is live: scan your brand and see answer drift and confidence ranges at work with your own answers, competitors and sources.
Google AI OverviewsGoogle AI Mode