What is AI answer sampling?
AI answer sampling is asking the same questions on each AI engine more than once and reading every visibility figure as an estimate from that sample, not a fixed fact. Ask an AI engine the same question twice and you can get two different shortlists. So no single answer is a measurement. Sampling means asking a fixed set of questions repeatedly, on each engine, and treating the share of answers that name you as an estimate with a range.
Why do AI answers have to be sampled?
Because the engines are not deterministic. OpenAI's documentation states that Chat Completions are non-deterministic by default, so outputs may differ from request to request (OpenAI API docs, read Oct 15, 2026).
Even at temperature zero the same prompt drifts. Thinking Machines Lab sampled 1,000 completions of one prompt from an open model at temperature zero and got 80 different completions; it traced the main cause to server load changing the batch size each request lands in (September 10, 2025, Thinking Machines). Part of what one answer says is how busy the server was.
How many answers are enough?
Enough that the interval is narrower than the change you want to detect. If a model truly names you 60% of the time, here is how wide the 95% Wilson interval stays at each sample size (the arithmetic):
| Answers | Named | 95% interval | Width |
|---|---|---|---|
| 5 | 3 | 23.1% to 88.2% | 65 points |
| 20 | 12 | 38.7% to 78.1% | 39 points |
| 40 | 24 | 44.6% to 73.7% | 29 points |
| 100 | 60 | 50.2% to 69.1% | 19 points |
How do you sample AI answers fairly?
Fix the method before you look at results, and keep it the same every period:
- The same questions, in the same words, on every engine, so a gap between engines is a real difference.
- Questions that do not name you, with brand questions kept in their own set.
- Runs spread across days, not one burst, because answers drift with load and model updates.
- Word-for-word repeats of an earlier answer dropped: a replay is not a second draw.
- No-answer cases, such as no AI Overview shown, left out rather than counted as misses.
What does sampling look like in a real example?
The public Tesla example (Oct 4, 2026) asked 25 buyer questions once on each of four engines: 100 answers overall, but only 25 per engine. That is why ChatGPT's 64% carries a range of about 45% to 80%.
Repeating questions changes the picture. Over three scan days (October 4, 8 and 15), five phrasings of one Tesla grid storage question gave 12 fresh answers each: two phrasings named Tesla 12 of 12 times, one 7 of 12, and two never (the post). One run of one phrasing could have reported any of those.
How does Proofsource sample?
A Proofsource plan runs daily or weekly. On a daily plan each tracked question is asked once on each engine per UTC day, and every rate carries a Wilson interval, so a move has to clear the noise before it counts. See how often scans run and Prompts.
Common questions about AI answer sampling
Is one ChatGPT screenshot enough to know if AI recommends us?
No. One answer that names you is consistent with a true mention rate as low as about 21%, and one miss with a rate as high as about 79%. A screenshot shows that something can happen, not how often it does.
Should I ask the same question several times in a row?
Spread the repeats across days instead. Back-to-back requests can be cached or replayed, and a word-for-word replay is not a new sample.
Does a seed or temperature zero make answers repeatable?
Only partly. OpenAI describes seeded outputs as mostly deterministic, and the consumer apps do not expose either setting. Treat every engine as a source of samples.
Daily or weekly sampling?
Pick by the decision. Weekly suits a trend on a large, fixed question set. Daily helps when the set is small or you shipped a change and want to see which day the answers moved and whether it held.
Related terms
- Wilson interval
A Wilson interval is the range a measured percentage could plausibly sit in, given how many answers it is based on.
- AI answer volatility
AI answer volatility is how much the brands an AI engine names change when the same question is asked again, or asked in different words.
- Mention rate
Mention rate is the share of AI answers to a question that name your brand.
- Prompt
In AI visibility, a prompt is a question tracked across AI engines, written the way a buyer would ask it.
Worked example: Tesla, public AI answers, Oct 4, 2026: Proofsource's first scan of Tesla, 25 buyer questions and 5 brand questions, one answer per question on each of four engines. A small sample from one brand, shown as an example, not a benchmark.
See where AI leaves you out, and what to fix.
See which buyer questions name your brand, who gets named instead, and what to fix. 200 answers. 25 Prompts/Question. Top AI engines. 2 days. Free. No card.
Google AI OverviewsGoogle AI Mode
