Why two AI visibility tools give you two different numbers

You trial two AI visibility tools in the same week. Same brand, same competitors, roughly the same questions. One says your ChatGPT visibility is 34%. The other says 11%.

So you ask both. Both answer immediately, both sound like they have done this before, and neither answer resembles the other. One talks about prompt coverage. One talks about sampling depth. Somebody mentions that the other tool “scrapes”.

The instinct at this point is to work out which one is telling the truth. That instinct is the problem. Both numbers can be produced honestly, from working software, by people who are not trying to mislead you. They disagree because they are not measurements of the same thing, and there is no third number sitting underneath that either of them is approximating.

This is not a story about vendors being sloppy. It is a story about what happens when a question shaped for search engines gets pointed at something that is not one.

Google had a results page. This has a sample.

Google rank was never the fixed thing people remember. It moved with location, device, personalisation and time, and Search Console reported an average position rather than a standing. But underneath all that wobble there was still an ordered results page. A thing existed. You could screenshot it, two people could argue about it, and the argument could in principle be settled.

An answer engine hands you nothing of the kind. It rewrites your question, decides whether to search, picks what to read, and writes prose over the top. What comes back is one sample from a distribution, not a position in a list.

Microsoft, of all people, makes this point in its own product. The AI Performance report in Bing Webmaster Tools reports citation counts from Microsoft’s own AI surfaces, which is about as close to first-party data as this field gets, and the documentation attaches a warning to it: the report “does not indicate ranking, authority, importance, or a page’s role within an individual AI answer.” Microsoft has the data and still declines to call it a rank. It is worth asking why a third-party tracker, working from the outside, would be more confident.

So when two tools disagree, the disagreement is not error bars around a hidden true value. It is two different experiments pointed at a moving target, and the gap between them is mostly explained by five choices.

1. The answer moves when nothing else does

Start with the floor. Two tools could make every other decision identically and still disagree, because the engines do not repeat themselves.

SparkToro ran the cleanest public version of this test in January 2026. 600 volunteers put 12 prompts through ChatGPT, Claude and Google’s AI, 60 to 100 times each, for 2,961 responses in total. The result: asked a hundred times, ChatGPT or Google’s AI will give you the same list of brands twice fewer than one time in a hundred. The same list in the same order is closer to one in a thousand. Not the same wording, the same list.

An academic group at St. Gallen measured the same instability from the source side, and controlled it properly. Their paper, Don’t Measure Once, compares repeated runs of identical prompts across four engines. Between consecutive days, the sets of cited sources overlapped by only 34% to 42%. Roughly two thirds of an answer’s sources change overnight. Brand sets held up better, at 45% to 59%, but not much better.

The part that matters most is what they did next. They re-ran identical prompts simultaneously, within minutes of each other, so that nothing about the web, the index or the news cycle could have moved. Source overlap came out at 32% to 43%: the same range as the day-apart comparison. The variation is not the world changing underneath the measurement. It is the answer being written fresh every time.

They also found that how noisy a prompt is depends on the prompt. Some prompts came back with overlap above 80% run after run. Others sat below 20%. So the amount of disagreement two tools will show you is partly a property of the questions each one happened to pick, which brings us to the next thing.

2. They are not talking to the same machine

“We track ChatGPT” describes at least four different experiments.

A tool can drive a browser against the chat interface, or it can call the documented API. Those are different systems. The consumer product carries hidden instructions, its own choice of which model to answer with, interface features, whatever else is in that conversation, and, if the account has memory switched on, whatever it has learned about that user. The API carries none of that unless you put it there yourself.

Martin Kulawik laid the trade-off out more clearly than most vendors will. Browser automation gets you closer to what a person actually sees, at the cost of session state, interface changes and bot detection you do not control. An API gets you a documented measurement you can record and repeat, at the cost of not reproducing any real user’s experience. His framing is the right one: not “which is more real”, but which of those two systems am I trying to make claims about.

Underneath that sits a bigger fork: did the model search at all? A prompt answered without searching measures what the model already believes about your category. A prompt answered after searching measures what the web currently offers on the query the engine invented. These are two different systems on two different clocks, and averaging them together produces a number that describes neither.

That fork is not always the tool’s to make. Kulawik notes that OpenAI and Anthropic let you require a web search, and Perplexity’s Sonar always searches, but Gemini decides for itself whether searching would improve the answer. A tool measuring Gemini is therefore reporting a blend of searched and unsearched answers unless it explicitly filters, and most do not tell you which.

3. “Visibility” is not one metric

Even given identical answers, two tools can score them differently, because the word is doing different jobs.

The most defensible definition is a rate: the share of tracked answers that mention you at least once. That is what we use, and it has the virtue of a denominator you can state out loud. Plenty of tools do something else. Some publish a composite scored out of 100, blending mention rate, citation rate, source authority and consistency under a set of weights chosen by the vendor. Some count raw mentions.

Raw counts break in a specific and avoidable way. In Kulawik’s data, average response length between 1 July and 8 August 2026 ran to 3,363 characters for Gemini against 1,462 for Claude: Gemini’s answers were about 2.3 times longer. A longer answer has more room to name brands. Add up absolute mentions across engines and you have measured verbosity as much as visibility. His is a small sample, six brands, and he says so, but the mechanism does not need a big sample to be real.

Then there is the retrieved and cited distinction. A page can be fetched and used without ever appearing as a footnote, and a tool that counts only visible citations will systematically undercount influence compared with one that counts retrieval. Neither is wrong. They are different columns.

4. The prompt sets are different

Most tools will generate a starting prompt set for you. Those generators do not agree, so before anything else happens, the two tools are asking a different universe of questions.

That decides more of the final number than anything downstream. A set weighted towards category questions (“best CRM for agencies”) will show a lower visibility than one weighted towards comparison questions naming you, and both are legitimate views of a business. Language and country do the same thing again: the engine’s own fan-out queries can leave the language you asked in, and where the tool places itself geographically changes what comes back.

If you are comparing two dashboards, the prompt list is the first thing to compare, and it usually explains more of the gap than the methodology argument you were having.

5. The measurement changed, not the visibility

The last one is the quietest, and it corrupts history rather than the current reading.

On 6 August 2026, Kulawik tightened his inclusion rule for Gemini so that only responses with a verifiable web search counted as visibility measurements. The share of Gemini responses carrying source URLs went from 84.9% to 100%. Gemini did not improve. The definition moved.

He published that, which is the point. Every tool in this category is constantly adjusting how it reads answers, how it matches brand names, which engines it covers and what counts as a measurement at all, because the engines keep changing underneath them. A vendor who ships those changes silently hands you a chart where your own performance and their release history are mixed together and cannot be separated afterwards.

The question to ask instead

“Which tool is right” has no answer. These do:

1. API or browser, and for which engines? Then: is that consistent across the engines I care about, or mixed?
2. How many samples per prompt, per day, and is a single day’s figure reported as one number or as a range?
3. What exactly is the denominator of the headline score? If the answer is a composite out of 100, ask for the weights.
4. Are answers without a web search included, excluded, or blended in?
5. When you change how you measure, where is that written down, and do the past numbers get recalculated to match?

A vendor who answers all five plainly is more useful than one whose number happens to be higher. The number you cannot interrogate is the one to distrust, and that holds for ours.

What survives all this

None of the above means the measurement is fake. It means a single answer is the wrong unit to read.

Rand Fishkin went into the SparkToro research expecting to conclude that tracking was useless, and says he came out the other side having changed his mind. The ordering is noise. The aggregate is not. In his data, City of Hope appeared in 69 of 71 ChatGPT answers about West Coast cancer hospitals, a 97% visibility rate, while topping the list in only 25 of them. The rank was a lottery. The presence was a fact about the model.

The St. Gallen team put numbers on where that fact becomes trustworthy. Repeat the same prompt seven times and how often a brand appears is pinned down to roughly plus or minus 16 points; at eight runs it is plus or minus 12, and the curve flattens from there. Sample once a day instead and you reach about plus or minus 10 points after three weeks, and plus or minus 7 after four. Those are two separate analyses in the paper rather than a like-for-like race, but the lesson is the same either way: precision comes from accumulating observations, and accumulating them daily also shows you drift, which a same-day burst cannot.

That has a consequence we should state about our own product rather than leave implied. Voxoria captures one answer per prompt, per engine, per day. A single day’s visibility figure for a single prompt is a coin flip and should not be read as anything else. What the design is built for is the trend line: enough prompts, observed long enough, that a real movement separates from the noise. If you need a defensible read on one prompt this afternoon, daily sampling is the wrong instrument, and we would rather say so than sell you a screenshot.

Which is the honest version of the answer to the 34% and the 11%. Both tools showed you a sample. The useful question was never which sample was correct. It was whether either of them was still going to mean the same thing in three months.