AI Answer Consistency: What 968 Repeated Prompt Runs Show

AI answers to the same prompt change often, and the sources they cite change far more than the brands they mention. Across 968 repeated runs of 42 prompt-engine pairs tracked in AI Rank Lab between February 13 and October 14, 2026, 18 of the 42 pairs (42.9%) switched between mentioning and not mentioning the tracked brand at least once, and two consecutive runs of the same prompt had a median cited-domain overlap of just 0.41 (Jaccard). In plain terms: a single run is a sample, not a measurement.

This is a first-party data study. Below you get the numbers by engine, the method and its limits, how our results line up with published work from Profound, Otterly, SparkToro and researchers at the University of St. Gallen, and a practical answer to the two questions every AI visibility report runs into: how often should you track, and how many runs do you need before you trust a change?

What did we measure, and how?

We measured how much the answer to the same prompt, on the same engine, changes from one run to the next. The data comes from scheduled and manual prompt runs in AI Rank Lab's brand visibility tracker, which runs a user's prompts on AI engines and records whether the tracked brand was mentioned and which domains the answer cited. Disclosure: AI Rank Lab is our product, and this data was collected by it.

Four metrics, each computed per pair and then summarized as a median (or a count) across pairs:

• Mention flip: the pair's runs did not all agree on whether the brand was mentioned. Pairs that never flipped are either "always mentioned" or "never mentioned".
• Consecutive-run Jaccard: for each pair of back-to-back runs, the number of cited domains they share divided by the number of distinct domains either one cited. 1.0 means identical source lists; 0 means no overlap. We average this within a pair, then take the median across pairs.
• Domains present in every run: the share of all domains a pair ever cited that appeared in every single one of its runs.
• Distinct domains across runs vs per run: how many distinct domains the pair cited over its whole history, divided by how many a single run cited on average.

Runs go through each engine's API with web search or search grounding switched on, not through the consumer chat apps, and the scoring instruction names the tracked brand so the answer can be scored. Both choices matter for interpretation, and we return to them in the limitations section.

How often do AI answers change for the same prompt?

Often enough that one run tells you little. Across all 42 pairs, 18 (42.9%) flipped brand mention at least once, 19 were mentioned in every run and 5 were never mentioned. On the source side, two back-to-back runs of the same prompt shared a median of 41% of the domains either run cited, so roughly three in five domains in any two consecutive answers appeared in only one of them. That is the core finding on AI answer consistency in our sample: the brand verdict is unstable for a large minority of prompts, and the source list is unstable for almost all of them.

Headline findings by engine

*One pair with five runs is an anecdote, not a finding. We show it for completeness only. The all-engine Jaccard (0.41) sits below three of the four main engines because it is the median of all 42 pairs pooled, and Gemini's 13 low-overlap pairs pull it down.

Which AI engine gives the most consistent answers?

It depends on which kind of consistency you mean, and in our data the two kinds pointed in opposite directions.

• Most consistent sources: Perplexity. Its back-to-back runs shared a median 0.59 of cited domains, the highest of any engine. One caveat: Perplexity pairs were run more densely, about 37 runs each over a median 12.8 days, versus roughly 20 to 22 runs over a median 25.3 days for the other engines. Runs closer together in time have less chance to drift, which may flatter its number.
• Most consistent brand verdict: Gemini. Only 3 of its 13 pairs (23.1%) ever flipped, and 10 were mentioned in every run. We think this mostly reflects the prompt mix that happened to be tracked on Gemini rather than a property of Gemini, and our sample cannot separate the two.
• Least consistent sources: Gemini. A median Jaccard of 0.26, only 2.1% of cited domains present in every run, and 3.5 times as many distinct domains over time as in a typical run. Gemini kept naming the brand while swapping the pages it pointed to.
• Highest share of "permanent" sources: Claude. A median 10% of the domains Claude cited for a pair showed up in every run, the highest of the four main engines.
• ChatGPT sat in the middle on source overlap (0.54) and had the most pairs that never mentioned the brand (4 of 11).

Are brand mentions more stable than cited sources?

Yes. In our data, 24 of 42 pairs (19 always mentioned plus 5 never mentioned) gave the same brand verdict on every run, while only a median 3.5% of cited domains survived every run. Published work agrees: Otterly's stability study found 12% to 19% of brands appeared on only one day of a 30-day window, against up to 67% of cited sources, and the St. Gallen paper measured day-to-day brand Jaccard of 0.45 to 0.59 against 0.34 to 0.42 for sources. If you report one number, a brand mention rate over a window is steadier than any "which pages got cited" snapshot. Our explainer on brand mentions vs citations covers why the two need separate KPIs.

Why do AI engines give different answers to the same question?

Because an AI answer is the end of a chain of steps, and almost every step has some randomness or some moving input. The table below lists the main causes and whether a tracking setup can control them.

Sampling. Language models choose each next token from a probability distribution, so the same prompt can produce different text. Even setting temperature to zero does not make hosted models fully repeatable: Thinking Machines Lab ran the same prompt 1,000 times at temperature 0 on Qwen3-235B and got 80 unique completions, which it traced to inference kernels whose results depend on batch size and therefore on server load.

Retrieval. Search-enabled answers depend on what the engine fetches. Google's own documentation on AI features says AI Overviews and AI Mode may use "query fan-out", issuing multiple related searches across subtopics, and that the two features "may use different models and techniques, so the set of responses and links they show will vary." Small differences in which sub-queries run, and what they return, change the citation list. Our query fan-out explainer covers the mechanism.

Real change over time. Some movement is genuine: new pages get published, rankings shift, and models get updated. Profound's volatility study found that 40% to 60% of cited domains changed between June and July 2025 for the same prompts, and its later citation half-life study found the median cited page loses half its peak citation share within 11 days of peaking. Our window spans eight months and model versions were not pinned, so some of our churn is this kind of real drift, not noise.

Different surfaces. Even within one company, products differ. Ahrefs compared 540,000 query pairs and found AI Mode and AI Overviews cited the same URLs only 13.7% of the time, even though their answers were semantically similar 86% of the time.

How do these results compare with Profound, Otterly, SparkToro and St. Gallen?

Broadly, they agree: source lists churn heavily, brand presence is steadier, and single snapshots mislead. The table puts the published numbers side by side. Note that the metrics differ, so compare directions, not decimals.

Every study in this table except the St. Gallen paper comes from a company that sells AI visibility tracking, including ours. That is worth keeping in mind when any of them, including us, concludes that you need more tracking.

Where we agree. Our overall source overlap of 0.41 sits inside the St. Gallen range (0.34 to 0.42 day to day). Perplexity being the most source-stable engine matches Otterly (about 75% day-to-day reuse) and Profound (lowest monthly drift at 40.5%). Brand presence being steadier than sources matches Otterly, St. Gallen and SparkToro.

Where we disagree. In the St. Gallen within-24-hour repeats, Gemini was the most source-stable engine (Jaccard 0.505) and Perplexity among the least (0.282), the opposite of our ranking. Their runs were in German, from Swiss servers, over a few days; ours are mostly English-language marketing and software prompts spread across weeks. Engine rankings for stability are not portable across markets and setups, which is a reason to build your own baseline per engine rather than borrow anyone's, ours included.

What about Peec AI? As of October 18, 2026 we did not find a published repeat-run stability figure from Peec AI. Its pricing page describes tracking coverage as tracked prompts multiplied by active models and tracking frequency, and its visibility overview as tracked daily, which implies the same window-based approach.

Is a single AI visibility snapshot reliable?

No. In our sample, 43% of prompt-engine pairs would have given a different brand verdict depending on which run you happened to look at, and a typical single run showed well under half of the domains that pair cited over time (the distinct-domain ratio of 2.68x means one run captured roughly 37% of the pair's eventual source pool). SparkToro's volunteers found ChatGPT and Google's AI returned the same brand list less than once in 100 runs. A screenshot proves an answer happened once. It does not tell you your visibility.

What should AI visibility reports show instead of snapshots?

Rates over windows. Replace "were we mentioned?" with "in what share of runs were we mentioned over the last N days, out of how many runs?" Here is the reporting setup we recommend, and the one this data came from:

1. Fix the prompt set. Keep a stable list of prompts per engine so the denominator does not move. Changing the prompts changes the metric more than most real shifts do; Profound's once-a-day test found prompt-portfolio choice moved citation share about an order of magnitude more than day-to-day drift.
2. Run on a schedule. Daily is the common default. In AI Rank Lab, scheduled prompt runs do this automatically for ChatGPT, Claude, Gemini and Perplexity and store every run, which is how the data in this article accumulated.
3. Report mention rate over a window. Mentioned runs divided by total runs, over 14 to 28 days, with the run count printed next to the rate.
4. Report citation share separately. Track which of your URLs get cited across the window, not in one run. Expect the list to churn; look at the share, not the membership. See our citation tracker for one way to do this.
5. Keep engines separate. Our engines disagreed on which kind of stability they had. A blended score hides that.
6. Annotate model and product changes. When an engine ships a new model, mark the date so a step change is not mistaken for a content win or loss.
7. Act on persistent changes. Treat a change as real when it holds across two consecutive windows and shows up in more than one prompt.

If your team uses a single AI visibility number, our guide to how an AEO score is measured explains what should sit underneath it.

How many runs do you need before trusting a change?

More than most dashboards imply, especially at the level of a single prompt. A mention rate is a proportion, so its uncertainty shrinks with the number of runs behind it. The table uses the standard normal approximation for a 50% mention rate (the worst case) and assumes independent runs. Real runs are correlated, so treat these as optimistic floors.

Two things follow. First, a single prompt's mention rate moving from 50% to 65% across two three-week windows is well inside the noise. Second, the clean way to get more runs is usually more prompts, not more repeats of one prompt. The St. Gallen paper reaches a similar conclusion from its own data: the standard error of a per-brand detection rate fell below 0.10 at 7 runs per prompt, and a rolling window brought per-brand error below 0.10 at 10 days and below 0.05 at 24 days, which is why the authors recommend two to four-week windows.

Vendor tests disagree on daily repeats, but they measure something different. Profound found once-daily runs landed within about 2 points of ten-times-daily runs on visibility (78.7% vs 80.4%) across 753 prompts, and Meikai found 1 run a day gave the same pooled signal as 10 across 500 prompts. Both are portfolio-level results with hundreds of prompts behind them. The reconciliation is simple: with a large prompt set, once a day is enough; to diagnose one prompt, you need many runs of it.

How many prompts do you need to track?

Enough that no single prompt's quirks drive the number, and in practice that means dozens per engine, not a handful. Otterly found that with 10 prompts, 90% of its brand coverage results fell anywhere between 39% and 71%; at 100 prompts the range narrowed to 51% to 60%, about 73% less variation. The St. Gallen authors saw prompt-level Jaccard range from below 0.2 to above 0.8 within a single campaign. Our own sample shows the same spread: within one engine, some pairs never flipped while others flipped repeatedly.

Our practical floor: at least 50 prompts per engine for a brand-level number, grouped into topic clusters, with 100+ if you want to compare engines or segments. This is a judgment call informed by the studies above, not something our 42-pair sample can prove on its own. Choosing the prompts matters as much as the count.

How often should you track AI visibility?

Run prompts daily and report on rolling 14 to 28-day windows. The cadence of running and the cadence of reporting are different decisions: runs need to be frequent enough to fill the window with samples, reports need to be slow enough that a window holds enough runs to mean something. In our data, ChatGPT, Claude and Gemini pairs accumulated about 20 to 22 runs over a median 25 days, close to one run a day, and even that left single-prompt mention rates with wide error bars.

For the bigger picture on what to track and why, see our AI visibility tracking guide.

What are the limitations of this study?

Plenty, and they bound what you should take from it.

• Small, uneven sample. 42 pairs and 968 runs, from 12 tracked domains skewed toward marketing and software. Perplexity has 6 pairs, Google AI Overview 1, Google AI Mode none. Per-engine percentages on 6 to 13 pairs can swing with a single pair.
• Not a random sample of prompts. Prompts were chosen by users to track their own brands. Gemini's high always-mentioned count (10 of 13) most likely reflects that mix.
• API, not consumer apps. Runs used each engine's API with web search or grounding on, and no personalization. A logged-in user may see more or less variation.
• The brand is named in the scoring instruction. That lets the tracker score the answer, but it can raise absolute mention rates compared with a blank chat. We therefore focus on stability, not on how often brands are mentioned.
• Automated scoring. Mentions and cited domains are detected automatically from the answer text and its citations; we did not hand-check every run.
• Uneven spacing and unpinned models. Runs were not on a fixed clock (Perplexity ran densest), and model versions were not pinned for the whole window, so our churn mixes noise with real drift.
• Domains, not URLs, and no ranking. Overlap is measured on cited domains, which is kinder than URL-level overlap, and we did not analyze answer position.
• No significance tests. These are descriptive medians and counts. Do not extrapolate them to other industries, languages or engines.

We will rerun this analysis as the sample grows and publish the updated numbers with the same definitions.

The bottom line

AI answer consistency is low enough that any one run can mislead you, and source lists churn far more than brand mentions do. The fix is not more screenshots; it is a fixed prompt set, daily runs, and rates reported over two to four-week windows, per engine, with the run count shown. If you want that setup without building it, AI Rank Lab's brand visibility tracking runs your prompts on a schedule and keeps every run, so you can read trends instead of snapshots.