How many runs does an AEO tracker need before a citation-share change counts?

Neither KACI nor WashU sets a fixed number. KACI ran two and found same-provider overlap between 0.319 and 0.664. Ask for three or more runs per prompt per period so a spread exists on your own panel, then count only changes above that band.

What is rerun overlap?

The share of cited domains two identical runs have in common, measured as Jaccard. KACI reports 0.664 for OpenAI, 0.515 for Claude, and 0.319 for Gemini.

Do the three AI providers cite the same sites?

Rarely. KACI's cross-provider overlap averaged about 0.06. Track each surface on its own line.

Does an AI citation mean traffic or pipeline?

A citation shows your domain appeared as a source in one answer. Measure referral traffic and pipeline separately in your own analytics.

← Benchmark

Ask the Same AI the Same Question Twice. How Many Runs Before a Citation Shift Counts as Proof?

Require a run count plus a variance band before any vendor citation-share delta counts as a result.

Jonathan Kvarfordt · Published October 6, 2026 · Last reviewed October 16, 2026 · 8 min read

Why trust this analysis?

The short answer

How many runs does our AEO tracker need before a citation-share change counts as proof?

One AEO tracker run tells you what one model cited once. Kingy.ai asked OpenAI, Claude, and Gemini the same 1,200 questions twice through their official APIs. Each provider's second run kept only part of its first run's cited sites: overlap of 0.664 for OpenAI, 0.515 for Claude, and 0.319 for Gemini [Verified primary]. Across providers, overlap fell to about 0.06 [Verified primary]. A citation-share "gain" smaller than that rerun noise is noise.

Decision rule

Require a run count plus a variance band before any vendor citation-share delta counts as a result.

Evidence

Same-provider cited-site overlap: OpenAI 0.664, Claude 0.515, Gemini 0.319 [Verified primary].
Across providers, overlap fell to about 0.06 [Verified primary].

Operator action

Ask your AEO vendor for its rerun overlap on your own prompt panel. If the vendor has no number, you have no baseline.

Last reviewed October 16, 2026

Sources

Kingy.ai, Curtis Pyke. KACI Wave One, v2026.10.05, published Oct 5, 2026. CC BY 4.0 (Kingy-authored analysis). https://kingy.ai/research/kaci-wave-one-ai-citation-study/
Zenodo evidence record and data ZIP, DOI 10.5281/zenodo.23170591 (license field: cc-by-4.0). https://zenodo.org/records/23170591 · https://doi.org/10.5281/zenodo.23170591
Xu, Iqbal, Montgomery (Washington University in St. Louis). Measuring Google AI Overviews: Activation, Source Quality, Claim Fidelity, and Publisher Impact. arXiv 2605.14021 (v2), to appear at ACM IMC '26. arXiv listing license: arXiv non-exclusive distribution license 1.0. https://arxiv.org/abs/2605.14021
Sara Savat, Washington University, via Digital Information World, Oct 5, 2026. https://www.digitalinformationworld.com/2026/10/washu-research-finds-gaps-in-google-ai-overview-citations-claims.html
RAR, Sep 25, 2026: Which AI Surfaces Actually Cite Us on Commercial-Intent Prompts. https://www.therevenueaireport.com/blog/b2b-ai-discovery-surface-specific-citations-vs-visibility-score
Zeno Rocha (Resend founder and CEO) on X, Oct 6, 2026. x.com
Zeno Rocha on X, Aug 28, 2026. x.com
Agentic AI Foundation (AAIF), "MCP Usage Surged as the Protocol Went Stateless," Sep 1, 2026. aaif.io
"LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?" Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '26), published Aug 8, 2026. DOI 10.1145/3770855.3817478. dl.acm.org
Jason Lemkin on X, Oct 6, 2026. x.com
SaaStr (Jason Lemkin), recap of Maia Josebachvili's SaaStr AI session. saastr.com
Guillermo Rauch (Vercel CEO) on X, Oct 9, 2026. x.com

The argument

How this benchmark breaks down

A map of the sections ahead, in the order the case is made. The sourced charts are in the research library.

Confidence tags used: [Verified primary] = checked against the study page and its published data files. [Reported] = from secondary press coverage. [RAR analysis] = our reasoning, built for buyers to test.

The short answer

Benchmark

Ask the Same AI the Same Question Twice. How Many Runs Before a Citation Shift Counts as Proof?

Source-cited charts live in the research library.

Decision rule: Require a run count plus a variance band before any vendor citation-share delta counts as a result.

Operator action this week: Ask your AEO vendor for its rerun overlap on your own prompt panel. If the vendor has no number, you have no baseline.

Why now

KACI Wave One went live Oct 5, 2026, with a DOI, a public data ZIP, and a CC BY 4.0 license on Kingy's analysis. It is the first public dataset we have seen that reruns the same panel on the same providers and publishes the overlap. The same day, Washington University in St. Louis published a 40-day audit of Google AI Overviews. Both point at the same buyer problem: a citation dashboard reports one draw from a moving system.

Our Sep 25 piece asked which AI surface cites you. This piece asks a prior question: does one run on any surface measure anything stable?

What KACI measured

Scope: 1,200 designed English questions, asked twice, on three official APIs = 7,200 completed answers. Returned models: gpt-6.1-sol, claude-sonnet-5-5, gemini-3.8-flash, each with native search and default settings [Verified primary].
Rerun overlap (same provider, two runs): OpenAI 0.664 (95% interval 0.645–0.682), Claude 0.515 (0.494–0.537), Gemini 0.319 (0.292–0.345) [Verified primary]. These averages cover only questions where at least one run cited a domain: 867, 652, and 337 of 1,200.
Cross-provider overlap: OpenAI/Claude 0.061, OpenAI/Gemini 0.060, Claude/Gemini 0.065 [Verified primary].
Answers with any citation: OpenAI 71.6%, Claude 53.7%, Gemini 25.2% [Verified primary].
Top cited domain: GitHub for OpenAI, arXiv for Claude, Reddit for Gemini, counted by citation occurrences. Kingy calls these rankings exploratory [Verified primary].

The metric is Jaccard: shared domains divided by all domains cited across both runs. A score of 0.319 means about a third of Gemini's combined cited-site list showed up in both runs.

Bar chart of cited-domain overlap between two runs of the same 1,200 questions: OpenAI 0.664, Claude 0.515, Gemini 0.319. Dashed line marks cross-provider overlap near 0.06. Source: Kingy.ai KACI, CC BY 4.0. — Same-provider rerun overlap (Jaccard), questions with at least one cited domain. Source: Kingy.ai KACI, CC BY 4.0. Redrawn by The Revenue AI Report.

Bar chart of the share of API answers with any native citation: OpenAI 71.6%, Claude 53.7%, Gemini 25.2%. Source: Kingy.ai KACI, CC BY 4.0. — Share of 2,400 completed API answers per provider with a native citation. Source: Kingy.ai KACI, CC BY 4.0. Redrawn by The Revenue AI Report.

The Proof Gap: one run is one sample

Most AEO dashboards show citation share as a single number per week. KACI shows the floor under that number moves on its own. On the provider with the lowest rerun overlap, two identical runs shared about a third of their cited domains. A vendor that reports "your share rose four points" from one run per period has given you a number with no error bar.

Gemini adds a second trap. It cited a source in about one answer in four. A thin citation base makes every share figure swing harder.

Google AI Overviews show the same pattern from another angle. WashU issued 55,393 trending queries over 40 days. 29.8% of AI Overview-cited domains were absent from the page-one results [Verified primary]. 11.0% of 98,020 extracted claims lacked support in the cited text the team could collect, a figure the authors treat as an upper bound [Verified primary]. 41.9% of verified overviews were fully grounded [Verified primary]. A cited link and a supported claim are two separate checks.

Agents are starting to pick vendors

Resend, an email API company, says AI agents called its MCP server 106,719 times in April, 1,062,650 times in August, and 3,003,809 times in September [Vendor-reported, unaudited, Zeno Rocha on X, Oct 6]. The Agentic AI Foundation placed Resend's August pass of one million monthly calls inside a wider surge: MCP tool calls from ChatGPT users more than doubled during August [Reported, AAIF, Sep 1].

Tool calls are how agents get work done. Retrieval decides which tool they reach. LiveMCPBench gave agents 527 tools across 70 MCP servers, and retrieval errors accounted for nearly half of all failures [Verified primary, ACM SIGKDD '26, Aug 8]. If the agent's search misses you, it moves on to someone else.

On Oct 6, Jason Lemkin asked his agent, live at SaaStr AI Connect, where to move SaaStr's hosting. It picked Render over Railway and Vercel. Lemkin said Render was new to him and his team would test it first [Reported, Jason Lemkin on X, Oct 6]. That is a lead created by an agent. It is also one run, the same limit this page sets for citation trackers.

Agents also read the docs before anyone books a demo. Vercel CEO Guillermo Rauch posted on Oct 9: "58.18% traffic is bot-originated (last 30d). This was 32% in Jan 2024." "60%+ of deployments on Vercel are now agentic, up from ~3% in Jan 2026." "Up to 83% of pageviews are now coming from agents to Vercel's own documentation sites." [Vendor-reported, unaudited, Guillermo Rauch on X, Oct 9]. Read these with care. Vercel does not define "bot-originated" or "agentic." The 83% is a ceiling, and it covers only Vercel's own docs. Vercel sells infrastructure for agents, so it gains when agent traffic looks big. Stripe's CRO of AI, Maia Josebachvili, said at SaaStr AI that agent traffic to Stripe's docs grew 10x in 2025, and predicted agents would read more Stripe docs than humans by the end of 2026 (company-stated, via SaaStr).

The buyer question: when agents read the docs first, what does the vendor shortlist rest on? On this evidence, it rests on what your docs say and whether an agent can parse them [RAR analysis].

What it means for a revenue leader: track whether agents pick you. Write the buying prompts your customers would hand an agent. Run each one several times per period on each agent that matters. Log which vendor each run picks and which pages it read. Report pick share with a run count and a variance band, the same rule as citation share [RAR analysis].

Order Form and pilot lines

Paste these into the AEO tracker contract or pilot plan. Fill the blanks with your vendor. [RAR analysis]

Run count: "Vendor runs each prompt [N] times per measurement period on each named surface." Two runs give one overlap number. Three or more give a spread.
Variance band: "Vendor reports the rerun overlap and the run-to-run range of citation share for our panel, per surface, before any delta is reported."
Delta threshold: "A citation-share change counts as a result only when it exceeds the baseline variance band for [X] consecutive periods."
Same prompt panel: "Prompt panel is frozen and versioned. Any change to wording, count, or language resets the baseline."
Reruns per period: "Reruns happen on a fixed schedule within a stated time window, with timestamps exported."
Surfaces named: "Report each surface on its own line: ChatGPT, Claude, Gemini, Perplexity, Google AI Overview, Google AI Mode, or the subset we name. API or consumer app stated per surface."
Citation, traffic, and pipeline on separate lines: "Citation share, referral sessions, and sourced pipeline are reported as three separate metrics. Vendor makes no claim converting one into another."
Raw export: "Buyer receives per-run answer text, cited URLs, and timestamps on request."

Eight-Seat Read: one decision each

CMO / marketing: Drop any board slide that shows a citation-share change without a run count and band.
Demand gen: Fund content off a share gain only after it holds above the band for the agreed periods.
RevOps: Own the frozen prompt panel and the baseline band. Version both.
Sales: Treat "AI cites us" from a single screenshot as an anecdote.
Customer success: Track citation on support and how-to prompts per surface. Expect Gemini to cite least often.
Finance: Approve AEO renewals on band-adjusted deltas, plus referral traffic and pipeline from your own analytics.
Product marketing: Check which domain types each surface favors (GitHub, arXiv, Reddit led in KACI) before choosing where to publish.
CEO / strategy: Ask one question in every AEO review: how many runs, and what is the band?

Why we might be wrong

KACI used official APIs. Consumer ChatGPT, Claude, and Gemini apps may behave differently.
The 1,200-question panel was designed for the study. It was never sampled from real buyer demand.
One author ran KACI, with Codex assisting on collection, analysis, and drafting. Curtis Pyke funded it for Kingy.ai.
Kingy.ai itself appears in eight Claude answers and three Gemini answers. Kingy says it counted its own domain by the same rules.
Two runs in one collection window, with gaps of up to 348.7 minutes between matched archives. Search indexes and batch timing may explain part of the drift.
WashU covers consumer Google AI Overviews on trending queries from one U.S. vantage point. B2B buying prompts may behave differently.
A citation measures answer visibility only. Traffic and pipeline need their own measures.

Soft close

New benchmarks land in the weekly issue first. Subscribe.

Take it to the room

The short list from this piece

Pulled from the argument above, written so you can read it out in a pipeline or board review.

Frequently asked questions

How many runs does an AEO tracker need before a citation-share change counts?: Neither KACI nor WashU sets a fixed number. KACI ran two and found same-provider overlap between 0.319 and 0.664. Ask for three or more runs per prompt per period so a spread exists on your own panel, then count only changes above that band.
What is rerun overlap?: The share of cited domains two identical runs have in common, measured as Jaccard. KACI reports 0.664 for OpenAI, 0.515 for Claude, and 0.319 for Gemini.
Do the three AI providers cite the same sites?: Rarely. KACI's cross-provider overlap averaged about 0.06. Track each surface on its own line.
Does an AI citation mean traffic or pipeline?: A citation shows your domain appeared as a source in one answer. Measure referral traffic and pipeline separately in your own analytics.

Share this page

Posting to Instagram or TikTok? Copy the link, it carries the title, summary and share image.

Get the next benchmark, with the sample size attached.

Keep reading

Benchmark

Which AI Surfaces Actually Cite Us on Commercial-Intent Prompts, and Why Is One Blended AI Visibility Score the Wrong Instrument for That Decision?

Foundation x AirOps measured citation share across six AI surfaces. One blended AI visibility score averages those systems into a vanity number. Track citation rate and citation share per surface. Soft subscribe only.

Benchmark

The AI Adoption Curve for Revenue Teams: L1 to L6, and Where Most Teams Stall

Six levels from one tool with no workflow change to a rebuilt process. Each level has a failure mode and an exit criterion. Most revenue teams are at L2 and reporting L4.