How to Read an AI Citation Count When the Cited Sources Rotate Weekly

Weekly turnover in AI-cited sources runs from 3% to 74% across four datasets that count different objects, and the academic work treats a citation figure as a distribution with an interval

*Image: Brave SERP and search engine by Fabriemo, licensed under CC0.*

On September 2, Google put Gemini 3.8 Flash into AI Mode, where AI Pro and Ultra subscribers could select it from the model menu. Answers produced with that model stopped linking to or citing their sources. Practitioner reports from Gagan Ghotra and Glenn Gabe came first, and Search Engine Land reported the missing citations across AI Mode answers, especially on top-of-funnel queries. Robby Stein, Google's VP of Product for Search, replied publicly that it was not working as intended and that a fix would roll out soon. The fix landed the following morning.

Coverage recorded the event, the acknowledgement and the repair. It contains no citation rate, no prompt count and no sample size for the window in which source links were missing. The change appeared for subscribers on AI Pro and Ultra who selected the new model, and the default AI Mode experience was reported as unaffected.

Nothing on the publisher side changed on September 2, and no publisher-facing report marked the difference. The set of sources an engine cites is produced inside that engine's pipeline, and it moves for reasons the publisher does not control. Four datasets published this year measure how far that set moves. Their headline turnover rates span 3% to 74%, and each one is a rate over a different object.

What the turnover numbers are counting

SISTRIX analyzed 82,619 prompts across three platforms, six countries and 17 weeks. Google's AI Mode replaced 56% of the sources it cited each week; ChatGPT replaced 74%. BrightEdge, tracking thousands of prompts a week across five engines and nine industries, reported that 96.8% of the cited domains it follows showed no week-over-week change, with 87% of the movement it did record going downward and most of it binary, a domain going from cited to not cited on a given prompt. Alex Birkett compared the 500 most-cited URLs for roughly 100 prompts at two points a month apart: 245 appeared in both snapshots, a Jaccard similarity of 0.32. Orbit Media's Bill Widmer, running 72 prompts weekly across four models, puts his churn at 30% to 65% and advises judging any trend on three weeks of data or more.

Each of those figures is a rate over a different population. SISTRIX counts the share of cited source positions replaced between runs. BrightEdge counts the share of tracked domains whose citation share moved, a population that includes domains cited once in the window. Birkett counts the overlap of a fixed top-500 URL list over a month. Orbit counts repeat runs of its own prompt set. Anyone comparing a turnover figure with a vendor's published rate needs to know the population it is a rate of, and SISTRIX describes its own domain-level numbers as conservative, reporting higher drift at the URL level.

One practitioner ran seven trackers against his own site for the same twelve queries across the same fifteen days in May and recorded citation counts from 38 to 312. He traced the spread to whether a tool counts a clickable source link or any brand mention, to de-duplication rules, and to engine coverage: only one of the seven sampled all five engines he cared about. The test covers one site and twelve self-selected queries, so the spread it found describes those twelve queries and the counting rules of the seven tools that answered them.

Engines rarely agree on which domains they cite

Across 1,792 query-and-domain combinations, Orbit's four engines agreed on citing the same domain 30 times, or 1.7% of the time. SISTRIX found that Google's AI Overviews and AI Mode cited different domains for the same prompt 83% of the time, a Jaccard overlap of 0.17, and that AI Mode and ChatGPT overlap at 0.125. Language contributes to the difference: for German queries, AI Mode drew on German-language sources 80% of the time while ChatGPT Core drew on English sources 68% of the time.

Source volume per answer varies as widely as the domain sets. Orbit measured an average of 19.2 sources per answer for Perplexity, 8.1 for Gemini, 4.5 for ChatGPT and 3.6 for Claude. SISTRIX counted 3 to 4 domains per ChatGPT response against 14 to 16 in AI Mode. Every one of those comparisons puts Perplexity and Google above ChatGPT on sources per answer, and a blended share-of-citations figure is therefore an average over populations of different sizes, different languages and different de-duplication rules.

Orbit's two B2B question sets shared 7.6% and 5.4% of their cited domains between ChatGPT and Perplexity, and at the exact-URL level the overlap between engines fell to somewhere between 1% and 5%. Citation usually followed mention in that dataset: of 1,052 query runs, 17 cited a brand's page without also naming the brand.

What persists across runs

For 86% of prompts, SISTRIX found that an AI Mode response carried a stable core of a few domains while the remaining citations rotated at 89% per week. The median AI Mode prompt held about two domains that appeared in all 17 weeks; the median ChatGPT prompt held none. Survival varied by domain type, with YouTube domains at a 24% core rate in AI Mode, large technology companies at 16%, Wikipedia at 12%, marketplaces at 10%, forums and user-generated content at 3%, and news and media at 1.4%. For brand queries, the brand's own domain survived all 17 weeks in 43% of cases and appeared in at least 80% of weeks for 66%.

A count that adds persistent and rotating citations together returns one figure for two behaviors with different consequences. Orbit's repeat runs show a similar split: running the same 72 questions through the same models two days apart, Claude kept 30% of the URLs it had cited, Gemini kept 38% of its domains, and Perplexity was the most stable of the four.

When the model changes between runs

SISTRIX evaluated 38 daily samples of 100,000 German-language ChatGPT responses, 3.8 million responses carrying more than 100 million source mentions. On May 23, ChatGPT moved from GPT-5 mini to GPT-5.5. Within 48 hours the domain distribution in German-language responses shifted by 47%, against 1% to 2% in the preceding days, and the average number of sources per response fell from 30.9 to 28.4. The team verified the model version with a daily prompt asking the system which model it was, and its own note states what the data cannot establish: from outside, a model swap cannot be separated from a retrieval-layer or prompt-layer change running in parallel. A second event in the same window, inline image embeddings between May 7 and 11, raised the share of responses carrying inline elements from about 15% to about 35% and left citations roughly where they were.

Neither change came with a publisher-facing announcement. The Gemini 3.8 Flash window was confirmed by practitioner reports and a public acknowledgement, and it closed in about a day, with no measured citation rate attached to it. The coverage does not describe how a publisher would detect that class of change before it lands.

What a citation count includes

A critical survey posted to arXiv on September 6 argues that a visibility score is assembled from a prompt corpus and a set of weights, and that the two together define an answer market which need not correspond to real user demand. Prompt wording changes which documents are retrieved and which sources compete. When a language model performs the citation-detection step, changing that model's instruction can move the score even when the answers being scored are identical. Two weighting schemes applied to the same three subcorpora of an earlier study produced generative-answer activation rates of 39.7% and 70.5% without changing any rate inside those subcorpora; those two figures measure only whether a generative answer appears at all. The paper states that a citation alone does not establish a source's contribution, and where weights are unknown it recommends reporting a set of admissible scores. It is a critical survey reporting no new experiments, and its author states that the framework's empirical validity remains to be assessed.

A second framework separates citation selection from citation absorption, meaning whether a cited page's language, evidence or structure reaches the answer at all. Across 602 prompts on three platforms, the researchers counted mean citations per prompt of 6.88 for ChatGPT, 12.06 for Google and 16.35 for Perplexity, and mean influence among fetched pages highest for ChatGPT at 0.2713 against 0.0584 for Google and 0.0646 for Perplexity. Q&A formatting on its own did not raise absorption. The authors work from a static snapshot and decline causal claims.

Trellner Research, an independent firm that states it takes no payment from the subjects it covers and publishes no rankings, asked two Perplexity models for the best products in 380 software categories, one prompt per category, 760 calls, and collected 7,534 citations across 2,055 domains. Of those citations, 59.8% point at domains ranked worse than #100,000 in Tranco's top-million list and 23.4% at domains outside the top million altogether, with a median rank of 71,611 among the ranked ones. Wikipedia was cited three times. A vendor's product-marketing blog drew 194 citations across 96 of the 380 categories, ahead of Gartner at 158. Three sites under apparently common control had published 215,128 machine-generated best <category> pages between them, none of the three predating December 2023, and two gave their homepages the HTML title "Facts & Grounding Page". Domains outside the top million were newer as well, with a median first Wayback capture of 2020 against 2011 for the ranked ones. Two recommendation slots resolved to the wrong domain, with dryad.co redirecting to an online-gambling portal and montecarlo.com resolving to the Monaco casino group. Only Perplexity was measured, one run per category with no repeat sampling, and Google was left out because grounding a Gemini model through OpenRouter would have described that router's search plugin.

A citation count treats pages engineered to be read by models, vendor blogs, review directories and reference works as interchangeable, though those categories are not held to a shared accuracy standard.

Measuring a fixed prompt corpus over time

The published datasets support repeat measurement of a corpus the reader controls, with the domains that persist separated from those that rotate. Prompt wording changes which documents are retrieved and which sources are in competition, so a list is comparable with itself only while the wording stays fixed. Widmer draws his questions from sales calls and support conversations, which keeps the corpus close to questions real buyers ask. Engines belong in separate series, because an average across four engines that agree 1.7% of the time describes none of them.

Sielinski's statistical framework treats citation visibility as a sample estimate of a distribution. In its motivating example, a 200-query sample on one platform gave a running-gear domain a citation share near 9.5% with a bootstrap interval of 5.5% to 12.5%, against a competitor near 6.0% with an interval of 4.0% to 8.0%. The intervals overlap, and the paper reports that across its platforms and topics, domains differing by less than roughly five to seven percentage points were usually indistinguishable. Rankings were unstable throughout the frequently cited set, not only at the top of it. The paper is a preprint, it covers three consumer product topics, and it publishes sample-size guidance for tighter intervals.

Each run should be logged with the same fields, so that a series is reproducible before anything is compared across it.

Fields to log on every run

• run_date: date of the run.
• engine: engine name.
• model_version: the version string the engine reports, if it reports one.
• prompt_id: stable identifier for the question.
• prompt_text: the exact wording used, unchanged between runs.
• cited_urls: every cited URL, in order.
• cited_domains: domain per URL, deduplicated.
• brand_mentioned: true or false.
• brand_cited: true or false.
• note: model, UI or retrieval changes observed before this run.

One release moved the cited set by 47% within 48 hours, so a series that spans a release describes two systems, and logging the model version keeps the two periods distinguishable. The note field is where an unannounced change gets recorded on the day it is noticed.

A mention rate and a citation rate are separate measurements, and the gap between them varies by engine. In Orbit's data it ran from 9 points for ChatGPT (56% mentioned, 47% cited) to 31 points for another model (72% mentioned, 41% cited), with Perplexity at 23 points. For Semrush, the largest brand in the study, the gap was 50 points, against 5 and 4 points for the two smaller sites: between 83% and 100% of relevant answers mentioned the brand while its citation rate ranged from 83% down to 0%. Widmer's reading is that models recommend well-known brands from training data while sourcing the answer from third-party pages, which would make a citation count understate visibility for an established brand and track a smaller one more closely. That reading comes from 72 prompts and applies to those questions.

Before renewing an AI visibility subscription, ask the vendor for five things.

Ask the vendor

• The prompt list and the weights applied to it.
• Which engines and which model versions produced the number.
• Whether a citation means a clickable link, a brand mention, or a share of voice.
• The de-duplication rule and the sampling window.
• The number of runs behind the figure, and the interval around it.

To run a baseline test, put a dozen of your own questions to two engines on the same day, list the domains each one cites, and count how many appear in both lists. The 0.125 overlap SISTRIX measured between AI Mode and ChatGPT, and the 30 agreements out of 1,792 in Orbit's data, both predict a small number, and the exercise shows what a blended score gives up. Two gaps remain in the public record: no measured citation rate for the stretch when Gemini 3.8 Flash served answers without source links, and no shared definition of what counts as a citation, which is how seven trackers return seven counts for one site over one fortnight.

Disclosure: GeoAnalyzer publishes this blog and is the subject of this article.