Repeat-Aware Audit Exposes Accuracy-Blind Answer Churn in RAG Index Expansion

A repeat-aware audit of retrieval-augmented QA shows that expanding a frozen corpus prefix from one to seven shards induces answer churn of 6–10 percentage points that is invisible to aggregate exact-match accuracy, which changes by only -1.5 to +3.0 points. The paper proposes the Snapshot Compatibility Audit to separate corpus-induced churn from generation variability.

Hugging Face Daily Papers — Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

August 26, 20261 source74% confidence

80Strategic

Strategic score

Evidence strength76

Technical novelty82

Commercial potential70

A retrieval-augmented question-answering system can change its answers on the same questions when the retrieval corpus is expanded, even though the model identifier, prompt, retrieval policy, evidence depth, rendering, and generation controls are held fixed. The authors call this 'accuracy-blind answer churn' because aggregate accuracy can stay flat or even improve while the answer distribution shifts underneath it. To measure it, they propose a Snapshot Compatibility Audit that subtracts same-snapshot repeat disagreement from cross-snapshot disagreement, isolating churn attributable to the corpus update from ordinary generation variability.¹

Context

The constraint this moves is in how RAG releases are evaluated. Standard practice treats exact-match or semantic accuracy as the release gate; the paper shows that two snapshots can be indistinguishable on aggregate accuracy yet incompatible at the answer level, with 40/400 repeat-stable semantic flips on Natural Questions. The requirement shifts from 'utility must not decrease' to 'answer compatibility must be audited, and if churn is excess, the update must be justified by utility gains.'¹

Assessment

The evidence is methodologically careful: both main studies are preregistered; the replication is outcome-blind and uses a different generator and serving configuration. Effect sizes are consistent across benchmarks—semantic excess churn of 10.25 pp and 8.75 pp—while exact-match accuracy moves in opposite directions (-1.5 and +3.0 pp). The main unaddressed limitation is that the audit was applied to one index expansion protocol (FineWeb prefix 1-to-7 shards) and two datasets, so external validity across corpus types and model families remains open.¹

Caveats

AI hypothesisThis analysis would change if independent replications on broader tasks found that the same-snapshot baseline overcorrects or undercorrects for generation variability, or if answer-level churn could be shown to be utility-irrelevant because the changed answers are semantically equivalent under the task's intended scoring. Also, the current evidence does not establish a cost threshold; a deployment could accept compatibility failure if utility gains are large and explicit. Without such a trade-off framework, the paper's recommendation to audit compatibility alongside utility is a safe but incomplete release policy.¹

Source text

Hugging Face Daily Papers — Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA · August 24, 2026

Computer Science > Information Retrieval

Title:Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

Abstract:A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval policy, evidence depth, rendering, and exposed generation controls are held fixed. Aggregate accuracy may hide these changes when gains and losses cancel, while ordinary generation variability makes one-shot comparisons overstate update effects. We call the hidden phenomenon accuracy-blind answer churn and introduce the \emph{Snapshot Compatibility Audit}, which estimates excess answer churn by subtracting same-snapshot repeat disagreement from cross-snapshot disagreement. We instantiate it by expanding one frozen FineWeb prefix from one to seven shards. In a preregistered 400-question Natural Questions study, normalized-exact and blinded-semantic excess churn are 6.44 and 10.25 percentage points while exact-match accuracy changes by only $-1.50$ points. A post-hoc analysis finds repeat-stable semantic flips on 40/400 questions. A separately preregistered 200-question TriviaQA study yields smaller, directionally consistent excess churn while exact-match accuracy moves in the opposite direction. An outcome-blind post-hoc 100-question subset replication with a second DeepSeek generator and serving configuration finds 8.75 pp of semantic excess churn even as exact match rises by 3.00 percentage points. Answer-level compatibility can therefore fail without a conspicuous or consistently directed utility shift. Retrieval-augmented releases should audit compatibility alongside utility.

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The term 'same-snapshot repeat disagreement' is the key control: it measures how often the same system gives different answers to the same question when the corpus snapshot is held fixed. By subtracting this from cross-snapshot disagreement, the audit assumes the repeat disagreement fully captures generation variability. If temperature or decoding nondeterminism interacts with the snapshot change, the subtraction could misattribute churn.

The reported 40/400 repeat-stable semantic flips means that on 40 questions the model's answer changed after the index expansion even though repeated runs on the same snapshot gave a stable answer; these are therefore not generation noise. This number is the core evidence that churn is real and not a measurement artifact.

The outcome-blind replication with a second DeepSeek generator and serving configuration is a deliberate attempt to rule out the objection that the main study's churn was specific to the exact decoding pipeline. Its finding of 8.75 pp semantic excess churn while exact match rose 3.00 pp is the strongest single observation that utility and answer compatibility are separable.

The source text above remains the property of its publisher and is reproduced for commentary.

Hugging Face Daily Papers — Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

August 24, 2026endistinct angle

Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

↑ Back to top