RLVR Coverage Loss Is Located at the First Decision, Not in Downstream Reasoning

Using exhaustively enumerated Countdown solution families, the paper shows that RLVR contracts solution coverage mainly by suppressing access to alternative entrance choices rather than by degrading later computation. Coverage falls by up to 67%, entrance-prefix injection raises completion from 0.018 to 0.212, and late-layer interpolation recovers 37% of coverage without sacrificing pass@1.

Hugging Face Daily Papers — Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

September 5, 20261 source70% confidence

75Strategic

Strategic score

Evidence strength64

Technical novelty78

Commercial potential40

Contraction caused by reinforcement learning with verifiable rewards has been previously observed, but this paper gives the loss a precise location: it is almost entirely at the entrance of a reasoning trajectory, before the first arithmetic operation is emitted. On Countdown, across both training setups studied, coverage of valid solution families drops by up to 67%, and per-token likelihood shifts are 11x to 16x larger at the entrance than during downstream reasoning.¹

Context

The work defines an entrance family by the first operand and operator on the Countdown task, making the initial choice the partition of the solution space. Exhaustive enumeration then lets the authors separate a model that never accesses a valid family from a model that starts a family but fails to execute it. The contrast is tested with PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct, with coverage tracked across checkpoints.¹

Assessment

The most decisive evidence is restoration after the entrance decision has been forced. Supplying only a non-sampled entrance prefix raises completion rates in low-access families by more than an order of magnitude (0.018 to 0.212 under PPO), which shows that the downstream computation is usable even when the policy no longer initiates it. The later finding that an entrance-targeted parameter interpolation recovers 37% of coverage with no loss in pass@1 turns that diagnostic into a practical intervention. The binding constraint therefore moves from total solution coverage to the first decision, rather than requiring preservation of broad support over the entire reasoning chain.¹

Caveats

AI hypothesisThe full enumeration evidence is restricted to Countdown; the six-benchmark result is about entropy collapse at the very first decoding steps, not about the complete entrance-family decomposition. Replication that showed a mid-trajectory intervention restoring coverage just as cheaply, or a failure of the interpolation intervention at larger scale, would force the localization claim to be weakened from a general property of RLVR to a task-specific effect.¹

Source text

Hugging Face Daily Papers — Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space · August 29, 2026

Computer Science > Machine Learning

Title:Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

Abstract:Reinforcement learning with verifiable rewards (RLVR) substantially improves single-sample accuracy (pass@1) but causes the policy's solution space to contract, diminishing the returns of test-time scaling. In this work, we investigate where inside a reasoning trajectory this breadth is lost: does the policy fail to access a valid solution family, or does it fail to execute computation once initiated? To disentangle access from execution, we analyze the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator, across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct. Across both training setups, solution coverage falls by up to 67%, halving even on problems solved across all checkpoints. We show that this contraction is heavily concentrated at the entrance: per-token likelihood shifts are 11x--16x larger prior to the first arithmetic operation than during downstream reasoning. Supplying only an unselected entrance prefix restores completion rates in low-access families by over an order of magnitude (0.018 -> 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated. Guided by this localization, we find that while surface prompting fails to recover diversity, entrance-targeted interventions succeed: late-layer parameter interpolation with early checkpoints increases solution coverage by 37% at no loss in pass@1. Finally, we show that early-step entropy collapse recurs across six math benchmarks with 7B and 14B models, but is not an inevitable byproduct of reasoning optimization: an SFT baseline preserves more than double the coverage, and staged SFT--DPO--RLVR pipelines retain early-step entropy. In summary, reasoning breadth is lost at the door, not inside the room. Code: this https URL.

Access Paper:

Current browse context:

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Pass@1 is per-sample accuracy and is not a measure of reasoning diversity; it can stay equal while the policy has collapsed onto one reasoning family. Reading the abstract's accuracy gain and coverage contraction as consistent requires keeping that distinction explicit, since the article's entire evaluation depends on coverage as an orthogonal outcome.

The claim that 'alternative solutions remain executable but are no longer initiated' relies on the assumption that a forced entrance prefix affects only selection of the solution family. If the prefix also changed sequence-level reward pressure, the access-versus-execution split would be less clean; the authors do not present an ablation that removes only initiation without altering later token distributions.

Late-layer parameter interpolation with early checkpoints is a parameter-space operation, not a decoding or prompting strategy. The abstract's wording leaves it ambiguous whether interpolation is applied only to later layers' parameters or all layers after an early checkpoint; the term signals an intervention that directly changes the policy weights, which is why its 37% coverage gain with zero pass@1 loss is not a prompting result.

The source text above remains the property of its publisher and is reproduced for commentary.

Hugging Face Daily Papers — Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

August 29, 2026endistinct angle

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

↑ Back to top