The central new move is to separate an evaluation artifact's forward computation from the claim attached to its metric. The paper introduces a claim-replay layer formalized as a frozen substrate D, a grounded family F, a claim query q, and the resulting identified set; only when these are bound can an eval be said to license a claim.1
Context
In this frame, the constraint that moves is the unit of evidence. A reported metric no longer suffices to justify an evaluator's advertised result; instead, the historical evidence and semantic alternatives needed to replay the claim must be bound. The authors apply this stronger requirement to all 124 mechanically eligible Inspect Evals units at one pinned commit.1
Assessment
The abstract reports a sharp empirical pattern: 110 of 124 units stop before deterministic inference because required historical evidence or semantic grounding is unavailable. For the units where execution closes, exact values, winners, complete orders, and pairwise relations split according to claim resolution and by primary versus review family. The output is therefore a set of typed stops, instability witnesses, and stable substructure.1
Caveats
This would change if the denominator or the stop classification were altered: the 110-stops numerator depends on the unspecified mechanical eligibility filter and on how grounding and historical evidence are operationalized. The result is also commit-bound, so a new pinned commit could move units across the stop/close boundary or change the stable substructure.1
