Evaluation Validity Infrastructure

As model and agent evaluation artifacts multiply, their claims are increasingly not transportable across commits, runtimes, attempt counts and corpus snapshots; a market exists for infrastructure that makes evaluation claims auditable and repeatable — pin the artifact, specify the claim, and report the operating point.

StrengthenedConfidence 68%September 21, 2026

Who it is for

Model and agent evaluation teams, procurement reviewers, and platform vendors whose release gates currently rest on a single reported score.

Why now

This window produced direct measurements of non-transportability rather than warnings about it: one-try success diverged from twenty-try reliability in stateful workflows, live-runtime and frozen-holdout rankings of the same agent configurations agreed at Spearman 0.13, an audit found most eligible eval units cannot support deterministic claim inference at a pinned commit, and a corpus expansion produced 6-10 point answer churn invisible to aggregate accuracy. A year ago the question was whether an eval pipeline could be run at all; now the question is whether its output licenses the claim attached to it.

What would kill it

Evidence that a single published operating point, at a fixed commit and runtime, predicts production reliability and procurement outcomes well enough that claim-relative auditing is unnecessary overhead — for example a vendor contract settled on pass@1 alone that later matched observed field reliability.

The events this rests on

Thinkingbox: Benchmarking Reliability in Stateful Business Workflows
Thinkingbox-bench shows strongest model at 65.36% pass@1 but only 25.25% pass^20, exposing a reliability gap invisible in single-try evals.
ClawProBench: Trace-Aware Runtime-Native Agent Evaluation Finds Weak Alignment Between Live and Frozen Holdout Rankings
ClawProBench finds weak alignment (Spearman 0.13) between live native-runtime and frozen holdout rankings, making eval results non-transferable across runtimes.
What Does an Evaluation License? A Commit-Bound Census of Claim-Relative Inference in Inspect Evals
Commit-bound census shows 110 of 124 Inspect Evals units cannot support deterministic inference, so eval claims require a frozen substrate and identified claim.
Greek-Language Reasoning Reveals What Accuracy Benchmarks Cannot See
The Greek-language reasoning study finds accuracy benchmarks noise-dominated while reasoning language, answer-channel leakage and format adherence separate models, and that RL with verifiable rewards rather than SFT enforces those properties—adding claim dimensions outside accuracy.
Repeat-Aware Audit Exposes Accuracy-Blind Answer Churn in RAG Index Expansion
Expanding a frozen corpus prefix from one to seven shards causes 6-10 percentage points of answer churn while aggregate exact-match moves only -1.5 to +3.0, showing that accepted release metrics can be blind to the thing users experience.
BenchMIRT: Multidimensional IRT for Prompt-Level Auditing of LLM Benchmarks
Prompt-level multidimensional IRT shows prompts inside a labeled benchmark load on different latent dimensions (BBQ on reasoning; WMDP negatively because refusal is scored correct), so aggregate benchmark labels misattribute what they claim to measure.

How it has moved

2026-08-31Open
2026-09-03Strengthened
2026-09-17Strengthened
2026-09-21Strengthened

↑ Back to top