Per-Field Selective Risk Control for Document Extraction: Three Failure Modes and a Validity Ladder

The paper diagnoses three failure modes in per-field selective risk control using 13,859 claude-sonnet-5 fields from 800 CORD receipts (49.0% correct): document clustering, score-refit leakage, and a tie-mass pathology. It proposes a validity ladder with tiered guarantee forms, showing that a fit/val split restores expected-selective-risk control (coverage 0.318, risk 0.096) but is not a certificate, while Mondrian Learn-then-Test yields per-group PAC certificates that are near-vacuous. A support-bin provenance taxonomy wins on the primary corpus but not on other models, and a human-gold audit verifies practical-tier risk at 1.3% against a 10% budget.

Hugging Face Daily Papers — Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays

August 20, 20261 source84% confidence

76Strategic

Strategic score

Evidence strength65

Technical novelty78

Commercial potential70

Per-field accept/review with selective risk at most alpha is the trust contract document-extraction systems need. The paper shows that the natural procedure silently violates this contract on real documents: on 13,859 genuine claude-sonnet-5 fields from 800 CORD receipts (49.0% correct), coverage 0.416 at risk 0.127 violates nominal alpha=0.10 in 95% of resplits. This is new because prior work assumed that per-field conditional calibration suffices; here it does not.¹

The diagnosis identifies three failure modes: document clustering (design effect 1.84–2.45), score-refit leakage, and a tie-mass pathology in which a degenerate score collapses the threshold grid (0.030 to 0.001). Each failure mode corresponds to an unstated assumption in the standard protocol: sample independence, use of the same data for scoring and threshold selection, and continuous non-degenerate scores.¹

The validity ladder organizes guarantee forms by tier. A fit/val split restores expected-selective-risk control for a learned fusion: coverage 0.318 at risk 0.096 at nominal alpha=0.10, but the realized risk exceeds alpha in 47.5% of resplits, so it is an on-average point, not a certificate. The only tier matched to documents is Mondrian Learn-then-Test with exact binomial tails, yielding per-group PAC certificates (field-iid 0.171 at risk 0.068, cluster-corrected 0.140, doc-iid 0.060), honestly near-vacuous today.¹

The constraint that moves is the target of inference: from expected selective risk over resplits to per-group certificates that hold simultaneously. Concurrently, the paper asks when conditioning pays. The pre-specified support-bin taxonomy wins every rigor tier on the sonnet CORD capture (Bonferroni-corrected p<1e-4), yet the win does not replicate on the same documents under haiku or qwen, and pooled thresholds win on higher-accuracy corpora. The evidence thus suggests conditioning helps exactly where pooled cannot certify.¹

A frozen-configuration confirmation on selection-untouched claude-haiku-4-5 held at both risk levels, and a blind three-annotator human-gold audit verifies the practical tier's accepted-set risk at 1.3% against its 10% budget (Fleiss' kappa=0.83). What would change our mind: a demonstration on multiple model families and corpora that the validity ladder transfers, or a failure of the frozen-configuration confirmation under larger scale.¹

Context

The paper is a methodological study in selective prediction, evaluated on a real document-extraction task. The dataset comprises 800 CORD receipts with fields extracted by claude-sonnet-5. The three failure modes are defined by contrast with the assumptions of the standard selective-risk estimator, and the validity ladder is the paper's organizing contribution: each rung adds a stricter guarantee form, from in-sample to PAC.¹

Caveats

The central caveat is that the primary guarantee is only expected-selective-risk, and the realized risk exceeds alpha in 47.5% of resplits; the PAC certificates, while valid, are near-vacuous. The support-bin win does not replicate on other model families, and the entire evaluation rests on a single corpus. A change of mind would require diverse replication and a certificate tier that is not vacuous in practice.¹

Source text

Hugging Face Daily Papers — Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays · July 28, 2026

Computer Science > Machine Learning

Title:Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays

Abstract:Per-field accept/review with selective risk at most alpha -- accept a field only if the error rate among accepted fields is controlled -- is the trust contract document-extraction systems need, and the natural procedure silently violates it on real documents. On 13,859 genuine claude-sonnet-5 fields from 800 CORD receipts (49.0% correct) we diagnose three failure modes: document clustering (design effect 1.84-2.45), score-refit leakage (coverage 0.416 at risk 0.127, violating alpha=0.10 in 95% of splits), and a tie-mass pathology (a degenerate score collapses the threshold grid, 0.030 to 0.001). We organize the fixes as a validity ladder, guarantee form stated per tier. A fit/val split protocol restores expected-selective-risk control for a learned fusion: coverage 0.318 at risk 0.096 at nominal alpha=0.10, no tolerance band (production variant 0.326) -- an on-average point whose realized risk exceeds alpha in 47.5% of resplits, not a certificate. Mondrian Learn-then-Test with exact binomial tails yields per-group PAC certificates: field-iid 0.171 at risk 0.068, cluster-corrected 0.140, doc-iid 0.060 -- the only tier matching documents, honestly near-vacuous today. Support-bin, the pre-specified provenance taxonomy, wins every rigor tier on the sonnet CORD capture (p<1e-4, Bonferroni-corrected) -- a win that does not replicate on the same documents under haiku or qwen -- while on higher-accuracy corpora pooled thresholds win: conditioning helps exactly where pooled cannot certify, subsumed by a learned score elsewhere. A frozen-configuration confirmation on selection-untouched claude-haiku-4-5 held at both risk levels, and a blind three-annotator human-gold audit verifies the practical tier's accepted-set risk at 1.3% against its 10% budget (Fleiss' kappa=0.83; labels err one-sidedly pessimistic). Released Apache-2.0 with seed-pinned, regression-gated procedures.

Access Paper:

Current browse context:

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The design effect of 1.84–2.45 quantifies variance inflation from document-level clustering relative to field-level independence. The unstated assumption in the standard protocol is that fields are iid; the design effect shows the effective sample size is over half smaller than the field count.

Score-refit leakage (coverage 0.416 at risk 0.127) is a data-leakage effect in which the score model is refit on the same data used for threshold selection. The paper demonstrates a violation of alpha=0.10 in 95% of resplits; that is a strong empirical demonstration but on one model-corpus pair.

The support-bin taxonomy's win at p<1e-4 Bonferroni-corrected is conditional: it holds on the sonnet CORD capture but not on haiku or qwen outputs, so assertions of its superiority are model- and corpus-specific. The evidence does not carry a general claim.

The human-gold audit's accepted-set risk of 1.3% against a 10% budget should be read with the caveat that labels err one-sidedly pessimistic, meaning human-label errors would inflate measured risk; the verified risk is therefore an upper bound in expectation.

The source text above remains the property of its publisher and is reproduced for commentary.

Hugging Face Daily Papers — Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays

July 28, 2026endistinct angle

Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays

↑ Back to top