QuoteBench: Isolating the Generation–Transport Boundary in Command-Path Failures

QuoteBench scores 56 one-shot Bash command-issuing tasks across eight execution-transport configurations and shows that matched execution scores conceal large opposing generation and boundary-adaptation errors. Raw generation is nearly saturated at the frontier, while boundary adaptation separates models, and the paper argues that command-agent evaluation must report configuration, generation contract, execution path, operating point, and final-state validator.

Hugging Face Daily Papers — QuoteBench: How Matched Scores Can Hide Command-Path Failures

August 22, 20261 source65% confidence

68Strategic

Strategic score

Evidence strength48

Technical novelty82

Commercial potential55

QuoteBench tests whether a matched command-execution score can be treated as a property of an LLM coding agent. It assembles 56 one-shot Bash tasks derived from 14 incident families and crosses each model's raw generation with an execution transport that deliberately adds one unescaped parser before replay. The central result is not a new state-of-the-art number but a decomposition: across eight configurations, replaying identical replies through the added parser lowers success by 55.4–73.2 percentage points, while disclosing the boundary recovers 30.4–60.7 points in six of the eight configurations. The benchmark's apparently small 'matched gap' for a frontier model can therefore hide two much larger opposing terms.¹

Context

Prior practice typically reports a matched execution score as if it measured the model alone. QuoteBench's move is to make the generation contract and execution transport explicit and to measure their interaction. The deliberately unescaped added parser is not a naturalistic deployment bug but a controlled instrument: escaping at the interpolation point reproduces the raw-path outcome, so any recovery when the boundary is disclosed must come from changed generation rather than from transport noise. That changes the constraint from 'maximize matched success on an assumed interface' to 'specify the interface and the operating point, and score both raw generation and boundary adaptation.'¹

Assessment

The claim that boundary adaptation, not raw generation, separates frontier agents is the load-bearing result. It would be weakened if the disclosed-boundary recovery were attributable to task memorization or if the 56 tasks, grouped into 14 incident families, lacked sufficient independence to support the reversal analysis. The evidence for a configuration-induced model reversal rests on one unambiguous reversal among 26 comparable pairs, with four more on single-task margins; more pairs and larger task pools would be needed to treat reordering as a robust finding.¹

Caveats

The source is a single arXiv abstract; no code, data, or full evaluation protocol is visible in the excerpt. The reported percentages are derived from an undisclosed model set and configuration definitions, and the 'same-window' qualifier means the comparisons hold within a single training-data or evaluation window. The authors' prescriptive conclusion—that command-agent evaluations should report configuration, generation contract, execution path, operating point, and final-state validator—is a reporting standard proposed from one benchmark, not yet a consensus practice.¹

Source text

Hugging Face Daily Papers — QuoteBench: How Matched Scores Can Hide Command-Path Failures · August 13, 2026

Computer Science > Artificial Intelligence

Title:QuoteBench: How Matched Scores Can Hide Command-Path Failures

Abstract:LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.

Access Paper:

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The abstract's phrase 'GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation' needs decoding. The matched gap is the net difference in success between the model's own raw-path output and the same output after the added parser at the default operating point; -64.3 measures the success drop caused by replaying the reply through the altered transport, and +60.7 measures the recovery obtained when the boundary is disclosed, i.e., the model can generate commands that survive the parser. Only the net -3.6 would appear in a conventional matched-score comparison.

The number 56 one-shot tasks from 14 incident-derived families matters because the paper's reversal claim is based on 26 comparable model pairs and one unambiguous reversal. With only 56 tasks, a single-task margin can determine pair order, so the effective sample size behind the reorder finding is small.

The phrase 'deliberately unescaped added parser' identifies the experimental manipulation: a synthetic execution transport, not a bug in a real shell, that re-parses generated commands at an interpolation point. The control—escaping at the interpolation point reproduces the raw-path outcome—is what lets the authors attribute success differences to the parser rather than to the model's raw output.

The source text above remains the property of its publisher and is reproduced for commentary.

Hugging Face Daily Papers — QuoteBench: How Matched Scores Can Hide Command-Path Failures

August 13, 2026endistinct angle

QuoteBench: How Matched Scores Can Hide Command-Path Failures

↑ Back to top