ClawProBench: Trace-Aware Runtime-Native Agent Evaluation Finds Weak Alignment Between Live and Frozen Holdout Rankings

ClawProBench evaluates 68 agent configurations on a 102-scenario live OpenClaw runtime and 37 on a frozen 68-scenario holdout, scoring execution traces via a safety-gated formula. Native-runtime tasks score lower than workspace-live tasks, pass@k-any exceeds strict three-trial pass, and full-profile/holdout rankings align weakly (Spearman 0.13).

Hugging Face Daily Papers — ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts

August 26, 20261 source70% confidence

69Strategic

Strategic score

Evidence strength58

Technical novelty78

Commercial potential55

ClawProBench proposes that agent benchmarks should score a declared model-plus-runtime configuration from execution traces rather than final answers. The new object of measurement is the configuration, with failures localized to evidence acquisition, runtime routing, safety boundaries, or repeated execution. The benchmark is instantiated on OpenClaw, a live agent runtime exposing workspace tools and surfaces for browsing, memory, messaging, scheduling, skills, and subagents.¹

Context

ClawProBench defines a 102-scenario full profile with live workspace and native-runtime routing tasks, plus a 68-scenario frozen holdout with closed-world JSON output contracts. Scoring is applied to traces with a safety-gated formula combining correctness, process quality, and efficiency, and failure evidence is preserved for audit. This shifts the constraint from validating final answers to auditing the whole execution path under declared safety limits.¹

Assessment

The reported results quantify the gap between evaluation regimes. The top safety-gated average trace score is 0.7671 across 68 configurations on the full profile; native-runtime tasks average 0.5238 versus 0.6415 for workspace-live tasks. On the frozen holdout, pass@k-any reaches 0.6638 while strict three-trial pass is 0.2890, indicating that one-off successes are common. Full-profile and holdout ranks align only weakly (Spearman 0.1300), so live open-world ranking does not transfer to frozen closed-world ranking.¹

Caveats

The abstract does not disclose the safety-gate definition, scoring weights, task composition, or inter-run variance, so the quantitative findings should be read as existence proofs rather than calibrated comparisons. The weak cross-track alignment might stem from the format difference (live workspace vs JSON contracts) rather than from a general failure of live evaluation. The claim would be overturned by evidence that final-answer leaderboards correlate strongly with trace-aware rankings, or that safety-gated scoring does not change configuration ranks.¹

Source text

Hugging Face Daily Papers — ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts · August 23, 2026

Computer Science > Artificial Intelligence

Title:ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts

Abstract:Agent benchmarks often evaluate only final answers even when agents run on stateful runtimes. We argue this under-specifies what is being evaluated: the proper unit is a declared model-plus-runtime configuration whose failures can occur in evidence acquisition, runtime routing, safety boundaries, or repeated execution. We present ClawProBench, a trace-aware benchmark for runtime-native agent evaluation instantiated on OpenClaw, a live agent runtime with workspace tools and native surfaces for browsing, memory, messaging, scheduling, skills, and subagents. ClawProBench defines two tracks: a 102-scenario full profile with live workspace and native-runtime routing tasks, and a frozen 68-scenario holdout with closed-world JSON output contracts for robust ranking. Trials are scored from execution traces via a safety-gated formula combining correctness, process quality, and efficiency, preserving failure evidence for audit. Our anonymous artifact includes benchmark definitions, scoring code, manifests and sanitized traces. We evaluate 68 configurations on the full profile and 37 on holdout. The top safety-gated average trace score is 0.7671. Native-runtime tasks underperform workspace-live tasks (0.5238 vs. 0.6415). On holdout, pass@k-any outperforms strict three-trial pass (0.6638 vs. 0.2890), while full-profile and holdout rankings show weak alignment (Spearman 0.1300). Rankings based purely on correctness differ substantially from process-aware, safety-gated and strict-pass views. Final-answer leaderboards may hide native-surface weaknesses, one-off successes and trace-local agent failure modes.

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The phrase 'model-plus-runtime configuration' is not a rhetorical broadening; it makes runtime version and tool surface part of the score. Consequently, a benchmark result cannot be cited as a model property without stating the runtime, and comparisons across runtimes require shared task contracts.

A 'safety-gated formula' is a scoring policy, not a metric: some trace outcomes are filtered or down-weighted when safety boundaries are violated. The reader should not interpret 0.7671 as raw task accuracy.

The pass@k-any vs strict-three-trial comparison is not apples-to-apples: any success in k runs counts in the former, while the latter demands three consecutive successes. The large spread implies heavy run-to-run variance and sparse successes, which is a known failure mode in tool-using agents.

Spearman 0.1300 is cited as weak alignment, but the two tracks differ in both scenario set and response contract. The abstract treats this as evidence about live versus frozen evaluation; the design conflates task domain with format.

The source text above remains the property of its publisher and is reproduced for commentary.

Hugging Face Daily Papers — ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts

August 23, 2026endistinct angle

ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts

↑ Back to top