Thinkingbox: Benchmarking Reliability in Stateful Business Workflows

Thinkingbox-bench evaluates stateful business-workflow agents with executable terminal-state checks; the strongest model scores 65.36% pass@1 but only 25.25% pass^20, evidencing a gap between one-try success and reliable completion.

Hugging Face Daily Papers — One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows

August 26, 20261 source68% confidence

72Strategic

Strategic score

Evidence strength45

Technical novelty80

Commercial potential65

Thinkingbox is a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state; Thinkingbox-bench contains 507 policy-conditioned workflows across retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support.¹

In this benchmark, task-specific executable checks accept valid trajectories while rejecting wrong, missing, or extra effects, and designated tasks also check final response properties. The strongest evaluated model reaches 65.36% pass@1 but only 25.25% pass^20.¹

Context

The central constraint shift is from one-off success to repeated success under sampling: pass^20 measures whether at least one of twenty attempts succeeds, while the terminal-state checks penalize collateral effects that output-level metrics usually ignore. This makes reliability measurable in settings where a plausible answer or valid-looking tool call is insufficient.¹

Assessment

The reported 65.36% pass@1 versus 25.25% pass^20 supports the title's thesis that one success is not reliability, especially because many failed trials show clean termination and valid state-changing actions. The evidence is suggestive rather than conclusive: pass^20 is a weak bound, and the abstract does not report whether some tasks are systematically unsolvable by the tested models.¹

Caveats

AI hypothesisThe results come from a single arXiv abstract without peer review, model details, or error analysis. The reliability claim would be strengthened by showing that models with high pass@1 still produce wrong terminal state on repeated runs and that the state-effect checks reorder model rankings; it would be weakened if tool-call validity strongly predicted state correctness or if pass^20 approached pass@1 with more capable models.¹

Source text

Hugging Face Daily Papers — One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows · August 20, 2026

Computer Science > Computation and Language

Title:One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows

Abstract:Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, Thinkingbox-bench contains 507 policy-conditioned workflows across numerous scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Across proprietary and open-weight models, the strongest achieves 65.36% pass@1, but only 25.25% pass^20. Moreover, many failed trials show clean termination and valid state-changing actions, showing that response or tool-call-level signals are not clear proxies for end-to-end task completion. Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both Thinkingbox and Thinkingbox-Bench: this https URL

Access Paper:

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

This sentence uses pass^20, the pass@k convention from code generation, denoting the probability that at least one of 20 independently sampled trajectories passes. It is therefore not a per-attempt reliability measure; a task counted as passed by pass^20 can still fail on 19 of 20 tries. The abstract's inference from pass^20 to 'reliability' rests on equating finding-a-success with reliable completion.

The evaluation 'checks that accept valid trajectories while rejecting wrong, missing, or extra effects' is a distinctive design choice: rejecting extra effects means penalizing collateral state mutations even when the requested outcome is achieved, which is stronger than typical output or tool-call validation. This is what lets the benchmark distinguish end-to-end task completion from plausible but sloppy agent behavior.

The observation that failed trials show 'clean termination and valid state-changing actions' contains the paper's key evidence for the proxy-failure claim. The abstract does not provide rates, so the frequency of such failures is unknown; without it, the claim that response/tool-call signals are 'not clear proxies' is supported only qualitatively.

The source text above remains the property of its publisher and is reproduced for commentary.

Hugging Face Daily Papers — One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows

August 20, 2026endistinct angle

One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows

↑ Back to top