SWE-bench Science: From Aggregate Success to Failure Mechanisms in Scientific Coding Agents

SWE-bench Science introduces a repository-level benchmark of 119 tasks from 98 repositories across 20 scientific domains. The best evaluated agent, Claude Code with Opus-5 (max), achieves pass@1 below 50%. Four recurring failure mechanisms are identified, and a paired ablation shows that scientific guidance is beneficial when well-grounded but can induce anchoring when poorly aligned.

August 22, 20261 source60% confidence

68Strategic

Strategic score

Evidence strength68

Technical novelty72

Commercial potential55

SWE-bench Science marks a shift from asking whether coding agents can fix software in the abstract to asking why they fail when the code participates in scientific evidence. Its contribution is not another leaderboard: the benchmark isolates three task paradigms, provides a failure taxonomy, and reports an ablation that treats scientific guidance as a variable rather than an always-positive input. The most consequential constraint change is evaluative: the authors move from aggregate resolution rates to the conditions under which domain knowledge helps, and they explicitly find that it can hurt when poorly aligned. I would change my view if the best agent approached or exceeded 50% pass@1, or if the ablation showed that scientific guidance improves exact repair success uniformly; either result would contradict the reported conditional picture.¹

Context

Existing evaluations of coding agents largely emphasize aggregate task success, which gives little insight into why agents fail when repairing scientific software. This benchmark is organized around Issue-driven, Expert-exploratory, and Engineering-integration paradigms, moving beyond conventional bug-fixing to tasks that require scientific exploration or system-level integration.¹

Assessment

The reported best pass@1 is below 50%, with four recurring failure mechanisms: deficits in scientific knowledge or abstraction; misguided exploration or surface-level repair; incomplete repair coverage or system integration; and failures to generalize beyond observed cases. The paired ablation strengthens the assessment: removing explicit scientific guidance while preserving executable context shows that well-grounded scientific information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance induces anchoring and does not necessarily improve exact repair success.¹

Caveats

The evidence is currently limited to the abstract-level description; no task distribution or per-paradigm breakdown is available, and the ablation's effect size is not reported. Consequently, the four failure mechanisms may overlap in individual instances, and the claim that scientific knowledge 'can constrain' repair is a directional average rather than a proven causal mechanism across all 119 tasks.¹

Source text

Hugging Face Daily Papers — SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science? · August 20, 2026

Computer Science > Computation and Language

Title:SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?

Abstract:Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce \textbf{SWE-bench Science}, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, \textbf{Claude Code with Opus-5 (max), achieves a pass@1 below 50\%}, highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.

Access Paper:

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The source text above remains the property of its publisher and is reproduced for commentary.

↑ Back to top