StarHarness introduces a distinct optimization axis for tool-using agents: instead of adapting model weights or hand-crafting prompts per task, it evolves the agent's harness—prompt framing, tool interfaces, MCP-backed providers, subagent structure, and loop configuration—against a stratified pool of tasks. This is new relative to prior work that treats the harness as fixed and the model as the only tunable element; here the model is deliberately frozen.1
Context
The central constraint shift is that the search space moves from parameters to programmatic and textual scaffolding. The paper also imposes a methodological constraint: proposer-visible search tasks are separated from proposer-hidden selection tasks, and held-out tasks measure generalization. This changes what counts as evidence of improvement, since performance gains are judged on tasks never used during evolution.1
Assessment
The reported gains of 20–35 percentage points over the default harness after 4–12 accepted changes per environment are substantial, and the claim that they persist on excluded tasks and transfer across GPT and Qwen model families is the strongest evidence for harness-level rather than model-specific shortcuts. What would change my mind is evidence that these gains depend on careful per-environment acceptance criteria, that the baselines are unusually weak, or that transfer requires re-evolution outside the two model families tested.1
Caveats
Caveats: the analysis rests on a single abstract, so evaluation protocols, selection-task construction, and the definition of an 'accepted change' are not independently verifiable. The trace analysis attributing gains to interface repairs and compressed search is described only at a high level.1
