We summarize a single arXiv paper proposing EASEL, a benchmark for dexterous visual tool use where agents incrementally paint a canvas to match a reference image. The core new contribution is not another QA set but a closed-loop environment in which the model's parameterized visual actions directly control the final result, and a 440k-sample two-stage curriculum dataset (EASEL-Data) plus EASEL-9B used to test whether dedicated trajectory supervision improves the capability. Across 25 models, reconstruction similarity bottlenecks at 0.40-0.54 and trajectory diagnostics show early saturation or post-peak degradation.1
Context
The paper positions EASEL against existing agentic benchmarks that emphasize web navigation, GUI operation, and software engineering; it argues these rarely require fine-grained, closed-loop parameterized visual action tied to visual evidence. The proxy task—reference-guided visual reconstruction—is a controlled instance: each paint action changes the canvas, so the model must continuously re-evaluate visual state. This selection makes the benchmark sensitive to closed-loop stability rather than hold-out accuracy alone.1
Assessment
The headline results indicate that adding trajectory supervision helps only marginally: EASEL-9B improves by a relative 6.3% over its base model but still ranks third, and all models fall far short on reconstruction similarity. The closed-loop instability finding—saturation early or degradation after peak—suggests the binding constraint is not single-step perception but iterative error recovery under self-produced visual states. If this is robust, it would change the field's priorities from static vision benchmarks toward agentic closed-loop training.1
Caveats
AI hypothesisThese conclusions rest on an abstract; the exact metric definitions, curriculum structure, and model families are not reported in the available text. What would change my mind: if the trajectory diagnostics are shown to be an artifact of the paint simulator (e.g., action discretization) rather than the model's closed-loop ability, or if the 6.3% improvement is not statistically significant or limited to a subset of tasks. Without these details, the benchmark's generalizability beyond painting remains plausible but unproven.1
