Direct Generation Outperforms Diff-Based Editing for Flutter/Dart Code Models, Except on Localized Edits

In a single empirical study, two code models—a 100M-parameter model trained from scratch (Rainbow-Pony-100M) and a fine-tuned Qwen2.5-Coder-0.5B—were trained in both direct whole-file generation and iterative diff-based edit regimes on a shared Flutter/Dart dataset. On a held-out set of approximately 1,790 tasks per model, direct generation outperformed diff-based generation on all measured aggregate metrics, with the gap persisting after matched-ID difficulty control and on code compiling on both sides. Diff-based generation was competitive on short, spatially localized edits, with category-level wins in refactoring and error-handling/edge-case fixes, the categories with the lowest mean edit-step count; the authors term this task locality.

Hugging Face Daily Papers — Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models

September 11, 20261 source60% confidence

65Strategic

Strategic score

Evidence strength65

Technical novelty55

Commercial potential60

The study trains two code models—a 100M-parameter model from scratch (Rainbow-Pony-100M) and a fine-tuned Qwen2.5-Coder-0.5B—in two output regimes: direct generation of the entire modified file and iterative diff-based generation ('steps') that emits localized search/replace edits until completion or step-budget exhaustion. Evaluation uses a held-out set of approximately 1,790 Flutter/Dart tasks per model. The new result is that direct generation substantially outperforms diff-based generation on every aggregate metric reported: compilation/static-analysis pass rate, bits-per-byte, character-level similarity to the reference, and blinded LLM-judge ratings of goal fulfillment, correctness, and code quality.¹

Context

The diff-based regime is motivated by mirroring how developers edit code and by requiring fewer generated tokens per turn. The study's constraint shift is that this motivation does not translate into general superiority: the direct-generation gap persists after controlling for task difficulty via a matched-ID comparison and when restricting to code that compiles on both sides. Diff-based generation is competitive only on short, spatially localized edits, and its category-level wins concentrate in refactoring and error-handling/edge-case fixes.¹

Assessment

The abstract supports the broad comparative claim with a multi-metric, multi-model design and two robustness checks. However, the evidence for the proposed mechanism—'task locality'—is correlational at the category level: the two winning categories are identified as those with the lowest mean edit-step count in the dataset. The abstract does not report a controlled manipulation of locality, effect sizes, or per-category uncertainty, so the mechanism remains a plausible but not decisively isolated explanation.¹

Caveats

AI hypothesisThe conclusion would be weakened if task locality is confounded with category-specific content or if the step budget and edit-step counting favor direct generation on longer tasks for reasons unrelated to locality. It would be strengthened by an experiment that varies locality within category while holding task identity constant, by reported effect sizes and confidence intervals, and by independent replication. Because only the abstract is available, the architecture-independence claim ('single, architecture-independent mechanism') is not yet demonstrated across a broad model family.¹

Source text

Hugging Face Daily Papers — Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models · September 5, 2026

Computer Science > Software Engineering

Title:Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models

Abstract:Large language models used for code editing can be trained and deployed in at least two output regimes: direct generation, where the model emits the entire modified file in one shot, and iterative diff-based generation ("steps"), where the model emits a sequence of localized search/replace edits applied one at a time until it signals completion or a step budget is exhausted. The diff-based regime is attractive because it mirrors how developers edit code and should require far fewer generated tokens per turn. We train two code models - a 100M-parameter model trained from scratch (Rainbow-Pony-100M) and a fine-tuned Qwen2.5-Coder-0.5B - in both regimes on a shared Flutter/Dart dataset, and evaluate all four resulting models on a held-out set of approx 1,790 tasks per model. Direct generation substantially outperforms diff-based generation on every metric we measure - compilation/static-analysis pass rate, bits-per-byte, character-level similarity to the reference, and blinded LLM-judge ratings of goal fulfillment, correctness, and code quality - and the gap persists after controlling for task difficulty via a matched-ID comparison and when restricting to code that compiles on both sides. We then identify a single, architecture-independent mechanism behind the conditions where diff-based generation does win: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in our dataset. We term this task locality and discuss its implications for when an edit-based training regime is and is not the right choice for a code-editing model.

Access Paper:

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The count 'approx 1,790 tasks per model' is a per-model held-out set, not a single shared evaluation set; the abstract does not state whether the four models see identical task instances or whether the per-model sets differ. This matters for interpreting the matched-ID comparison and the compile-both restriction.

'Direct generation substantially outperforms diff-based generation on every metric we measure' is an aggregate claim; the same abstract reports that diff-based generation has category-level wins. The phrase should be read as 'every aggregate metric,' not as 'every task category.' Without per-category aggregates, the size of the overall gap is not recoverable from the abstract.

'Task locality' is introduced as a term of art defined operationally by low mean edit-step count. It is a proposed explanatory construct, not a pre-existing standard metric. The abstract equates locality with membership in two categories (refactoring and error-handling/edge-case fixes) rather than measuring locality directly per task.

The 'architecture-independent mechanism' claim is supported by only two models of different sizes and training histories (one from scratch, one fine-tuned). That is a narrow basis for independence across architectures, and the abstract does not specify how the mechanism was tested for invariance.

Blinded LLM-judge ratings are one of the four metric families, but the abstract does not report judge model, prompt, agreement, or calibration. The 'blinded' adjective addresses rater knowledge of condition, not the reliability of LLM judgments.

The source text above remains the property of its publisher and is reproduced for commentary.

Hugging Face Daily Papers — Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models

September 5, 2026endistinct angle

Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models

↑ Back to top