Harness-Aware Training Shifts the Update Constraint from Weights to Runtime Harness

This report introduces Harness-Aware Training (HAT), a three-stage procedure that trains compact streamer-agent models to remain effective when the surrounding harness—Skills, Hooks, prompts, and tools—is modified without touching model weights. The mechanism called Harness-State Augmentation (HSA) creates task-preserving variations of harness components so the model learns from strong-model trajectories across many valid configurations. The target problem is a latency/adaptability trade-off: large models can zero-shot use a new harness but miss latency targets on the deployment GPU, while compact models hit latency but overfit their original harness.¹

The report claims HAT achieves 94.8 on Live-Stream QA (base model: 80.3; strongest general LLM: 93.0) and 94.6 on Harness-Variant QA (base: 75.4). It also states that fixed-harness SFT lowers IFEval by 7.7 points, whereas HAT reaches 83.5 without that regression, and that the Taobao Live deployment yielded positive online A/B results for GMV and item-page views.¹

The material constraint moved by this work is the unit of iterative control: strategy updates can now be shipped as harness edits to a compact model that has been trained to follow those edits, instead of as weight updates. The claim would weaken if independent evaluation found that a latency-equated general LLM matches HAT on the same harness variants, or if the online A/B effect was not reproducible outside the described deployment.¹

Context

This work is set in live-streaming commerce, where agents answer product questions and execute marketing strategies in real time. The paper treats the harness as the deployable surface: Skills, Hooks, prompts, and tools can be updated independently of model weights, enabling rapid iteration. On one NVIDIA H20 GPU, the optimized system reports P50 and P95 latencies of 3.4 s and 8.1 s, the latency envelope that the compact model is meant to satisfy.¹

Assessment

The central evidence is a set of point estimates from four evaluation sets whose construction is not described in the abstract. The comparison against the 'strongest general LLM' is a single number with no variance, so the reported margin is not clearly reliable. The Harness-Variant QA score is the most direct evidence for the method's core claim, but without information on how far the variants are from the training distribution, the generalization claim is incompletely supported. The online A/B result is reported only as 'positive', so it cannot serve as a precise commercial validator.¹

Caveats

Only the abstract is available: no code, weights, training hyperparameters, evaluation splits, or statistical details are provided. The latency figures are tied to a specific optimized system on one H20 GPU. The causal attribution of the online A/B improvements to HAT, as opposed to other system components in Taobao Live, is not established by this document alone.¹