FIRM-Video: Check-before-Score Reward Modeling for Text-to-Video

FIRM-Video introduces a check-before-score principle for training text-to-video reward models: instead of asking a judge to score holistically, it builds dimension-specific checklists, verifies each criterion against temporal visual evidence, and aggregates only verified decisions. The paper reports a unified construction covering instruction following, world coherence, and perceptual quality, and introduces a 90K training set and a 750-annotation benchmark. The headline result is an 8B Qwen3-VL-based reward model with the best reported MAE on FIRM-Video-Bench and the highest VBench scores in Best-of-8 sampling across three generators.¹

Context

Prior text-to-video reward models are said to rely on holistic judges with fixed rubrics or open-ended reasoning, which the authors diagnose as incomplete inspection, unfaithful justification, and entangled attribution. FIRM-Video moves the binding constraint from judge architecture or prompting strategy to checklist construction and verification: instruction-following prompts are decomposed into weighted atomic requirements, world-coherence checks are calibrated to visible entities and actions, and perceptual quality follows a generic defect taxonomy. The reported design makes verification before scoring the methodological bottleneck, which is the main constraint shift claimed by this work.¹

Assessment

The contribution appears to be a supervision-quality intervention rather than a novel reward-model architecture. Its plausibility rests on the claim that per-criterion verification yields cleaner training labels than holistic scoring; the accompanying 90K instance dataset and 750-point benchmark provide concrete artifacts for testing that claim. However, the abstract alone does not establish how verification is automated, whether human raters adjudicate criteria, or how the natural-language analyses are generated from verified scores.¹

Caveats

AI hypothesisThis assessment depends on assumptions that the full paper has not yet confirmed. If checklist verification is largely human-supervised, the efficiency advantage over holistic judges would be much smaller. If FIRM-Video-Bench annotations or the VBench Best-of-8 protocol are tightly coupled to the training data, the reported performance advantage could be overfit. A full comparison table with baselines, variance, and significance tests would be needed to confirm the 'best' claims.¹