VGI-Bench: Structured Evaluation of Visual Reasoning in Video Generation Models

VGI-bench introduces a 27-task, 810-instance benchmark for evaluating zero-shot visual reasoning in video generation models. Current models, including Seedance 2.0, reach at most 51.0% accuracy under criteria emphasizing valid evolving processes and calibrated difficulty. The paper documents limited self-correction in denoising and analyzes failure modes, input-condition sensitivity, and transfer from synthetic fine-tuning.

August 27, 20261 source60% confidence

65Strategic

Strategic score

Evidence strength32

Technical novelty78

Commercial potential44

VGI-bench is a new evaluation instrument for visual intelligence in video generation models. Its contribution is not a new generation model but a benchmark with 27 tasks and 810 instances, constructed around three design requirements: inputs must align with the visual priors of current video models, tasks must require valid evolving processes rather than only a plausible final state, and difficulty must be calibrated to remain challenging yet partly feasible. This moves the evaluation constraint from static output quality to process validity and task calibration.¹

Context

The source reports that current generative systems solve only a subset of visually grounded reasoning tasks; the strongest tested model, Seedance 2.0, achieves 51.0% under VGI-bench's evaluation criteria. The benchmark is organized by a two-level taxonomy of task domains and skill tags.¹

Assessment

The paper's analysis spans four dimensions: output failure modes, input condition sensitivity, transfer boundary from synthetic fine-tuning, and internal denoising behavior. The most consequential finding is limited self-correction during denoising: later steps mostly refine early hypotheses rather than correct reasoning errors. If robust, this mechanism-level explanation identifies a target for architectural change, not just for data scaling.¹

Caveats

The evidence is an abstract only. The 51.0% figure is relative to unspecified scoring criteria, and the definition of 'valid evolving process' is not formalized. The self-correction claim is presented without quantitative support in the abstract. Changes to my assessment would follow from seeing the task-level breakdown, the exact grading protocol, and evidence that denoising trajectory analysis controls for alternative explanations.¹

Source text

Hugging Face Daily Papers — VGI-BENCH: Probing Visual Intelligence in Video Generation Models · August 20, 2026

Computer Science > Computer Vision and Pattern Recognition

Title:VGI-Bench: Probing Visual Intelligence in Video Generation Models

Abstract:Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags for fine-grained evaluation of visual reasoning capabilities of video generation models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our analysis further explore the output failure modes, input condition sensitivity, performance transfer boundary from synthetic fine-tuning, and internal denoising perspective revealing limited self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-bench will help stimulate the development of next-generation video generation models. Website: this https URL

Access Paper:

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The phrase 'inputs aligned with the visual priors of current video models' assumes a shared input distribution for video generation models. The abstract does not specify how alignment is measured or verified, leaving open the possibility that poor scores reflect input mismatch rather than reasoning failure.

The '51.0% under our evaluation criteria' result should be read as a composite pass rate across 27 tasks with strict process-validity requirements, not as ordinary top-1 accuracy. The denominator (per-instance vs per-task) and tolerance for admissible evolving processes are not given in the abstract.

The self-correction limitation is inferred from denoising-step analysis, but the abstract does not report the measure or the comparison baseline. It is a diagnostic hypothesis: later steps optimize the already-formed initial guess rather than revisit it. The claim would need intervention experiments to establish causality.

The source text above remains the property of its publisher and is reproduced for commentary.

↑ Back to top