OmniAssistBench: Constraining Open Interaction for Omni-LLM Evaluation

OmniAssistBench converts open-ended assistant-style video interactions into fixed-route tasks using source-video priors, addressing the dynamic-path problem that static offline benchmarks cannot evaluate. On it, Gemini-3-Pro scores 66.4/100 and Qwen3-Omni-Instruct 51.2/100, highlighting failures in visual prompts, multi-turn context, and temporally delayed responses.

August 25, 20261 source75% confidence

72Strategic

Strategic score

Evidence strength65

Technical novelty85

Commercial potential55

OmniAssistBench is new not because it adds another video benchmark, but because it forces an interactive assistant to follow the exact solution path encoded in source video, rather than allowing any successful route. The old constraint was static offline evaluation: it could check final captions or QA answers, but could not respond to a model that changes the user's next action. The new constraint is a fixed, prior-injected interaction graph, which makes dynamic interaction testable offline.¹

Context

The authors reverse-engineer existing web videos into multi-turn clips, deduce user goals, and require assisted guidance along exactly the same route. Construction consumed over 1000 expert hours. Reported scores are 66.4 for Gemini-3-Pro and 51.2 for Qwen3-Omni-Instruct on a 100-point scale.¹

Assessment

The scores are useful as empirical anchors, but the decisive evidence lies in the error analysis: loss of visual prompt comprehension, lost history over turns, and premature responding. These three failure modes are consistent with models that understand semantics but cannot align to the assistant role's timing and referential constraints. The abstract does not provide per-category scores, so those failures are not yet precisely quantified.¹

Caveats

AI hypothesisThe design's main risk is construct validity: forcing one predefined solution path may conflate goal planning with route compliance. If variations in path optimality or multiple equally valid instructions are penalized, benchmark scores could underestimate assistant usefulness. What would change my mind is evidence that fixed-route scores predict open-ended deployment success, or a released dataset allowing the constraint to be relaxed.¹

Source text

Hugging Face Daily Papers — OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs · August 21, 2026

Computer Science > Computer Vision and Pattern Recognition

Title:OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs

Abstract:Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and guide users to achieve specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help. Evaluating this is rather challenging, as the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, we introduce OmniAssistBench. To solve the issue of diverging interaction paths where the same user goal can be achieved through various methods, we provide models with predefined priors derived from the source video, requiring them to guide users along the exact same routes. Since real interaction videos are rare, we construct the dataset by reverse-engineering existing Internet videos. We deduce logical user goals and segment the videos into multi-turn clips to simulate continuous interactions. This rigorous pipeline required over 1000 expert person-hours to build the dataset. Results show that the proprietary Gemini-3-Pro reaches 66.4 out of the max point of 100, while the open-source Qwen3-Omni-Instruct achieves 51.2. Although current models generally understand user inputs, they frequently provide incorrect or incomplete answers. Specifically, they struggle with visual prompts (e.g., hand gestures), fail to maintain historical context during multi-turn interactions, and fail to delay response until the target event. Results indicate substantial room for improvement before models can become reliable assistants.

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The source text above remains the property of its publisher and is reproduced for commentary.

↑ Back to top