MOSS-VL: Making Real-Time Interaction a First-Class Capability in an Open Vision-Language Model

MOSS-VL is an open vision-language model family that treats real-time interaction as a first-class capability, using gated cross-attention so the decoder can perceive while speaking. It reports leading streaming benchmark results, including 66.0 vs. 37.5 on OmniMMI Proactive Alerting, and widens the time-to-first-token advantage over same-backbone Qwen3-VL-8B from 2.8x to 5.1x as visual context grows. Five checkpoints, the training curriculum, and real-time inference code are released.

August 20, 20261 source62% confidence

74Strategic

Strategic score

Evidence strength45

Technical novelty84

Commercial potential77

MOSS-VL is an open vision-language model family that makes real-time interaction — perceiving while speaking — a first-class capability rather than an add-on. The design removes the usual turn-taking constraint: the language decoder attends to visual input only through gated cross-attention, so newly arriving frames can be consumed while text is generated. The authors report that this widens the time-to-first-token advantage over a same-backbone Qwen3-VL-8B from 2.8x to 5.1x as visual context grows.¹

Context

The work addresses a known limitation of streaming vision-language models, which typically alternate between perception and generation or buffer visual tokens inside the decoded sequence. By keeping visual tokens outside the decoded sequence and adding a synthesized interaction corpus to supervise when to speak, stay silent, or revise, MOSS-VL shifts the constraint from sequential turn-taking to concurrent perception-and-generation. This is a stack-level co-design: architecture, data, and curriculum are all shaped for real-time interaction.¹

Assessment

The strongest reported evidence is MOSS-VL-Realtime leading three of four streaming benchmarks and scoring 66.0 versus 37.5 for the best baseline on the OmniMMI Proactive Alerting subset, which squarely tests proactive behavior; the offline model also leads temporal-reasoning video sets at comparable scale. However, the abstract does not specify the metric, dataset splits, or baseline list, so the magnitude of these gains cannot be fully verified from this document alone.¹

Caveats

AI hypothesisThe release of five checkpoints, the training curriculum, and real-time inference code makes the claims independently checkable. But this is a technical report, not a peer-reviewed paper, and central details — gating mechanism specifics, corpus construction, and the evaluation protocol — are absent from the abstract. The assessment would change if the open evaluation code shows smaller margins against non-proactive baselines, or if the gated cross-attention design fails to generalize beyond the five released checkpoints.¹

Source text

Hugging Face Daily Papers — MOSS-VL Technical Report · August 15, 2026

Computer Science > Computer Vision and Pattern Recognition

Title:MOSS-VL Technical Report

Abstract:We present MOSS-VL, an open vision-language model family that treats real-time interaction -- perceiving while it speaks -- as a first-class capability. It is co-designed across the stack: the language decoder attends to vision only through gated cross-attention, so the model can naturally see incoming frames while generating; a synthesized interaction corpus supervises when to speak, when to stay silent, and when to revise; and a staged curriculum concentrates all real-time-specific training in one light final stage over a strong offline foundation. Offline, MOSS-VL-Instruct is competitive at comparable scale and leads temporal-reasoning video sets. Across four streaming benchmarks, MOSS-VL-Realtime posts the best average on three (second on the fourth) among open-source streaming models, sweeping the three subsets that squarely test proactive behavior -- 66.0 vs. 37.5 for the best baseline on OmniMMI Proactive Alerting. With 11.3B parameters but visual tokens outside the decoded sequence, MOSS-VL widens its time-to-first-token advantage over same-backbone Qwen3-VL-8B from 2.8x to 5.1x as visual context grows. We release all five checkpoints, the training curriculum, and the real-time inference code at this https URL.

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

'Gated cross-attention' is a term of art: it implies that visual information enters the decoder only through gating functions on cross-attention, so visual tokens are not part of the autoregressively decoded sequence. The unstated assumption is that new frames can update these gated representations without recomputing past token embeddings, which is the mechanism behind the reported TTFT scaling.

The abstract gives no metric for OmniMMI Proactive Alerting. The 66.0 vs. 37.5 comparison is reported as a single unbracketed pair of numbers, likely a task-specific accuracy or F1, but without the metric and baseline list the 28.5-point margin cannot be interpreted from the abstract alone.

Time-to-first-token is not directly measured in the abstract; the 2.8x-to-5.1x scaling is a derived comparison with Qwen3-VL-8B under growing visual context. The stated basis is that visual tokens outside the decoded sequence avoid prefix re-encoding per new frame, but the exact benchmark configuration and hardware are left unspecified.

The source text above remains the property of its publisher and is reproduced for commentary.

↑ Back to top