Motion-Omni Unifies Spoken Dialogue and Full-Body Motion in One End-to-End Model

Motion-Omni is an end-to-end framework that makes a spoken dialogue model also output facial, hand, upper-body, and lower-body motion directly from the hidden states used to generate speech, eliminating the separate motion-model inference pass of cascade systems. Instantiated with Qwen2.5-7B, it reports RTF=0.78 (faster than real time), a 5.4x latency reduction versus the same-audio teacher cascade, within-2% reference-free motion agreement, and a 2.62% word error rate.

Hugging Face Daily Papers — Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

September 8, 20261 source70% confidence

75Strategic

Strategic score

Evidence strength55

Technical novelty80

Commercial potential75

Motion-Omni replaces the usual two-stage pipeline for speaking avatars by embedding full-body motion generation directly inside an end-to-end spoken-dialogue model. Rather than feeding completed audio to a separate co-speech motion model, it emits facial expressions and hand, upper-body, and lower-body motion from the same hidden states that generate the spoken reply. The published Qwen2.5-7B instantiation reports RTF=0.78, is 5.4x faster than the same-audio teacher cascade, matches that cascade within 2% on reference-free motion metrics, and reaches a 2.62% word error rate.¹

Context

Previously, conversational content and body movement were solved by separate model families: spoken dialogue models output no motion, while co-speech motion models require an already generated audio track. The standard remedy is a cascade that first synthesizes speech and then runs a motion model over the finished audio, which imposes a second full inference pass and prevents any joint optimization between linguistic content and physical gesture. Motion-Omni is positioned not as a better cascade but as a single joint model for both abilities.¹

Assessment

The key technical claim is that joint training is necessary: freezing the speech pathway leaves motion misaligned, while co-adapting the LLM, Speech Generator, and Motion Generator under both objectives restores alignment without collapsing spoken-dialogue ability. If true, this demonstrates that the latency and quality limitations of cascades are not inherent to the task but result from decoupling decision and motion. The abstract, however, does not quantify the frozen-speech misalignment nor define what level of alignment was considered acceptable.¹

Caveats

AI hypothesisAll performance figures are self-reported in an arXiv abstract without an accompanying study to inspect. The definitive test would be a stronger teacher plus a frozen speech backbone: if that combination matches or beats Motion-Omni-Q7 on human evaluation, the necessity of joint training would be weakened. In that case the reported 2% reference-free gap would also have to be shown perceptually irrelevant before concluding that the cascade constraint has truly been removed.¹

Source text

Hugging Face Daily Papers — Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue · August 28, 2026

Computer Science > Sound

Title:Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

Abstract:An avatar that holds a conversation should decide what to say and to move while saying it, yet these abilities live in separate model families: spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them. The standard remedy is a cascade that first generates the spoken response and then runs a motion model over the finished audio, which requires a second full inference pass and precludes any joint optimisation between the two. We present Motion-Omni, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression together with hand, upper-body and lower-body motion, generated directly from the hidden states that produce the speech. Joint training is not optional here: with the speech pathway frozen, motion remains misaligned with the audio, and co-adapting the LLM, Speech Generator and Motion Generator under both objectives is what recovers alignment while retaining spoken-dialogue ability. Supervision comes from a scalable, model-agnostic pipeline that pseudo-labels consistent-voice speech responses with a replaceable motion teacher, yielding 422,856 quality-ranked pairs (1,402 hours). We further release SwDA-500 and, to our knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems while unifying rendering, automatic metrics, human evaluation, and latency measurement. Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding 5.4 x faster (RTF=0.78, faster than real time), surpasses all non-teacher cascades on beat correlation and diversity, and reaches a 2.62% word error rate, the lowest among the omni-modal systems compared.

Current browse context:

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

RTF is the real-time factor: compute time divided by generated audio duration, so RTF=0.78 means the entire speech-plus-motion system produces output faster than the audio would play. The reported 5.4x speedup is the ratio of the teacher cascade's total inference time to Motion-Omni's, so it depends on the unstated choice of speech synthesizer and motion teacher used in the cascade.

The 'same-audio teacher cascade' is an evaluation setup in which the teacher model that pseudo-labeled the training data is given the same speech audio generated by Motion-Omni, producing reference motion from identical audio. This isolates motion quality from speech quality, but the within-2% agreement measures how closely the student reproduces the teacher's motion distribution; it does not by itself establish naturalness to human observers because reference-free metrics are not specified.

The 422,856 quality-ranked pairs from 1,402 hours of pseudo-labeled speech responses are not a simple collected corpus: the word 'ranked' implies a filtering or ordering step, but the abstract does not state how many raw candidates were generated, how many survived, or how ranking quality was measured. The average pair length implied by these numbers is about 11.9 seconds, which matters if the evaluation protocol is intended to measure sustained full-body conversational behavior.

The source text above remains the property of its publisher and is reproduced for commentary.

Hugging Face Daily Papers — Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

August 28, 2026endistinct angle

Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

↑ Back to top