VibeWorlding Introduces Benchmark and RL Gym for 3D Open-World Agents; RL-Trained Open-Source Models Achieve Best Pass@1
An arXiv preprint introduces VibeWorlding, a framework for benchmarking and training multimodal agents that construct interactive 3D open worlds from user queries. It includes VWE-BENCH (2,616 assets, 323 seed worlds, 6,828 reverse-synthesized queries) and VibeWorlding-Gym, a multimodal RL post-training environment with MCP tools and rubric-based verification. In experiments, frontier MLLMs GPT-5.5 and Qwen3.8-Max achieve below 60% success; the VibeWorlder-30B-A3B model attains the best overall Pass@1, indicating RL training can let open-source models surpass closed-source frontiers.
VibeWorlding claims to be the first unified framework for benchmarking and training 'vibe worlding agents' that construct interactive 3D open worlds from user queries in a multi-turn loop. Rather than relying on idealized simple queries, it introduces VWE-BENCH, containing 2,616 3D assets, 323 human-annotated seed worlds, and 6,828 reverse-synthesized multimodal user queries.1
Context
The work targets a known gap: existing evaluations of multimodal agents in 3D environments use overly simple queries and lack systematic comparison. By providing a benchmark and a sandbox environment that unifies asset retrieval, editing, and image rendering as MCP tools, VibeWorlding-Gym supports both evaluation and multimodal RL post-training with a rubric-based verifier.1
Assessment
The paper reports that frontier models, including GPT-5.5 and Qwen3.8-Max, score below 60% success, and identifies precise 3D world editing as the main bottleneck. The central finding is that RL post-training alleviates this weakness: the open-source VibeWorlder-30B-A3B achieves the best overall Pass@1, while the 8B variant is comparable to frontier models.1
Caveats
The results are self-reported from a single arXiv preprint and have not been independently replicated; the design of the verifier and reverse-synthesized queries could influence Pass@1. Confirmation would require open-sourced benchmark, models, and independent evaluation. If re-evaluation shows RL gains come from rubric gaming rather than genuine world-editing ability, the constraint-shift claim would be weakened.1
Title:VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?
Abstract:Constructing an interactive 3D open world from a user query is important. However, existing methods are primarily evaluated on idealized, simple queries, making it difficult to systematically analyze and compare how multimodal agents understand user intent, use 3D tools, and reason over textual and visual 3D world information. To this end, we propose VibeWorlding, a unified framework for benchmarking and training vibe worlding agents: a multimodal agent that can autonomously infer user intent, plan scene layout, invoke 3D tools, and reflect on the multimodal feedback in a multi-turn agent-environment interaction process. To achieve this, we first build VWE-BENCH, a benchmark of 2,616 high-quality 3D assets, 323 human-annotated seed 3D worlds, and 6,828 reverse-synthesized multimodal user queries, split into verified queries with ground-truth and unverified queries with carefully designed rubrics. Moreover, we develop VibeWorlding-Gym, a joint multimodal RL post-training framework that integrates (1) a sandbox environment unifying asset retrieval, editing, and image rendering as MCP tools, and (2) a rubric-based verifier that combines physical feasibility and intent fulfillment verification, supporting both fair model evaluation and scalable multimodal RL reward service. Our experiments show that current frontier MLLMs are far from solving the vibe worlding agent task, with even GPT-5.5 and Qwen3.8-Max reaching below 60% success rate, and trace the bottleneck to precise 3D world editing. We further find that RL training can ease this weakness and enable open-source MLLMs to even surpass closed-source frontiers: our VibeWorlder-8B is comparable to frontier MLLMs, while our flagship VibeWorlder-30B-A3B attains the best overall Pass@1 among all evaluated models.
Access Paper:
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
The source text above remains the property of its publisher and is reproduced for commentary.
