REFACTOR-VLA: Wake/Sleep Typed Motor-Program Library Learning with a Behavioral-Equivalence Kernel
REFACTOR-VLA replaces monolithic VLA policies with a wake/sleep architecture that discovers reusable typed motor programs using a behavioral-equivalence kernel over a learned latent world model and samples a library-conditioned rectified-flow action decoder. The publication reports that increasing world-model size hurt all four LIBERO suites and that adding an InfoNCE auxiliary loss improved skill-clustering NMI under n=3 multi-seeding.
REFACTOR-VLA is presented as an alternative to monolithic vision-language-action models. Instead of decoding raw actions, it learns a library of typed motor programs via a wake/sleep loop, admitting only abstractions that pass a Minimum Description Length criterion and a return-preservation gate, and it selects among those programs through a library-conditioned rectified-flow action decoder.1
Context
Prior decomposition methods cluster action embeddings or rely on LLMs to judge behavioral equivalence, which the authors state are not calibrated to robot dynamics. REFACTOR-VLA defines behavioral equivalence through rollouts in a learned latent world model, making the equivalence criterion dynamics-aware rather than purely linguistic or embedding-based.1
Assessment
The strongest reported findings are unusually non-monotonic: enlarging the world model from 188M to 430M parameters degraded results on all four LIBERO suites, while adding an InfoNCE contrastive objective during world-model pretraining improved skill-cluster quality as measured by NMI. However, the supplied evidence is clustering NMI under n=3 multi-seeding rather than end-to-end action success.1
Caveats
AI hypothesisDirect claims of improved long-horizon behavior would be more persuasive with downstream task-success comparisons against monolithic baselines and ablations that isolate world-model capacity from auxiliary-loss effects. Without such numbers, the inferred practical advantage over existing VLA models remains a hypothesis rather than an established result.1
Most current vision-language-action (VLA) models—such as OpenVLA, π0, RT-2, and RDT-1B—are “monolithic.” This means they generate raw motor commands or very short sequences of actions, without organizing behaviors into reusable, well-defined abstractions. As a result, these models perform poorly on long-horizon (multi-step) tasks, and it’s difficult to interpret what they have learned. Existing approaches for discovering skills often avoid the core problem of deciding when two action sequences are “behaviorally equivalent.” For example, AtomicVLA and AtomSkill group action sequences by clustering their contrastive embeddings. In contrast, BLADE and LRLL rely on a large language model (LLM) to judge whether two sequences are equivalent, but these LLMs are not calibrated to the robot’s own dynamics. We introduce REFACTOR-VLA, a system that learns reusable skills using a “wake/sleep” architecture. In the sleep phase, the system clusters segments of motor programs using a Behavioral-Equivalence Kernel (BEK). This BEK is based on the outcomes of rolling out actions in a learned latent world model, Mφ. In the wake phase, the system generates typed lambda terms (simple, structured programs) from a vocabulary inspired by the Hindley–Milner type system. These lambda terms are then used by a library-conditioned rectified-flow action decoder to produce actions. Only abstractions that pass both a Minimum Description Length (MDL) criterion and a return-preservation gate are accepted as skills. To train REFACTOR-VLA, we use a three-phase schedule: • Phase A (World-model warmup): The latent world model Mφ is trained. • Phase B (Wake-phase policy optimization): The policy that uses the library of skills is optimized. • Phase C (Sleep-phase skill discovery): The system clusters action fragments into reusable skills. We evaluated REFACTOR-VLA on the full LIBERO benchmark suite. Our results show two main findings. First, simply increasing the size of the world model—from 188 million to 430 million parameters—worsened performance on 4 out of 4 benchmark suites, disproving the idea that just making the world model bigger always helps. Second, changing the training objective makes a big difference: adding an auxiliary supervised contrastive loss (specifically, InfoNCE loss) during the world-model warmup (Phase A) greatly improved the quality of skill clustering in the sleep phase (Phase C). We measured this using Normalized Mutual Information (NMI) under n = 3 multi-seeding: • Object suite: 0.462 ± 0.021 • Spatial suite: 0.867 ± 0.025 • Goal suite: 0.915 ± 0.013 • LIBERO-10 suite: 0.754 ± 0.010
Related readings and updates.
Grounding Multimodal Large Language Models in Actions
February 20, 2025research area Computer Vision, research area Methods and Algorithmsconference NeurIPS
Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains, including Embodied AI. In this work, we study how to best ground a MLLM into different embodiments and their associated action spaces, with the goal of leveraging the multimodal world knowledge of the MLLM. We first generalize a number of methods through a unified architecture and the lens of action space adaptors. For continuous actions,…
KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation
May 14, 2024research area Methods and Algorithms, research area Speech and Natural Language Processingconference ICML
Large Language Model or LLM inference has two phases, the prompt (or prefill) phase to output the first token and the extension (or decoding) phase to the generate subsequent tokens. In this work, we propose an efficient parallelization scheme, KV-Runahead to accelerate the prompt phase. The key observation is that the extension phase generates tokens faster than the prompt phase because of key-value cache (KV-cache). Hence, KV-Runahead…
The source text above remains the property of its publisher and is reproduced for commentary.
