NVIDIA Validates Day-0 Serving of 2.4T-Parameter Qwen3.8 on GB300 NVL72

Alibaba released the open-weight Qwen3.8-2.4T-A95B, a 2.4T-parameter MoE model with 95B activated parameters, and NVIDIA reports Day-0 multinode inference on GB300 NVL72 achieving over 4K tokens/s/GPU and over 350 tokens/s/user, making data-center-scale deployment practical.

What happened

Source factAlibaba released the open weights for Qwen3.8-2.4T-A95B, also called Qwen3.8-Max, and NVIDIA published a blog describing optimized serving on GB300 NVL72. The model has 2.4T total parameters, 95B activated parameters per token, a hybrid of full and linear attention, and a context window up to one million tokens.

Architecture

Source factThe model uses a fine-grained mixture-of-experts architecture with a larger population of smaller experts, activated by a learned router per token. It alternates between full-attention layers and linear-attention layers, where the latter uses a bounded recurrent state instead of a growing KV cache, keeping memory costs bounded as context scales.

Performance claims

Source factNVIDIA reports that without additional tuning, the model achieves over 4,000 tokens per second per GPU and over 350 tokens per second per user on GB300 NVL72 in FP8 precision on Day 0. It also states that further optimizations with NVFP4 precision are expected to deliver enhanced performance gains over time.

Ecosystem

Source factThe model is supported by open-source inference stacks including SGLang, vLLM, and NVIDIA Dynamo, and is also available via model-free NVIDIA NIM. Fine-tuning support is available through NVIDIA NeMo AutoModel, including full SFT and LoRA. Hosted APIs are provided by DeepInfra, DigitalOcean, Fireworks AI, Modal, and OpenRouter.

Analysis

AI analysisThe combination of fine-grained MoE and hybrid linear attention represents a practical solution to the two main scaling bottlenecks: parameter-accounting and KV-cache growth. By activating only 95B parameters, serving cost scales with active compute rather than total parameters. The GB300 NVL72, with 72 GPUs interconnected by a 130 TB/s NVLink domain, is positioned to remove the communication bottleneck that would otherwise plague expert routing across GPUs.

Constraint shift

AI analysisThis event shifts the practical constraint for large open-weight models from raw GPU count to the efficiency of the inter-GPU network and memory architecture. The 2.4T-parameter model is no longer theoretically deployable but is claimed to run on 72 GPUs with high throughput. This validates rack-scale co-design as a critical enabler for open-weights adoption.

Limitations

AI hypothesisIf the claimed throughput is not independently reproducible, the commercial impact could be significantly lower. Also, the long-term viability of linear attention for tasks requiring exact token-level recall remains uncertain, and the model's performance on complex reasoning benchmarks is not yet established with third-party validation.