vLLM's Transformers Backend Reaches Native Speed Through Runtime Graph Rewriting
The Hugging Face team announces a new version of the vLLM Transformers backend that, using torch.fx static analysis and AST-based source rewriting, reaches native vLLM throughput on three Qwen3 benchmarks with a single --model-impl transformers flag, eliminating hand-written vLLM ports for compatible architectures.
The Hugging Face team reports that vLLM's Transformers modeling backend now matches or exceeds the throughput of vLLM's hand-written model implementations on three Qwen3 configurations, after a change that rewrites Transformers model code at runtime with optimization fusions. This is the first statement in the post that places the general-purpose backend on par with custom ports for compatible architectures.1
Context
Constraint moved: previously, performance parity required a separate vLLM implementation; the attention-based backend could run Transformers models, but custom ports remained necessary to unlock tensor/expert parallelism, fusions, and compilation. The updated backend uses torch.fx static graph analysis and AST rewriting to apply vLLM kernels and infer parallelism, so the same Transformers code can serve at native speed.1
Assessment
Evidence strength is moderate: the post provides a single-flag API, a reproducible benchmark script, and a before/after comparison, but the actual throughput numbers appear only in an image, not in text, and no independent validation is included. If the gist reproduces the claimed parity, the change removes a major porting bottleneck; if not, the headline claim is overreach.1
Caveats
The claimed scope excludes linear-attention models and non-compliant custom Hub models; the team says linear attention support is upcoming. A detailed technical post is promised. These limitations mean the 'any model' framing is narrower than it appears, and an external reproduction would materially change confidence.1
# Upgrade the vllm pip package uv pip install --upgrade vllm --torch-backend auto
The transformers library has become the reference modeling library for Machine Learning. It supports 450+ architectures through consistent APIs, and is designed with the main goal that model implementations are self contained and easy to understand. Going through transformers code makes it easy for contributors to learn how an architecture works, and then port it to other frameworks such as vLLM, SGLang, MLX, llama.cpp, and many others.
We have fully embraced this role in the ecosystem and are investing a lot of effort to make it easier. A big step in this direction was the integration last year of transformers as a modeling backend in vLLM. This has been allowing model authors to run transformers models (LLMs and VLMs alike) inside vLLM, without having to port anything. Transformers provides the modeling code, and vLLM provides extremely optimized inference techniques such as continuous batching and custom attention kernels.
This integration gets better now 🚀!
We put the transformers modeling backend for vLLM head to head with vLLM's hand written native implementations across three very different Qwen3 models:
4B dense model on a single GPU
32B dense model on tensor parallelism
235B-parameter FP8 Mixture-of-Experts on data + expert parallelism on the same 8×H100 node
The result: the transformers modeling backend now meets or beats native throughput on every one of them.
Running any* Hugging Face model through the transformers modeling backend is a single flag — --model-impl transformers . It composes with the usual parallelism options, so nothing about your serving setup changes:
# Qwen3-4B dense, single GPU vllm serve Qwen/Qwen3-4B --model-impl transformers # Qwen3-32B dense, tensor-parallel across 2 GPUs vllm serve Qwen/Qwen3-32B --model-impl transformers --tensor-parallel-size 2 # Qwen3-235B-A22B-FP8 MoE, data-parallel + expert-parallel across 8 GPUs vllm serve Qwen/Qwen3-235B-A22B-FP8 --model-impl transformers --data-parallel-size 8 --enable-expert-parallel # add --max-model-len 8192 if your node is memory constrained
*Models that use linear attention are not currently supported, but they will be soon! Custom models where the code lives in a Hub repo are unlikely to work as they will not have been written compliantly.
Each model is compared under three conditions that are identical in every way except the code path:
native — --model-impl vllm , vLLM's hand-written model (the bar to match)
after — --model-impl transformers with the PR
before — --model-impl transformers without the PR
The full, reproducible runner is available as a gist: benchmark.sh
The transformers modeling backend for vLLM used to focus on attention as the bottleneck for inference. By plugging vLLM’s attention implementation at runtime, we could make a transformers model run efficiently inside the vLLM engine. But there are many dimensions to deployments that only a custom port can target to extract maximum inference performance. Parallelization across GPUs, compilation, fused kernels, and many more, all contribute to leveraging your hardware to achieve ultra-fast inference.
A new model used to be integrated once for transformers, and once for vLLM with custom optimizations
When model authors wanted the absolute best performance, they were still writing custom vLLM implementations.
A new model once integrated to transformers, can now be immediately used in vLLM with native vLLM implementation speed
The latest iteration of the transformers modeling backend for vLLM dynamically applies inference specific layer fusions at runtime to match the speed of custom code implementations, for compatible architectures.
The transformers modeling backend for vLLM now uses torch.fx to perform static analysis on the model’s graph. This process searches for known patterns that can be optimised. After any patterns have been identified, it uses ast (abstract syntax tree) to manipulate the source code and rewrite some of the operations in place.
What can we achieve with this?
Fused operations that are many-to-one mapped to (ultra) optimized vLLM kernels, such as the ones used for Expert Parallelization (EP) in Mixture-of-Experts (MoE) models.
The main other fused operations are vLLM's MergedColumnParallelLinear and QKVParallelLinear . These blocks allow us to infer parallel plans for TP (tensor-parallel). PP (pipeline-parallel) plans can also be inferred if the decoder block list is easily identifiable.
The manipulated models are still fully (torch) compilable, being passed through torch.compile and CUDA Graphs, just the same as a dedicated vLLM model implementation.
Unlike vLLM model implementations, Transformers model implementations can be used in training. So you can use the same model code for training/evals/RL rollouts.
As shown above, this results in native vLLM inference speed for compatible models, without having to write a single line of code to optimize the model for inference.
We are in the process of writing a detailed blog post to dive deep inside these optimized inference methods and explain in detail how we manipulate the model to adapt to them.
The source text above remains the property of its publisher and is reproduced for commentary.
