Compact 2026 Reasoning Models Become Serviceable on Jetson with NVFP4 and Speculative Decoding
NVIDIA reports that 2026 compact open reasoners such as Nemotron 3.5 Lightning and Qwen3.8-27B can now be deployed locally on Jetson AGX Thor and AGX Orin using NVFP4 quantization and speculative decoding, with up to 6.28x reported decode-throughput speedup. Evidence is vendor-only: no accuracy data or independent reproductions are supplied.
Until recently, running a multi-step reasoning or agentic model at the edge forced developers to accept a datacenter dependency. NVIDIA's release note identifies a break in that constraint: 2026 compact open models, named as Nemotron 3.5 Lightning and Qwen3.8-27B, can now be served from NVIDIA Jetson AGX Thor or AGX Orin, enabling on-device agents and anomaly detection without an active connection.1
The removed bottleneck is not simply model scale but total decode cost per token. Two configurable techniques act as the lever: NVFP4 lowers precision to reduce work and memory traffic per pass, while speculative decoding lets a smaller draft model propose tokens that the target model verifies together. In the source's measured comparison the combination reaches up to 6.28x decode throughput over BF16; equally important, the fastest draft method differs by model (DSpark for Nemotron, DFlash2 for Qwen), so the recipe is not one-size-fits-all.1
The assessment should change if the result cannot be reproduced outside NVIDIA's lab. Figure 2 is a staged maximum from a vendor blog with no accuracy table and no latency distribution behind it. A replicated run on real Jetson hardware using representative prompts, with NVFP4 accuracy compared against BF16, would determine whether this shift is a capability milestone or an edge-infrastructure convenience.1
Context
Both example models are recent open designs with different memory and performance profiles. Nemotron 3.5 Lightning is a 30B-parameter Mixture-of-Experts model that activates 3B parameters per token, positioning it for response-heavy loops; Qwen3.8-27B is a dense 27B model that must activate all parameters, positioning it for fewer, higher-value decisions. The source recommends benchmarking representative decisions before choosing a model and treats agent loops, not single-turn chat, as the target workload.1
The default serving path uses vLLM containers and quantized checkpoints, with JetPack 7.2, enough storage, Docker configured, and license acceptance as prerequisites. NVIDIA further identifies Gemma 4 E4B as a strong candidate for the smaller Jetson Orin Nano and positions NVFP4 and speculative decoding as complementary rather than alternatives. This context matters because the blog's speed claims only apply inside that setup.1
Assessment
The decisive evidence is a staged model benchmark rather than an application benchmark: it begins with BF16, then adds NVFP4, then adds the fastest tested speculative configuration. The workload extension reports token throughput between 123.01 and 138.02 tokens/s for Nemotron 3.5 Lightning and between 27.69 and 34.44 tokens/s for Qwen3.8-27B across writing, reasoning, summarization, and retrieval-augmented generation. The graphs support the claim that the fastest method remains stable by model, but they do not report accuracy or acceptance rate.1
Because the two models differ by a factor of roughly four in tokens/s, reasoning at the edge is not a uniform property; it is an engineering result with a per-deployment configuration. If an application demands high output volume, Qwen's lower throughput may negate the benefit of a cleaner decision profile; if it demands hard per-token reasoning, Nemotron's speed does not by itself show it preserves quality. The blog's own advice to validate with representative prompts is therefore the correct reading of its evidence.1
Running reasoning and agentic AI at the edge has been harder than it needs to be. Until recently, models capable of multi-step reasoning were too large to run locally on edge hardware. Developers building agents have had to route inference through a data center, adding network dependency, increasing costs, and exposing data that may need to stay on device. That constraint is lifting. Several model families released throughout the summer have collectively marked a turning point for edge AI. This new generation of compact open models now delivers reasoning and agentic capabilities that required large data center systems only a few months ago, and NVIDIA Jetson can run them today.
This could power in-cab assistants, real-time anomaly detection, and robots working in harsh or remote environments. On-site experts could spend less time troubleshooting, and critical systems could keep running when connectivity is limited or unavailable. This post walks through what you need to know to deploy this new generation of open models on Jetson, using Nemotron 3.5 Lightning and Qwen3.8-27B as examples. You’ll learn what to look for when comparing model architectures, how to apply inference optimization techniques to get the most out of your hardware, and how to validate the configuration for your workload.
Specifically, this post answers the following developer questions:
How do you choose a reasoning model for Jetson?
How do NVFP4 quantization and speculative decoding improve inference performance?
How do you serve Nemotron 3.5 Lightning and Qwen3.8-27B with vLLM?
How do you validate the configuration for your application?
Figure 1, below, shows this shift. It plots the Artificial Analysis Intelligence Index by model size and release date. Open models released in 2026 now reach scores similar to leading models from 2025, while using far fewer parameters.
Figure 1. Edge-capable 2026 models (green) reach intelligence scores comparable to 2025 frontier models at a fraction of the parameter count
Better training methods and more efficient architectures are driving this change. For example, distillation transfers some of Nemotron 3 Ultra’s capabilities into the smaller Nemotron 3.5 Lightning model. Different architectures also create different tradeoffs in capability, memory use, and generation speed.
Qwen3.8-27B is a dense model, so it activates all 27 billion parameters for each token. Nemotron 3.5 Lightning uses a mixture-of-experts (MoE) architecture. It has 30 billion total parameters but activates only 3 billion for each token. This makes the two models suitable for different types of workloads.
These differences become especially important for long-running agents. For example, an agent can monitor a system using live sensor data and device logs, take approved corrective actions, verify the results against predefined tests, and escalate to an expert only when needed. All of this runs locally at the edge without an internet connection, keeping latency low.
Nemotron 3.5 Lightning is a good fit for these response-heavy workflows, where faster token generation can shorten the overall process. Qwen3.8-27B is a better fit for tasks that require fewer, harder decisions and allow the agent to spend more time generating each response.
Benchmark both models on the decisions, tools, and response patterns required by your application before selecting one.
On Jetson, these agent loops can run next to the sensors and systems they work with. You can deploy models locally through popular frameworks, including vLLM and llama.cpp, so the reasoning loop does not depend entirely on the data center.
Gemma 4 E4B is a strong starting point for Jetson Orin Nano. For Jetson AGX Orin and Jetson AGX Thor, Nemotron 3.5 Lightning and Qwen3.8-27B are strong options. These model families have high-quality quantized checkpoints and optimized deployment options across popular inference engines.
Two complementary techniques can improve inference performance: NVFP4 quantization reduces the work and memory required for model operations, while speculative decoding can generate multiple accepted tokens per verification step.
Model architecture sets the starting point, but serving choices also affect performance. Figure 2, below, compares both models in BF16, NVFP4, and NVFP4 with the fastest speculative decoding configuration we tested for each model.
Figure 2. NVFP4 quantization and speculative decoding together deliver up to 6.28x decode throughput speedup over BF16 on Jetson
As shown in Figure 2, above, we add two optimizations, one at a time. BF16 provides the baseline. NVFP4 adds quantization. The final configuration combines NVFP4 with the fastest speculative decoding configuration we tested for each model.
During decoding, a model generates its response one token at a time. Each token usually requires another pass through the model. This gives you two ways to improve performance: reduce the work in each pass or produce more tokens from each pass.
Quantization takes the first approach. Lower-precision values reduce the amount of data the GPU must move and process during each pass. With a format such as NVFP4, you can improve generation speed and reduce memory use while keeping quality close to BF16.
Speculative decoding takes the second approach. A smaller draft model proposes several tokens, and the main model verifies them together. The main model still makes the final decision. If it accepts several proposed tokens, generation advances by several tokens in one verification step.
Video 1, below, shows a response generation with and without speculative decoding, illustrating the resulting speedup.
Several methods can produce these drafts, including MTP, DFlash, and DSpark. All three run on Jetson, but they generate and evaluate proposals differently. We tested the available methods and draft checkpoints instead of assuming that one configuration would perform best for every model.
The two optimizations complement each other. NVFP4 reduces the cost of each pass, while speculative decoding increases the number of accepted tokens produced from each pass. Together, they improve performance more than either optimization alone.
The fastest speculative decoding configuration differed between the two models. Nemotron 3.5 Lightning performed best with DSpark, while Qwen3.8-27B performed best with DFlash2. Test the method and draft checkpoint with the model you plan to deploy instead of assuming that one configuration will work best for every model.
Before running the following commands, confirm that you have:
A Jetson AGX Thor or Jetson AGX Orin
JetPack 7.2 with NVIDIA Container Runtime and Docker configured.
Enough storage for the model and draft checkpoints
Accepted the license terms for NVIDIA Nemotron and Qwen3.8 checkpoints
For Nemotron 3.5 Lightning, you can run the fastest configuration we tested, which combines NVFP4 with DSpark, on Jetson AGX Thor or Jetson AGX Orin.
First, start the vllm/vllm-openai:v0.28.0 container:
For Qwen3.8-27B, you can use the same container launched with the command above and run the fastest configuration we tested, which combines NVFP4 with DFlash2, on Jetson AGX Thor or Jetson AGX Orin using the following command:
Each method creates and evaluates draft tokens differently, which affects the cost and accuracy of its proposals.
MTP uses prediction heads trained with the main model to propose several future tokens. You can now use MTP with many mainstream model families, including Qwen, Gemma, and Nemotron.
DFlash uses a separate diffusion-based draft model to propose a block of tokens in parallel. It currently supports the widest selection of compatible draft checkpoints. DSpark builds on DFlash by correcting drafts and stopping weak proposals early. DSpark can be faster when a matching checkpoint is available, but it supports fewer checkpoints.
A model-level benchmark can help you identify a strong speculative decoding configuration. However, applications generate different kinds of text, and performance can change with the workload. To measure this effect, we kept the fastest configuration for each model fixed and tested four SpeedBench categories: writing, reasoning, summarization, and retrieval-augmented generation.
Figure 3. Speculative decoding speedup varied by workload, with RAG and writing benefiting most for both models
The same method remained the fastest for each model across the categories we tested, but throughput still varied by workload. Nemotron 3.5 Lightning with DSpark ranged from 123.01 to 138.02 output tokens/s, while Qwen3.8-27B with DFlash2 ranged from 27.69 to 34.44 output tokens/s.
Validate your speculative decoding configuration with prompts that represent the intended application. A representative dataset will help you choose the method and draft checkpoint that work best for your model and use case.
For most applications, start with an existing quantized checkpoint and draft model. This is usually enough to get strong accuracy and a useful speedup without training anything yourself. Before deploying the configuration, test it with prompts from your application. General benchmarks cannot tell you whether a checkpoint preserves the behavior that matters for your data.
If quantization reduces accuracy, use NVIDIA Model Optimizer to fine-tune the quantized model with quantization-aware training or distillation. Quantization-aware training simulates lower precision during training. Quantization-aware distillation also uses a higher-precision teacher model to help the quantized model retain the original model’s behavior.
This additional tuning is most useful for specialized workloads where small accuracy changes matter. To learn how to train a quantized model with either approach, follow the NVIDIA Model Optimizer QAT and QAD tutorial.
Apply the same approach to speculative decoding. A public draft checkpoint can be compatible with your model but still provide less acceleration than expected. The speedup depends on how often the main model accepts the proposed tokens. When the acceptance rate is low, the cost of producing and verifying drafts can reduce the benefit. If this happens, follow the vLLM Speculators training guide to train a compatible speculator with representative application data. Speculators supports methods including MTP, EAGLE-3, DFlash, and DSpark. After training, measure both the draft-token acceptance rate and decode throughput on your own prompts.
Most applications will not need custom training. Start with the available checkpoints, measure their accuracy and performance, and train your own only when those results show a clear gap.
Jetson supports the latest open models with optimized runtimes, quantized checkpoints, and speculative decoding. With this foundation in place, you can move from testing models to building and shipping edge applications.
To learn more about these models, see how to benchmark them, find recommended recipes, and compare performance across Jetson platforms, visit the Jetson AI Lab Models page.
For more hands-on guidance, see our tutorials on running LLMs and VLMs on Jetson, benchmarking generative AI models, and getting started with speculative decoding on popular frameworks.
The source text above remains the property of its publisher and is reproduced for commentary.
