FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

FlashPrefill V2 adds a mean correction term, a sparse attention operator aligned with FlashAttention-3/4 and FP8, and native paged KV cache and continuous batching support. It reports up to 47.26x and 27.19x speedups over FlashAttention-2 on NVIDIA H20 at 128K context under FP8 and BF16, and a 30.49x FP8 speedup over a dense FA3/4-aligned baseline.

August 22, 20261 source58% confidence

68Strategic

Strategic score

Evidence strength42

Technical novelty74

Commercial potential78

FlashPrefill V2 is a deliberate move from sparse-prefill algorithm to deployable attention backend: it adds a mean correction term to bound approximation error, redesigns the operator for FA3/4-aligned execution with FP8 support, and integrates native paged KV cache with continuous batching. The novelty is not the sparse idea but the simultaneous removal of memory-access, precision, and framework-integration barriers.¹

Context

The quadratic cost of attention during prefill is the target bottleneck. FlashPrefill's prior prototype addressed cost through pattern discovery and max-based dynamic thresholding; V2 retains that lineage but frames itself as a backend for inference frameworks such as SGLang, evaluated on NVIDIA H20 GPUs. These are background facts from the abstract.¹

Assessment

The strongest evidence is the reported 47.26x/27.19x speedup over FlashAttention-2 at 128K in FP8/BF16, with a 30.49x FP8 speedup over a dense FA3/4-aligned baseline. Read literally, this means the sparse operator is competitive even against a modern dense kernel baseline, but because the comparison is 'up to' and against FA2 for the headline numbers, the margin over FA3/4 sparse kernels remains unquantified.¹

Caveats

The interpretation would change with artifact inspection or an independent benchmark that reproduces the speedups under continuous batching, varied batch sizes, and measured output quality. The abstract gives no sparsity ratios, correction-term formula, or quality-loss curve, so the error-suppression claim is not yet falsifiable from the text.¹

Source text

Hugging Face Daily Papers — FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving · August 20, 2026

Computer Science > Computation and Language

Title:FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

Abstract:Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase. Our previous work, FlashPrefill, mitigates this cost through instantaneous pattern discovery and max-based dynamic thresholding; however, it remains an algorithmic prototype that is still distant from production deployment. In this paper, we present FlashPrefill V2, which evolves FlashPrefill from a prototype toward practical long-context serving along three dimensions. First, we introduce a mean correction term that effectively suppresses the approximation error, keeping performance degradation manageable even at extreme sparsity levels. Second, we redesign the sparse attention operator with PackGQA memory access, warp specialization, and pingpong pipelining, fully aligning with the latest FlashAttention-3/4 implementations and supporting FP8 inference to meet practical quantization requirements. Third, FlashPrefill V2 natively supports paged KV cache and continuous batching, allowing integration as an attention backend in modern inference frameworks such as SGLang. Extensive evaluations on NVIDIA H20 GPUs---among the most widely deployed inference accelerators---demonstrate that FlashPrefill V2 delivers up to 47.26x and 27.19x speedups over FlashAttention-2 at 128K context length under FP8 and BF16 precision, respectively, and, in FP8, still achieves a 30.49x speedup against an FA3/4-aligned dense baseline.

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The source text above remains the property of its publisher and is reproduced for commentary.

↑ Back to top

FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

August 22, 20261 source58% confidence

68Strategic

Strategic score

Evidence strength42

Technical novelty74

Commercial potential78

Context

Assessment

Caveats

Source text

Hugging Face Daily Papers — FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving · August 20, 2026

Computer Science > Computation and Language

Title:FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

Abstract:Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase. Our previous work, FlashPrefill, mitigates this cost through instantaneous pattern discovery and max-based dynamic thresholding; however, it remains an algorithmic prototype that is still distant from production deployment. In this paper, we present FlashPrefill V2, which evolves FlashPrefill from a prototype toward practical long-context serving along three dimensions. First, we introduce a mean correction term that effectively suppresses the approximation error, keeping performance degradation manageable even at extreme sparsity levels. Second, we redesign the sparse attention operator with PackGQA memory access, warp specialization, and pingpong pipelining, fully aligning with the latest FlashAttention-3/4 implementations and supporting FP8 inference to meet practical quantization requirements. Third, FlashPrefill V2 natively supports paged KV cache and continuous batching, allowing integration as an attention backend in modern inference frameworks such as SGLang. Extensive evaluations on NVIDIA H20 GPUs---among the most widely deployed inference accelerators---demonstrate that FlashPrefill V2 delivers up to 47.26x and 27.19x speedups over FlashAttention-2 at 128K context length under FP8 and BF16 precision, respectively, and, in FP8, still achieves a 30.49x speedup against an FA3/4-aligned dense baseline.

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The source text above remains the property of its publisher and is reproduced for commentary.

↑ Back to top