TurboQuant: Extreme 3-bit KV Cache Quantization with Zero Accuracy Loss

Google Research presents TurboQuant, QJL, and PolarQuant, theoretically grounded vector quantization algorithms that eliminate per-block memory overhead. TurboQuant compresses the LLM KV cache to 3 bits without training or accuracy loss, achieves up to 8x attention logits speedup, and improves vector search recall, according to ICLR/AISTATS-bound research.

background

Source factGoogle Research introduced TurboQuant, QJL, and PolarQuant on March 24, 2026. The algorithms target memory bottlenecks in large language model key-value caches and high-dimensional vector search by performing vector quantization with zero memory overhead.

problem

Source factVector quantization reduces high-dimensional vector size but traditional methods incur 1–2 extra bits per number for storing quantization constants, partially defeating the purpose. The new methods claim to optimally solve this overhead problem.

methods

Source factQJL applies a Johnson-Lindenstrauss transform to reduce each vector number to a single sign bit while preserving distances, and uses a special estimator balancing high-precision queries with low-precision data.

Source factPolarQuant maps vectors to polar coordinates, replacing Cartesian coordinates with a radius and angle. Because angle patterns are concentrated on a fixed circular grid, it eliminates the normalization step and the associated memory overhead.

evaluation

Source factTurboQuant quantizes the key-value cache to 3 bits without training or fine-tuning and without accuracy loss on LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, and L-Eval, using Gemma and Mistral. It achieves up to 8x faster attention logits over 32-bit keys on H100 and reduces KV memory by at least 6x.

Source factIn high-dimensional vector search, TurboQuant showed better recall@1 than PQ and RabbiQ baselines while avoiding large codebooks and dataset-specific tuning, and is data-oblivious.

analysis

AI analysisThe significance of TurboQuant is that the memory overhead of quantization constants, often treated as an unavoidable cost, is claimed to be eliminated with theoretical guarantees. This makes 3-bit KV cache practical for serving long contexts, and the data-obliviousness simplifies deployment without per-dataset codebooks.

Implications

AI hypothesisIf the results generalize to larger models and production workloads, TurboQuant could substantially lower inference memory costs, increase serving throughput, and enable longer context windows. It might also accelerate semantic search index construction, but these impacts depend on replication and deployment details beyond the blog.