TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration
TileMix proposes a training-free, tile-centric mixed-precision attention kernel that routes hardware-aligned score tile groups through FP16 or INT8 within fused dense attention, using compact bitmask routing and a shared online-softmax state. It claims to recover long-context quality lost under uniform INT8 and improve prefill throughput over FP16 on LongEval and LV-Eval with LLaMA, Qwen, and Vicuna on A100. Without further methodological details or independent validation, the reported performance is preliminary.
TileMix introduces tile-centric mixed-precision attention, making numerical precision a spatial decision within fused dense attention rather than a global uniform setting. It routes score-tile groups through FP16 or INT8 paths while sharing a single online-softmax state, preserving dense token connectivity without retraining.1
Context
The constraint shifted is the granularity of precision control: prior work applies uniform low precision to all attention scores or selects token interactions, trading away either quality or full connectivity. TileMix's bitmask routing over hardware-aligned tiles permits per-tile precision selection with compact metadata, a new point in the accuracy-efficiency design space.1
Assessment
The abstract reports benchmarks on LongEval and LV-Eval with LLaMA, Qwen, and Vicuna on A100 prefill, claiming recovery of long-context quality lost under uniform INT8 and higher prefill throughput than FP16. These are promising but self-reported results without full methodological detail.1
Caveats
AI hypothesisWhat would change my mind is public evidence on the overhead of routing and bitmask handling, and sensitivity to hardware and variable batch sizes. If tile-level routing gains shrink on non-A100 accelerators or at very long contexts, the advantage narrows; without ablations and reproducibility details, the controllable frontier claim remains a hypothesis.1
Title:TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration
Abstract:Long-context prefill in large language models (LLMs) incurs substantial computation and memory traffic because dense self-attention computes quadratic query-key scores. Existing methods either use a uniform low-precision path or select token interactions, leaving spatial precision routing over hardware-aligned score tiles outside fused dense attention. We introduce TileMix, a tile-centric precision-routing kernel that makes numerical precision an executable spatial decision over score-tile groups within fused dense attention. TileMix partitions the attention matrix into hardware-aligned score tiles, packs routing decisions into compact bitmasks, and dispatches each tile group through FP16 or INT8 score computation while both paths update a shared online-softmax state. Scalable precision grouping lets each routing bit govern multiple adjacent key tiles, preserving hardware-aligned compute tiles and compact metadata at long contexts. By routing all legal tile groups, TileMix preserves dense token connectivity, requires no training, and supports grouped-query attention, variable-length batches, and INT8 key/value caches. Across LongEval, LV-Eval, and A100 prefill benchmarks on LLaMA, Qwen, and Vicuna, TileMix recovers long-context quality lost under uniform INT8 and improves prefill throughput over FP16, yielding a controllable accuracy-efficiency frontier across model families. The implementation is available at this https URL.
Access Paper:
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
The source text above remains the property of its publisher and is reproduced for commentary.
