Research Editor selected
LoopVQ: Sub-1-bit KV Cache Quantization for Looped Transformers
LoopVQ is a vector quantization method for KV caches in looped transformers, exploiting redundancy along the loop axis and variable sensitivity to quantization. It achieves 0.5 bits per element, outperforming existing KV quantization methods on reasoning and retrieval tasks, and combined with TaSQ retains near-BF16 accuracy at 0.25 bits. Implemented in SGLang, it nearly halves decode latency at 32K context and boosts serving throughput up to 6.1x.
Why it matters: This paper presents a concrete, novel optimization for an emerging architecture class, with measurable efficiency gains and practical deployment in SGLang.
Looped transformersKV cache compressionModel quantization