Live intelligence

Latest AI updates

1 updates
Research Editor selected

BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design

BitDecoding-FP4 is an algorithm-system co-design for accurate and efficient native FP4 KV cache decoding in long-context LLM serving. It combines dual-residual query quantization, underflow-aware probability scaling, and outlier-aware key quantization with system optimizations including transposed softmax, fused probability quantization, and stacked query MMA on paged KV caches. The authors report accuracy within 0.5 points of BF16 KV with NVFP4 (1 point with MXFP4) across dense and MoE models using GQA and MLA, and 2.4-3.0x higher peak serving throughput than BF16 KV on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B. The implementation is publicly available on GitHub.

Why it matters: It offers a concrete FP4 KV-cache decoding method with reported accuracy and serving-throughput gains, useful for engineers working on efficient long-context LLM inference.

BitDecoding-FP4FP4 KV cacheLLM inference serving