BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design
BitDecoding-FP4 is an algorithm-system co-design for accurate and efficient native FP4 KV cache decoding in long-context LLM serving. It combines dual-residual query quantization, underflow-aware probability scaling, and...