Research Editor selected
FOSCAR Accelerates 2- and 4-Bit KV Cache Decode Attention via Affine Folding
FOSCAR is a GPU kernel library that speeds up decode attention over 2-bit and 4-bit quantized KV caches by applying scale and offset through affine folding rather than per-value conversion. It uses bit manipulation and warp-specialized pipelines for Hopper H100 and Blackwell B200, targeting wgmma and tcgen05. The authors report geo-mean kernel speedups over FlashInfer BF16 and TensorRT-LLM FP8 baselines, plus throughput gains when integrated into SGLang for long-context serving. FOSCAR is released as an open-source kernel library.
Why it matters: It offers a concrete, open-source kernel-level optimization for memory-bound LLM decode attention, with reported latency and serving-throughput gains on current NVIDIA GPUs.
KV cache quantizationLLM inference optimizationGPU kernels