Live intelligence

Latest AI updates

4 updates
Research Editor selected

BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design

BitDecoding-FP4 is an algorithm-system co-design for accurate and efficient native FP4 KV cache decoding in long-context LLM serving. It combines dual-residual query quantization, underflow-aware probability scaling, and outlier-aware key quantization with system optimizations including transposed softmax, fused probability quantization, and stacked query MMA on paged KV caches. The authors report accuracy within 0.5 points of BF16 KV with NVFP4 (1 point with MXFP4) across dense and MoE models using GQA and MLA, and 2.4-3.0x higher peak serving throughput than BF16 KV on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B. The implementation is publicly available on GitHub.

Why it matters: It offers a concrete FP4 KV-cache decoding method with reported accuracy and serving-throughput gains, useful for engineers working on efficient long-context LLM inference.

BitDecoding-FP4FP4 KV cacheLLM inference serving
Research

Only Accepted Tokens Count: Energy Accounting for Speculative Decoding Under Latency Targets

The paper measures the energy cost of speculative decoding under production serving conditions on H100 and H200 GPUs using SGLang with CUDA graphs, at batch sizes up to 64 across five corpora. It finds that on dense models verification, not drafting, dominates energy, and that the saving shrinks as batch size grows because batching has already shared the weight read that speculation would divide among accepted tokens. At large batch sizes block drafters push verification past the arithmetic-memory balance point into a compute-bound regime where the saving becomes a loss, and low acceptance rates raise energy per token. Meeting a tighter latency target with speculation therefore costs additional energy, while larger batches lower energy per token without speculation.

Speculative DecodingLLM Inference ServingEnergy Efficiency
Tutorial Editor selected

Push-based KV transfer in vLLM cuts TTFT up to 3x for disaggregated inference

AWS describes a push-based NIXL KV connector for vLLM's prefill-decode disaggregated serving, contributed upstream as an accelerator-agnostic feature. Instead of the decode instance pulling KV blocks after prefill, the decode side registers target blocks in advance and the prefill side issues an NIXL WRITE from a background thread, removing the router round trip and setup work from the TTFT critical path. On a 1P1D deployment with EC2 P5en instances and EFA, the Qwen3-32B benchmark showed up to

Why it matters: It gives concrete benchmark data and implementation guidance for a real vLLM latency optimization that teams running disaggregated inference can adopt.

vLLMLLM inference servingAWS
Tutorial Editor selected

Bi-directional KV transfer in vLLM cuts TTFT for multi-turn agentic workloads

AWS details a bi-directional KV cache transfer capability in vLLM that lets decode nodes return retained KV blocks to prefill nodes on later conversation turns, avoiding repeated prefill computation in P-D disaggregated serving. It uses an extended NIXL KV connector and a KV-cache-aware router that tracks block metadata across turns, with configurable block retention TTL and a recompute threshold. On a 1P1D Amazon EC2 P5en deployment with EFA, AWS reports TTFT reductions of up to 1.6x for Qwen3-

Why it matters: Gives LLM serving teams a concrete open-source vLLM optimization, with configuration guidance and measured latency data, for reducing time to first token in long multi-turn agent sessions.

vLLMKV cache transferLLM inference serving