Live intelligence

Latest AI updates

3 updates

Research Editor selected

Workload-Static Optical Circuit Switching for Scalable LLM Inference Decode

This paper proposes using workload-static optical circuit switching (OCS) for LLM inference decode, exploiting the fixed communication structure of decode to pre-configure circuits per deployment. Co-designed with model parallelism and topology-aware collective schedules, the approach turns spare accelerator ports into fewer communication phases or higher bandwidth. Evaluations on a 576-accelerator scale-up domain serving 5T-parameter MoE models show up to 65% better interactivity and 119% better energy efficiency over oversubscribed packet-switched fabrics.

Why it matters: Offers a concrete, novel networking architecture for LLM inference with significant measured gains, relevant to scalable AI systems.

arxiv2d agoScore 85

LLM inferenceoptical networkingmodel parallelism

Research Editor selected

Query Routing Enables Power Flexibility in LLM Datacenters

This paper proposes using query routing across LLM instances to provide power demand flexibility in datacenters without deferring requests. The authors develop an analytical framework linking query arrival rates and sequence lengths to token iteration time, residence time, capacity, and power, validated on four GPU platforms with 2.8% mean absolute error. A trace-driven simulation of a 10,000 H100 GPU datacenter shows up to 16% total power flexibility, with 90% of desired reduction achieved in ~10 seconds.

Why it matters: Provides a concrete, validated mechanism for datacenter power flexibility, directly addressing the growing energy demands of LLM inference.

arxiv2d agoScore 82

LLM inferencedatacenter power managementquery routing

Research

Only Accepted Tokens Count: Energy Accounting for Speculative Decoding Under Latency Targets

The paper measures the energy cost of speculative decoding under production serving conditions on H100 and H200 GPUs using SGLang with CUDA graphs, at batch sizes up to 64 across five corpora. It finds that on dense models verification, not drafting, dominates energy, and that the saving shrinks as batch size grows because batching has already shared the weight read that speculation would divide among accepted tokens. At large batch sizes block drafters push verification past the arithmetic-memory balance point into a compute-bound regime where the saving becomes a loss, and low acceptance rates raise energy per token. Meeting a tighter latency target with speculation therefore costs additional energy, while larger batches lower energy per token without speculation.

arxivOct 30, 2026Score 76

Speculative DecodingLLM Inference ServingEnergy Efficiency