Research Editor selected
Query Routing Enables Power Flexibility in LLM Datacenters
This paper proposes using query routing across LLM instances to provide power demand flexibility in datacenters without deferring requests. The authors develop an analytical framework linking query arrival rates and sequence lengths to token iteration time, residence time, capacity, and power, validated on four GPU platforms with 2.8% mean absolute error. A trace-driven simulation of a 10,000 H100 GPU datacenter shows up to 16% total power flexibility, with 90% of desired reduction achieved in ~10 seconds.
Why it matters: Provides a concrete, validated mechanism for datacenter power flexibility, directly addressing the growing energy demands of LLM inference.
LLM inferencedatacenter power managementquery routing