Query Routing Enables Power Flexibility in LLM Datacenters
This paper proposes using query routing across LLM instances to provide power demand flexibility in datacenters without deferring requests. The authors develop an analytical framework linking query arrival rates and sequ...