Only Accepted Tokens Count: Energy Accounting for Speculative Decoding Under Latency Targets
The paper measures the energy cost of speculative decoding under production serving conditions on H100 and H200 GPUs using SGLang with CUDA graphs, at batch sizes up to 64 across five corpora. It finds that on dense mode...