arXiv:2611.18392v1 Announce Type: new
Abstract: Interactive LLM services must meet latency service-level objectives (SLOs) for the time to first token (TTFT) and the time between tokens (TBT), as streaming and soft real-time applications require predictable token delivery. We identify decode jitters in both aggregated and disaggregated serving. A jitter delays every request in the batch, and long responses span hundreds of steps, so even rare overruns cause many SLO violations. In aggregated serving, SLO-aware schedulers set each step's prefill budget until its predicted latency just meets the TBT target, leaving no room for mispredictions or host stalls. Disaggregation meets average TBT targets at a cost in capacity, yet the steps that admit new requests still run late.
We present \sysname, a scheduler that calibrates how much prefill work each decode step carries using the measured tail of its own iteration latency. The SLO determines how rare a late step must be, and \sysname leaves enough headroom in each iteration to prevent overruns and meet the target decode latency. This headroom slows prefill, so \sysname serves the shortest waiting prompt first. When TTFT deadlines grow with prompt length, the long prompts it delays have the most time to spare. Under disaggregation, it spaces admissions so that no request waits on more slow admission steps than its P99 target allows. On three attention families and varying workloads, \sysname sustains 2.4--2.9$\times$ the SLO goodput of the strongest baseline on Qwen3-8B and DeepSeek-V4-Flash, and at least 4.4$\times$ on DeepSeek-V2-Lite. With prefill and decode disaggregated on eight GPUs, \sysname meets the SLO for more than twice as many requests as state-of-the-art baselines.
arxiv
VelServe: Tail-Calibrated Scheduling Reduces Decode Jitters in LLM Inference
VelServe is a new scheduler that calibrates prefill work per decode step using measured tail latency to meet TBT SLOs. It leaves headroom to prevent overrunschers, serving shortest waiting prompts first and spacing admissions in disaggregated setups. On Qwen3-8B and DeepSeek models, it achieves 2.4–2.9× SLO goodput over baselines and over 2× more requests under disaggregation.
Provides a concrete, recent technique for improving LLM inference latency predictability, with measurable gains over strong baselines.
Comments & Questions (0)
No comments yet
Be the first to comment!