Research
VelServe: Tail-Calibrated Scheduling Reduces Decode Jitters in LLM Inference
VelServe is a new scheduler that calibrates prefill work per decode step using measured tail latency to meet TBT SLOs. It leaves headroom to prevent overrunschers, serving shortest waiting prompts first and spacing admissions in disaggregated setups. On Qwen3-8B and DeepSeek models, it achieves 2.4–2.9× SLO goodput over baselines and over 2× more requests under disaggregation.
LLM inference optimizationlatency SLOsscheduling algorithms