VelServe: Tail-Calibrated Scheduling Reduces Decode Jitters in LLM Inference
VelServe is a new scheduler that calibrates prefill work per decode step using measured tail latency to meet TBT SLOs. It leaves headroom to prevent overrunschers, serving shortest waiting prompts first and spacing admis...