Live intelligence

Latest AI updates

1 updates
Research

VelServe: Tail-Calibrated Scheduling Reduces Decode Jitters in LLM Inference

VelServe is a new scheduler that calibrates prefill work per decode step using measured tail latency to meet TBT SLOs. It leaves headroom to prevent overrunschers, serving shortest waiting prompts first and spacing admissions in disaggregated setups. On Qwen3-8B and DeepSeek models, it achieves 2.4–2.9× SLO goodput over baselines and over 2× more requests under disaggregation.

LLM inference optimizationlatency SLOsscheduling algorithms