Validation Pipeline for Deploying LLM Inference Platforms on Agentic Workloads
The paper presents an end-to-end evaluation pipeline for agentic LLM serving, spanning small-scale testing, sizing, pre-production and production validation. Experiments show that replaying captured traces reproduces per-session token load within 1.18x at the 95th percentile, while live re-runs reach 4.98x, making recorded sessions a repeatable load source. A code-coverage analysis of vLLM finds live, replayed and synthetic workloads exercise complementary Python serving paths. In a case study rolling out four large mixture-of-experts models on llm-d, the target models reached about half the optimal concurrency and roughly 60% of the throughput on captured workloads compared with public traces; the authors open-source the tools and release a trace dataset.
It offers practitioners a validated methodology and open tooling for sizing and validating self-hosted agentic LLM inference deployments.
Comments & Questions (0)
No comments yet
Be the first to comment!