Back to AI News
arxiv

Validation Pipeline for Deploying LLM Inference Platforms on Agentic Workloads

The paper presents an end-to-end evaluation pipeline for agentic LLM serving, spanning small-scale testing, sizing, pre-production and production validation. Experiments show that replaying captured traces reproduces per-session token load within 1.18x at the 95th percentile, while live re-runs reach 4.98x, making recorded sessions a repeatable load source. A code-coverage analysis of vLLM finds live, replayed and synthetic workloads exercise complementary Python serving paths. In a case study rolling out four large mixture-of-experts models on llm-d, the target models reached about half the optimal concurrency and roughly 60% of the throughput on captured workloads compared with public traces; the authors open-source the tools and release a trace dataset.

AI Brief

It offers practitioners a validated methodology and open tooling for sizing and validating self-hosted agentic LLM inference deployments.

arXiv:2611.25905v1 Announce Type: new Abstract: Enterprises are moving agentic workloads from closed APIs to their own open-weight model deployments. For this reason, lifecycle-wide validation, from small-scale testing through sizing and pre-production to production, becomes all the more important. Typically, rollouts are evaluated with ad hoc tools whose accuracy is rarely checked. We present an end-to-end evaluation pipeline for agentic LLM serving and empirically ground the design choices that connect its stages. Our experiments show that replaying captured traces reproduces the per-session load of the original agent sessions: at the 95th percentile replay stays within 1.18x of the recorded input tokens while live re-runs reach 4.98x, so a small set of recorded sessions yields a repeatable per-session token load. A code-coverage analysis of vLLM further shows that the evaluated live, replayed, and synthetic workloads are complementary in this configuration: each exercises Python serving paths not reached in the measured runs of the others. In a case study rolling out four large mixture-of-experts (MoE) models on llm-d, experiments show that the target models reached about half the optimal concurrency, and correspondingly about 60% of the throughput, on captured workloads compared with public traces. We open-source the relevant tools and release a trace dataset.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!