Back to AI News
arxiv

Cost-Aware Agentic System Optimizes Features Across Fleet-Scale Ranking Models

Researchers describe an end-to-end agentic system that optimizes feature selection for large fleets of ranking models drawing from a shared pool of tens of thousands of features, under both fleet-wide and per-model resource constraints such as pipeline cost and serving QPS. LLM agents propose hypotheses while deterministic tools execute and verify experiments; for each model an LLM generates and shortlists cross features, importance methods rank candidates, and feature blending merges rankings via Pareto fronts for multi-task models. An LLM Researcher agent then tunes selection across 10-25 asynchronous trials within resource caps, followed by a trimming step to hold pipeline cost to budget. The authors report an average 0.31% gain in normalized entropy, a 2.3% pipeline cost reduction from trimming with no regression beyond noise, and a 0.20% improvement in a primary online metric in A/B tests on launched models.

AI Brief

It offers a concrete, production-validated blueprint for combining LLM agents with deterministic tooling to optimize ML features under cost and throughput constraints.

arXiv:2611.26531v1 Announce Type: cross Abstract: Large Web platforms operate large fleets of ranking models, each selecting features from a shared pool of tens of thousands. We present an end-to-end agentic system that optimizes features across the fleet under resource constraints shared across models and those set per model, such as feature pipeline cost and serving throughput (QPS), respectively. In this system, LLM agents propose hypotheses, and deterministic tools execute experiments and verify them. For each model, an LLM generates and shortlists cross features from a combinatorial candidate space, and all features are then ranked by several importance methods. Feature blending mixes these rankings into a single selection, using Pareto fronts across tasks for multi-task models. An LLM Researcher agent then tunes the feature selection in 10--25 asynchronous trials under resource caps, and a trimming step keeps the pipeline cost within budget. Across production models, the system improves normalized entropy by 0.31% on average within each model's budget, and the trimming step cuts pipeline cost by 2.3% with no regression beyond measurement noise. The optimized models were launched, and in online A/B tests they improved a primary online metric by 0.20%.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!