Back to AI News
arxiv

Inference-Environment Coordination Cuts LLM Agent Batch Makespan by Up to 65%

The paper targets batch makespan for LLM agent tasks that alternate between inference and environment interaction, a pattern common in benchmark evaluation, synthetic-data generation and RL rollouts. It introduces a workload-based resource model for GPU memory allocation between KV cache and recurrent state in hybrid LLMs, plus trajectory-aware routing and scheduling for cache reuse and load balancing. Under constrained environment capacity, it decouples sample admission from environment execution and adds Chain-Advancing Action Scheduling, which prioritizes long chains to shorten the batch tail. Live evaluations with Ling-3.0-Flash and Ling-3.1-Flash on BFCL, SWE-bench Multilingual, WideSearch and MCP-Atlas report makespan reductions of 10.4% to 65.0% with aligned evaluation scores.

AI Brief

It offers concrete systems-level techniques and measured makespan reductions for teams running LLM agent batches at scale.

arXiv:2611.16661v1 Announce Type: cross Abstract: Efficient batch execution of agent tasks is critical to LLM training and development workflows, including benchmark evaluation, synthetic-data generation, and RL rollouts. Since these tasks alternate between LLM inference and environment interaction, we present coordinated optimizations to reduce batch makespan, the time required to complete all tasks in a batch. For hybrid LLMs, a workload-based resource model guides GPU-memory allocation between key-value (KV) cache and recurrent state and provides a reference sample concurrency to exploit parallelism while limiting recomputation. Trajectory-aware routing and scheduling promote cache reuse while balancing worker load. Under constrained environment capacity, we decouple sample admission from environment execution and introduce Chain-Advancing Action Scheduling (CAAS), which prioritizes long chains using observed progress to shorten the batch tail. Controlled replay combines actual inference with recorded environment responses and durations for ablations of individual optimizations. Live evaluations with Ling-3.0-Flash and Ling-3.1-Flash on the BFCL, SWE-bench Multilingual, WideSearch, and MCP-Atlas benchmarks demonstrate makespan reductions of 10.4% to 65.0%, while evaluation scores remain aligned. Furthermore, these methods have been successfully applied to training and evaluation workflows for the Ling-3.x model series, accelerating the model-development cycle.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!