Inference-Environment Coordination Cuts LLM Agent Batch Makespan by Up to 65%
The paper targets batch makespan for LLM agent tasks that alternate between inference and environment interaction, a pattern common in benchmark evaluation, synthetic-data generation and RL rollouts. It introduces a workload-based resource model for GPU memory allocation between KV cache and recurrent state in hybrid LLMs, plus trajectory-aware routing and scheduling for cache reuse and load balancing. Under constrained environment capacity, it decouples sample admission from environment execution and adds Chain-Advancing Action Scheduling, which prioritizes long chains to shorten the batch tail. Live evaluations with Ling-3.0-Flash and Ling-3.1-Flash on BFCL, SWE-bench Multilingual, WideSearch and MCP-Atlas report makespan reductions of 10.4% to 65.0% with aligned evaluation scores.
It offers concrete systems-level techniques and measured makespan reductions for teams running LLM agent batches at scale.
Comments & Questions (0)
No comments yet
Be the first to comment!