Timeout Cascade Strategy Cuts Coding Agent Inference Costs by Up to 33%
A new arXiv paper investigates whether smaller, cheaper models can reduce the GPU compute cost of AI coding agents without sacrificing accuracy. The authors evaluate 36 popular open-weight large and small models released since 2024 on a software engineering benchmark, finding that recent small models are narrowing the accuracy gap and that the remaining gap is concentrated on hard tasks. They propose a hybrid inference strategy that tries a small model first and escalates to a large one only when needed, reporting up to 33% reduction in total GPU-hour costs and 16.4% on all tasks from SWE-bench Verified without sacrificing accuracy. The simple cascade reportedly nearly matches an oracle-style task router, suggesting a practical path to cost-aware coding agents without learned task routers.