Accelerate Bench: Evaluating Agents for Application-Level Program Optimization
Researchers introduce Accelerate Bench, a benchmark of 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions. Each task supplies a working program plus workload, correctness requirements, timing boundaries and permitted edits, with public testing for iteration and a separate verifier scoring the final submission. On version 1.1, 16 coding agents (model plus harness pairings) score between 25.5% and 67.4% Score@1, with most programs correct and speedup separating the leaders. The two leading agents reach at least 2x speedup on 22 or more of the 32 tasks, while the other fourteen hit 2x on only 8 to 17 tasks, leaving 125 of their 412 correct outcomes below 1.25x. Controlled replays show build and runtime settings alone can swing identical source code between 0.29x and 2.12x, highlighting the need for coordination across a full application.
It gives a concrete, executable benchmark and scored agent results for application-level code optimization, a task where gains depend on whole-program coordination rather than isolated edits.
Comments & Questions (0)
No comments yet
Be the first to comment!