The Price of an Explanation: What Demanding Verifiable Artifacts Costs, and What It Buys Back
This arXiv paper measures the cost of requiring LLMs to produce executable, verifiable artifacts on the 120-task ARC-AGI-2 evaluation set. Demanding a verifiable artifact reduced the score from 93 to 68 tasks, and 21 of the 25 lost tasks were not reached by any program-producing configuration at any budget. For 27 of the 28 tasks solved only by the answer-only arm, some program arm had a program that reproduced every demonstration but still failed the test, indicating the bottleneck is generalizing explanations rather than finding them. A short evolutionary search over two already-sampled programs improved over best-of-2 sampled code by 9.2 ± 1.4 tasks and enabled early termination, cutting budget use by up to 49% versus up to 38% for best-of-n sampling.
It provides concrete measurements and a method for the trade-off between verifiable program explanations and raw LLM answer quality, with implications for evaluation and compute-efficient reasoning.
Comments & Questions (0)
No comments yet
Be the first to comment!