Back to AI News
arxiv

The Price of an Explanation: What Demanding Verifiable Artifacts Costs, and What It Buys Back

This arXiv paper measures the cost of requiring LLMs to produce executable, verifiable artifacts on the 120-task ARC-AGI-2 evaluation set. Demanding a verifiable artifact reduced the score from 93 to 68 tasks, and 21 of the 25 lost tasks were not reached by any program-producing configuration at any budget. For 27 of the 28 tasks solved only by the answer-only arm, some program arm had a program that reproduced every demonstration but still failed the test, indicating the bottleneck is generalizing explanations rather than finding them. A short evolutionary search over two already-sampled programs improved over best-of-2 sampled code by 9.2 ± 1.4 tasks and enabled early termination, cutting budget use by up to 49% versus up to 38% for best-of-n sampling.

AI Brief

It provides concrete measurements and a method for the trade-off between verifiable program explanations and raw LLM answer quality, with implications for evaluation and compute-efficient reasoning.

arXiv:2611.25896v1 Announce Type: new Abstract: Generative models produce many candidate solutions but provide no intrinsic signal by which to rank them. One mitigation is to demand a \emph{verifiable artifact}: a program that can be executed against the available evidence. A common concern is that requiring executable verification incurs costs in both solution quality and computational budget. We measure these two costs on the $120$-task ARC-AGI-2 evaluation set. Demanding a verifiable artifact from an LLM reduced the score from $93$ to $68$ of $120$ tasks ($77\%$ to $57\%$). Of those $25$ tasks, $21$ were reached by no program-producing configuration we ran, at any budget, by sampling or by search; yet for $27$ of the $28$ tasks that only the answer-only arm solved, some program arm held a program that reproduced every demonstration and still failed the test. The bottleneck is obtaining an explanation that \emph{generalizes}, not finding one. A short evolutionary search over two already-sampled programs, at no additional model cost, beats best-of-$2$ sampled code by $9.2 \pm 1.4$ tasks, and over a single sampled program beats $1$-shot sampling by $9.9 \pm 1.3$. Moreover, because a program can be checked against the training examples, any program-producing protocol can terminate early once a task is explained: under the same stopping rule, best-of-$n$ sampling declines up to $38\%$ of its budget and the evolved arm up to $49\%$, whereas an answer-only baseline cannot terminate early. Applying the same methodology to decoder synthesis for quantum error correction yields interpretable decoders evaluated against a certified panel of published decoders on held-out exam shots.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!