Back to AI News
arxiv

A Full-Ledger Counterfactual Audit of Adaptive Reasoning

The paper introduces the Full-Ledger Counterfactual Value Audit (FLCVA), a method for comparing an immediate answer with four fixed-length continuations from the same frozen prefix under one answer procedure while accounting for all signal costs. On GSM8K and MATH-500, a random 256-token continuation recovered wrong answers in 26.6% of Qwen3-8B cases and 18.9% of DeepSeek-R1-Distill-Qwen-7B cases, while damaging correct answers in 1.4% and 1.9%. Correctness labels from independently sampled continuations predicted a separate continuation's outcome in both primary models, and offline replay improved performance on DeepSeek but was inconclusive for Qwen3. Under full cost accounting, the tested lightweight myopic controllers mainly recognized current errors and added little continuation-specific value beyond that shortcut.

AI Brief

It provides a concrete evaluation framework and empirical findings on whether extra reasoning compute actually improves LLM answers, which is directly useful to researchers working on adaptive inference.

arXiv:2611.22778v1 Announce Type: new Abstract: Adaptive reasoning must decide whether to answer now or spend more compute. Predicting that the current answer is wrong does not establish whether further reasoning will improve it. We introduce the Full-Ledger Counterfactual Value Audit (FLCVA), which compares an immediate answer with four fixed-length continuations from the same frozen prefix under one answer procedure and accounts for the cost of every signal used by a policy. On GSM8K and MATH-500, a random 256-token continuation recovers a wrong answer in 26.6% of Qwen3-8B cases and 18.9% of DeepSeek-R1-Distill-Qwen-7B cases, while damaging a correct answer in 1.4% and 1.9%. Correctness labels from independently sampled continuations predict a separate continuation's outcome in both primary models. In offline replay, using these labels to allocate compute improves performance on DeepSeek; the Qwen3 gain remains inconclusive. Under full cost accounting, the tested lightweight, myopic controllers mainly recognize current errors and add little continuation-specific value beyond that shortcut. A Llama-8B replication and action/cost ablations examine the robustness of these findings. FLCVA separates measuring local continuation value, predicting it from observable signals, and using those predictions to allocate computation.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!