Back to AI News
arxiv

Budget Internalization: Fixed-Budget RLVR Trades Test-Time Scaling for In-Budget Accuracy

A new arXiv paper finds that the fixed rollout cap used in RL with verifiable rewards (RLVR) becomes a learned constraint, an effect the authors call budget internalization. After training, models use few additional tokens beyond the training budget, outperform base models near that budget, but stop improving and fall below the base model at larger budgets. The effect is observed across Qwen3.5-4B, Gemma4-E2B, and Nemotron-3-Nano-4B on mathematical reasoning and procedurally generated problems, and transfers to science and coding tasks. Ablations link the effect to length-censored post-training rather than an RL-specific objective, with SFT on artificially truncated traces reproducing it while naturally short traces do not.

AI Brief

It reveals that a common RLVR training budget can silently limit test-time scaling and reduce accuracy when more inference compute is available, which matters for reasoning-model training and deployment.

arXiv:2611.26172v1 Announce Type: cross Abstract: Reasoning models often become more accurate when they can generate longer answers at test time. This behavior is known as test-time scaling. However, RL with verifiable rewards (RLVR) usually trains them with a fixed rollout cap. We find that this cap changes model behavior, not only training cost. After training, models use few additional tokens beyond the training budget. They outperform the base models near that budget, but their accuracy stops improving and falls below the base model at larger budgets. The rollout cap thus becomes a learned constraint, that limits model's test-time scaling. We call this effect budget internalization. We observe it across Qwen3.5-4B, Gemma4-E2B, and Nemotron-3-Nano-4B and two reasoning training tasks: mathematical reasoning and diverse, procedurally generated problems. This effect also transfers from math training to science and coding tasks. Training-objective level ablations on Qwen3.5-4B show that most changes recover little test-time scaling. Stronger KL regularization and lower training-data truncation preserve more scaling, but stronger KL reduces in-budget accuracy. Forcing an answer on truncated training rollouts largely removes budget internalization, but also removes most of the in-budget gain. SFT on artificially truncated traces reproduces budget internalization, whereas SFT on naturally short traces does not, which shows that the effect is not specific to RL. This result links budget internalization to length-censored post-training rather than to an RL-specific objective term. Thus, a rollout cap does not only control training cost. It can trade low-budget gains for lower performance when additional inference compute is available.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!