Budget Internalization: Fixed-Budget RLVR Trades Test-Time Scaling for In-Budget Accuracy
A new arXiv paper finds that the fixed rollout cap used in RL with verifiable rewards (RLVR) becomes a learned constraint, an effect the authors call budget internalization. After training, models use few additional tokens beyond the training budget, outperform base models near that budget, but stop improving and fall below the base model at larger budgets. The effect is observed across Qwen3.5-4B, Gemma4-E2B, and Nemotron-3-Nano-4B on mathematical reasoning and procedurally generated problems, and transfers to science and coding tasks. Ablations link the effect to length-censored post-training rather than an RL-specific objective, with SFT on artificially truncated traces reproducing it while naturally short traces do not.
It reveals that a common RLVR training budget can silently limit test-time scaling and reduce accuracy when more inference compute is available, which matters for reasoning-model training and deployment.
Comments & Questions (0)
No comments yet
Be the first to comment!