Not All Entropy Is Equal: Reward-Conditioned Entropy Allocation in Reinforcement Learning with Verifiable Rewards
The paper argues that global token entropy is an unreliable signal for reasoning performance in reinforcement learning with verifiable rewards (RLVR), because it mixes uncertainty from successful and failed trajectories, variation across correct solution strategies, and variation within a single strategy. It introduces Reward-Conditioned Entropy Allocation (RCEA), a hierarchical diagnostic framework that separates trajectory-level from average token entropy, conditions uncertainty on verifier outcome and process validity, and measures retention of correct solution modes. Across six RLVR objectives at two model scales, entropy trajectories did not produce a stable ordering of reasoning accuracy, and token entropy was higher for verifier-failed than successful trajectories. A local covariance approximation predicted the direction of adjacent-checkpoint entropy change in 95.5% of nonfailed transitions, and every reportable trained endpoint showed some contraction in audited correct-solution-mode breadth. The authors conclude that entropy collapse should be diagnosed by which uncertainty is removed rather than by global entropy alone.