Back to AI News
arxiv

Not All Entropy Is Equal: Reward-Conditioned Entropy Allocation in Reinforcement Learning with Verifiable Rewards

The paper argues that global token entropy is an unreliable signal for reasoning performance in reinforcement learning with verifiable rewards (RLVR), because it mixes uncertainty from successful and failed trajectories, variation across correct solution strategies, and variation within a single strategy. It introduces Reward-Conditioned Entropy Allocation (RCEA), a hierarchical diagnostic framework that separates trajectory-level from average token entropy, conditions uncertainty on verifier outcome and process validity, and measures retention of correct solution modes. Across six RLVR objectives at two model scales, entropy trajectories did not produce a stable ordering of reasoning accuracy, and token entropy was higher for verifier-failed than successful trajectories. A local covariance approximation predicted the direction of adjacent-checkpoint entropy change in 95.5% of nonfailed transitions, and every reportable trained endpoint showed some contraction in audited correct-solution-mode breadth. The authors conclude that entropy collapse should be diagnosed by which uncertainty is removed rather than by global entropy alone.

AI Brief

Offers RLVR practitioners a concrete diagnostic framework and empirical evidence that commonly used entropy metrics can mislead when judging reasoning capability.

arXiv:2611.09448v1 Announce Type: cross Abstract: Policy entropy is widely used to diagnose exploration and collapse in reinforcement learning with verifiable rewards (RLVR), yet its magnitude does not consistently track reasoning performance across optimization objectives. We argue that this ambiguity arises because global token entropy conflates uncertainty in successful and failed trajectories, diversity across correct solution strategies, and variation among realizations of the same strategy. We introduce \textbf{Reward-Conditioned Entropy Allocation (RCEA)}, a hierarchical diagnostic framework that separates complete-trajectory entropy from average token entropy, conditions uncertainty on verifier outcome and process validity, measures correct-solution-mode retention, and connects these population-level quantities to local covariance-driven updates. In controlled sweeps over six RLVR objectives at two model scales, entropy trajectories do not induce a stable ordering of reasoning accuracy. Token entropy is higher for verifier-failed than successful trajectories under both broad separately aggregated estimates and a common-question paired analysis, whose confidence intervals exclude equality at 10 of 12 displayed endpoints. A local covariance approximation predicts the direction of adjacent-checkpoint entropy change in 95.5\% of nonfailed transitions; a reward-stratified endpoint proxy further shows that positive chosen-token covariance is more tail-concentrated within failed prefixes. Every reportable trained endpoint exhibits some contraction in independently audited correct-solution-mode breadth, whose positive association with large-\(k\) coverage remains exploratory. These results show that entropy collapse should be diagnosed by which uncertainty is removed, rather than by global entropy alone.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!