Back to AI News
arxiv

MCPO: Margin-Guided Token Credit for RL with Verifiable Rewards

Researchers propose MCPO (Margin-Credit Policy Optimization), which redistributes a solution's advantage across tokens according to next-token top-2 margin hesitation rather than equally as in GRPO. The margin is read from the trainer's existing forward pass, requiring no extra model evaluation or batch statistics. Training MiniCPM5-2B-Base with LoRA for 200 steps on 2,048 DAPO-Math problems, MCPO beat GRPO on all five data seeds by 6.55 points of macro avg@32 on average; a Qwen3-4B-Base replication improved on all three seeds by 4.75 points. Gains were largest on problems GRPO solves only sometimes and grew with k in pass@k, though MCPO models also wrote two to three times longer answers and the experiments do not separate the two effects.

AI Brief

It introduces a low-overhead credit-assignment change for verifiable-reward RL and reports multi-seed gains over GRPO on competition math, useful for researchers tuning LLM post-training.

arXiv:2611.14015v1 Announce Type: cross Abstract: In reinforcement learning with verifiable rewards, a language model is rewarded for a correct final answer, and GRPO passes that reward to every token of the solution in equal measure. We instead place the credit where the model hesitated, in a method we call MCPO (Margin-Credit Policy Optimization). We measure hesitation by the top-2 margin $M=p_1-p_2$ of the policy's next-token distribution, map it to a score with two fixed thresholds, and spread each solution's advantage over its tokens so that the mean stays unchanged. The margin is read from a forward pass the trainer already performs, so the method needs no extra model evaluation and no batch statistics. We train MiniCPM5-2B-Base with LoRA for 200 steps on 2,048 DAPO-Math problems and evaluate on six competition-math benchmarks. MCPO beats an unmodified GRPO baseline for all five data seeds, by 6.55 points of macro avg@32 on average (95\% CI $[4.1, 9.0]$), and a Qwen3-4B-Base replication improves for all three seeds ($+4.75$). The gain is largest on problems that GRPO solves only some of the time and grows with $k$ in pass@$k$, reaching 13.8 points at pass@32. MCPO models also write two to three times longer answers, and runs that write longer score higher; our experiments do not separate the two effects.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!