MCPO: Margin-Guided Token Credit for RL with Verifiable Rewards
Researchers propose MCPO (Margin-Credit Policy Optimization), which redistributes a solution's advantage across tokens according to next-token top-2 margin hesitation rather than equally as in GRPO. The margin is read from the trainer's existing forward pass, requiring no extra model evaluation or batch statistics. Training MiniCPM5-2B-Base with LoRA for 200 steps on 2,048 DAPO-Math problems, MCPO beat GRPO on all five data seeds by 6.55 points of macro avg@32 on average; a Qwen3-4B-Base replication improved on all three seeds by 4.75 points. Gains were largest on problems GRPO solves only sometimes and grew with k in pass@k, though MCPO models also wrote two to three times longer answers and the experiments do not separate the two effects.
It introduces a low-overhead credit-assignment change for verifiable-reward RL and reports multi-seed gains over GRPO on competition math, useful for researchers tuning LLM post-training.
Comments & Questions (0)
No comments yet
Be the first to comment!