AgentOPSD Assigns Turn-Level Credit in Multi-Turn RL via Recursive Belief Revision
AgentOPSD is a critic-free reinforcement learning scheme that assigns credit one interaction turn at a time instead of spreading a single advantage across an entire trajectory. Within each turn, token-level log-probability gaps between a skill-conditioned self-teacher and the student are summed and used as evidence in a recursive Bayesian update in log-odds space, producing bounded revisions that preserve the sign of the outcome-derived advantage. On ALFWorld with Qwen2.5-3B and Qwen2.5-7B backbones, the method improves consistently over GRPO and self-distillation baselines, reaching 89.1% success at the 7B scale. Ablations attribute the gains to aggregating evidence at turn boundaries and making belief revision history-dependent rather than to the local gap alone.
It offers a concrete, ablation-backed credit-assignment method for multi-turn agent RL, giving practitioners a denser alternative to trajectory-level advantage spreading.
Comments & Questions (0)
No comments yet
Be the first to comment!