Back to AI News
arxiv

AgentOPSD Assigns Turn-Level Credit in Multi-Turn RL via Recursive Belief Revision

AgentOPSD is a critic-free reinforcement learning scheme that assigns credit one interaction turn at a time instead of spreading a single advantage across an entire trajectory. Within each turn, token-level log-probability gaps between a skill-conditioned self-teacher and the student are summed and used as evidence in a recursive Bayesian update in log-odds space, producing bounded revisions that preserve the sign of the outcome-derived advantage. On ALFWorld with Qwen2.5-3B and Qwen2.5-7B backbones, the method improves consistently over GRPO and self-distillation baselines, reaching 89.1% success at the 7B scale. Ablations attribute the gains to aggregating evidence at turn boundaries and making belief revision history-dependent rather than to the local gap alone.

AI Brief

It offers a concrete, ablation-backed credit-assignment method for multi-turn agent RL, giving practitioners a denser alternative to trajectory-level advantage spreading.

arXiv:2611.12587v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards ordinarily spreads a single advantage over a whole trajectory, a granularity too coarse to isolate the few decisions on which success in a long, multi-turn interaction actually turns. Privileged self-distillation supplies denser supervision, but it leaves unanswered how a set of local teacher-student discrepancies should be converted into credit along a sequence. We introduce AgentOPSD, a critic-free scheme that assigns credit recursively, one interaction turn at a time. Inside a turn, the token-level log-probability gaps between a skill-conditioned self-teacher and the student are summed, and that sum acts as evidence in a recursive Bayesian update carried out in log-odds space; the resulting revisions reallocate the outcome-derived advantage across turns through a multiplier that is bounded and preserves its sign. With Qwen2.5-3B and Qwen2.5-7B as policy backbones on ALFWorld, the method improves consistently on GRPO and on self-distillation baselines, and attains 89.1% success at the 7B scale. Ablations attribute the gain to aggregating evidence at turn boundaries and to making the belief revision depend on history, rather than to the local gap by itself.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!