Back to AI News
arxiv

WM-PPO Uses World-Model-Predicted Successor Values to Improve Sparse-Reward RL

Researchers present WM-PPO, a method that couples a world model with PPO so the critic values predicted successor observations for each sampled action, whether executed or not. These value comparisons provide auxiliary advantages to the policy and an auxiliary target to the critic, with a shared gate weighting both channels by how reliably earlier value comparisons predicted real outcomes. The approach requires no teacher or privileged information, and only the policy is used at inference. With comparable real environment steps, WM-PPO raises success from 63.5% to 87.5% on ALFWorld and from 39.3% to 52.9% on ScienceWorld, matching the strongest tested PPO refinements on ALFWorld and exceeding the best by 6.5 points on ScienceWorld.

AI Brief

It offers a concrete sparse-reward reinforcement-learning technique with measured gains on two agent benchmarks and a clear mechanism for using world models beyond observation prediction.

arXiv:2611.12200v1 Announce Type: new Abstract: A sparse terminal reward tells an agent little about its steps. Much of the missing information is implicit in the environment's dynamics, because each state transition can change the conditions for success long before the reward arrives. A world model trained on real transitions can learn to predict what each action changes, but a predicted observation does not by itself show whether success has become more likely. With a binary terminal reward, PPO's critic makes this judgment by estimating the current policy's probability of success, but only for states the agent reaches. Our method, WM-PPO, elegantly brings the two together. For each action the policy samples, executed or not, the world model predicts the next observation and the critic values the predicted successor. In the policy channel, comparisons among these values give the policy auxiliary advantages; in the value channel, their mean gives the critic an auxiliary target. A shared gate weights both channels according to how reliably comparisons with similar margins predicted real outcomes earlier in training. The policy, critic and world model all learn from the agent's own interaction, with no teacher or privileged information, and only the policy is used at inference. With a comparable number of real environment steps, WM-PPO raises success from PPO's 63.5% to 87.5% on ALFWorld and from 39.3% to 52.9% on ScienceWorld. It matches the strongest PPO refinements we tested on ALFWorld and exceeds the best of them by 6.5 points on ScienceWorld. The trained policy more often completes prerequisite steps before placing an object or committing to an answer.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!