WM-PPO Uses World-Model-Predicted Successor Values to Improve Sparse-Reward RL
Researchers present WM-PPO, a method that couples a world model with PPO so the critic values predicted successor observations for each sampled action, whether executed or not. These value comparisons provide auxiliary advantages to the policy and an auxiliary target to the critic, with a shared gate weighting both channels by how reliably earlier value comparisons predicted real outcomes. The approach requires no teacher or privileged information, and only the policy is used at inference. With comparable real environment steps, WM-PPO raises success from 63.5% to 87.5% on ALFWorld and from 39.3% to 52.9% on ScienceWorld, matching the strongest tested PPO refinements on ALFWorld and exceeding the best by 6.5 points on ScienceWorld.
It offers a concrete sparse-reward reinforcement-learning technique with measured gains on two agent benchmarks and a clear mechanism for using world models beyond observation prediction.
Comments & Questions (0)
No comments yet
Be the first to comment!