Research Editor selected
Sampling-Based Control Can Exploit Human-Preference Rewards, Cutting True Return by up to 76%
This paper investigates how reward models learned from human preferences can be exploited by sampling-based control methods like MPPI and best-of-N selection. Experiments on HalfCheetah, Hopper, and Meta-World tasks show that harder search can reduce true return by up to 76%, with drawer-open failing in all 20 runs despite high held-out label accuracy. The authors propose a chi-squared-regularized MPPI rule calibrated from oracle rankings that preserves 86-94% of cloned policy return, though it is cautious where search helps.
Why it matters: Provides concrete evidence and a practical fix for a critical failure mode in preference-based reward optimization, relevant to robotics and RL safety.
reward exploitationpreference-based learningsampling-based control