Live intelligence

Latest AI updates

1 updates
Research Editor selected

Sampling-Based Control Can Exploit Human-Preference Rewards, Cutting True Return by up to 76%

This paper investigates how reward models learned from human preferences can be exploited by sampling-based control methods like MPPI and best-of-N selection. Experiments on HalfCheetah, Hopper, and Meta-World tasks show that harder search can reduce true return by up to 76%, with drawer-open failing in all 20 runs despite high held-out label accuracy. The authors propose a chi-squared-regularized MPPI rule calibrated from oracle rankings that preserves 86-94% of cloned policy return, though it is cautious where search helps.

Why it matters: Provides concrete evidence and a practical fix for a critical failure mode in preference-based reward optimization, relevant to robotics and RL safety.

reward exploitationpreference-based learningsampling-based control