Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts
The paper proposes multi-expert replay policy distillation (MRPD), which consolidates multiple RL-trained domain experts into a single student model by replaying the experts' existing training logs instead of resampling from environments. At each step it recomputes the policy gradient on logged scored trajectories and distills the final experts' distributions at logged prefixes, keeping the replay quasi-on-policy. MRPD requires no environment or reward model and is reported to cost about one-seventh of the compute used to train the experts. On three long-horizon agentic domains (reasoning, search and SQL) it produced the best student model, beating online RL baselines and surpassing multi-teacher on-policy distillation from the same experts at lower cost, with analysis suggesting the order of the logs matters more than their amount or rewards.
Why it matters: It offers a concrete, cheaper alternative to multi-teacher on-policy distillation for consolidating RL experts, with evidence that replayed training logs act as a curriculum for long-horizon agentic post-training.