Back to AI News
arxiv

Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts

The paper proposes multi-expert replay policy distillation (MRPD), which consolidates multiple RL-trained domain experts into a single student model by replaying the experts' existing training logs instead of resampling from environments. At each step it recomputes the policy gradient on logged scored trajectories and distills the final experts' distributions at logged prefixes, keeping the replay quasi-on-policy. MRPD requires no environment or reward model and is reported to cost about one-seventh of the compute used to train the experts. On three long-horizon agentic domains (reasoning, search and SQL) it produced the best student model, beating online RL baselines and surpassing multi-teacher on-policy distillation from the same experts at lower cost, with analysis suggesting the order of the logs matters more than their amount or rewards.

AI Brief

It offers a concrete, cheaper alternative to multi-teacher on-policy distillation for consolidating RL experts, with evidence that replayed training logs act as a curriculum for long-horizon agentic post-training.

arXiv:2611.24059v1 Announce Type: cross Abstract: Frontier post-training follows a common recipe: train one expert per domain with reinforcement learning (RL), then consolidate the experts into one model via multi-teacher on-policy distillation (MOPD). Such consolidation is expensive: the student samples in every environment again, an expert scores every sampled token, and the student inherits the drift of the final experts. Yet we argue that the experts' RL runs have already produced what consolidation needs: rollouts with their sampling probabilities and rewards, in the order the experts learned from them. We propose multi-expert replay policy distillation (MRPD), which replays these training logs through the student in the experts' order. At each step, the policy gradient is recomputed on these scored trajectories, and the final experts' distributions are distilled at the logged prefixes. Following the experts' own trajectory reduces the training-inference mismatch and keeps the student within the clip range, so the replay is off-policy in origin but quasi-on-policy throughout consolidation. MRPD samples nothing and calls no environment or reward model: the trainer is the sole process, and consolidating the experts costs only about a seventh of the compute used to train them. On three long-horizon agentic domains (reasoning, search and SQL), MRPD yields the best student model, substantially outperforming online RL baselines, and surpassing MOPD from the same experts at a fraction of MOPD's cost. Our analysis on the same logs shows that the replayed trajectories carry most of the gain, and that both the policy gradient and the distillation term add to it. Most interestingly, the order of the logs matters more than their amount or rewards, suggesting that expert training logs are a curriculum, not a data mixture.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!