Live intelligence

Latest AI updates

2 updates
Research Editor selected

Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts

The paper proposes multi-expert replay policy distillation (MRPD), which consolidates multiple RL-trained domain experts into a single student model by replaying the experts' existing training logs instead of resampling from environments. At each step it recomputes the policy gradient on logged scored trajectories and distills the final experts' distributions at logged prefixes, keeping the replay quasi-on-policy. MRPD requires no environment or reward model and is reported to cost about one-seventh of the compute used to train the experts. On three long-horizon agentic domains (reasoning, search and SQL) it produced the best student model, beating online RL baselines and surpassing multi-teacher on-policy distillation from the same experts at lower cost, with analysis suggesting the order of the logs matters more than their amount or rewards.

Why it matters: It offers a concrete, cheaper alternative to multi-teacher on-policy distillation for consolidating RL experts, with evidence that replayed training logs act as a curriculum for long-horizon agentic post-training.

reinforcement learningmodel distillationagentic AI
Research Editor selected

RL Post-Training Improves Ranking but Hurts Recall in Generative Recommenders

A new arXiv paper studies generative recommenders that decode a user's next item semantic ID with a language model and are post-trained with reinforcement learning after supervised fine-tuning. Across five datasets, the authors find RL post-training raises hit@K for small K but lowers hit@K when K is in the hundreds, meaning the model becomes a better ranker and a worse retriever. They trace the loss to negative advantages assigned to sampled rollout candidates, since only one next item is logged per user, and propose out-of-rollout anchoring, an added loss term that preserves the SFT model's ordering of items outside the rollout. The remedy restores candidate-list recall to SFT levels on all five datasets while keeping top-of-list gains, and also works under other RL configurations.

Why it matters: It exposes a concrete, previously unmeasured failure mode of RL post-training in recommender systems and offers a simple, tested fix that practitioners can apply.

generative recommendationreinforcement learningrecommender systems