Back to AI News
arxiv

RL Post-Training Improves Ranking but Hurts Recall in Generative Recommenders

A new arXiv paper studies generative recommenders that decode a user's next item semantic ID with a language model and are post-trained with reinforcement learning after supervised fine-tuning. Across five datasets, the authors find RL post-training raises hit@K for small K but lowers hit@K when K is in the hundreds, meaning the model becomes a better ranker and a worse retriever. They trace the loss to negative advantages assigned to sampled rollout candidates, since only one next item is logged per user, and propose out-of-rollout anchoring, an added loss term that preserves the SFT model's ordering of items outside the rollout. The remedy restores candidate-list recall to SFT levels on all five datasets while keeping top-of-list gains, and also works under other RL configurations.

AI Brief

It exposes a concrete, previously unmeasured failure mode of RL post-training in recommender systems and offers a simple, tested fix that practitioners can apply.

arXiv:2611.22838v1 Announce Type: cross Abstract: Generative recommenders decode the semantic ID of a user's next item with a language model, and post-training them with reinforcement learning (RL) after supervised fine-tuning (SFT) has become standard practice. The gains are mostly reported on the first ten results, yet in deployment such models hand hundreds of candidates to a downstream ranker or reward model. We show that RL post-training has a hidden cost: it improves accuracy at the top of the list (hit@K for small K) but, on all five datasets we study, lowers the recall of the candidate list (hit@K for K in the hundreds). The model becomes a better ranker and a worse retriever. Controlled experiments trace the loss to the negative advantages that RL assigns to the candidates it samples, the rollout. The root cause is that only one next item is logged per user, so RL treats every other candidate as wrong. Yet these candidates are the model's most plausible alternatives, and often the next items of other users with similar histories. Penalising them pushes down exactly what similar users choose next, and these items drop out of those users' candidate lists. This diagnosis points directly to a remedy: out-of-rollout anchoring, a single additional loss term that preserves the SFT model's ordering of the items outside the rollout. It keeps the gain at the top and restores recall to the SFT level on all five datasets, and it is effective under other RL configurations as well. To our knowledge, this is the first systematic study of this recall loss: we measure it, identify its cause, and show that, once the cause is known, a simple remedy suffices.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!