Back to AI News
arxiv

On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning

This arXiv paper analyzes the reliability of majority-vote pseudo-labels in test-time reinforcement learning with online Direct Preference Optimization, decomposing the error into population-majority bias and prompt-dependent finite-rollout variation. It proposes Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which conditions within-prompt robust reweighting on observed vote reliability so updates from sampling-sensitive prompts are more conservative, and adds a cache-based mechanism to reduce self-supervised forgetting. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings. The work targets finite-rollout variation because population-majority bias is generally unidentifiable without external supervision.

AI Brief

It offers a concrete method and empirical evaluation for making self-generated pseudo-label training of LLMs more reliable, relevant to researchers working on test-time adaptation and preference optimization.

arXiv:2611.09872v1 Announce Type: new Abstract: Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose \emph{Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO)}, which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative. We further introduce a cache-based mechanism to reduce self-supervised forgetting during online adaptation. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!