On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning
This arXiv paper analyzes the reliability of majority-vote pseudo-labels in test-time reinforcement learning with online Direct Preference Optimization, decomposing the error into population-majority bias and prompt-dependent finite-rollout variation. It proposes Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which conditions within-prompt robust reweighting on observed vote reliability so updates from sampling-sensitive prompts are more conservative, and adds a cache-based mechanism to reduce self-supervised forgetting. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings. The work targets finite-rollout variation because population-majority bias is generally unidentifiable without external supervision.
It offers a concrete method and empirical evaluation for making self-generated pseudo-label training of LLMs more reliable, relevant to researchers working on test-time adaptation and preference optimization.
Comments & Questions (0)
No comments yet
Be the first to comment!