Shuffled OPSD: When Semantic Consistency Hurts On-Policy Self-Distillation?
This paper introduces Shuffled OPSD, a variant of on-policy self-distillation that shuffles the words of the shared problem statement while keeping the teacher-only worked example intact. Across three math reasoning benchmarks and three Qwen3 model scales, it improves mean accuracy over both intact-target and Standard OPSD, with the largest gain of 3.52 points on Qwen3-1.7B. Controls show that shuffling the teacher-only example hurts accuracy, while word-level shuffling of the problem statement helps, challenging the assumption that semantic coherence is always beneficial.
Why it matters: Provides concrete, recent experimental evidence that disrupting input structure can improve self-distillation, offering a novel insight for AI training methodology.