Bootstrapped On-Policy Distillation
The paper analyzes on-policy self-distillation (OPSD), where a reference-conditioned copy of the model acts as teacher, and finds that continued training can degrade performance due to growing mismatch between the evolvi...