Speculative Self-Distillation Enables Efficient Knowledge Internalization
The paper proposes Speculative Self-Distillation (SSD), a per-token mixed-policy method for teaching language models knowledge absent from pretraining or updated facts. The student generates rollouts by default, while the teacher intervenes only where its next-token distribution diverges substantially. On three closed-book QA benchmarks, SSD nearly matches on-policy distillation while using 38-69% fewer supervised tokens and outperforms off-policy distillation on held-out tests. A forgetting stress test shows SSD retains the most general capability while achieving the best in-domain accuracy with less supervision than the on-policy baseline.
It introduces a practical self-distillation method that reduces supervision cost while improving the accuracy-efficiency-forgetting tradeoff for knowledge internalization.
Comments & Questions (0)
No comments yet
Be the first to comment!