Turing MoE Study Reports Strong Specialist Integration and 2.86× Faster Consolidation
A new paper combines calibrated multi-teacher on-policy distillation with execution optimizations for dynamically routed mixture-of-experts models. On Turing-20B-A2B, the authors report retaining at least 90.16% of measured specialist gains in every evaluated domain and outperforming shared supervised fine-tuning and Qwen3-8B on four reasoning-enabled mathematics and code benchmarks at a matched response cap. A PPU-based profile shows consolidation step time falling from 398.7 to 139.3 seconds, with the results attributed to the complete training and execution framework rather than a single component.
Why it matters: The study offers concrete evaluation and execution results for researchers seeking to combine specialist models while reducing MoE post-training costs.