When Does Expert Sharing in MoE Models Pay Off? New Theory Links Local Error to Final Outputs
An arXiv paper develops a theory of when sharing experts across layers or steps in mixture-of-experts (MoE) models translates local prediction gains into better final outputs. In specified linear regression settings, the authors derive exact sharing and merging criteria, characterize expected final-output risk under a fixed linear propagation map, and construct cases where sharing lowers average local error yet increases final-output error. Experiments on low-data symbolic tasks with trained Transformers show statistically supported teacher-forced one-step gains without statistically resolved free-running final-output gains, while separate intervention tests distinguish genuine multi-layer expert actions from single-expert attribution. The work also derives conditional end-to-end error bounds accounting for distribution shifts, surrounding component errors, and changes in expert selection.
It gives architects and interpretability researchers concrete criteria for when MoE expert sharing and routing actually improve end-to-end reasoning, plus cautionary evidence on attributing capabilities to individual experts.
Comments & Questions (0)
No comments yet
Be the first to comment!