Back to AI News
arxiv

When Does Expert Sharing in MoE Models Pay Off? New Theory Links Local Error to Final Outputs

An arXiv paper develops a theory of when sharing experts across layers or steps in mixture-of-experts (MoE) models translates local prediction gains into better final outputs. In specified linear regression settings, the authors derive exact sharing and merging criteria, characterize expected final-output risk under a fixed linear propagation map, and construct cases where sharing lowers average local error yet increases final-output error. Experiments on low-data symbolic tasks with trained Transformers show statistically supported teacher-forced one-step gains without statistically resolved free-running final-output gains, while separate intervention tests distinguish genuine multi-layer expert actions from single-expert attribution. The work also derives conditional end-to-end error bounds accounting for distribution shifts, surrounding component errors, and changes in expert selection.

AI Brief

It gives architects and interpretability researchers concrete criteria for when MoE expert sharing and routing actually improve end-to-end reasoning, plus cautionary evidence on attributing capabilities to individual experts.

arXiv:2611.17879v1 Announce Type: cross Abstract: Mixture-of-experts (MoE) models can share experts across layers or computational steps, pooling training information from different contexts. This can reduce local prediction error, but multi-step systems are ultimately evaluated by their final outputs. We study when these local gains survive composition. In specified linear regression models, we derive exact sharing and merging criteria by balancing variance savings against bias from sharing across roles with different target behavior. Under a common fixed linear propagation map, we characterize expected squared final-output risk, retaining error directions and correlations. We construct cases where sharing lowers average local error yet increases final-output error. Within this setting, we also identify when orthogonal full pooling with isotropic noise preserves every local improvement at the final output. For general routed computations, we derive conditional end-to-end error bounds that account for distribution shifts, errors in surrounding components, and changes in expert selection. Separately, we show that correct predictions and route labels alone do not establish that individual experts implement the attributed operations. Controlled experiments realize a strict local-to-final risk reversal. In low-data symbolic settings, trained Transformers exhibit statistically supported teacher-forced one-step gains without statistically resolved free-running final-output gains. Separate intervention tests distinguish successful multi-layer actions from single-expert attribution and generalization to unseen transitions. Together, these results distinguish when sharing improves local estimation, when those gains survive to the final output, and what evidence is needed to attribute reusable operations to individual experts.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!