Live intelligence

Latest AI updates

1 updates
Research

When Does Expert Sharing in MoE Models Pay Off? New Theory Links Local Error to Final Outputs

An arXiv paper develops a theory of when sharing experts across layers or steps in mixture-of-experts (MoE) models translates local prediction gains into better final outputs. In specified linear regression settings, the authors derive exact sharing and merging criteria, characterize expected final-output risk under a fixed linear propagation map, and construct cases where sharing lowers average local error yet increases final-output error. Experiments on low-data symbolic tasks with trained Transformers show statistically supported teacher-forced one-step gains without statistically resolved free-running final-output gains, while separate intervention tests distinguish genuine multi-layer expert actions from single-expert attribution. The work also derives conditional end-to-end error bounds accounting for distribution shifts, surrounding component errors, and changes in expert selection.

Mixture-of-Experts modelsmulti-step reasoningerror propagation