Neuron-Level Specialization in Mixture-of-Experts Language Models
This arXiv paper introduces Routing-Aware Magnitude Profiling (RAMP) to study domain-specialized computation at the neuron level in Mixture-of-Experts models. It reports domain-specialized neuron sets with 58-74% overlap across same-domain samples versus 15-24% across different domains. The analysis traces neuron importance through pre-training and post-training across three open MoE families, finding continued reorganization between pre-training checkpoints but largely preserved rankings after post-training. It also presents RAMP-based Pruning (RAMPr), which preserves 91-98% of full performance at 75% neuron retention across six models and, at 50% retention, reduces resident GPU memory by 36-43% and improves batched prefill throughput by 17-39%.
It offers a concrete neuron-level method for understanding and pruning MoE models, with reported efficiency and performance trade-offs relevant to model researchers and engineers.
Comments & Questions (0)
No comments yet
Be the first to comment!