Mixture of Experts

Understand Mixture-of-Experts Models

A neural network does not have to use every parameter for every input. A mixture-of-experts (MoE) layer takes advantage of this idea by keeping several expert networks and using a router to select only a small subset for each token. This creates an important distinction: a model can have a large total parameter count while activating far fewer parameters for one token. That can increase model capacity without making the arithmetic performed for every token grow in direct proportion to the total number of expert parameters.