A sparse Mixture-of-Experts layer can contain many expert networks while activating only a small subset for each token. That conditional computation depends on a router, but router scores alone do not determine the executed graph. In implementations with bounded expert batches, each expert also has a finite number of token slots.
This creates a second boundary after expert selection: a token can prefer an expert that has no remaining capacity. The handling of that overflow is an implementation and architecture choice with direct consequences for training and serving.
Router scores produce a dispatch decision
Consider a router that maps token representation x_t to logits over E experts:
r_t = W_r x_t
p_t = softmax(r_t)A top-1 router selects one expert:
e_t = argmax_e p_t[e]Top-k routing generalizes the selection to several experts, usually with combination weights derived from router scores. The exact normalization and combination rule varies by architecture.
This selection is sparse because only selected experts evaluate the token. It does not imply that expert demand is balanced. Many tokens in the same batch can select the same expert while another expert receives few tokens.
Capacity converts imbalance into an execution constraint
Distributed MoE systems often pack routed tokens into fixed or bounded expert buffers. A common conceptual capacity is proportional to the average token count per expert:
capacity ≈ (tokens / experts) * capacity_factorThe precise rounding, grouping scope, and token accounting are implementation-specific. The key property is that capacity is finite.
If routing were perfectly balanced, each expert would receive about tokens / experts assignments. Router outputs need not have that distribution. Once assignments to an expert exceed its available slots, the runtime needs an overflow policy.
Some MoE designs can drop excess assignments for that layer. Other systems reroute, increase capacity, alter batching, or use routing schemes intended to avoid overflow. These behaviors are not interchangeable, so a statement about “MoE routing” is incomplete unless the capacity and overflow policy are known.
Dropped routing is not equivalent to selecting another expert
Suppose a top-1 router assigns token t to expert e, but e is full. Dropping that expert computation does not mean the token was processed by its second-ranked expert. A fallback requires an explicit routing rule that selects and dispatches the alternative.
This distinction matters when reading model code. A mask that zeros an overflowed dispatch weight can leave the surrounding residual path intact while removing the expert contribution for that token. Another implementation may send the token elsewhere. Both can preserve tensor shapes while implementing different functions.
Capacity behavior therefore belongs to model semantics at the MoE layer boundary, not merely to memory allocation.
Load balancing acts before overflow becomes severe
Routers can concentrate assignments because token representations are not uniformly distributed across experts. Training objectives often include an auxiliary balancing term or another routing constraint intended to spread traffic.
A balancing objective does not create a hard guarantee that every expert receives the same count. It changes optimization pressure on router behavior. The hard capacity limit remains a separate mechanism.
This separation is useful operationally. Router balance can be measured from assignment statistics, while overflow can be measured from assignments that cannot enter the selected expert buffer. A model can have nonuniform routing without overflow when spare capacity is sufficient, and it can overflow when a local batch or routing group becomes concentrated even if aggregate traffic looks balanced.
Capacity factor trades spare slots against overflow pressure
Increasing the capacity factor allocates more token slots per expert relative to the even-share baseline. That gives skewed routing more room before an expert fills. It can also increase padded computation, communication volume, or buffer footprint depending on the distributed implementation.
Reducing capacity tightens those resources but leaves less room for assignment skew. The resulting effect is conditional: the same capacity factor can behave differently under different batch composition, routing-group size, number of experts, and router distribution.
For that reason, capacity factor is not a model-quality knob in isolation. It participates in a system-level contract connecting router statistics, tensor shapes, communication, and overflow handling.
Training and serving can expose different routing pressure
MoE routing is often described from the training graph, where batches contain many tokens and balancing losses can influence router parameters. Serving has a different traffic shape. Prefill may present many prompt tokens together, while autoregressive decode can process small token groups across concurrent requests.
The model parameters may be identical, yet the runtime’s packing and expert-parallel strategy can change the local set of tokens competing for expert slots. Capacity defined per device, per routing group, or per batch therefore needs to be interpreted in that scope.
A serving engine that uses dynamic token scheduling may also encounter demand patterns that differ from static training batches. This does not change router logits for a fixed hidden state, but it can change which other tokens contend for bounded execution resources at the same time.
Expert count does not equal active compute per token
Adding experts increases the parameter pool, but sparse routing keeps per-token activation tied to the selected expert count rather than all experts. Capacity adds another dimension: total expert parameters, selected experts per token, and available token slots per expert are separate quantities.
That separation is central to deployment planning. Parameter memory depends on how experts are stored and sharded. Communication depends on where selected experts reside. Compute depends on dispatched tokens and expert structure. Overflow depends on assignment concentration relative to capacity.
A sparse MoE layer is therefore not defined by router top-k alone. Its executed behavior also depends on the capacity boundary and the policy applied when token demand crosses it. Any implementation that changes those rules can preserve the same expert weights while changing which expert computations actually contribute to a token.