A sparse mixture-of-experts layer can contain many expert networks while evaluating only a small subset for each token. The router makes that sparsity possible: it assigns scores to experts, selects a limited set, and sends each token through the selected computation paths.
That selection is not only an optimization detail. If many tokens concentrate on a few experts, some devices can receive much more work than others, capacity limits can discard or redirect assignments, and experts that receive little traffic get fewer task gradients. Router balance therefore affects both computation and the function represented by the model.
Routing turns one dense layer into conditional computation
Consider a layer with E experts. For token representation x, a router produces one score per expert. A common form uses a linear projection followed by a softmax:
p = softmax(W_r x)The model then selects the top k experts according to p. With top-1 routing, one expert processes the token. With top-2 routing, two experts can contribute, typically with weights derived from their router scores.
The expert networks can share the same architecture while holding separate parameters. The router determines which parameter subset participates in the token’s forward path. As a result, two tokens at the same layer can invoke different expert parameters even though they pass through the same model block.
Sparse activation reduces expert computation relative to evaluating every expert for every token. It does not make routing free. Implementations must score experts, group token assignments, move token representations to the devices that host selected experts, execute expert computation, and combine the returned outputs.
Router preference and expert load are different quantities
A router can assign probabilities that look moderately distributed while discrete top-k choices remain concentrated. The reverse can also occur near decision boundaries. Monitoring only mean router probabilities therefore does not fully describe the work sent to each expert.
For a batch containing T routed tokens, define an assignment count for expert i:
n_i = number of tokens assigned to expert iThe corresponding load fraction is n_i / T for top-1 routing. With multiple selected experts per token, the denominator and counting rule need to match the implementation because there are more assignments than tokens.
Load counts expose operational imbalance directly. Router probabilities expose the model’s preference distribution before or around discrete selection. Both signals matter during training because an auxiliary balancing objective may be computed from router probabilities, assignment frequencies, or a combination of the two.
A perfectly uniform load is not inherently the target for every model state. Expert specialization can produce nonuniform traffic. The engineering problem appears when concentration is large enough to create capacity pressure, poor device utilization, or experts with too little traffic to receive useful updates.
Capacity creates a hard boundary after routing
Distributed implementations commonly allocate a finite token capacity per expert for a routing group. A conceptual capacity can be written as:
capacity = ceil(capacity_factor * expected_assignments_per_expert)The exact formula varies across implementations, especially for top-k routing and different grouping schemes. The key property is fixed: each expert has a bounded number of token slots for the group being dispatched.
If an expert receives more assignments than its capacity, the system needs an overflow policy. An implementation may drop excess expert assignments, route them through another choice, or use another explicit fallback. These choices are not equivalent. Dropping an assignment changes the computation performed for that token, while rerouting changes which expert parameters process it.
Increasing the capacity factor gives routing more room to absorb imbalance but reserves additional expert-buffer space and can increase communication or computation associated with padded capacity. Decreasing it tightens resource use while making overflow more likely under concentrated routing. Capacity is therefore part of the model-serving and training configuration, not merely a memory constant.
Balance losses influence the router rather than equalizing experts directly
A routing objective can add an auxiliary term that discourages persistent concentration. Such a term changes router gradients so that expert usage becomes less skewed according to the chosen statistic.
The coefficient on that auxiliary term sets its influence relative to the primary model objective. If it is too small for the scales involved, the primary objective can dominate and routing can remain concentrated. If it is made very large, the router can be pushed toward balanced assignments even when stronger specialization would reduce the primary loss.
This makes the raw auxiliary-loss value insufficient as a diagnostic. It should be paired with observed expert loads, overflow counts, router-score statistics, and the primary task metric. A falling balance loss does not establish that dispatch is free of hotspots, and uniform assignment counts do not establish that the model is using experts effectively.
The balancing formulation also needs to match the routing scope. Statistics computed per local batch, per device, or across a larger routing group can produce different gradients because each scope sees a different token distribution. In distributed training, the scope of aggregation is part of the algorithmic definition.
Small routing groups make balance estimates noisy
Expert load is discrete. When a routing group contains only a small number of tokens relative to the number of experts, even an unbiased routing process cannot divide assignments smoothly across every expert in every group.
For example, 32 top-1 assignments across 16 experts have an average load of only two assignments per expert. A difference of one token already represents a large relative change from that average. Treating each small group as if it should be nearly uniform can overstate imbalance.
Larger routing groups provide more assignments over which load can average out, but they can require broader communication or different batching. The useful diagnostic is therefore the load distribution over the same grouping boundary used by dispatch, followed by aggregation across enough groups to distinguish persistent concentration from ordinary finite-sample variation.
Sequence composition can also shift routing. A batch dominated by one token pattern may legitimately produce a different expert distribution from a heterogeneous batch. Comparing load traces without recording the routing-group construction can make a data-composition change look like a router regression.
Expert traffic affects optimization as well as throughput
An expert receives primary-task gradients only from tokens routed through its active path. An expert that receives little traffic has fewer such contributions during that interval. An overloaded expert, in contrast, can receive many assignments but may also encounter capacity truncation depending on the dispatch policy.
This creates feedback between routing and expert parameters. Router choices affect which data each expert sees; expert updates alter the outputs that can make those routes useful later. The router and experts are therefore coupled during optimization even when their parameters are stored separately.
A load-balancing term addresses only part of that coupling. It can encourage traffic distribution, but it does not guarantee semantic specialization, equal gradient magnitudes, or equivalent expert quality. Those properties require separate measurements if they matter to the application.
Router instability can also appear as frequent changes in token-to-expert assignments near score boundaries. Two experts with close router scores can swap order after a small parameter update. Hard top-k selection then changes the active computation path even when the underlying score change is small. Tracking routing entropy or assignment consistency can expose this behavior, but neither metric alone defines a fault.
Serving constraints can expose training-time assumptions
A model trained with a particular expert layout may be served under different batch sizes, routing-group sizes, device topology, or capacity settings. Those changes can alter the operational effect of the same router outputs.
If serving uses tighter expert capacity than training, token concentration that was harmless during training can begin to trigger overflow. If experts are distributed across devices, skew can create stragglers because a request batch cannot finish its expert phase until required dispatch work completes. A router distribution that is acceptable from a model-loss perspective can still be expensive under the serving topology.
For that reason, evaluation of a sparse mixture-of-experts model benefits from keeping routing measurements beside ordinary model metrics. Expert assignment histograms, overflow rates, dispatched token counts, and per-expert execution load describe behavior that aggregate accuracy or loss cannot expose.
The central boundary is that sparse experts save computation by making execution conditional. Once execution is conditional, the distribution of those conditions becomes part of the system. Router scores, discrete assignments, expert capacity, and overflow policy together determine which expert computation actually reaches each token.