A mixture-of-experts layer can contain far more parameters than it evaluates for each token. A router scores a set of experts, selects a small subset, and sends each token only to those selected computation paths. The parameter count can grow without making every token execute every expert.
That sparse structure creates a separate systems problem: the router decides where computation lands. Two models with the same experts and the same nominal top-k routing can have very different behavior if one spreads tokens across experts and the other concentrates them on a few paths.
Mixture-of-experts routing is therefore more than a selection rule. Expert capacity, routing losses, token dispatch, and numerical details all affect whether sparse activation produces usable computation or persistent bottlenecks.
Sparse activation changes the unit of computation
Consider a feed-forward block replaced by eight experts. Each expert has the same input and output dimensions, but separate parameters. For token representation x, a router produces logits over the experts:
x -> router -> [1.2, -0.4, 2.1, 0.7, -1.0, 0.3, 1.5, 0.1]With top-2 routing, only the two selected experts process that token. If experts 3 and 7 receive the highest routing scores, the layer can be represented conceptually as:
x -> expert 3 --+
+-> weighted combination -> output
x -> expert 7 --+The exact weighting and normalization depend on the architecture. The core property is sparse activation: most expert parameters are inactive for this token.
This differs from an ordinary dense feed-forward block, where every token traverses the same parameter set. In an expert layer, the router makes the active parameter set data-dependent.
Top-k selection does not guarantee balanced traffic
A router can satisfy top-k selection for every token and still send most of a batch to the same experts. Suppose a batch contains 1,024 token positions and uses top-1 routing across eight experts. An even assignment would place about 128 tokens on each expert, but the router is not inherently constrained to produce that distribution.
If 700 tokens select one expert, the layer has a hot path. Other experts may be lightly used even though their parameters occupy memory. On distributed hardware, the overloaded expert can also determine the completion time of the expert computation phase because other workers may finish earlier and wait.
The routing distribution matters at more than one scale. A batch can look balanced in aggregate while particular devices receive uneven traffic. A model can also use experts evenly across a large corpus yet produce severe concentration for a narrow request class. Measurements need to match the actual dispatch boundary used by the serving or training system.
Capacity turns imbalance into a concrete constraint
Many mixture-of-experts implementations assign a finite token capacity to each expert for a routing group. Capacity is commonly related to the number of tokens, number of experts, and a capacity factor. A simplified form is:
capacity_per_expert = ceil(tokens * capacity_factor / experts)This expression is a useful model, not a universal interface contract. Specific architectures can group tokens differently, route to multiple experts, reserve extra capacity, or apply distinct rules during training and inference.
Finite capacity bounds the amount of work allocated to an expert. It also creates a policy question for tokens that select an expert whose capacity is already full. An implementation may drop the expert contribution, redirect traffic, use a fallback path, or avoid fixed dropping through a different dispatch design. Those choices are not interchangeable.
A larger capacity factor reduces overflow pressure but can increase allocated buffers and communication work. A smaller value limits per-expert work more tightly but gives concentrated routing less room. Capacity is therefore tied to both model behavior and the mechanics of token exchange.
Auxiliary objectives can discourage router collapse
Task loss alone does not necessarily reward balanced expert use. If one expert becomes slightly preferred early in training, more tokens can reach it, which changes the data and gradient it receives. That feedback can reinforce concentration.
Mixture-of-experts architectures often add an auxiliary routing objective that penalizes uneven assignment or relates routing probabilities to observed expert usage. The precise formula varies across model families, so an implementation should follow the objective defined by its architecture rather than treating all load-balancing losses as equivalent.
The coefficient on such an objective matters. If it is too small for the observed routing dynamics, it may have little effect on concentration. If it dominates the task objective, the router can be pushed toward uniform traffic even when specialization would support the model objective. The target is not uniformity for its own sake; it is enough routing diversity to keep sparse capacity usable without erasing useful expert specialization.
Some designs also regularize router logits directly. A penalty on large logits addresses a different mechanism from a load-balancing term: it constrains router score magnitude rather than directly prescribing token counts. Combining these ideas requires keeping their roles separate during diagnosis.
Routing probabilities and dispatch decisions expose different signals
Router probabilities are continuous, but expert selection is discrete. That distinction matters when inspecting a model.
Suppose two experts receive probabilities 0.51 and 0.49 under top-1 routing. Only the first expert receives the token even though the scores are close. A small parameter update can flip the dispatch decision. In contrast, probabilities 0.99 and 0.01 produce the same selected expert but indicate a much larger routing margin.
Token counts alone cannot show that difference. Router entropy, top-k margins, expert assignment counts, overflow counts, and per-expert token volumes describe different parts of the routing process. No single statistic captures both selection confidence and systems load.
This also makes aggregate averages easy to misread. An expert receiving 12.5 percent of tokens across a large run may still receive bursty traffic that exceeds local capacity in individual routing groups. Capacity failures are local events, so local distributions need direct inspection.
Distributed execution makes routing a communication problem
Experts are often partitioned across accelerators. Once the router chooses experts, token representations must be grouped and transferred to the devices that own those experts. Expert outputs then need to return to the original token order before the surrounding transformer computation continues.
Conceptually, the layer contains four distinct phases:
route -> dispatch -> expert compute -> combineThe arithmetic inside the experts is only one part of the cost. Dispatch can involve collective communication, temporary buffers, permutation of token representations, and synchronization across devices. A routing pattern that looks acceptable from a parameter-count perspective can still produce poor device utilization if communication or imbalance dominates the expert compute.
Top-k also changes this cost. Routing each token to two experts can provide a richer combination than top-1 routing under a given model design, but it also creates more expert assignments per token. That increases expert work and can increase communication volume when selected experts live on remote devices. The practical effect depends on placement, batching, interconnect characteristics, and the implementation of dispatch.
Router precision can affect discrete expert choices
The router sits at a sensitive boundary: small score changes can alter a discrete top-k decision. Reduced numerical precision can therefore matter even when the router is a small fraction of total model computation.
This does not imply that every router requires the same precision policy. It means router precision should be treated as an architecture and implementation choice rather than copied automatically from the expert matrix multiplications. If two candidate experts have nearly equal logits, rounding can change their ordering. Once selection changes, a token follows different parameters and may generate different downstream activations.
For debugging, it is useful to separate score instability from expert behavior. If expert assignments change before expert computation begins, investigating only expert outputs misses the source of the divergence.
Evaluate routing at the boundary that can fail
A useful routing evaluation connects model statistics to the actual resource boundary. If capacity is enforced per routing group, inspect expert occupancy per routing group. If experts are sharded across devices, inspect traffic per device as well as per expert. If latency matters, correlate routing concentration with dispatch and expert-compute timing rather than relying only on average expert usage.
The same principle applies to offline model evaluation. A balanced router on general text may concentrate heavily on code, a particular language, or repetitive structured input. Slices that resemble expected application traffic can expose routing patterns hidden by corpus-wide averages.
Sparse expert models gain their computational character from conditional execution, and conditional execution makes routing part of the model’s effective resource policy. Expert quality still matters, but the router determines which quality is available to each token and where the corresponding computation occurs. Treating routing metrics, capacity behavior, and dispatch cost as first-class model signals makes that boundary visible before it becomes a throughput or model-quality problem.