A sparse Mixture-of-Experts layer can route many tokens toward the same expert even when every expert has identical nominal capacity. The router makes token-dependent choices, while distributed execution commonly allocates bounded token slots per expert. When those two mechanisms disagree, an expert can receive more assignments than its execution buffer admits.

That boundary is not an inherent property of every MoE architecture. It is a property of capacity-constrained routing designs, including the routing formulation described for Switch Transformers. In such systems, expert capacity converts an uneven routing distribution into an operational event: some assignments fit, while assignments beyond the capacity limit require an explicit overflow policy.

Routing sparsity does not imply balanced traffic

An MoE layer contains multiple parameterized expert networks and a router that selects a small subset for each token. With top-1 routing, a token is assigned to the expert with the strongest router score. Sparse activation keeps the number of active experts per token small even when the model contains many experts.

Sparse activation alone says nothing about how evenly tokens are distributed. If N tokens are routed across E experts, the average assignment count is

average tokens per expert = N / E

but the actual count for expert e is determined by routing decisions:

n_e = count(tokens assigned to expert e)

There is no general requirement that every n_e equal N / E. Correlated token representations or router preferences can concentrate assignments on a subset of experts.

This separates two forms of sparsity that are easy to conflate. Parameter activation is sparse because each token touches only a small part of the expert pool. Traffic can still be highly skewed because many tokens may choose the same part.

Capacity places a systems bound on router output

The Switch Transformer formulation defines expert capacity from the token count, expert count, and a capacity factor:

expert capacity =
    (tokens per batch / number of experts) * capacity factor

Ignoring integer rounding, a capacity factor of 1.0 provisions each expert near the average assignment count. A value above 1.0 reserves additional slots for routing imbalance.

The capacity factor does not force the router to distribute tokens evenly. It changes the amount of imbalance the execution layout can absorb before an expert exceeds its allotted slots.

For example, with 1,024 routed tokens and 16 experts, the mean is 64 tokens per expert. A conceptual capacity factor of 1.25 provides about 80 slots per expert before implementation-specific rounding. An expert receiving 76 assignments fits. One receiving 93 does not.

The numerical boundary is therefore a systems constraint layered on top of a data-dependent routing decision.

Overflow policy changes the layer computation

Once assignments exceed capacity, the implementation must decide what happens to the excess. The Switch Transformer design describes overflowed tokens as not being processed by the selected expert in that layer. Other MoE systems can use different routing or capacity policies, so token dropping is not a universal MoE guarantee.

This distinction matters because the router’s preferred assignment and the executed assignment are no longer equivalent after overflow. A router score can identify an expert as the preferred destination while capacity prevents that expert computation from occurring for every selected token.

Overflow is consequently more than a utilization metric. Under a dropping policy it changes the computation graph experienced by affected tokens. Under rerouting designs it can instead change which expert performs the computation. The exact semantic effect belongs to the routing implementation, not to the abstract idea of MoE.

Extra capacity trades overflow risk for padded work

Increasing capacity gives each expert more room for bursts of assignments. In a fixed-shape distributed implementation, however, those extra slots can also become padding when traffic is balanced or an expert receives few tokens.

That creates a direct tradeoff. Small capacity limits keep expert buffers compact but make overflow easier to trigger. Larger limits tolerate more skew but reserve more space and can increase computation or communication associated with padded capacity.

The tradeoff cannot be reduced to parameter count. Sparse MoE models may hold many expert parameters while activating only a subset per token, yet execution efficiency still depends on token movement, expert placement, buffer shapes, and load distribution.

A capacity factor is therefore partly an execution parameter. Its useful value depends on routing statistics and the distributed implementation around the experts.

Load-balancing objectives act before the capacity boundary

Capacity handles imbalance after routing scores have produced assignments. A load-balancing objective addresses a different part of the system: it adds training pressure intended to keep traffic from collapsing onto a small set of experts.

Switch Transformers use an auxiliary load-balancing loss based on router probabilities and token fractions assigned to experts. Its purpose is to encourage more balanced expert use. The loss does not make equal assignment a hard architectural guarantee.

That difference is operationally important. A soft training objective can reduce skew in aggregate while individual batches still cross an expert’s capacity. Capacity remains a hard execution boundary for implementations that allocate a fixed number of slots.

The two controls therefore should not be treated as interchangeable:

load-balancing objective -> influences routing behavior
expert capacity          -> bounds admitted assignments
overflow policy          -> defines excess-assignment handling

Each acts at a different stage.

Batch composition affects overflow pressure

Expert capacity is often computed from the number of tokens participating in a routing group. The observed assignment distribution depends on those tokens, so changing batch composition can change which experts become crowded even with fixed model parameters.

This makes overflow a property of both the router and the workload presented to it. Two batches with the same token count can produce different expert histograms. A capacity factor that leaves ample headroom for one routing distribution can be tight for another.

Distributed implementations add another boundary: routing may operate within groups rather than across every token in a global training step. The relevant N for capacity calculations is then the implementation’s routing group, not necessarily the global batch size shown by a training configuration.

Any capacity analysis therefore has to use the actual grouping and dispatch semantics of the serving or training stack.

Expert utilization and useful specialization are different measurements

A balanced assignment histogram can be attractive for hardware utilization, but equal traffic does not by itself establish that experts have developed useful specialization. Conversely, specialization does not automatically produce a distribution that maps efficiently onto fixed-capacity hardware.

Router quality, expert specialization, overflow rate, padding rate, communication volume, and device utilization measure different properties. Collapsing them into a single notion of “balanced MoE” hides the boundary between model behavior and execution behavior.

For capacity-constrained MoE layers, the central systems fact is narrower: token-dependent routing produces a variable load, while expert buffers impose a bounded load. Capacity factor determines the available headroom, load-balancing pressure can alter the assignment distribution, and the overflow policy determines what the layer does when demand crosses that bound.

Artificial Intelligence Mixture of Experts Sparse Models Routing Transformers