A Switch-style sparse layer can send many tokens toward the same expert even when every expert has identical compute capacity. The router makes token-level choices from model-produced scores; it does not inherently produce an even partition. A fixed expert capacity therefore creates a hard boundary between routing preference and the amount of expert computation admitted for a batch.
In the Switch Transformer formulation, each token is routed to the expert with the highest router probability. Each expert receives a fixed token capacity derived from the token count, expert count, and a capacity factor. When assignments exceed that capacity, excess tokens overflow instead of enlarging the expert batch without bound.
Top-1 routing makes activation sparse
Let a sparse layer contain E experts. For token representation x, a router produces logits over those experts and converts them to probabilities:
p = softmax(W_r x)
e* = argmax_e p_eThe selected expert e* processes the token. This top-1 choice differs from a dense feed-forward layer in which the same parameter set processes every token. The model may contain many expert parameter sets while activating only one expert for a token at this layer.
Sparse activation does not imply balanced activation. Two tokens can select the same expert, and an entire batch can produce a skewed assignment distribution. Router probabilities express model preferences; they are not a scheduling guarantee.
Expert capacity bounds admitted tokens per expert
A common Switch capacity definition is
C = (T / E) * capacity_factorwhere T is the number of tokens being routed and E is the number of experts. Concrete implementations must map this quantity to an integer capacity according to their tensor-shape rules.
A capacity factor of 1.0 allocates capacity corresponding to an even token split. A factor above 1.0 adds buffer for routing imbalance. The extra slots reduce overflow risk but can increase padding, computation, and communication associated with the expert batches.
Capacity is a property of the execution arrangement, not a constraint forcing the router to choose each expert exactly T / E times. The router can still prefer one expert for more than C tokens.
Overflow appears when preference exceeds capacity
Consider T = 12 tokens, E = 3 experts, and a capacity factor of 1.0. Each expert has capacity:
C = (12 / 3) * 1.0 = 4Suppose top-1 routing produces assignment counts:
expert 0: 7 tokens
expert 1: 3 tokens
expert 2: 2 tokensExpert 0 has three assignments beyond its four-token capacity even though five slots remain unused across experts 1 and 2. Fixed per-expert capacity does not automatically transfer idle slots to an overloaded expert.
In the Switch Transformer design, overflow tokens are not processed by the selected expert in that sparse layer. This behavior is specific to that routing design; other mixture-of-experts systems can use different overflow policies, rerouting rules, multiple selected experts, or dynamic capacity schemes.
Capacity factor trades overflow against reserved work
Increasing the capacity factor changes the execution envelope without changing the basic top-1 selection rule. With the same 12-token, 3-expert example, a factor of 1.5 gives a nominal capacity of six tokens per expert:
C = (12 / 3) * 1.5 = 6The seven-token assignment to expert 0 still exceeds capacity, but only one token now overflows. The system reserves more expert-batch space to absorb skew.
That buffer has a cost. Static expert batches can contain unused slots when routing is more balanced than the reserved capacity anticipates. In distributed execution, larger capacity can also enlarge the data movement associated with dispatching token representations to experts and combining results. Capacity therefore controls a systems tradeoff rather than acting as a model-quality guarantee.
Router balancing pressure is separate from the hard capacity
Switch Transformer training uses an auxiliary load-balancing objective to encourage routing across experts. That objective changes optimization pressure on the router; it does not replace the fixed capacity boundary.
The distinction matters because soft optimization pressure and hard execution limits operate at different stages. A balancing objective can make severe skew less attractive during training, while capacity still determines what happens when the realized assignments are uneven for a particular batch.
Perfect balance is not required for sparse routing to function, and the auxiliary objective cannot guarantee identical expert counts for every batch. Capacity remains necessary in implementations that require bounded expert tensor shapes or bounded per-expert work.
Token count and routing group define the capacity calculation
The symbol T needs an implementation-level definition. A distributed system may route tokens within a local group rather than across every token in a global batch. In that case, capacity is computed for the routing group used by the implementation, and imbalance is observed within that boundary.
This distinction affects both overflow and utilization. Two systems with the same global batch size, expert count, and nominal capacity factor can expose different effective routing conditions if their dispatch groups differ.
Padding tokens also require explicit treatment. Tokens excluded from routing should not silently be counted as ordinary workload unless the implementation defines them that way. The capacity equation is only meaningful when its token population matches the actual dispatch semantics.
Sparse parameters do not make routing free
Top-1 routing reduces the number of experts activated per token, but the sparse layer still carries router computation, token dispatch, expert execution, and result combination. When experts reside on different devices, dispatch and combine operations can become communication-sensitive.
Expert capacity gives these operations a bounded shape in the Switch design. It also exposes imbalance as a measurable event: overflow indicates that router demand for an expert exceeded the space reserved for that expert.
The useful boundary is precise. Top-1 routing determines the preferred expert, the capacity factor determines reserved per-expert space, and overflow handles assignments that cross that space. Load-balancing pressure can influence the assignment distribution, but it does not erase the capacity boundary.