Use Gated Feed-Forward Blocks in Transformers
A transformer block does more than attention. After attention mixes information across token positions, a feed-forward block transforms each token representation independently. That second stage often accounts for a large share of the model’s parameters and computation.
A gated feed-forward block changes this stage by adding a second projection that controls how much of another projected signal passes through. GLU, GEGLU, and SwiGLU are common names for members of this family.
For developers reading model definitions or implementing transformer variants, the useful mental model is simple: a standard feed-forward block transforms one hidden stream, while a gated block creates a value stream and a gate stream, combines them element by element, then projects the result back to the model width.
Start with the ordinary feed-forward block
Let a token representation be a vector (x) with model width (d). A common two-layer feed-forward block has the form
[ \operatorname{FFN}(x) = \phi(xW_1 + b_1)W_2 + b_2 ]
where (W_1) expands the representation from width (d) to a larger hidden width (d_{ff}), (\phi) is a nonlinear activation, and (W_2) projects the result back to width (d).
Attention can connect one token to information from other positions. The feed-forward block then applies the same transformation to each position separately. It does not mix token positions by itself.
For a single token, imagine the first projection produces four hidden values:
[ [1.2,\ -0.5,\ 0.8,\ 2.0] ]
An activation changes those values, and the second projection mixes the resulting hidden features back into a vector of width (d).
The gated version keeps the broad expand-transform-contract pattern but changes the middle.
A gate adds a second hidden stream
A GLU-style block computes two projections from the same input. One becomes the value stream. The other becomes the gate stream.
A useful generic form is
[ h = (xW_v) \odot \phi(xW_g) ]
followed by
[ y = hW_o ]
Here, (W_v) produces values, (W_g) produces gate activations, (W_o) projects back to model width, and (\odot) means element-wise multiplication.
Suppose the value stream is
[ [2.0,\ -1.0,\ 0.5] ]
and the activated gate stream is
[ [0.8,\ 0.1,\ 1.2]. ]
Their product is
[ [1.6,\ -0.1,\ 0.6]. ]
The gate does not select an entire token or attention head. It scales hidden features independently. A gate value near zero suppresses its corresponding value feature. A larger positive value gives that feature more influence. Depending on the activation, gate values can also be negative, so “gate” should not be interpreted as a strict switch constrained to the interval from zero to one.
This distinction matters when reading implementations. A GLU-style gate is multiplication between hidden vectors, not necessarily a sigmoid mask.
GLU, GEGLU, and SwiGLU differ mainly in the gate activation
The original gated linear unit uses a sigmoid gate. In a simplified notation,
[ \operatorname{GLU}(x) = (xW_v) \odot \sigma(xW_g). ]
Related variants replace the sigmoid with another activation.
GEGLU uses GELU on the gate branch:
[ \operatorname{GEGLU}(x) = (xW_v) \odot \operatorname{GELU}(xW_g). ]
SwiGLU uses the SiLU function, also commonly associated with the Swish family:
[ \operatorname{SwiGLU}(x) = (xW_v) \odot \operatorname{SiLU}(xW_g). ]
After this gated product, another linear projection returns the hidden vector to model width.
Names and projection ordering can vary across codebases. One implementation may call the two branches gate and up; another may reverse the written multiplication order. Since element-wise multiplication is commutative, branch naming alone does not establish a behavioral difference. Check which activation is applied and inspect the tensor dimensions.
Gating changes the parameter budget
A common implementation mistake is to replace a two-projection feed-forward block with a three-projection gated block while keeping the same intermediate width, then compare the models as if their sizes were equivalent.
Ignoring biases, a standard feed-forward block with model width (d) and hidden width (d_{ff}) contains approximately
[ 2dd_{ff} ]
weights: one matrix going up to the hidden width and one coming back down.
A gated block with gate width (d_g) uses three major matrices:
[ 3dd_g. ]
There are separate value and gate projections plus the output projection.
To make those leading parameter counts equal,
[ 3dd_g = 2dd_{ff}, ]
so
[ d_g = \frac{2}{3}d_{ff}. ]
As a teaching example, if a standard block uses (d_{ff}=3072), a parameter-matched gated block would use (d_g=2048):
[ 2d(3072) = 3d(2048) = 6144d. ]
Real architectures may choose widths for hardware alignment, established model recipes, or other design constraints, so the exact ratio is not a requirement. It is a useful baseline for fair comparisons.
Parameter matching also does not guarantee identical runtime. Kernel fusion, memory traffic, tensor shapes, numeric precision, accelerator characteristics, and framework implementation can change actual latency and throughput.
Implement the tensor flow explicitly
Framework APIs differ, but the core data flow is small. Pseudocode for a bias-free SwiGLU block looks like this:
value = value_projection(x)
gate = gate_projection(x)
hidden = value * silu(gate)
output = output_projection(hidden)If x has shape [batch, sequence, d], both value and gate typically have shape [batch, sequence, d_g]. Their element-wise product therefore has the same shape. The output projection maps the last dimension from d_g back to d.
The two input projections are often computed together for efficiency:
value_and_gate = input_projection(x)
value, gate = split(value_and_gate)
hidden = value * silu(gate)
output = output_projection(hidden)This fused representation does not change the mathematical role of the two branches. It is an implementation choice that can reduce overhead on some systems.
When porting weights between implementations, tensor layout becomes significant. A combined projection may store value and gate weights in a particular order. Swapping the halves still produces tensors with valid shapes, but the model’s outputs will be wrong.
Residual connections and normalization stay outside the gate
The gated feed-forward block is usually only one subcomponent of a transformer layer. Residual connections and normalization are separate architectural decisions.
Conceptually, a layer may contain operations such as
x = x + attention(normalize(x))
x = x + gated_ffn(normalize(x))but transformer families differ in normalization placement, residual structure, bias use, and related details. Replacing the feed-forward function does not imply that those surrounding choices should also change.
This separation is useful during implementation. Test the gated block first as a shape-preserving function from model width back to model width. Then integrate it into the layer’s existing residual and normalization structure.
Common mistakes are mostly structural
The formulas are short, but several implementation errors can survive basic shape checks.
Applying the activation to the wrong tensor. A named architecture such as SwiGLU specifies which branch receives SiLU before multiplication. If weights come from an existing checkpoint, branch semantics must match the checkpoint layout.
Forgetting the extra projection in size estimates. A gated block has two input-side projections rather than one. Keeping the old hidden width can increase parameter count and compute.
Treating the gate as a probability. SiLU and GELU gates are not bounded to ([0,1]). Their values should not be interpreted as calibrated probabilities or binary routing decisions.
Splitting a fused projection incorrectly. A fused matrix can place the two branches in different orders across implementations. Confirm the checkpoint convention rather than inferring it from variable names.
Comparing architectures at unequal budgets. If one model has materially more parameters or computation, an observed quality difference cannot be attributed to gating alone.
These errors are easier to catch with small deterministic tests. Feed a fixed tensor through a reference implementation and the ported implementation, then compare intermediate value, gate, hidden, and output tensors rather than checking only the final shape.
Gating is not sparse expert routing
The word “gate” also appears in mixture-of-experts systems, but the mechanisms serve different roles.
A GLU-style feed-forward block computes dense hidden vectors for each token and multiplies corresponding features. It does not, by itself, choose a subset of expert networks.
A mixture-of-experts router typically produces scores used to select or weight separate expert modules. That routing decision can change which parameter subsets process a token.
Both mechanisms involve multiplicative control, but they operate at different structural levels. Keeping them separate prevents misleading assumptions about conditional computation or sparse execution in an ordinary SwiGLU block.
Decide based on the full architecture and budget
A gated feed-forward block is a reasonable choice when you are implementing an architecture that specifies one, reproducing a compatible checkpoint, or evaluating feed-forward variants under controlled parameter and compute budgets.
It is not a drop-in guarantee of higher quality. Changing the feed-forward block changes parameterization and can interact with initialization, optimizer settings, model width, training duration, and surrounding architecture. A comparison is most informative when those factors are controlled or explicitly accounted for.
For an existing deployed model, compatibility can matter more than architectural preference. Changing a standard feed-forward block to SwiGLU changes weight shapes and semantics, so old checkpoint tensors cannot generally be copied into the new block without a deliberate conversion or new training.
If the current architecture already meets its quality, latency, and memory targets, leaving it unchanged may be the simpler engineering decision.
A practical checkpoint for implementation
When a transformer specification says GLU, GEGLU, or SwiGLU, translate the name into three concrete questions: which activation is applied to the gate branch, what width the two branches use, and how the resulting product is projected back to model width.
Then verify tensor shapes and parameter counts before training or loading a checkpoint. That small check catches many expensive mistakes early and makes architecture comparisons much easier to interpret.