Attention gets much of the attention in transformer explanations, but every transformer layer also contains a feed-forward network that performs substantial computation on each token representation. The activation function inside that network is a small-looking design choice with an important job: it introduces nonlinearity so the network can learn transformations that stacked linear projections alone cannot express.
Understanding this part of a transformer helps when reading model architectures, comparing implementations, estimating parameter and compute costs, or deciding whether two feed-forward designs are actually equivalent.
Start with the feed-forward network
After an attention sublayer has mixed information across token positions, a transformer typically applies a feed-forward transformation independently to each position. A simple form is:
FFN(x) = W2 activation(W1 x + b1) + b2Here:
xis one token representation;W1projects it into a usually wider hidden space;- the activation function transforms that intermediate representation;
W2projects it back toward the model dimension.
The same learned weights are applied at every sequence position. If the input contains 1,000 tokens, the feed-forward network performs the same kind of transformation for each token separately. Attention is the component that mixes information between positions; the feed-forward network transforms the representation at each position.
A useful mental model is:
attention: exchange information between tokens
feed-forward: transform each token's representationResidual connections and normalization surround these sublayers in common transformer architectures, although their exact placement varies by model.
Why an activation function is necessary
Suppose the activation were removed and biases were ignored for simplicity:
FFN(x) = W2(W1 x)Matrix multiplication is associative, so the two projections can be combined:
FFN(x) = (W2 W1)xThe result is still one linear transformation. Making the intermediate layer wider would add parameters, but the composed mapping would remain linear.
An activation function breaks that collapse:
FFN(x) = W2 activation(W1 x)Because the activation is nonlinear, the second projection receives a transformed signal that cannot generally be represented by simply multiplying W1 and W2 together. This gives the network the ability to build richer input-dependent transformations.
This is the same basic reason nonlinear activations matter in neural networks outside transformers.
ReLU provides the simplest mental model
The rectified linear unit, or ReLU, is:
ReLU(z) = max(0, z)For a single intermediate value:
z = -2.0 -> ReLU(z) = 0
z = 0.5 -> ReLU(z) = 0.5
z = 3.0 -> ReLU(z) = 3.0ReLU is easy to reason about. Negative values become zero while positive values pass through unchanged.
Early transformer architectures used ReLU in their position-wise feed-forward networks. It remains useful as a teaching model because the role of the nonlinearity is visible immediately.
ReLU also has limitations. Its output changes abruptly at zero, and units receiving negative pre-activation values have zero local gradient through the ReLU itself. Other activation functions use smoother transitions or different gating behavior.
GELU changes values smoothly
The Gaussian Error Linear Unit, or GELU, is another activation used in transformer models. Unlike ReLU, GELU does not simply keep every positive input and discard every negative one.
Conceptually, GELU scales an input according to its magnitude using the standard normal cumulative distribution function:
GELU(x) = x * Phi(x)where Phi(x) is the probability that a standard normal random variable is less than or equal to x.
This means the gate is smooth rather than binary. Large positive inputs are mostly preserved, values near zero are scaled, and some negative inputs remain negative instead of being forced immediately to zero.
The important practical point is not that GELU is universally better than ReLU. The activation is part of a model’s trained architecture. Replacing it in an already trained network changes the function the network computes and is not a free inference optimization.
Gated feed-forward networks use two learned paths
Many transformer architectures use a gated feed-forward network rather than the simple two-projection form.
A generic gated form can be written as:
gate = activation(Wg x)
value = Wv x
hidden = gate * value
output = Wo hiddenThe * represents element-wise multiplication.
This design creates two learned views of the same input. One path determines a gate; the other produces values to be modulated by that gate. The multiplication lets the network control which transformed features pass forward and by how much.
The structure is different from merely changing activation() in a conventional feed-forward network. A gated block has additional projections and therefore different parameter and compute characteristics for a given intermediate width.
SwiGLU combines SiLU with gating
One widely used gated variant is SwiGLU. Its gate uses the SiLU activation, also called the swish activation in closely related formulations:
SiLU(x) = x * sigmoid(x)A simplified SwiGLU-style block is:
gate = SiLU(Wg x)
value = Wv x
hidden = gate * value
output = Wo hiddenThe name reflects the combination of a swish-like activation with a gated linear unit structure.
It is useful to separate two decisions that are sometimes discussed as though they were one:
- which nonlinear function is used, such as ReLU, GELU, or SiLU;
- whether the feed-forward block uses a simple activation path or a gated architecture.
SwiGLU changes both the nonlinear behavior and the block structure relative to a basic ReLU feed-forward network.
Intermediate width affects the comparison
A conventional feed-forward network with model dimension d and intermediate dimension m has two main weight matrices:
W1: d x m
W2: m x dIgnoring biases, that is approximately:
2dm parametersA gated block with separate gate and value projections has three main matrices:
Wg: d x m
Wv: d x m
Wo: m x dor approximately:
3dm parametersTherefore, comparing two architectures at the same intermediate width does not compare equal parameter counts. A model designer can choose a smaller gated intermediate dimension to compensate for the extra projection.
This is why statements such as “SwiGLU has more parameters” need a condition attached. It has an extra projection relative to a simple two-matrix block at the same dimensions, but complete model architectures can choose different widths to target similar parameter or compute budgets.
Activation choice is tied to training
It can be tempting to treat activation functions like runtime settings. They are not comparable to generation controls such as temperature.
During training, optimization adapts all surrounding weights to the architecture that is actually present. A model trained with a GELU feed-forward network has learned parameters in the context of GELU. A model trained with a SwiGLU block has learned a different arrangement of projections and nonlinear operations.
Changing the activation after training generally changes model behavior. Changing a simple feed-forward block into a gated one also requires parameters that the original model does not contain.
For developers using pretrained models, the practical rule is straightforward:
Reproduce the feed-forward architecture defined by the model rather than substituting an activation because another model uses it.
This matters when implementing a model from a configuration file, porting weights between frameworks, or writing a custom inference runtime.
Numerical details can affect implementations
The mathematical name of an activation does not always specify every implementation detail.
For example, GELU may be computed from its exact definition or from an approximation. Frameworks can expose more than one mode. Two implementations that both say “GELU” may therefore produce slightly different floating-point results.
Precision also matters. Transformer inference commonly uses reduced-precision arithmetic, and kernels may fuse projections, activations, and element-wise operations for efficiency. Fusing operations can reduce memory traffic and kernel-launch overhead, but the exact numerical behavior depends on the implementation and precision.
If you are reproducing a pretrained model, use the activation variant and numerical conventions expected by that model and runtime. Do not infer them from the broad architecture name alone.
Do not judge an activation in isolation
It is difficult to make a useful architecture decision by asking only whether ReLU, GELU, or SwiGLU is “best.”
Model quality depends on the full training setup: architecture, width, depth, data, optimizer, learning-rate schedule, training budget, regularization, and many other choices. A result measured for one model scale or training recipe does not automatically transfer to another.
Likewise, inference performance depends on hardware and software. An activation that requires more arithmetic may still be efficient in a runtime with a highly optimized fused kernel, while an apparently simpler formulation can perform poorly if it causes additional memory movement or unsupported operations.
When comparing designs, hold the relevant budget constant and measure the property you actually care about. Depending on the project, that may be model quality at a fixed parameter count, training compute, inference latency, throughput, or memory use.
Common mistakes when reading transformer architectures
One mistake is to describe the entire feed-forward network by its activation name. Saying that a model “uses SwiGLU” communicates more than saying it “uses SiLU,” because SwiGLU implies a gated structure with multiple projections.
Another mistake is to compare intermediate dimensions without accounting for the number of matrices. Equal hidden widths do not imply equal parameter counts across simple and gated blocks.
A third mistake is to assume the feed-forward network mixes tokens. In the standard position-wise design, it does not. Each position is transformed independently with shared parameters. Token-to-token interaction occurs in attention or another sequence-mixing component.
Finally, do not swap activations in pretrained weights and expect equivalent output. The activation is part of the learned function.
What developers should remember
The feed-forward network is not a minor appendix to transformer attention. It is a core transformation in every layer, and its nonlinear activation prevents the block’s learned projections from collapsing into a single linear mapping.
ReLU gives the clearest starting point. GELU provides a smooth nonlinear gate. Gated architectures add a second learned path, and variants such as SwiGLU combine that structure with SiLU-like gating. Those choices affect parameterization, computation, and the function the model learns.
When using an existing model, match its architecture exactly. When designing or comparing models, evaluate the whole feed-forward block under a meaningful parameter, compute, and quality budget rather than choosing an activation by reputation alone.