Deep Learning

Understand Activation Functions in Transformer Feed-Forward Networks

Attention gets much of the attention in transformer explanations, but every transformer layer also contains a feed-forward network that performs substantial computation on each token representation. The activation function inside that network is a small-looking design choice with an important job: it introduces nonlinearity so the network can learn transformations that stacked linear projections alone cannot express. Understanding this part of a transformer helps when reading model architectures, comparing implementations, estimating parameter and compute costs, or deciding whether two feed-forward designs are actually equivalent.