Normalization

Understand Layer Normalization in Transformers

Transformer diagrams often contain small boxes labeled LayerNorm or Norm. They are easy to treat as plumbing between attention and feed-forward layers, but normalization has an important job: it controls the scale of hidden activations as information passes through many residual blocks. That matters because a transformer repeatedly adds new updates to an existing residual stream. If activation scales become poorly behaved, optimization can become harder and numerical problems can become more likely. Layer normalization gives each normalized hidden vector a predictable scale while preserving learnable degrees of freedom.