https://nalar.dev/tags/normalization/ Recent content in Normalization on Nalar Hugo en-us Fri, 04 Sep 2026 00:00:00 +0700 https://nalar.dev/understand-layer-normalization-in-transformers/ Fri, 04 Sep 2026 00:00:00 +0700 https://nalar.dev/understand-layer-normalization-in-transformers/ <p>Transformer diagrams often contain small boxes labeled <code>LayerNorm</code> or <code>Norm</code>. They are easy to treat as plumbing between attention and feed-forward layers, but normalization has an important job: it controls the scale of hidden activations as information passes through many residual blocks.</p> <p>That matters because a transformer repeatedly adds new updates to an existing residual stream. If activation scales become poorly behaved, optimization can become harder and numerical problems can become more likely. Layer normalization gives each normalized hidden vector a predictable scale while preserving learnable degrees of freedom.</p>