https://nalar.dev/tags/glu/ Recent content in GLU on Nalar Hugo en-us Mon, 01 Jan 0001 00:00:00 +0000 https://nalar.dev/use-gated-feed-forward-blocks-in-transformers/ Fri, 11 Sep 2026 00:00:00 +0700 https://nalar.dev/use-gated-feed-forward-blocks-in-transformers/ <h1 id="use-gated-feed-forward-blocks-in-transformers">Use Gated Feed-Forward Blocks in Transformers</h1> <p>A transformer block does more than attention. After attention mixes information across token positions, a feed-forward block transforms each token representation independently. That second stage often accounts for a large share of the model&rsquo;s parameters and computation.</p> <p>A gated feed-forward block changes this stage by adding a second projection that controls how much of another projected signal passes through. GLU, GEGLU, and SwiGLU are common names for members of this family.</p>