https://nalar.dev/tags/attention/ Recent content in Attention on Nalar Hugo en-us Thu, 03 Sep 2026 00:00:00 +0700 https://nalar.dev/understand-self-attention-in-transformer-models/ Thu, 03 Sep 2026 00:00:00 +0700 https://nalar.dev/understand-self-attention-in-transformer-models/ <p>Transformers can process relationships between tokens without stepping through a sequence one token at a time. The mechanism that makes this possible is self-attention: each token builds a weighted view of other tokens in the same context.</p> <p>The formula is compact, but using transformer models well becomes easier when you understand what the calculation is doing, why masking matters, and where the computational cost comes from.</p> <h2 id="start-with-token-representations">Start with token representations</h2> <p>Before attention runs, each input token is represented by a vector. Let the matrix <code>X</code> contain those token representations. A transformer layer applies learned projections to produce three matrices:</p>