Transformers can process relationships between tokens without stepping through a sequence one token at a time. The mechanism that makes this possible is self-attention: each token builds a weighted view of other tokens in the same context.

The formula is compact, but using transformer models well becomes easier when you understand what the calculation is doing, why masking matters, and where the computational cost comes from.

Start with token representations

Before attention runs, each input token is represented by a vector. Let the matrix X contain those token representations. A transformer layer applies learned projections to produce three matrices:

Q = XW_q
K = XW_k
V = XW_v

They are called queries, keys, and values.

A useful mental model is:

  • a query describes what a token is looking for;
  • a key describes what a token can be matched on;
  • a value contains information that can be passed forward.

These are not manually assigned meanings. The projection matrices are learned during training, so useful matching patterns emerge from the training objective.

Compute attention scores

For each query, the model compares it with every eligible key using a dot product. In matrix form:

scores = QK^T

A larger dot product means the query and key are more aligned in the learned representation space. The raw scores are scaled before softmax:

scaled_scores = QK^T / sqrt(d_k)

Here, d_k is the key dimension. Without scaling, dot products tend to grow in magnitude as the dimension increases. Very large values can push softmax toward extremely sharp distributions and make optimization less stable.

Softmax converts the scores for each query into non-negative weights that sum to one:

weights = softmax(scaled_scores)

The output is then a weighted combination of the value vectors:

output = weights V

Each token therefore receives a new representation influenced by the tokens it attended to.

Follow a small conceptual example

Consider the sentence:

The server rejected the request because it was malformed.

When building a representation for it, an attention head may assign a relatively high weight to request. Another head may focus on grammatical or positional relationships.

The important point is not that one attention head always resolves pronouns. Different heads can learn different patterns, and those patterns are distributed across layers. Attention provides a mechanism for combining information; it does not guarantee a human-readable explanation of the model’s reasoning.

Understand why multiple heads help

Transformers normally use multi-head attention. Instead of performing one large attention operation, the layer performs several attention operations with separate learned projections.

Conceptually:

head_1 = Attention(XW_q1, XW_k1, XW_v1)
head_2 = Attention(XW_q2, XW_k2, XW_v2)
...
combined = concat(head_1, head_2, ...)

The concatenated result is projected again before moving through the rest of the transformer block.

Multiple heads allow the layer to represent several relationships at the same time. One head can become sensitive to nearby syntax while another captures longer-range associations, although individual heads should not automatically be treated as clean semantic modules.

Distinguish self-attention from cross-attention

In self-attention, queries, keys, and values originate from the same sequence representation. Every token can therefore build its output from other eligible tokens in that sequence.

Cross-attention uses different sources. Queries may come from one sequence while keys and values come from another representation. Encoder-decoder transformers commonly use this pattern so a decoder can attend to encoder output.

This distinction matters when reading model architectures. The attention equation can be nearly identical even though the information sources are different.

Apply masking before softmax

Not every token should always be allowed to attend to every other token.

Autoregressive language models use a causal mask. When predicting the token at position i, the model must not read future positions. Their attention scores are effectively removed before softmax:

score[i, j] = -infinity  when j > i

After softmax, those positions receive zero attention weight.

Padding masks serve a different purpose. They prevent artificial padding tokens in a batch from influencing meaningful token representations.

Masking is therefore part of the model’s information boundary, not just an implementation detail.

Account for position

Self-attention by itself compares vector content and does not inherently encode sequence order. Transformers add positional information so the model can distinguish arrangements such as:

dog bites person
person bites dog

Architectures use different positional techniques, including learned position embeddings and relative or rotary position representations. The implementation varies, but the goal is similar: make token relationships sensitive to where tokens occur in the sequence.

Recognize the context-length cost

For a sequence of n tokens, ordinary full attention constructs an n × n score matrix for each head. Its attention-score computation and storage therefore grow quadratically with sequence length.

Doubling the number of tokens can create roughly four times as many query-key relationships. This is one reason long-context inference can require substantially more compute and memory than short prompts.

Implementations can reduce memory overhead, and some architectures use sparse, local, grouped, or other attention variants. Those optimizations change practical costs, but they do not make context length free.

Do not interpret attention weights as certainty

An attention weight tells you how strongly one value contributes through a particular attention operation. It is not a probability that a fact is true, and it is not a calibrated confidence score.

Likewise, a visually strong attention link does not prove that the linked token caused the final answer. Transformer outputs also depend on other heads, residual connections, feed-forward layers, normalization, and later transformer blocks.

Attention visualizations can be useful diagnostic tools, but they should not be presented as complete explanations of model decisions.

Connect attention to practical LLM behavior

Understanding self-attention clarifies several behaviors developers encounter when building with large language models.

Long prompts are expensive because many token relationships must be processed. Repeated or irrelevant context can consume capacity without adding useful information. Causal masking explains why an autoregressive model generates from the prefix available so far rather than reading its future output. Positional mechanisms help explain why ordering and prompt structure can affect results even when the same words are present.

It also explains why simply placing information somewhere inside a large context does not guarantee that the model will use it effectively. The model must transform that information through many learned attention and feed-forward operations before it influences the final prediction.

Use the right mental model

Self-attention is best understood as learned information routing. Queries determine what representations seek, keys determine how representations can be matched, and values carry the information that is mixed according to those matches.

That mental model is more useful than treating attention as a database lookup or as a direct explanation of reasoning. It captures both the strength of transformers—the ability to model relationships across a context—and their practical trade-offs in compute, memory, interpretation, and context design.

Artificial Intelligence Transformer Attention Neural Networks