Transformer batches often contain sequences with different lengths. To store them in one rectangular tensor, shorter sequences are usually extended with padding tokens. Padding solves a shape problem, but it creates a modeling problem: those extra positions are not part of the original input.

If attention treats padding like ordinary content, real tokens can assign probability to positions that carry no useful information. The result may be wasted attention, representations that depend on how much padding was added, and training behavior that differs unnecessarily across batches.

An attention padding mask tells the attention operation which key positions are real and which should be unavailable. This article develops the mental model from a small example, shows why masking belongs before softmax, and explains the implementation mistakes that commonly make padding masks ineffective.

Padding changes shape, not meaning

Suppose a batch contains two tokenized sequences:

A: [red, fox, runs]
B: [blue, bird]

A rectangular batch needs the same sequence length for both rows, so the second sequence can be padded:

A: [red,  fox,  runs]
B: [blue, bird, PAD ]

The PAD position exists so the tensors line up. It does not add a third word to sequence B.

That distinction matters because self-attention compares query positions with key positions. Without a mask, the query for blue can score all three keys:

query: blue
keys:  blue   bird   PAD
score:  2.0    1.0   0.5

Softmax converts those scores into positive weights that sum to one. The padding position therefore receives some attention even though it represents no input content.

A learned padding embedding being small or fixed does not make this safe in general. Attention scores depend on the model’s projected query and key vectors, and model architecture details vary. The robust rule is simpler: if a position is semantically absent, exclude it explicitly rather than expecting the model to ignore it.

Mask invalid keys before softmax

Scaled dot-product attention can be written as:

scores = Q K^T / sqrt(d_k)
weights = softmax(scores)
output = weights V

Q, K, and V are the query, key, and value matrices. d_k is the key dimension used for scaling.

A padding mask modifies the scores before softmax. Conceptually, valid positions keep their scores while padded key positions receive negative infinity:

original scores: [2.0, 1.0, 0.5]
masked scores:   [2.0, 1.0, -inf]

Softmax then gives the invalid position zero probability:

softmax([2.0, 1.0, -inf])
~= [0.731, 0.269, 0.000]

The useful property is not the exact numbers. It is the normalization behavior: probability is distributed only across valid keys.

In implementation code, frameworks may represent the mask as booleans, zeros and ones, or an additive bias containing a very negative value. The public API determines the required convention. Do not assume that 1 means masked or that True means allowed across every library.

Why masking after softmax is easy to get wrong

A tempting implementation is to compute ordinary attention weights and then multiply the padding positions by zero:

weights = softmax(scores)
weights = weights * valid_mask

This removes the padded value contribution, but the remaining weights no longer necessarily sum to one.

Using the earlier example, suppose ordinary softmax produced:

[0.629, 0.231, 0.140]

Zeroing the final entry gives:

[0.629, 0.231, 0.000]

The valid weights now sum to 0.860, not 1.0. Part of the probability mass was allocated to padding before it was discarded.

You could renormalize after zeroing, but that recreates work already handled naturally by masking before softmax. Standard attention implementations therefore incorporate the mask into the attention-score calculation or an equivalent fused operation.

Mask keys and queries for different reasons

Padding masks are most often described as preventing attention to padded keys. That is the essential operation for protecting real query positions from padded content.

Consider sequence B again:

positions: [blue, bird, PAD]
valid:     [yes,  yes,  no ]

For the query at blue, the PAD key should be unavailable. The same is true for the query at bird.

But what about the query located at the padded position itself?

Some implementations still compute an output for padded query rows. That output may contain finite values because the padded query can attend to valid keys. This is not necessarily a problem if downstream computation also knows which positions are padding and those outputs never affect the objective or later semantic aggregation.

For example, a token-level loss should ignore padded target positions. A pooling operation should also exclude padding rather than averaging every row blindly.

This leads to an important engineering distinction:

mask padded keys    -> real queries cannot attend to nonexistent content
ignore padded queries -> padded output rows do not affect the task

A particular model or framework may combine these concerns differently. Inspect the API contract instead of assuming one attention mask automatically handles every downstream use of padded positions.

Keep padding masks separate from causal masks

Padding masks and causal masks solve different problems.

A padding mask says:

this position does not contain input content

A causal mask says:

this query must not see a future position

For an autoregressive sequence, a causal pattern might look like this, where 1 means visible:

      k0 k1 k2
q0     1  0  0
q1     1  1  0
q2     1  1  1

If the sequence is also padded, both constraints may apply. A position is available only when it is allowed by the causal rule and is a real token.

Conceptually:

allowed = causal_allowed AND key_is_valid

Many attention APIs accept separate causal and padding information, while others expect a combined mask or additive attention bias. The mathematical intent is the same even when tensor shapes and argument names differ.

Confusing the two masks can produce subtle bugs. A causal mask alone does not identify padding, and a padding mask alone does not prevent an autoregressive decoder from seeing future real tokens.

Broadcast the mask over the correct dimensions

Attention tensors commonly contain batch, head, query, and key dimensions. Exact layouts vary, but an attention-score tensor often has a logical shape similar to:

[batch, heads, query_length, key_length]

A basic padding mask is usually defined per sequence and key position:

[batch, key_length]

The implementation then broadcasts or expands that mask so every relevant attention head and query position sees the same invalid keys.

For a batch of two sequences:

valid keys:
A: [1, 1, 1]
B: [1, 1, 0]

The mask for B must hide its third key for every real query in B. It must not accidentally hide the third key in A.

This is why a mask with the right number of elements can still be wrong. A silent broadcasting mistake may apply a sequence’s mask across the wrong axis or across every batch item.

When implementing attention manually, verify shapes at the point where the mask meets the score tensor. For a small test batch, inspect the resulting attention probabilities and confirm that masked key positions receive zero probability for the intended rows.

Avoid numerical shortcuts that leak probability

Teaching examples often say to replace masked scores with -inf. That expresses the mathematics clearly because exp(-inf) = 0.

Production kernels may instead use a sufficiently negative finite value for the computation’s numeric type, or they may implement masking internally without materializing such a tensor. Follow the framework’s supported masking mechanism when one exists.

Using an arbitrary constant such as -10 is risky. Whether it is negligible depends on the other logits and numeric behavior. A masked position should be structurally excluded, not merely discouraged by a score that happens to look small in one example.

There is also a boundary case: a query row for which every key is masked. Mathematically, softmax over a row containing only negative infinity is not a normal probability distribution. Depending on the implementation, this situation can produce non-finite values or framework-specific behavior.

Avoid constructing fully masked rows unless the attention implementation explicitly defines their behavior. If fully padded sequences are possible, decide how they should be handled before they enter attention rather than relying on accidental numerical results.

Test masking with invariants, not only model quality

Masking bugs can survive ordinary training because a model may partially adapt to them. Direct tests are faster and more diagnostic.

A useful invariant is padding invariance. For the same real sequence, adding extra padding should not change the representations of its real positions beyond expected numerical tolerance, assuming all other inputs and position handling are equivalent.

For example, compare:

[red, fox, runs]

with:

[red, fox, runs, PAD, PAD]

After correctly masking the added keys, the attention calculation for the three original positions should not gain probability mass from those padding positions.

Other useful checks include:

  • masked key positions receive zero attention probability;
  • each non-degenerate attention row sums to approximately one;
  • one batch item’s padding pattern does not affect another item;
  • causal and padding constraints both hold when they are combined;
  • padded positions are excluded from token losses or pooling when the task requires it.

These checks target the mechanism directly. An end-to-end accuracy metric is still important, but it is a poor first detector for a tensor-shape or mask-convention bug.

Know when padding masks are unnecessary

Not every attention workload needs a padding mask.

If every sequence in a batch already has the same meaningful length, there may be no padding to exclude. Some systems also use packed or specialized variable-length attention representations that avoid materializing ordinary padded positions. In those cases, masking behavior depends on the representation and kernel rather than a conventional [batch, length] padding mask.

Do not add a mask merely because Transformers often have one. Add the constraint that matches the data representation.

Conversely, if rectangular batching introduces positions that are not part of the input, make their treatment explicit. Correct masking is usually cheaper and easier to reason about than asking the model to learn that arbitrary filler positions should not matter.

Conclusion

Padding is a batching device, not model input. Transformer attention should preserve that distinction by preventing real queries from assigning attention probability to padded keys.

The core mental model is straightforward: identify invalid key positions, exclude their scores before softmax, and make sure downstream losses or pooling also ignore padded outputs where necessary. Keep padding constraints conceptually separate from causal constraints, verify mask shapes and API conventions, and test simple invariants such as zero probability on masked keys and invariance to added padding.

Once those rules are explicit, padding becomes an implementation detail rather than an accidental source of signal.

Artificial Intelligence Machine Learning Transformers Attention