Transformer inputs rarely have identical lengths. One sentence may contain 8 tokens while another contains 30, yet efficient training and inference usually process multiple sequences in rectangular tensors. The usual solution is to add padding tokens to shorter sequences until their shapes match.

Padding solves the shape problem but creates a semantic one: the added positions are not real input. If the model treats them like ordinary tokens, they can influence attention, pooling, and training loss. A padding mask tells the computation which positions are valid and which exist only to make the batch rectangular.

This article builds a practical mental model for padding masks, shows where they belong in transformer computations, and explains the common mistakes that can make a model learn from tokens that were never part of the data.

Padding is a shape operation, not new information

Suppose a batch contains two already-tokenized sequences:

A: [red, fox, runs]
B: [blue, bird, flies, south, today]

A dense batch tensor needs one sequence length, so the shorter row can be padded to length 5:

A: [red,  fox,  runs,  PAD,   PAD]
B: [blue, bird, flies, south, today]

The corresponding validity mask can be written as:

A: [1, 1, 1, 0, 0]
B: [1, 1, 1, 1, 1]

Here 1 means a real position and 0 means padding. The exact convention is API-specific: some libraries use booleans, some use the opposite boolean meaning, and some expect an additive mask. The important idea is independent of representation: padded positions must be excluded wherever the task assumes that only real tokens contribute.

Padding itself does not tell a transformer that PAD is meaningless. A padding token normally has an embedding like other token IDs. Without the appropriate mask, later operations can use that representation.

Mask padding before attention probabilities are normalized

Self-attention compares a query at one position with keys from positions it is allowed to attend to. For one attention row, the unmasked scores can be written conceptually as:

scores_j = (q · k_j) / sqrt(d_k)
weights  = softmax(scores)

If positions 4 and 5 are padding, they should not receive attention probability. A common conceptual implementation changes their logits before softmax:

scores:       [ 1.2,  0.7, -0.1,  0.9,  0.2]
padding mask: [keep, keep, keep, mask, mask]
masked:       [ 1.2,  0.7, -0.1, -inf, -inf]

Softmax then assigns zero probability to the masked positions:

softmax(masked) -> [positive, positive, positive, 0, 0]

This ordering matters. Zeroing an attention weight after softmax is not generally equivalent, because the padded positions may already have consumed part of the normalized probability mass. If post-softmax weights are zeroed without renormalization, the remaining valid weights no longer sum to one.

Production kernels do not have to materialize literal negative infinity. They may use fused operations or finite sentinel values appropriate to the numerical format. The behavioral requirement is that masked keys contribute no attention probability under the implementation’s masking contract.

Distinguish padding masks from causal masks

Padding and causal masking solve different problems.

A padding mask hides positions that do not represent input data. A causal mask prevents an autoregressive position from attending to future positions.

For a padded autoregressive sequence, both restrictions may apply. Consider five tensor positions where only the first three contain real tokens. For the query at position 2, causal attention may permit positions 1 and 2 but reject position 3 because it is in the future. Positions 4 and 5 are rejected because they are padding.

Conceptually, the allowed keys are the intersection of both rules:

allowed = not_future AND not_padding

Do not assume that passing one mask automatically creates the other. Some model APIs construct a causal mask internally, some accept an explicit attention mask, and some combine several mask types. Check the model or framework contract rather than relying on a mask name alone.

Query padding and key padding are different concerns

Masking padded keys prevents real query positions from reading meaningless padded positions. That is the central requirement for attention.

Padded query positions can still produce hidden states in a dense implementation. Whether that matters depends on what happens next. If those outputs are discarded and never affect a valid position or objective, computing them can be harmless extra work. If they enter pooling, a loss, or another unmasked operation, they can contaminate the result.

This distinction explains why a model can appear to have correct attention masking while still train incorrectly. Attention may ignore padded keys, but the training objective may still score padded output positions.

Mask the loss for token-level objectives

Consider next-token training with a padded target row:

input:  [red, fox, runs, PAD, PAD]
target: [fox, runs, END, PAD, PAD]
valid:  [  1,   1,   1,   0,   0]

The last two target positions were introduced only by batching. They should normally contribute no token-level loss.

A simplified masked average is:

masked_loss = sum(loss_i * valid_i) / sum(valid_i)

The denominator is important. Dividing by the padded tensor length instead of the number of valid targets changes the effective loss scale according to how much padding happens to be in each batch.

Many training libraries provide an ignore index or an explicit loss mask for this purpose. Their exact behavior is framework-specific, so verify whether reduction happens over all tensor elements or only non-ignored targets.

For sequence-level classification, the issue appears in a different place. The model may need one representation for the whole sequence, often produced through a designated token or a pooling operation. If mean pooling is used, padded positions should not be included in the average:

pooled = sum(hidden_i * valid_i) / sum(valid_i)

Otherwise shorter sequences receive a larger fraction of artificial padding in their pooled representation.

Padding direction can interact with model assumptions

Two common layouts are right padding and left padding:

right: [A, B, C, PAD, PAD]
left:  [PAD, PAD, A, B, C]

With correct masking, both layouts can represent the same token content, but they are not automatically interchangeable in every model or serving stack.

Position IDs, generation logic, cached states, and model-specific preprocessing may assume a particular padding side. For example, an autoregressive generation implementation that selects logits from the final tensor position needs to know whether that position corresponds to the final real token or to padding. Libraries can account for this, but only when used according to their documented input conventions.

Treat padding side as part of the model interface rather than as a purely cosmetic tensor choice.

Padding also has a compute cost

A mask protects semantics, but it does not necessarily eliminate all computation associated with padded tensor positions.

Suppose a batch has sequence lengths:

[40, 39, 38, 7]

Padding every row to 40 creates 160 token slots for 124 real tokens. The difference is moderate. A less uniform batch such as:

[2000, 80, 60, 40]

creates 8000 slots for only 2180 real tokens when represented as a dense 4 x 2000 tensor.

Whether a particular kernel can skip some masked work is an implementation detail. Developers should not assume that masking alone removes the memory and compute cost of padding.

Length-aware batching can reduce waste by grouping sequences of similar lengths. Dynamic padding can also pad each batch only to its longest member instead of to a dataset-wide maximum. These techniques add batching complexity, so their value depends on sequence-length variation, workload size, and the capabilities of the training or serving framework.

Common masking mistakes are often silent

Padding bugs frequently produce plausible outputs instead of immediate exceptions, which makes explicit checks valuable.

Reversing mask semantics. An API may define 1 as valid, True as masked, or accept additive values. Passing the right shape with the wrong convention can mask real tokens instead of padding.

Masking attention but not loss. The model stops reading padded keys but is still optimized to predict padding targets. Attention masking and objective masking are separate responsibilities.

Using the wrong broadcast shape. Attention tensors can include batch, head, query, and key dimensions. A mask that broadcasts over the wrong axis can apply a valid-looking pattern to the wrong positions.

Forgetting pooling. Mean or sum pooling over all tensor positions can include padded hidden states even when attention itself is correct.

Assuming a padding token ID is enough. Giving padding a dedicated vocabulary ID identifies it; it does not by itself guarantee that every relevant operation ignores it.

Combining masks incorrectly. Adding or logically combining padding and causal masks with incompatible conventions can accidentally unmask forbidden positions or mask valid ones.

Test masking with invariants, not only model quality

A small deterministic test can catch mistakes earlier than a training curve.

Take one sequence, run it alone, then run the same sequence in a batch where extra padding is added. Under evaluation-mode conditions and a model/API that is expected to be padding-invariant, predictions for the real positions should agree up to normal numerical tolerance.

Also inspect the attention or mask construction directly when the framework exposes it. Useful invariants include:

real query -> padded key: disallowed
padded target -> token loss: ignored
mean pooling denominator: number of real tokens

Be careful when interpreting exact equality. Dropout, nondeterministic accelerator kernels, batch-dependent layers, or different execution paths can introduce differences unrelated to padding. The test should control those factors where practical.

When padding masks are not the whole solution

Padding masks are appropriate when rectangular batches contain positions that should have no semantic effect. They do not solve every sequence-layout problem.

Packed or concatenated training sequences may place multiple real examples in one token stream. Those boundaries can require an attention pattern that prevents one example from reading another, even though none of the tokens are padding. Sparse and block-structured attention can require still different masks.

Likewise, if every input has a fixed meaningful length, there may be no padding to mask. Adding masking machinery in that case provides little value.

The practical rule is to derive the mask from the information-flow constraint. Ask which query positions are allowed to use which key positions, and which outputs are allowed to affect the objective. Padding is one reason to restrict those paths, not the definition of masking itself.

Conclusion

Padding lets variable-length sequences share rectangular tensors, but padded positions are storage artifacts rather than model input. A correct transformer pipeline keeps that distinction explicit: mask padded keys before attention normalization, exclude padded targets from token-level loss, and prevent padded states from entering pooling or other downstream reductions when they should not contribute.

Keep padding masks conceptually separate from causal masks, follow the model’s documented padding-side and mask conventions, and test simple invariants around valid positions. Those checks turn masking from an invisible implementation detail into a verifiable part of model correctness.

Artificial Intelligence Transformers Attention Model Training