Variable-length text batches are commonly padded into rectangular tensors. The extra positions simplify batching, but they are not ordinary training targets. If padded target positions contribute to cross-entropy, the optimizer receives gradients for synthetic symbols that were introduced only to align tensor shapes.
Preventing that signal requires a loss mask. An attention mask can stop selected positions from participating in attention, but that does not by itself remove their target terms from the objective.
Attention masking and loss masking solve different problems
Consider two token sequences padded to the same width:
A: [a, b, c, d]
B: [e, f, PAD, PAD]A causal attention mask controls which source positions each query can attend to. A padding-aware attention mask can additionally prevent the real tokens in sequence B from attending to padded key positions.
The language model loss is a separate computation. For next-token prediction, each valid target position contributes a term such as:
loss_t = -log p(target_t | prefix)If the training code computes that term for PAD targets, those positions remain part of the objective even if attention to padding was blocked elsewhere. The model is then optimized to predict a batching artifact at sequence tails.
This separation is easy to miss because both mechanisms are often represented by tensors called masks. Their shapes can even look similar. Their destinations are different: one modifies attention computation, while the other determines which target positions contribute to the scalar loss.
Shifted labels move the masking boundary
Decoder-only language models usually align logits and labels with a one-token shift. Given tokens:
[x0, x1, x2, x3]the model state associated with x0 predicts x1, the state at x1 predicts x2, and so on. The exact shifting location can be inside the model loss implementation or in the input pipeline.
Padding masks must match that convention. Suppose a sequence is represented as:
[p, q, r, PAD, PAD]The prediction targets after shifting are conceptually:
[q, r, PAD, PAD]Only targets that correspond to real sequence content should contribute. Masking based on an unshifted position index without checking the loss implementation can leave one padded target active or suppress one valid target.
A reliable invariant is to inspect the actual label tensor consumed by cross-entropy. Every position that represents padding should carry the loss function’s ignored label value or be excluded through an equivalent explicit mask.
The padding token ID and ignored label value are separate concepts
A tokenizer may assign a normal vocabulary ID to its padding token. Cross-entropy implementations often use a separate sentinel label to mark positions that do not contribute to loss. These values serve different purposes.
For example, assume the padding token ID is 3 and the loss function ignores labels equal to -100. A padded input can retain token ID 3 because the embedding lookup requires a valid vocabulary index. Its corresponding target label can be replaced with -100 before loss computation:
input_ids: [18, 42, 3, 3]
labels: [18, 42, -100, -100]The exact ignored value is framework-specific and must match the loss configuration. Replacing padded input IDs with a negative sentinel would be a different operation and can make embedding lookup invalid.
This distinction also matters when a tokenizer reuses an existing vocabulary token as padding. Even if the input representation uses a valid token ID, padded target positions still need to be excluded from the objective.
Loss reduction should count valid targets
Masking changes the denominator of an averaged token loss. If a batch contains N tensor positions but only M valid targets, the token-level mean should be based on the valid contributions under the intended objective.
In simplified form:
mean_loss = sum(valid_token_losses) / count(valid_targets)Dividing by the padded tensor size instead can make loss magnitude depend on how much padding happened to be present. Two batches containing the same number of real target tokens can then receive different scaling solely because their rectangular shapes differ.
Many standard cross-entropy functions handle this denominator when an ignore index is configured, but custom loss code needs to preserve the same logic explicitly. This becomes especially relevant when per-token weights, multiple objectives, or distributed reduction are added around the base loss.
For distributed training, local batches can contain different numbers of valid targets. Averaging already-averaged local losses gives each worker equal weight rather than each valid token equal weight. If the intended objective is a global token mean, the reduction needs global loss sums and global valid-token counts before division.
Packed samples add another masking dimension
Padding is not the only artificial boundary in language model batches. Multiple independent samples can be packed into one token row to reduce unused space. In that case, every packed token may be a real vocabulary token, yet some next-token targets can still be invalid for the intended objective.
If sample A ends immediately before sample B begins, a naive shifted loss creates a target that asks the final state of A to predict the first token of B. That cross-sample transition did not exist in either original sequence.
A packing pipeline can mask the boundary target so it contributes no loss. Attention isolation may also be needed if the design requires each packed sample to remain independent. Once again, attention masking and target masking are related but distinct controls.
The same principle applies to prompt-response fine-tuning when only response tokens are intended to contribute to the objective. Prompt tokens can remain visible as context while their label positions are masked from loss. A target mask therefore expresses which observed tokens are training targets, not merely which tokens exist in the input.
Metrics can expose a masking error
A padding-loss bug can remain numerically quiet. Cross-entropy still produces finite values, gradients still flow, and training can appear to progress. The artifact is easier to detect when metrics are computed over explicit valid-token counts.
Useful checks compare the number of contributing labels with the number expected from sequence lengths after shifting. A tiny synthetic batch with known lengths can expose off-by-one errors without requiring a full training run. Per-position loss inspection can also confirm that padded targets contribute zero weight to the reduced objective.
Token accuracy or perplexity calculations need the same target mask if they are meant to describe real text positions. Including padding in a metric can make the reported value disagree with the objective even after training loss is masked correctly.
Padding is a tensor-shape device, not part of the text distribution being modeled. Keeping that boundary explicit in labels, reductions, and metrics prevents batch geometry from becoming an unintended source of supervision.