Short training examples can waste much of a fixed-length transformer batch on padding. Sequence packing reduces that waste by placing several examples into one token buffer, but concatenation alone changes the computation. A causal mask prevents a token from attending to future positions; it does not prevent that token from attending to an earlier, unrelated example.

The distinction matters whenever packed examples are intended to remain independent. The token buffer may be contiguous for storage and compute while attention, position handling, and loss accounting still need explicit example boundaries.

A causal mask does not create document boundaries

Consider two independent token sequences packed into one row:

[A0 A1 A2 | B0 B1 B2 B3]

A standard causal mask permits each position to attend to itself and earlier positions. Under that rule, B0 can attend to A0, A1, and A2. The model therefore receives context from example A while predicting tokens in example B.

If A and B were sampled as independent training examples, that context is artificial. The packed row has silently changed from two sequences into one longer sequence.

A boundary-aware causal mask keeps causality inside each segment:

        A0 A1 A2 B0 B1 B2 B3
A0       x
A1       x  x
A2       x  x  x
B0                x
B1                x  x
B2                x  x  x
B3                x  x  x  x

The mask is block diagonal, with a causal triangle inside each block. Tokens in B cannot read tokens in A even though both occupy the same physical row.

Segment identity is enough to express the mask

A packed row can carry a segment identifier beside each token:

tokens:   A0 A1 A2 B0 B1 B2 B3
segment:   0  0  0  1  1  1  1

For query position i and key position j, attention is permitted when both conditions hold:

j <= i
segment[i] == segment[j]

Padding positions, if any remain, add another exclusion condition.

This representation separates storage layout from semantic connectivity. The model can process a dense token buffer while the mask preserves the independence of the original examples.

Materializing a full boolean matrix is conceptually simple but can be costly for long sequences. Some attention implementations accept sequence-length metadata or block descriptions that encode the same boundaries without storing a dense mask. That is an implementation detail; the semantic requirement remains that cross-segment attention is excluded.

Position indices are a separate decision

Attention isolation does not determine position indices. A packed buffer can use positions that continue across segment boundaries:

A: 0 1 2
B: 3 4 5 6

or positions can restart for each segment:

A: 0 1 2
B: 0 1 2 3

Those choices are not interchangeable for every positional representation. Continuing positions makes the second segment appear to start later in the model’s positional coordinate system. Restarting positions makes each segment resemble an independently presented sequence more closely, but support depends on how position information is supplied to the model and attention implementation.

The attention mask and the position indices should therefore be validated independently. A correct block mask does not imply correct positions, and reset positions do not stop cross-segment attention.

Next-token labels need boundary handling too

Causal language-model training commonly shifts tokens so that each position predicts the next token. Naively shifting the entire packed row creates a target across the boundary:

input:   A0 A1 A2 B0 B1 B2
target:  A1 A2 B0 B1 B2 B3

Here A2 is asked to predict B0. If the examples are independent, that target is artificial even when the attention mask is block diagonal.

The loss at the final predictive position of each segment needs treatment consistent with the data format. If each example already contains an explicit end marker, predicting that marker can be a valid target. The transition from one example to the next should not become a target merely because packing placed the examples next to each other.

A loss mask can exclude boundary transitions:

loss:    on on off on on on

Exact indices depend on whether start markers, end markers, separators, or other control tokens are part of each example.

Separators do not replace isolation

A separator token can tell the model that one region ended and another began, but it does not prevent information flow. Under ordinary causal attention, tokens after the separator can still attend to all tokens before it.

That behavior can be intentional for formats where earlier segments are context for later segments. Prompt-response data is an obvious case: the response must attend to the prompt, so blocking that boundary would destroy the intended dependency.

Packing independent examples has a different contract. A separator may still be useful as content, yet isolation comes from the attention structure rather than from the separator token itself.

Packing changes batch geometry

Without packing, a batch often has one example per row and padding fills unused positions. With packing, one row can contain several examples. Batch size measured in rows then stops describing the number of independent sequences processed in that batch.

Token counts, example counts, and optimizer update boundaries can diverge in new ways. If batches are assembled to a nearly fixed token budget, the number of examples per batch varies with their lengths. Metrics averaged per row can become misleading because a row is now a container rather than a single example.

Loss reduction deserves the same attention. A token-mean loss weights examples in proportion to their number of contributing tokens. An example-mean objective requires different aggregation. Packing does not create this distinction, but dense packing can make it less visible.

Boundary tests catch silent leakage

Cross-example leakage may not trigger a shape error or numerical failure. The model can train normally while consuming context that should have been inaccessible.

A direct mask test is more reliable than inspecting only tensor dimensions. Construct two segments, select a query in the second segment, and verify that every key in the first segment is excluded. Then check the final predictive position of the first segment and confirm that its target does not become the first token of the second segment unless the data contract explicitly calls for that transition.

Packed-sequence code is correct only when the logical examples survive the packing operation. Dense storage is an optimization; it should not redefine which tokens can communicate or which transitions contribute to the objective.

Artificial Intelligence Transformers LLM Model Training