Transformer batches often waste token slots on padding when examples have uneven lengths. Sequence packing reduces that waste by placing several shorter samples into one fixed-length token block. The arithmetic is attractive, but concatenation alone changes the training problem: tokens from one sample can attend to tokens from another unless the packed representation preserves sample boundaries.

A correct packing scheme therefore has two jobs. It must fill token capacity more densely, and it must keep the model’s effective computation consistent with the intended independence of the original samples.

Packing changes the shape, not the sample relationship

Consider three tokenized samples with lengths 3, 2, and 3. A padded batch with width 5 contains unused positions:

A: [a0 a1 a2 -- --]
B: [b0 b1 -- -- --]
C: [c0 c1 c2 -- --]

A packed block can use eight consecutive positions:

[a0 a1 a2 b0 b1 c0 c1 c2]

The storage layout is denser, but the semantic boundaries still exist after a2 and b1. For causal language-model training, b0 should not gain access to a0..a2 merely because the two samples now share a tensor row. The same applies at the next boundary.

A standard causal mask only blocks future positions. On its own, it permits b0 to attend to every earlier token in the packed row. Packing independent samples therefore needs boundary-aware attention, an equivalent segmented attention operation, or an implementation whose packed-sequence primitive enforces the same isolation.

Conceptually, the allowed attention pattern is block diagonal:

      A A A B B C C C
A     x
A     x x
A     x x x
B           x
B           x x
C               x
C               x x
C               x x x

Each block is causal internally. Entries across sample boundaries remain inaccessible.

Position handling is a separate decision

Packing also changes physical token offsets. In the packed row above, b0 occupies tensor position 3 even though it is the first token of sample B. Whether that physical offset should become the model’s position index depends on the model and training design.

For models that expect each independent sample to start from its initial position, position identifiers can reset at each boundary:

tokens:     a0 a1 a2 b0 b1 c0 c1 c2
sample:      A  A  A  B  B  C  C  C
position:    0  1  2  0  1  0  1  2

This keeps position assignment aligned with processing the samples separately. It is not interchangeable with attention masking. A reset position index does not prevent cross-sample attention, and an isolated attention mask does not automatically reset positions.

Some architectures encode position through mechanisms such as rotary position embeddings rather than adding a position embedding vector to each token. The implementation detail changes, but the boundary question remains: the packed execution must supply position information that matches the intended per-sample computation.

If a training setup intentionally treats concatenated documents as one continuous stream, resetting positions may not be desired. That is a different objective from packing independent samples. The data model has to state which interpretation applies.

Loss boundaries can leak a different signal

Attention isolation is not the only boundary to preserve. Next-token targets also need deliberate handling.

For a causal objective, token a2 would normally predict the next token inside sample A, or an explicit end marker if the data format includes one. After raw concatenation, the next physical token is b0. Training a2 to predict b0 creates an artificial transition between unrelated samples.

One option is to include an end-of-sequence token as part of each sample before packing. Another is to mask the loss at boundary positions so that no prediction target crosses into the next sample. The correct choice depends on the tokenizer and objective, but the invariant is clear: packing should not silently manufacture target pairs that were absent from the un-packed dataset.

Loss masking and attention masking solve different problems. The first controls which predictions contribute to the objective. The second controls which token states can influence a prediction. Applying only one leaves the other boundary exposed.

Dense packing has a combinatorial side

Once boundaries are represented correctly, the remaining problem resembles bin packing. Given a maximum sequence length, the data pipeline assigns variable-length samples to token blocks while trying to leave little unused capacity.

Exact optimal packing is generally unnecessary for a training input pipeline. Simple length-aware heuristics can reduce padding substantially when sample lengths vary, but they also affect data ordering and batching behavior. A pipeline that groups similar lengths minimizes slack differently from one that combines short and long samples into near-full blocks.

The relevant metric is token utilization rather than the number of samples per batch:

utilization = non_padding_token_slots / allocated_token_slots

This metric describes layout density only. It does not establish higher model quality or lower wall-clock time. Actual throughput also depends on attention kernels, sequence metadata, device utilization, host-side packing cost, and the framework’s support for variable-length execution.

A packed representation can even lose its expected compute advantage if the implementation materializes a large dense attention mask and still executes attention over the full packed square. Kernel support matters because the logical block structure does not guarantee that computation skips disallowed regions.

Packed execution needs an equivalence check

A useful validation target is equivalence with separate execution under controlled conditions. Take a few short samples, run them independently, then run the same tokens through the packing path with matching positions and boundaries.

For deterministic evaluation settings, compare logits at corresponding token positions. Small numerical differences can arise from kernel choice, floating-point order, or batching details, so exact bit equality is not a universal requirement. Large or structured differences near sample boundaries are more informative: they can indicate cross-sample attention, shifted positions, or incorrect target alignment.

The comparison should include boundary tokens, since that is where packing bugs concentrate. Interior tokens can look correct even when the first token of a later sample can see an earlier sample.

This check also exposes an assumption that is easy to miss: equivalence is only expected when the packed and separate paths implement the same model semantics. If the model uses batch-dependent operations or the data objective intentionally carries context across documents, separate execution is not the right reference.

Metadata becomes part of model correctness

Packing moves information that was implicit in the batch dimension into explicit metadata. Sample boundaries, sequence lengths, cumulative offsets, position identifiers, and loss masks may all participate in reconstructing the original independent examples.

That metadata should be treated as model input rather than incidental loader bookkeeping. An off-by-one boundary can expose a token to the preceding sample. A mismatched cumulative length can assign a query to the wrong attention segment. A target mask shifted by one position can train artificial transitions while the attention path itself remains correct.

The exact metadata format is framework-specific. Some systems use dense masks, while optimized variable-length kernels often accept sequence lengths or cumulative offsets. The representation can vary as long as it expresses the intended segmentation consistently across attention, positions, and loss construction.

Sequence packing is therefore most useful when its invariant is explicit: changing tensor layout must not silently change which sample a token belongs to. Once that invariant is enforced and tested, padding reduction becomes an implementation optimization rather than a change to the training data semantics.

Artificial Intelligence Transformers Training Attention