Language-model training often processes sequences in fixed-size tensors. When examples have very different lengths, padding makes those tensors easy to batch but can leave many token positions doing little useful work. A batch that physically contains 8,000 positions may contain far fewer than 8,000 real training tokens.

Sequence packing reduces this waste by placing multiple shorter examples into the same fixed-length training sequence. The idea is simple; the semantics are not. If packing accidentally lets one example attend to another, predicts across boundaries that should be independent, or assigns incorrect position IDs, the training objective changes rather than merely becoming more efficient.

This article builds a practical mental model for packing, shows the smallest useful example, and explains the attention, loss, and position rules that make packed training equivalent to separate examples when that equivalence is the goal.

Padding spends capacity on empty positions

Suppose a training step uses sequences of length 8 and receives three examples containing 3, 5, and 2 tokens. Padding each example independently produces:

A: [a1 a2 a3  _  _  _  _  _]
B: [b1 b2 b3 b4 b5  _  _  _]
C: [c1 c2  _  _  _  _  _  _]

The tensor contains 24 positions, but only 10 hold example tokens. An attention mask can stop padding from affecting the model’s predictions, and a loss mask can stop padded targets from contributing to the objective. Those masks preserve correctness, but they do not necessarily remove all computation associated with the padded tensor positions.

Packing changes the layout instead:

P1: [a1 a2 a3 b1 b2 b3 b4 b5]
P2: [c1 c2  _  _  _  _  _  _]

The same 10 useful tokens now occupy 16 tensor positions instead of 24. With many short examples and a suitable packing strategy, utilization can improve substantially.

The important metric is therefore not just batch size. A more useful quantity is token utilization:

token utilization = non-padding token positions / total tensor positions

For the padded layout above, utilization is 10 / 24, or about 41.7%. For the simple packed layout, it is 10 / 16, or 62.5%.

This arithmetic is only a teaching example. Real throughput depends on the model, sequence length, attention implementation, hardware, data pipeline, and packing overhead. Higher token utilization does not guarantee the same proportional increase in training speed.

Packing is a layout transformation, not a new objective

The safest mental model is to separate two questions:

  1. Where are examples stored in the tensor?
  2. Which tokens are allowed to interact and contribute to loss?

Packing answers the first question. Masks and metadata answer the second.

If examples A and B are supposed to be independent training examples, placing them next to each other should not by itself make B part of A’s context. A correct packed representation therefore needs enough information to recover the intended boundaries.

One conceptual representation is:

tokens:      [a1 a2 a3 b1 b2 b3 b4 b5]
example_id:  [ 0  0  0  1  1  1  1  1]

The example_id values do not have to exist as a literal tensor with that name. They represent the boundary information from which an implementation can construct attention rules, position IDs, and loss masks.

Preserve attention boundaries when examples are independent

A causal language model normally allows each token to attend to earlier positions. On an ordinary sequence, that gives a lower-triangular attention pattern.

Naively concatenating A and B changes that pattern. Token b1 appears after a3, so ordinary causal attention would allow b1 and later B tokens to attend to A:

[a1 a2 a3 | b1 b2 b3]
             ^
             can see A under a plain causal mask

That may be correct when A and B are intentionally parts of one continuous document. It is not equivalent to training them as independent examples.

For independent examples, the logical attention mask must combine two conditions:

allow(query, key) = same_example(query, key)
                    AND key_position <= query_position

The resulting pattern is block-diagonal and causal: tokens attend backward within their own example but not across example boundaries.

For two examples with three tokens each, the conceptual mask is:

        a1 a2 a3 | b1 b2 b3
a1       1  0  0 |  0  0  0
a2       1  1  0 |  0  0  0
a3       1  1  1 |  0  0  0
----------------------------
b1       0  0  0 |  1  0  0
b2       0  0  0 |  1  1  0
b3       0  0  0 |  1  1  1

Some training stacks can express this structure efficiently without materializing a dense square mask. Others have specific packed-sequence or variable-length attention interfaces. The required API is implementation-dependent; the invariant is that the effective attention relationships must match the intended training semantics.

Decide what should happen at target boundaries

Autoregressive training commonly shifts tokens so that each position predicts a following token. Packing creates a boundary where the final token of one example is physically followed by the first token of another.

Consider:

packed tokens: [a1 a2 a3 | b1 b2 b3]

If A and B are independent, training a3 to predict b1 creates a target that did not exist when the examples were separate. The loss construction must prevent that artificial cross-example prediction.

A conceptual target mask could look like:

input:       a1  a2  a3 | b1  b2  b3
target:      a2  a3   - | b2  b3   -
loss valid:   1   1   0 |  1   1   0

Here - means that no cross-boundary next-token target is introduced. In a real language-model dataset, an example may already include an explicit end-of-sequence token. If predicting that token is part of the objective, its target should remain trainable; what must be avoided is accidentally treating the next independent example as its continuation.

This is why simply masking cross-example attention is not enough. Attention controls what information a prediction can use. The loss mask controls which predictions are optimized. Both need correct boundary semantics.

Position IDs need an explicit policy

Packing also raises a position question. In the physical tensor, B may begin at offset 3 even though it was position 0 when trained separately.

For independent examples, a common semantic goal is to restart positions at each boundary:

tokens:      [a1 a2 a3 | b1 b2 b3 b4]
position:    [ 0  1  2 |  0  1  2  3]

Whether this is necessary depends on the model’s positional mechanism and training implementation. Positional schemes are not interchangeable, and a framework may derive positions internally rather than accept explicit IDs.

The practical rule is stronger than any one implementation detail: compare the packed run with the unpacked run and ensure that packing does not silently alter positional inputs when independence is required.

Do not assume that resetting integer position IDs alone makes every positional mechanism equivalent. The model’s actual positional computation determines the behavior.

Pack examples without changing their contents

Once the semantics are clear, the packing problem resembles bin packing. Each example has a token length, and each training sequence has a capacity.

For a capacity of 10:

example lengths: 6, 4, 4, 3, 2

A useful arrangement is:

pack 1: 6 + 4 = 10
pack 2: 4 + 3 + 2 = 9

Only one position remains unused across the two packs.

Finding the mathematically optimal arrangement can be unnecessary. Practical pipelines often use simple heuristics such as grouping similar lengths or greedily placing examples into available space. The right strategy depends on whether examples arrive offline, in a stream, or under strict shuffling requirements.

Packing should not truncate examples merely to improve utilization unless truncation is already an intentional part of the data policy. Packing and truncation solve different problems: packing reduces empty space between examples, while truncation discards tokens.

Separate packing from concatenating related text

Two layouts can look identical while representing different training objectives.

Suppose two chunks come from adjacent parts of the same document:

[document chunk A][document chunk B]

If the goal is continuous language modeling, allowing B to attend to A and predicting across the boundary may be desirable. In that case, the boundary is not an independence boundary at all.

Now suppose the same physical layout contains two unrelated chat conversations:

[conversation A][conversation B]

Letting B see A creates information flow between examples that would otherwise be unrelated. The packed tensor needs boundary-aware attention and loss handling.

Calling both operations “packing” can hide this distinction. Define the semantic unit first: which tokens belong to one model context? Then optimize how those units occupy tensors.

Measure whether packing helps the real bottleneck

Packing adds complexity to dataset construction and model inputs, so evaluate it against the resource that actually limits training.

Useful measurements include:

  • token utilization before and after packing;
  • non-padding training tokens processed per second;
  • step time and end-to-end training throughput;
  • accelerator memory use;
  • CPU time spent constructing packs and masks;
  • distribution of examples per packed sequence.

If examples are already close to the fixed sequence length, padding waste may be small and packing may add little value. Length-based batching can also be a simpler alternative: putting similarly sized examples together reduces padding without placing independent examples in the same sequence.

Conversely, a dataset dominated by short, variable-length examples can leave enough empty space that packing is worth the additional machinery.

Quality metrics must also remain part of the comparison. A faster pipeline is not equivalent if a masking bug changes the learned task.

Validate equivalence before scaling up

A small deterministic test can catch many packing errors before an expensive training run.

Take two short examples and process them in two ways:

run A: examples processed separately
run B: the same examples packed together

When the intended semantics are independent, inspect these invariants:

  • tokens from one example cannot attend to tokens from another;
  • no loss term predicts the first token of the next independent example merely because it is adjacent;
  • padding positions do not contribute to loss;
  • position handling matches the intended unpacked behavior;
  • the number of valid target tokens is the same in both layouts.

Under deterministic model behavior and mathematically equivalent masking, corresponding logits and losses should agree up to the numerical differences expected from the implementation. Exact bitwise equality is not a universal requirement because kernels and execution order can differ.

This test is more informative than checking only whether training loss decreases. A model can learn successfully while still training on unintended cross-example context.

Common packing mistakes

The most damaging errors usually come from treating physical adjacency as semantic adjacency.

Using only a causal mask. A plain causal mask prevents looking into the future but does not prevent later examples from seeing earlier examples in the same packed sequence.

Masking attention but not boundary targets. This can still train the last token of one independent example to predict the first token of the next.

Assuming padding masks solve example isolation. Padding masks distinguish real positions from empty ones; they do not necessarily distinguish one real example from another.

Ignoring positional behavior. Packed offsets can change positional inputs unless the model and data pipeline deliberately preserve the intended policy.

Optimizing utilization instead of throughput. A sophisticated packer can produce nearly full sequences while becoming a CPU or synchronization bottleneck. Measure end-to-end performance.

Mixing incompatible semantics. Some data should be continuous across boundaries; other data should remain isolated. One global masking rule may be wrong for a heterogeneous dataset.

When sequence packing is worth using

Packing is a strong candidate when fixed-length training batches contain substantial padding, examples are much shorter than the model’s training length, and the training stack can represent boundaries correctly and efficiently.

Prefer a simpler approach when padding waste is already low, length-based batching solves most of the problem, or the attention implementation cannot preserve the required example isolation without excessive overhead. Simplicity has operational value: fewer boundary rules mean fewer ways to train on the wrong objective.

The central lesson is that sequence packing is not “remove padding and concatenate everything.” It is a storage optimization constrained by model semantics. First define which tokens may attend to one another, which targets should contribute to loss, and how positions should behave. Then pack examples around those invariants. When those rules are preserved, packing can spend more of each fixed-size training tensor on useful tokens without quietly changing what the model is learning.

Artificial Intelligence LLM Transformers Model Training