Pack Training Sequences Without Leaking Between Examples

Language-model training often wastes computation on padding. If a batch contains examples with very different lengths, shorter examples are extended with padding so tensors have compatible shapes. The model still has to move those tensor positions through parts of the training pipeline even though they contain no training content.

Sequence packing reduces that waste by placing multiple shorter examples into one fixed-length training sequence. The idea is simple; the boundary handling is not. If attention or loss masks are wrong, one example can accidentally use another example as context, or the model can be trained to predict tokens that should not count as targets.

This article develops a practical mental model for sequence packing, shows the smallest useful example, and explains how to decide whether the extra data-pipeline complexity is worth it.

Start with the padding you are paying for

Suppose a training configuration uses a sequence length of eight tokens. Three tokenized examples have lengths five, two, and one:

A: a1 a2 a3 a4 a5
B: b1 b2
C: c1

If each example occupies its own length-eight row, the batch looks conceptually like this:

A: a1 a2 a3 a4 a5  _  _  _
B: b1 b2  _  _  _  _  _  _
C: c1  _  _  _  _  _  _  _

Only eight of the 24 positions contain example tokens. The other 16 are padding.

Packing can instead combine the three examples into one row:

packed: a1 a2 a3 a4 a5 b1 b2 c1

Now all eight positions carry training content. This does not make each token mathematically cheaper. It improves useful token utilization: a larger fraction of the fixed-size tensor contains tokens that contribute to the intended training objective.

That distinction matters. Actual speedup depends on the model, kernels, batching strategy, hardware, data loader, and whether the un-packed implementation already avoids work on padded positions. Packing is a way to reduce padding waste, not a guarantee of a particular throughput gain.

Packing changes layout, not the meaning of the examples

A safe mental model is:

logical examples:  A      B      C
                    |      |      |
physical row:       AAAAA  BB     C

The physical layout is compact, but the logical training examples should retain the semantics required by the objective.

For causal language modeling, each target token is normally predicted from allowed earlier tokens. If B is an independent example, b1 should not suddenly depend on a1 ... a5 merely because the data loader placed them next to each other.

A naive causal mask over the whole packed row would allow exactly that:

b1 can attend to a1 a2 a3 a4 a5
b2 can attend to a1 a2 a3 a4 a5 b1

That may teach artificial cross-example relationships that will not exist when examples are used independently.

To preserve independence, the attention pattern must respect segment boundaries. Conceptually, the permitted attention matrix is block diagonal: tokens in A attend only within A, tokens in B only within B, and tokens in C only within C, while each block still follows the causal direction.

        keys
        A A A B B C
query A x . . . . .
      A x x . . . .
      A x x x . . .
      B . . . x . .
      B . . . x x .
      C . . . . . x

The diagram is simplified, but the rule is the useful part: packing should not silently create context across examples unless that context is intentionally part of the training data.

Attention boundaries and loss boundaries solve different problems

It is easy to treat “the mask” as one thing. In practice, two separate questions need answers.

The attention mask controls which positions a token may use as context. The loss mask controls which target positions contribute to the optimization objective. They can have different boundary rules.

Consider two independent causal examples packed together with an explicit end-of-sequence token:

A: user_a answer_a <eos>
B: user_b answer_b <eos>

A boundary-aware attention rule prevents tokens in B from reading tokens in A. Separately, the loss configuration decides which next-token predictions should be trained.

If the training objective is ordinary language modeling over every token in each example, most within-example next-token targets may contribute to loss. If the dataset is instruction tuning and only assistant responses are intended as targets, prompt tokens may be excluded from loss even though response tokens are allowed to attend to them.

Packing does not decide that policy. It has to preserve it.

A useful implementation design therefore keeps at least these concepts distinct:

• segment identity: which logical example owns each token;
• attention permission: which earlier tokens each query may read;
• target permission: which token predictions contribute to loss.

Conflating them is a common source of subtle training bugs.

Position handling depends on the model and implementation

Packed examples also raise a question about position identifiers. Should positions continue across the physical row, or restart at each logical example?

There is no universal answer that can be inferred from the word “packing” alone. Position handling must match the model architecture, positional encoding, training recipe, and attention implementation.

A boundary-aware packing scheme might conceptually assign:

A tokens: 0 1 2 3 4
B tokens: 0 1
C token:  0

Another implementation may keep physical positions increasing while using other metadata to represent segments. Some optimized attention kernels accept sequence-length metadata rather than a dense block-diagonal mask. Those are implementation choices, not guarantees of the packing concept itself.

The safe rule is to preserve the positional assumptions used by the model’s intended training setup. Do not reset or continue positions simply because one option looks more intuitive.

Build packing around token counts, not text lengths

Packing decisions have to happen after tokenization, because model sequence limits are measured in tokens rather than characters or words.

Suppose the maximum training length is 16 tokens and the next examples contain 9, 6, and 5 tokens. A simple greedy packer could place the 9-token and 6-token examples together, leaving one unused position, then start a new packed row for the 5-token example:

row 1: [9-token example][6-token example][_]
row 2: [5-token example][...]

A conceptual packing loop is:

for example in tokenized_examples:
    if example does not fit in current_pack:
        emit current_pack
        start new pack
    append example and record its segment boundary

Production packers often use more sophisticated ordering or buffering to reduce leftover space. The basic correctness requirement remains the same: every emitted row needs enough metadata to reconstruct the logical boundaries required by attention and loss.

Do not optimize bin filling before the boundary semantics are tested. A perfectly full sequence with the wrong mask is worse than a padded sequence with correct training behavior.

Long examples need an explicit policy

An example longer than the training sequence length cannot be fixed by packing. You still need a truncation, splitting, or filtering policy.

Those choices change the training data in different ways. Truncation can remove a target or evidence near the end of an example. Splitting can create chunks whose beginning lacks context from the original document. Filtering can bias the dataset toward shorter examples.

For instruction data, blindly truncating a complete conversation is especially risky if it removes the assistant response while leaving the prompt. For document language modeling, chunking may be entirely appropriate because contiguous chunks are themselves meaningful training sequences.

Handle long examples before or as part of packing, and record what the policy does. Packing should not become an accidental truncation mechanism.

When cross-example attention may be acceptable

Not every packed training recipe needs strict isolation. If the training objective intentionally treats concatenated documents as one continuous token stream, allowing later tokens to attend to earlier material can be part of the chosen objective.

For example, classic causal language-model pretraining can concatenate text into long streams separated by document markers. In that setting, the training recipe may deliberately allow attention across a document separator. The separator tells the model about a boundary, but it does not necessarily enforce computational isolation.

That is different from packing independent instruction examples while intending each example to behave as a separate conversation.

The right question is not “does packing require block-diagonal attention?” It is:

Would these tokens be valid context for one another if they had not been placed together merely to save padding?

If the answer is no, isolate them. If the answer is yes because concatenation is part of the actual objective, cross-boundary attention may be intentional.

Measure useful tokens before adding complexity

Packing is most attractive when length variation causes substantial padding. Before changing the pipeline, measure the fraction of token positions that represent useful, non-padding content.

A simple utilization metric is:

useful_token_fraction = non_padding_tokens / allocated_token_positions

If an un-packed batch allocates 8,192 positions and 5,120 are real tokens, the fraction is:

5120 / 8192 = 0.625

So 62.5% of allocated positions carry non-padding tokens.

After packing, also measure end-to-end training throughput rather than assuming the utilization improvement transfers directly to wall-clock performance. Boundary-aware masking, packing logic, irregular sequence metadata, or kernel limitations can add overhead. The metric that matters operationally is usually useful training tokens processed per unit time at an acceptable memory footprint and model quality.

Keep quality evaluation separate from throughput. A faster run is not an improvement if incorrect boundaries changed the objective.

Common packing failures are boundary failures

Most dangerous mistakes do not crash training. They produce plausible-looking batches with the wrong semantics.

Allowing independent examples to attend across boundaries. The model receives context that belongs to another sample. Loss can still decrease, so training curves may not expose the problem.

Training on separator or prompt tokens unintentionally. A separator may be useful as context without being a desired supervised target. Instruction-tuning datasets often have additional target-masking rules that packing must preserve.

Dropping the final partial pack. Requiring every row to be completely full can silently discard examples. Padding the final pack or carrying it into the next packing buffer may be preferable, depending on the pipeline.

Changing position semantics accidentally. Resetting position IDs, failing to reset them, or using metadata unsupported by the model can make packed training differ from the intended recipe.

Comparing only examples per second. Packing changes how many examples and useful tokens fit in a row. Tokens per second, target tokens per second, memory use, and model quality usually provide a clearer comparison.

Test the packer as part of the training objective

A packer deserves small deterministic tests before it touches a long training run.

Construct two tiny examples with clearly different token IDs. Verify that the emitted sequence contains every expected token exactly once, except for any documented truncation or padding. Then inspect the attention relation at the boundary: the first token of the second independent example must not see tokens from the first when isolation is required.

Do the same for the loss mask. Mark which positions should contribute targets before packing, pack the examples, and confirm those target permissions are unchanged afterward.

A particularly useful regression test compares packed and un-packed loss on the same examples under evaluation mode, with stochastic layers disabled. If the packing scheme is intended to preserve independent-example computation and the implementation handles positions equivalently, corresponding target losses should agree up to expected numerical differences. If they do not, investigate the masks, positions, separators, and target alignment before scaling up.

Use sequence packing when padding is the real bottleneck

Packing is a strong fit when training examples are frequently much shorter than the configured sequence length, padding occupies a meaningful share of allocated positions, and the model stack supports the boundary semantics you need.

It is less compelling when examples already fill most sequences, when variable-length or optimized attention already avoids much of the padding cost, or when implementing safe packed attention would force the training stack onto a slower path. In those cases, bucketing examples by similar length may recover much of the utilization with less complexity.

Bucketing and packing can also complement each other. Bucketing reduces length variance within batches; packing fills remaining space with multiple examples. The useful choice depends on the data distribution and the capabilities of the training implementation, not on packing as a technique in isolation.

Treat packing as a semantic-preserving optimization

The safest way to adopt sequence packing is to begin with a correct un-packed objective and treat packing as a change in physical layout. Write down which tokens may attend to which other tokens, which predictions count toward loss, and how positions should behave. Then make the packed representation preserve those rules.

After correctness tests pass, measure useful-token throughput and memory use on representative data. If padding was substantial, packing can make fixed-length training capacity much more productive. If it was not, the simpler pipeline may be the better engineering choice.