A padded batch is shaped by its longest sequence, not its average sequence. If one batch contains token counts of 120, 124, 131, and 900, every sequence may be represented at length 900. Most positions in the first three rows then carry padding rather than input tokens.
Length bucketing changes batch composition instead of changing the model. Examples with similar token counts are placed near each other before batches are formed. The maximum length inside each batch falls closer to the lengths of its members, reducing the number of padded positions processed by operations that still use the rectangular batch shape.
Padding cost follows the local maximum
For a batch of B sequences with lengths l_1 ... l_B, let
L = max(l_1, ..., l_B)A dense padded representation contains B * L token positions. The number of real token positions is
sum_i l_iso the padding count is
B * L - sum_i l_iThis arithmetic makes batch composition significant. Four sequences of lengths 120, 124, 131, and 136 require 544 padded positions in total, of which 511 contain actual tokens. Replacing the last sequence with one of length 900 raises the rectangular shape to 3600 positions while the actual token count becomes 1275.
Padding masks prevent padded positions from being treated as ordinary content, but a mask does not imply that every underlying kernel skips all arithmetic or storage associated with those positions. The exact savings from reducing padding depend on the model architecture, attention implementation, tensor shapes, compiler behavior, and hardware. Token occupancy is therefore a useful batching metric, not a direct promise of proportional runtime reduction.
A simple occupancy measure is
occupancy = real_tokens / padded_tokenswhere padded_tokens is B * L for a conventional dense batch. Tracking this value exposes a property that examples-per-batch alone hides.
Bucketing changes the distribution inside each batch
The most direct bucketing scheme sorts examples by token count and takes consecutive groups. That produces compact batches, but a global sort also creates a strong ordering pattern. Short examples appear together for an extended interval, followed by progressively longer examples.
That ordering can be undesirable during stochastic optimization because batch composition is then correlated with sequence length across a large portion of an epoch. Sequence length may also correlate with other properties of the data, so strict ordering can cluster more than just tensor shapes.
A common design keeps bucketing local. The data pipeline can shuffle examples, collect a larger temporary pool, sort or partition that pool by length, emit batches from nearby lengths, then continue with another shuffled pool. Another design assigns examples to coarse length ranges and shuffles within each range.
These approaches trade some packing tightness for less deterministic ordering. The correct bucket width is workload-dependent: narrow ranges reduce padding but constrain which examples can share a batch, while broad ranges increase mixing at the cost of more variation in the local maximum.
Token budgets behave differently from fixed batch sizes
Fixed example counts can still produce large variation in total tokens. A batch of eight 100-token sequences presents a very different tensor shape from a batch of eight 2,000-token sequences.
A token budget limits batch formation using sequence lengths rather than only the number of examples. One simple policy adds examples while the estimated padded shape remains below a configured bound:
candidate_cost = candidate_batch_size * candidate_max_lengthThe estimate matches the number of token positions in a dense rectangular input, though it does not represent every memory allocation made during a forward or backward pass.
Length bucketing and token budgets address related but distinct concerns. Bucketing reduces variation among examples placed together. A token budget adjusts the number of examples so long-sequence batches do not become arbitrarily large in token count. Combining them can produce small example counts for long sequences and larger example counts for short sequences while keeping padded token counts within a narrower range.
This changes the meaning of a training batch. If optimizer updates occur after every physical batch, variable example counts also produce variable numbers of examples per update. Systems that require a particular effective batch definition may instead accumulate gradients until a target token count, example count, or another explicit boundary is reached.
Length must be measured after the relevant transformation
Bucketing is only as accurate as its length estimate. Character count, byte count, word count, and token count are not interchangeable. For transformer inputs, the useful quantity is generally the sequence length that reaches batching after tokenization and any truncation or special-token insertion relevant to the model input.
Precomputing exact token counts can simplify batch planning when tokenized examples are stored. In pipelines that tokenize dynamically, an inexpensive estimate may avoid an extra preprocessing pass, but estimation errors reduce bucket quality. The effect is operational rather than semantic: examples still contain the same data, but poorly estimated groups can regain the padding spread that bucketing was meant to reduce.
Paired or multi-field inputs add another detail. If the tokenizer combines fields into one model sequence, the batching length should reflect the combined representation. Bucketing on only one field can be misleading when the other field varies substantially.
Truncation also changes the useful distribution. If all inputs above a configured limit are truncated to that limit, their effective lengths collapse to the same ceiling. Bucketing based on raw pre-truncation size can separate examples that ultimately produce identical tensor lengths.
Distributed training needs balanced batch assignment
In data-parallel training, each worker typically processes its local batch before workers synchronize for the next update. If one worker repeatedly receives longer padded batches than its peers, faster workers can spend time waiting at synchronization points.
Length-aware batching can reduce padding on each worker yet still create imbalance across workers if batches are assigned poorly. For example, placing several long batches on one worker and several short batches on another creates different local workloads even when each individual batch has high occupancy.
A distributed sampler therefore has two related jobs: construct batches with acceptable internal length variation and distribute those batches so workers receive comparable work over the synchronization interval. Exact balancing is not guaranteed by matching example counts, because equal counts can represent very different padded token counts.
Randomness also needs explicit handling. If every worker independently builds buckets from the same global data without coordinated partitioning, examples can be duplicated or omitted. The partitioning and shuffling rules should remain consistent with the framework’s distributed sampling contract.
Padding metrics need context
A lower padding ratio is useful evidence that bucketing changed tensor occupancy, but it does not establish an end-to-end speedup by itself. Input preparation can become more complex, dynamic shapes can affect compilation or kernel selection, and very narrow buckets can fragment available examples into awkward batch sizes.
Useful measurements separate these effects. Padding ratio shows how much of the rectangular input is real data. Tokens processed per unit time captures execution throughput. Peak device memory shows whether smaller padded shapes create usable memory headroom. Data-loader wait time can reveal whether sorting, tokenization, or queueing moved the bottleneck outside the model.
The comparison also needs the same effective workload. Measuring two runs with different truncation, optimizer update semantics, or token counts can make a batching change appear responsible for effects produced elsewhere.
Length bucketing is most valuable when sequence lengths vary enough for local maxima to create substantial padding. When inputs are already close in size, the batching machinery has little padding to remove. Treating the observed length distribution as part of the system design keeps the optimization tied to the workload rather than to a fixed batching recipe.