Batching

Bucket Sequence Lengths to Reduce Padding Waste

A padded batch is shaped by its longest sequence, not its average sequence. If one batch contains token counts of 120, 124, 131, and 900, every sequence may be represented at length 900. Most positions in the first three rows then carry padding rather than input tokens. Length bucketing changes batch composition instead of changing the model. Examples with similar token counts are placed near each other before batches are formed. The maximum length inside each batch falls closer to the lengths of its members, reducing the number of padded positions processed by operations that still use the rectangular batch shape.