https://nalar.dev/tags/batching/ Recent content in Batching on Nalar Hugo en-us Wed, 16 Sep 2026 00:00:00 +0700 https://nalar.dev/bucket-sequence-lengths-to-reduce-padding-waste/ Wed, 16 Sep 2026 00:00:00 +0700 https://nalar.dev/bucket-sequence-lengths-to-reduce-padding-waste/ <p>A padded batch is shaped by its longest sequence, not its average sequence. If one batch contains token counts of <code>120</code>, <code>124</code>, <code>131</code>, and <code>900</code>, every sequence may be represented at length <code>900</code>. Most positions in the first three rows then carry padding rather than input tokens.</p> <p>Length bucketing changes batch composition instead of changing the model. Examples with similar token counts are placed near each other before batches are formed. The maximum length inside each batch falls closer to the lengths of its members, reducing the number of padded positions processed by operations that still use the rectangular batch shape.</p>