Process Go Slices in Batches with slices.Chunk

Batching a slice sounds simple until the loop starts collecting edge cases: the final batch may be short, an empty input needs sensible behavior, and careless subslicing can leave each batch with capacity to overwrite later elements.

Go 1.23 added slices.Chunk, which handles that bookkeeping and exposes the batches as an iterator. If you already have the data in a slice and want to process consecutive groups without first building a [][]T, it’s a useful small tool.

What slices.Chunk returns

slices.Chunk takes a slice and a positive batch size. It returns an iter.Seq that yields consecutive subslices containing at most that many elements.

package main

import (
    "fmt"
    "slices"
)

func main() {
    jobs := []string{"a", "b", "c", "d", "e"}

    for batch := range slices.Chunk(jobs, 2) {
        fmt.Println(batch)
    }
}

The output is:

[a b]
[c d]
[e]

Every batch except possibly the last has the requested size. slices.Chunk doesn’t pad the final batch, so code consuming it should accept a shorter last group.

This API also means batching is lazy at the container level. slices.Chunk doesn’t allocate a [][]string containing every batch before the loop starts. The iterator yields each subslice as the loop advances.

Use slices.Chunk when the work is naturally batched

A common case is sending records to an API that accepts only a limited number per request. The batching logic can stay separate from the operation performed on each group:

func sendAll(ctx context.Context, users []User) error {
    for batch := range slices.Chunk(users, 100) {
        if err := sendBatch(ctx, batch); err != nil {
            return err
        }
    }
    return nil
}

There are two useful properties here. The original order is preserved, and processing can stop immediately when sendBatch fails. There is no need to calculate start and end indexes in the business logic.

The same pattern works for database writes, queue publishing, file processing, or any operation where the input is already materialized as a slice. If the input arrives as a stream instead, forcing the entire stream into a slice just to use slices.Chunk is usually the wrong trade-off; an iterator- or stream-oriented batching function fits that case better.

The chunks share elements with the original slice

The chunks are subslices, not copies. Mutating an existing element through a chunk therefore changes the corresponding element in the original slice.

numbers := []int{10, 20, 30, 40}

for batch := range slices.Chunk(numbers, 2) {
    batch[0] = -1
}

fmt.Println(numbers) // [-1 20 -1 40]

That behavior can be useful when processing data in place, but it’s easy to miss if a batch is handed to code that looks like it owns its input. If a consumer needs an independent batch, clone it explicitly:

for batch := range slices.Chunk(numbers, 2) {
    owned := slices.Clone(batch)
    processLater(owned)
}

Cloning is especially relevant when batches outlive the loop or cross an ownership boundary where mutation would be surprising.

Each chunk has its capacity clipped

Ordinary subslicing can expose more capacity than its visible length. For example, s[:2] may still have enough capacity for append to overwrite elements that appear later in s.

slices.Chunk avoids that particular trap. Every yielded subslice has its capacity clipped to its length. For a five-element slice chunked by two, the capacities are 2, 2, and 1.

That changes what happens when you append:

values := []int{1, 2, 3, 4}

for batch := range slices.Chunk(values, 2) {
    batch = append(batch, 99)
    fmt.Println(batch)
    break
}

fmt.Println(values) // [1 2 3 4]

Because the first batch has no spare capacity, append must obtain different backing storage instead of using the original slice’s following element as append space. Existing elements still alias the source; capacity clipping doesn’t turn the chunk into a deep or independent copy.

Empty input produces no batch

An empty slice yields an empty sequence. The loop body doesn’t run, and there isn’t a single empty batch to handle.

var values []int
count := 0

for range slices.Chunk(values, 3) {
    count++
}

fmt.Println(count) // 0

This is convenient for most processing loops: no input means no calls to the batch operation. If your application needs to perform an action once even when there are zero items, that policy belongs outside slices.Chunk.

A non-positive chunk size is a programming error

The chunk size must be at least one. Passing zero or a negative value causes slices.Chunk to panic.

That makes a fixed constant straightforward:

const batchSize = 100

for batch := range slices.Chunk(records, batchSize) {
    // process batch
}

If the size comes from configuration or user input, validate it before constructing the iterator rather than treating the panic as input validation:

if batchSize < 1 {
    return fmt.Errorf("batch size must be positive: %d", batchSize)
}

for batch := range slices.Chunk(records, batchSize) {
    // process batch
}

One subtle detail is worth knowing: iterator functions execute when iteration begins. Don’t build code around the exact moment a bad size panics; validate external values explicitly and keep the contract clear.

Don’t collect chunks unless you need to retain them

It can be tempting to turn every iterator into a slice immediately. For batch processing, that often removes the main benefit of the API.

Prefer processing directly:

for batch := range slices.Chunk(records, 50) {
    process(batch)
}

If you need a persistent [][]T for later random access, then collecting batches can make sense. Remember that collected chunks still refer to the original elements unless you clone each chunk. A two-dimensional slice of subslices is not automatically an ownership boundary.

There is also no requirement to consume every batch. A break stops iteration cleanly:

for batch := range slices.Chunk(records, 50) {
    if enough(batch) {
        break
    }
}

That makes slices.Chunk a better fit than eagerly constructing all batch descriptors when the consumer may stop early.

Prefer the standard helper over index arithmetic

A manual batching loop isn’t inherently bad. You may need one when boundaries depend on byte size, cost, timestamps, or another condition that isn’t a fixed element count.

For fixed-size groups, though, slices.Chunk expresses the rule directly. It also handles the short final batch, empty input, and capacity clipping consistently. That leaves the loop focused on what happens to a batch rather than how to calculate it.

Use it when the input is already a slice and batches are defined by element count. Validate dynamic batch sizes, clone when a consumer needs ownership, and keep streaming inputs streaming instead of materializing them solely to fit this API.