Processing data in groups is common in Python. An application may send records to an API 100 at a time, insert rows into a database in manageable groups, or divide a stream of identifiers into work units without first loading the whole input into memory.

Since Python 3.12, the standard library provides itertools.batched() for this pattern. It consumes an iterable lazily and yields tuples containing up to a requested number of items. Python 3.13 added a strict option for cases where an incomplete final batch should be treated as an error.

The helper removes a surprising amount of hand-written iterator code, but using it well still requires understanding what it guarantees. Batching controls grouping. It does not automatically add concurrency, retries, atomicity, rate limiting, or a hard limit on how much input will eventually be consumed.

Start with the basic operation

A simple call groups values into tuples:

from itertools import batched

values = range(10)

for batch in batched(values, 3):
    print(batch)

The batches are:

(0, 1, 2)
(3, 4, 5)
(6, 7, 8)
(9,)

The last tuple is shorter because the input length is not evenly divisible by three. That is the default behavior, not an exceptional case.

n must be at least one. Passing zero or a negative value raises ValueError.

Batching is lazy

One of the most useful properties of batched() is that it accepts any iterable and consumes only enough input to produce the next batch.

That means the input does not need to be a list:

from itertools import batched


def read_ids():
    for line in open("ids.txt", encoding="utf-8"):
        yield line.strip()


for group in batched(read_ids(), 500):
    process_ids(group)

Conceptually, each iteration asks the source for enough elements to fill one tuple. The next portion of the source is not consumed until the caller asks for the next batch.

This is different from first materializing all values and then slicing them. If the source contains millions of records, lazy batching can keep the grouping layer’s memory requirement proportional to the batch size rather than the total number of records.

The source itself can still buffer data, of course. Laziness in batched() cannot force an upstream library to use bounded memory if that library has already loaded everything.

Do not convert the result to a list unless you need all batches

It is easy to accidentally remove the memory advantage:

all_batches = list(batched(source, 1000))

That expression consumes the entire source and stores every resulting tuple in a list. It may be appropriate for a small finite input, but it is not streaming anymore.

Prefer direct iteration when each batch can be handled independently:

for batch in batched(source, 1000):
    write_batch(batch)

After write_batch() returns, the loop can move on without retaining previous tuples unless other code keeps references to them.

Decide whether a partial final batch is valid

For many workloads, the default partial final tuple is exactly what you want. If an API accepts at most 100 records per request, there is usually no reason to reject a final request containing 37 records.

from itertools import batched

for batch in batched(records, 100):
    send_records(batch)

Other formats require exact group sizes. Imagine a flat stream representing RGB pixels, where every logical pixel must contain exactly three channel values. Silently accepting one or two trailing values could hide malformed input.

On Python 3.13 and later, use strict=True:

from itertools import batched

for red, green, blue in batched(channels, 3, strict=True):
    process_pixel(red, green, blue)

If the input ends with an incomplete tuple, iteration raises ValueError instead of yielding that tuple.

Strictness is therefore a statement about the input shape: the total number of consumed elements must be divisible by the batch size.

Strict errors happen during iteration

Because batched() is lazy, an incomplete-input error cannot necessarily be known when the iterator is created.

from itertools import batched

batches = batched(range(5), 2, strict=True)

print(next(batches))  # (0, 1)
print(next(batches))  # (2, 3)
print(next(batches))  # raises ValueError

This timing matters when earlier batches have side effects. Suppose each full batch is immediately inserted into an external system. By the time strict mode discovers the incomplete tail, earlier inserts may already have succeeded.

for batch in batched(records, 100, strict=True):
    database.insert_many(batch)

If the source contains 250 records, two 100-record inserts can happen before the iterator discovers the remaining 50 and raises an error.

strict=True validates batch completeness, but it does not make the whole loop transactional. If all-or-nothing behavior is required, validate the complete finite input before producing side effects or use transaction support provided by the destination system.

Tuples give each batch a stable snapshot

Each yielded batch is a tuple. The tuple itself cannot be resized or have its positions reassigned:

batch = next(batched(["a", "b", "c"], 2))
assert batch == ("a", "b")

That makes batches convenient to pass to functions that should receive a fixed group of references.

Tuple immutability does not make the contained objects immutable. If a batch contains dictionaries, lists, or application objects, those objects can still be changed by other code.

records = [{"status": "new"}, {"status": "new"}]
batch = next(batched(records, 2))

records[0]["status"] = "sent"
assert batch[0]["status"] == "sent"

Treat the tuple as a stable grouping boundary, not as a deep copy of the data.

Iterators are consumed, not copied

Passing an iterator to batched() advances that iterator. If other code holds the same iterator, it observes the new position.

from itertools import batched

source = iter(range(10))
batches = batched(source, 3)

assert next(batches) == (0, 1, 2)
assert next(source) == 3

The batching iterator does not create an independent replayable view of the source.

This is normal iterator behavior, but it becomes important when a source represents a database cursor, network response, parser, or generator with side effects. Make ownership clear: if one component is responsible for consuming the source in batches, avoid having unrelated components pull values from the same iterator.

Batch size is not a total processing limit

A call such as:

batched(events, 100)

means “yield groups containing up to 100 events.” It does not mean “process at most 100 events.”

If events contains one million items and the caller iterates to exhaustion, all one million can eventually be consumed.

If the application needs an overall limit as well as batching, express both policies:

from itertools import batched, islice

limited_events = islice(events, 10_000)

for batch in batched(limited_events, 100):
    process(batch)

That loop consumes at most 10,000 values from events. Whether silently stopping at that point is correct depends on the application. For untrusted input, you may instead need to detect that more data exists and reject it explicitly.

Batching does not add concurrency

This loop is sequential:

for batch in batched(records, 100):
    send_records(batch)

The next call to send_records() does not begin until the previous call returns. batched() only shapes the input.

If parallel requests are appropriate, concurrency must be introduced separately with an executor, an asynchronous design, or another concurrency primitive. That also introduces new questions: maximum in-flight work, ordering, retries, cancellation, destination capacity, and partial failure.

Do not add concurrency simply because work has been divided into batches. A destination may have rate limits or transaction constraints that make sequential processing preferable.

Batch size belongs to the destination contract

There is no universally optimal batch size. Choose it according to the system receiving each group.

A database may have limits on statement size or parameter count. An HTTP API may document a maximum number of records per request. A worker may have a memory budget. A filesystem operation may become slower rather than faster if each unit of work is made too large.

For example:

API_MAX_ITEMS = 200

for batch in batched(records, API_MAX_ITEMS):
    send_records(batch)

Naming the constraint communicates more than a unexplained numeric literal and makes it easier to keep the code aligned with the external contract.

When batch size is only a performance tuning parameter, measure the real workload. Larger groups can reduce per-call overhead but can also increase latency, memory usage, retry cost, and the amount of work affected by one failure.

Handle failures at the correct boundary

Consider an API that accepts batches and can fail transiently:

for batch in batched(records, 100):
    send_records(batch)

If send_records() raises, iteration stops unless the exception is handled. The current tuple still exists, so retry logic can retry that same batch without asking the source to reconstruct it.

That can be useful:

for batch in batched(records, 100):
    send_with_retry(batch)

But retry safety depends on the destination. If the first request may have succeeded even though the client observed a timeout, blindly repeating it can create duplicates. Idempotency keys, unique constraints, upserts, or destination-specific transaction semantics may be necessary.

batched() gives you a convenient unit to retry. It cannot determine whether retrying that unit is safe.

Be careful with infinite iterables

Lazy batching works naturally with infinite iterables:

from itertools import batched, count, islice

numbers = count()
first_five_batches = islice(batched(numbers, 4), 5)

for batch in first_five_batches:
    print(batch)

This terminates because islice() limits the number of batches consumed.

Without an external stopping condition, iterating over batches from an infinite source never completes. That may be intentional for a long-running stream processor, but cleanup, cancellation, and shutdown then need to be part of the surrounding design.

Strict mode is generally meaningful only when an iterable eventually ends. An infinite iterable never has a final incomplete batch to validate.

Account for the Python version

itertools.batched() was added in Python 3.12. The strict keyword was added in Python 3.13.

Code that must support Python 3.11 or earlier cannot import batched from itertools. Code that runs on Python 3.12 can use batched() but cannot pass strict=True.

Check the project’s declared minimum Python version before adopting either feature. If older versions must remain supported, a small compatibility helper using itertools.islice() can implement the same grouping pattern, but keeping one local implementation is preferable to scattering subtly different batching loops throughout a codebase.

Test boundaries, not only the happy path

Batching logic is small enough that boundary tests provide most of the value. Test an empty input, an input shorter than one batch, an exact multiple, and a final partial batch.

For Python 3.13 strict mode:

from itertools import batched

assert list(batched([], 3, strict=True)) == []
assert list(batched(range(6), 3, strict=True)) == [
    (0, 1, 2),
    (3, 4, 5),
]

And verify the failure case:

import pytest
from itertools import batched

with pytest.raises(ValueError):
    list(batched(range(5), 3, strict=True))

For code with external side effects, also test what happens when processing fails on a middle batch. The important behavior is often not the grouping itself but whether retries, checkpoints, or partial results behave according to the application’s contract.

Conclusion

itertools.batched() is a focused standard-library tool for turning an iterable into consecutive tuples without materializing the complete input. Its lazy behavior makes it suitable for large and streaming sources, while the default partial final batch handles the common case where the input length does not divide evenly.

Use strict=True on Python 3.13 and later when an incomplete final group means the input is malformed, but remember when that validation occurs: a lazy consumer can already have processed earlier batches before the error is discovered.

Most importantly, keep batching separate from the policies around it. Batch size controls grouping; it does not impose a total input limit, make side effects atomic, create parallelism, or make retries safe. When those guarantees matter, design them explicitly around the simple iterator boundary that batched() provides.

Python Iterators Standard Library Data Processing