Building Memory-Efficient Iterator Pipelines with Python itertools

Python programs often transform data in stages: read records, discard unwanted items, reshape values, group adjacent entries, and stop after enough output has been produced. A straightforward implementation may build a new list after every stage. That is easy to understand, but it can also allocate intermediate collections that the next stage immediately consumes.

Iterator pipelines offer another model. Each stage requests values from the stage before it as needed. Python’s itertools module provides building blocks for this style, including tools for chaining inputs, taking slices from streams, computing running values, grouping consecutive records, and duplicating an iterator when two consumers genuinely need it.

Lazy processing is not automatically better. Iterators are stateful and usually single-pass, some tools buffer internally, and debugging a long pipeline can be harder than inspecting a concrete list. The goal is to use laziness where it matches the data flow, not to remove every list from a program.

Start with the single-pass model

An iterable can produce an iterator, while an iterator represents a position in a stream. Calling next() advances that position.

values = iter([10, 20, 30])

print(next(values))
print(list(values))

The output is:

10
[20, 30]

The call to next() consumed the first item. Converting the same iterator to a list afterward starts from its current position rather than rewinding it.

This is the first rule of iterator pipelines: consumption is part of program state. A function that accepts an arbitrary iterator should not assume it can traverse the input twice.

Iterables and iterators are not interchangeable concepts

A list is iterable and can create a fresh iterator each time:

values = [10, 20, 30]

print(list(values))
print(list(values))

Both traversals see all three values.

A generator object, file iterator, or many objects returned by itertools are themselves iterators. Once consumed, their previous elements are gone unless some layer explicitly stored them.

APIs that accept Iterable inputs should therefore avoid undocumented second passes. If repeated traversal is required, either require a reusable collection or materialize the input deliberately and document the memory trade-off.

Chain sources without concatenating them first

itertools.chain() presents several iterables as one continuous iterator:

from itertools import chain

primary = ["a", "b"]
fallback = ["c", "d"]

for item in chain(primary, fallback):
    print(item)

The first iterable is exhausted before iteration continues with the next one. chain() does not need to construct a combined list first.

This is useful when sources are already separate and downstream code only needs sequential access.

When the inputs themselves arrive as an iterable of iterables, use chain.from_iterable():

from itertools import chain

pages = [
    ["row-1", "row-2"],
    ["row-3"],
    ["row-4", "row-5"],
]

rows = chain.from_iterable(pages)
print(list(rows))

This flattens exactly one level. It is not a recursive flattening operation, which is usually a benefit: the structure being removed is explicit.

Limit a stream with islice

Normal sequence slicing assumes indexed access. itertools.islice() applies slice-like selection to an iterable by consuming it in order.

from itertools import islice

source = iter(range(100))
selected = islice(source, 10, 20, 3)

print(list(selected))

The selected positions are 10, 13, 16, and 19.

Unlike list slicing, islice() does not support negative start, stop, or step values. An iterator cannot generally jump backward to find an element relative to its end.

Slicing still advances the input

Using islice() does not preserve skipped values for later use:

from itertools import islice

source = iter(range(10))
head = list(islice(source, 3))
rest = list(source)

print(head)
print(rest)

The output is:

[0, 1, 2]
[3, 4, 5, 6, 7, 8, 9]

That behavior makes islice() useful for consuming a prefix, but it can surprise code that expects the original iterator to remain untouched.

Treat infinite iterators as resources that need a boundary

Some itertools functions can produce unbounded streams. count() is a common example:

from itertools import count, islice

ids = count(start=1000)
print(list(islice(ids, 5)))

The result is:

[1000, 1001, 1002, 1003, 1004]

An infinite iterator is safe when a downstream stage establishes a finite stopping condition. Without one, operations that try to consume the entire iterator never finish.

For example, never call list(count()). The problem is not count() itself; it is combining an unbounded producer with a consumer that requests exhaustion.

The same reasoning applies to cycle() and an unbounded repeat(). Keep the termination rule visible near the pipeline that uses them.

Compute running state with accumulate

itertools.accumulate() emits intermediate accumulated values rather than only a final reduction.

from itertools import accumulate

changes = [5, -2, 4, -1]
balances = accumulate(changes, initial=100)

print(list(balances))

The output is:

[100, 105, 103, 107, 106]

With the default operation, values are added. A custom two-argument function can define another accumulation rule.

The initial value is itself emitted first, so providing it makes the output one element longer than the input. This detail matters when accumulated values are zipped back together with source records.

Use sum() when only the final total matters. Use accumulate() when each running result is meaningful to downstream processing.

Group consecutive records with groupby

itertools.groupby() is often misunderstood as a general SQL-style grouping operation. It groups consecutive items that have the same key.

from itertools import groupby
from operator import itemgetter

records = [
    ("error", "disk full"),
    ("error", "timeout"),
    ("info", "started"),
    ("info", "ready"),
]

for level, group in groupby(records, key=itemgetter(0)):
    print(level, list(group))

Because equal keys are adjacent, the output contains one error group followed by one info group.

If matching keys are separated, they form separate groups:

records = [
    ("error", "first"),
    ("info", "middle"),
    ("error", "last"),
]

This input produces two different error groups.

Sort first only when global grouping is the requirement

If the requirement is to gather all equal keys together and reordering is acceptable, sort by the same key before calling groupby():

from itertools import groupby
from operator import itemgetter

records = [
    ("error", "first"),
    ("info", "middle"),
    ("error", "last"),
]

records.sort(key=itemgetter(0))

for level, group in groupby(records, key=itemgetter(0)):
    print(level, list(group))

Sorting changes the complexity and memory story. If records are already ordered by the grouping key, groupby() can process them incrementally. If they are not and global grouping is required, the sort may dominate the work.

Consume each group before advancing

The group iterator shares the underlying input with groupby(). Once the outer iterator advances, an earlier group cannot be treated as an independent stored collection.

If a group must outlive that iteration step, materialize it while it is current:

stored = []

for key, group in groupby(records, key=itemgetter(0)):
    stored.append((key, list(group)))

That copy is intentional: persistence requires ownership of the values rather than a transient view over the shared iterator.

Duplicate an iterator with tee carefully

Sometimes two consumers really do need to read the same source independently. itertools.tee() can create multiple iterators from one input:

from itertools import tee

source = iter([10, 20, 30])
left, right = tee(source)

print(next(left))
print(list(right))
print(list(left))

The output is:

10
[10, 20, 30]
[20, 30]

The two resulting iterators can advance at different rates, but that independence requires internal buffering. If one consumer runs far ahead while another lags, values must be retained for the slower consumer.

For a large stream with highly uneven consumers, that buffer can grow substantially. If one consumer will finish most or all of the data before the other starts, materializing a list may be simpler and can be more appropriate.

tee() also is not a general thread-synchronization mechanism. Do not use it as a substitute for a queue or other concurrency primitive.

Compose a bounded lazy pipeline

Iterator tools become most useful when the pipeline has a clear source, transformation, and boundary.

Suppose a service receives pages of integer measurements. The program needs the first five non-negative values across all pages and then wants their running totals:

from itertools import accumulate, chain, islice

pages = [
    [-3, 4, 2],
    [-1, 5, 7],
    [9, 11, 13],
]

values = chain.from_iterable(pages)
non_negative = (value for value in values if value >= 0)
first_five = islice(non_negative, 5)
running_totals = accumulate(first_five)

print(list(running_totals))

The result is:

[4, 6, 11, 18, 27]

The pipeline stops after five accepted values. It does not need a flattened list of every page or a separate list containing every non-negative value.

The final list() is appropriate here because the example explicitly wants a concrete result. In a real application, another streaming consumer could iterate over running_totals directly.

Know where laziness stops helping

Lazy iteration reduces intermediate materialization, but it cannot make every algorithm streaming.

A global sort needs to see all relevant values before it can produce a fully ordered result. Computing an exact median of arbitrary unsorted input generally needs retained state. Reusing the same data for many independent passes may be clearer with a concrete collection.

Laziness also does not guarantee lower peak memory when a pipeline includes buffering operations. tee() may retain a growing backlog, cycle() saves values from its input so it can repeat them, and application code may materialize groups or results later.

Evaluate the whole pipeline rather than labeling an individual function “memory efficient” in isolation.

Keep side effects out of fragile pipelines

Lazy stages run when values are requested, not necessarily when the pipeline is constructed.

def observe(value):
    print("seen", value)
    return value

mapped = map(observe, [1, 2, 3])

At this point observe() has not run. Its side effect occurs as mapped is consumed.

This deferred execution can make pipelines with logging, mutation, network calls, or other side effects harder to reason about. Prefer pure transformations where practical. When a side effect is the purpose of the operation, an explicit loop often communicates timing and error handling better.

Avoid hiding consumption in helper functions

A helper that peeks at an iterator by calling next() has consumed a value unless it deliberately returns that value to the caller’s flow.

For example, this function loses the first item:

def has_items(iterator):
    try:
        next(iterator)
    except StopIteration:
        return False
    return True

Calling it changes the iterator it examines.

If an API needs non-destructive emptiness checks, reconsider the interface. A collection can expose its length, while a one-pass stream generally cannot reveal whether it has a next item without attempting to retrieve that item. Designs that preserve the retrieved value explicitly are safer than pretending the check was observational.

Common pitfalls

Iterating twice over a single-pass input

The second pass sees only what the first pass left behind. Require a reusable collection or materialize deliberately when repeated traversal is part of the contract.

Assuming groupby gathers every matching key

groupby() groups adjacent equal keys. Sort first only if global grouping and reordered input are appropriate.

Letting tee consumers drift indefinitely

The faster consumer can force buffering for the slower one. Use tee() when consumption stays reasonably close or when the retained backlog is known to be acceptable.

Exhausting an infinite iterator

Functions such as count(), cycle(), and unbounded repeat() need a downstream limit or stopping predicate.

Expecting islice to preserve skipped input

islice() advances the source iterator as it finds the requested positions. It is a consuming view, not random access.

Building pipelines that are cleverer than their requirements

A list comprehension may be clearer for a small in-memory collection. Iterator composition is most valuable when input can be large, arrives incrementally, or can stop early.

Design around ownership and consumption

Iterator pipelines work well when each stage has a simple contract: receive the next value, transform or filter it, and pass results onward. chain() can join sources without concatenation, islice() can establish a finite boundary, accumulate() can carry running state, and groupby() can process already ordered groups incrementally.

The trade-off is that iteration has state. Values disappear as they are consumed, duplicated consumers may require buffering, and some algorithms eventually need concrete storage.

Keep those boundaries explicit. Use iterators when one-pass processing matches the problem, materialize data when ownership or repeated access requires it, and treat buffering as a real cost even when it is hidden behind a convenient standard-library function.