Reading a file in chunks is a small problem that appears in many larger tasks: hashing uploads, copying large files, parsing binary records, compressing streams, and sending data without loading everything into memory.

A common solution is a while loop that reads one chunk, checks for end-of-file, processes the chunk, and repeats. That loop is correct when written carefully, but Python has another standard-library pattern that expresses the same control flow as iteration:

iter(callable, sentinel)

In this two-argument form, iter() repeatedly calls a zero-argument callable. Each returned value becomes the next item until a value compares equal to the sentinel, a distinguished stop value. The sentinel itself is not yielded.

For ordinary binary files, this fits chunked reads well because read(size) returns b"" after end-of-file. The result is a compact loop whose control condition is explicit: keep processing chunks until the read returns the EOF value.

Start with the ordinary read loop

Suppose you want to compute a SHA-256 digest without reading an entire file into memory.

The straightforward form is:

import hashlib


def sha256_file(path: str) -> str:
    digest = hashlib.sha256()

    with open(path, "rb") as file:
        while True:
            chunk = file.read(64 * 1024)
            if chunk == b"":
                break
            digest.update(chunk)

    return digest.hexdigest()

This loop has four responsibilities:

  1. perform the read;
  2. recognize EOF;
  3. stop at EOF;
  4. process every non-empty chunk.

The code is valid and often the clearest choice when the loop needs additional state or several exit conditions. But when the repeated operation and the stop value are the entire loop condition, iter(callable, sentinel) can represent that structure directly.

The two-argument form of iter() is a loop adapter

Most Python developers first meet iter() through its one-argument form:

iterator = iter([10, 20, 30])

That asks an iterable object for an iterator.

The two-argument form has a different contract:

iterator = iter(callable, sentinel)

Here, the first argument must be callable with no arguments. Each time the iterator needs another item, it calls that callable. If the returned value compares equal to sentinel, iteration ends. Otherwise, the value is yielded.

A small example makes the behavior visible:

values = iter([3, 2, 1, 0, 99])

countdown = iter(lambda: next(values), 0)

assert list(countdown) == [3, 2, 1]

The callable returns 3, 2, 1, and then 0. Because 0 == sentinel, the iterator stops and never asks for 99.

The mental model is:

call -> result -> compare with sentinel
                 |
                 +-- equal     -> stop
                 +-- not equal -> yield result, then call again

This is useful whenever an API naturally signals completion by returning a particular value.

Adapt read(size) into a zero-argument callable

A file’s read() method accepts a size argument, but iter(callable, sentinel) needs a callable that takes no arguments.

One clear way to bridge that difference is functools.partial():

from functools import partial


with open("archive.bin", "rb") as file:
    read_chunk = partial(file.read, 64 * 1024)

    for chunk in iter(read_chunk, b""):
        process(chunk)

partial(file.read, 64 * 1024) creates a callable that behaves like this:

def read_chunk():
    return file.read(64 * 1024)

The important part is not partial() itself. The important part is adapting an operation that requires fixed arguments into a zero-argument operation that iter() can call repeatedly.

A lambda is equally valid:

with open("archive.bin", "rb") as file:
    for chunk in iter(lambda: file.read(64 * 1024), b""):
        process(chunk)

Choose whichever form is easier to read in the surrounding code. partial() can make the fixed arguments explicit by name; a short lambda can be easier to recognize without another imported name.

Rewrite the hashing example as iteration

The earlier hashing function becomes:

import hashlib
from functools import partial


def sha256_file(path: str) -> str:
    digest = hashlib.sha256()

    with open(path, "rb") as file:
        read_chunk = partial(file.read, 64 * 1024)

        for chunk in iter(read_chunk, b""):
            digest.update(chunk)

    return digest.hexdigest()

The data behavior is the same as the explicit while loop:

  • the file is opened in binary mode;
  • each call asks for at most 64 KiB;
  • each non-empty bytes object is passed to the hash;
  • b"" marks EOF and is not passed to digest.update().

This pattern bounds the amount requested from the file on each read. It does not guarantee that every returned chunk has exactly the requested length. The final chunk of a regular file is commonly shorter, and stream-like objects may return shorter reads for other reasons.

Code that processes chunks should therefore treat chunk length as variable unless the specific input API promises otherwise.

The sentinel is checked with equality

An important detail is that iter(callable, sentinel) stops when the returned value is equal to the sentinel. The comparison is not based only on object identity.

Conceptually, the behavior is close to:

while True:
    value = callable()

    if value == sentinel:
        break

    yield value

This matters when choosing a sentinel. If the callable can legitimately return a value that compares equal to the sentinel before the operation is actually finished, iteration will stop too early.

For ordinary binary file reads, b"" is a natural sentinel because a non-empty chunk contains at least one byte, while an empty bytes result indicates EOF for the blocking file reads used here.

For a different API, do not copy b"" mechanically. First identify that API’s actual completion signal and whether it can collide with legitimate data.

EOF is not the same as an error

The sentinel pattern handles a returned stop value. It does not turn failures into normal termination.

Consider the hashing loop again. If the underlying read raises an OSError, the read did not return b""; it failed. The exception leaves the loop unless surrounding code handles it.

That distinction is useful:

read returns b"" -> normal EOF -> iteration ends
read raises OSError -> read failure -> exception propagates

Do not catch broad exceptions merely to make every failure look like EOF. A truncated read caused by an I/O problem is operationally different from successfully reaching the end of a file.

If your application can recover from particular errors, handle those errors explicitly at the boundary where you have enough context to decide what recovery means.

Chunk size is a trade-off, not a magic constant

Using chunked I/O avoids constructing one Python object containing the entire file, which matters when input can be much larger than available memory.

The chunk size still involves trade-offs. Very small chunks increase the number of Python-level loop iterations and read calls. Larger chunks reduce that per-chunk overhead but keep more data live at once.

There is no universally optimal size. The best value depends on the storage layer, buffering already provided by Python and the operating system, the work performed per chunk, and the application’s memory constraints.

A value such as 64 * 1024 is reasonable for an example because it is large enough to demonstrate bounded block processing without pretending to be a performance recommendation. For performance-sensitive code, benchmark the complete workload with realistic input rather than optimizing the integer in isolation.

Keep processing separate from the read condition

The sentinel iterator is easiest to understand when it controls only the repeated read:

with open("events.bin", "rb") as file:
    for chunk in iter(lambda: file.read(8192), b""):
        parse_chunk(chunk)

Avoid hiding unrelated side effects inside the callable:

# Harder to reason about: reading and application state are coupled.
for chunk in iter(lambda: read_and_update_metrics(file), b""):
    parse_chunk(chunk)

The second version may be valid, but the stop condition is less obvious because the callable is doing more than producing the next value.

A useful design rule is: let the callable produce the next unit of work, and let the loop body process that unit. This preserves the same separation that makes normal for loops readable.

Be careful when records can cross chunk boundaries

Chunk iteration solves how to read bounded pieces. It does not automatically solve how to parse logical records.

Suppose a newline-delimited record is longer than one chunk. This is unsafe:

with open("events.log", "rb") as file:
    for chunk in iter(lambda: file.read(4096), b""):
        for line in chunk.splitlines():
            process_line(line)

A line can begin near the end of one chunk and continue in the next. Splitting each chunk independently can therefore manufacture incomplete records.

If the input is naturally line-oriented, iteration over the file object is usually simpler:

with open("events.log", "rb") as file:
    for line in file:
        process_line(line)

If you are parsing a binary or framed protocol, keep an explicit buffer for incomplete records and join them with subsequent chunks according to that format’s rules.

The read boundary is an implementation boundary. It is not necessarily a data-format boundary.

Do not use a sentinel loop when normal iteration already models the data

iter(callable, sentinel) is useful, but it is not a replacement for ordinary iteration.

For line-by-line text processing, prefer:

with open("access.log", encoding="utf-8") as file:
    for line in file:
        handle(line)

The file object already implements the iterator protocol around lines, so wrapping readline() in a sentinel iterator usually adds machinery without adding clarity.

Likewise, if a library already returns an iterator of messages, rows, or records, iterate over that object directly.

Use the callable-sentinel form when the API is fundamentally shaped like this:

call repeatedly -> receive a value
special returned value -> finished

It is most valuable when it turns that protocol into normal Python iteration.

A while loop is better when stopping is more complicated

The explicit loop remains the clearer option when there are multiple termination conditions:

while remaining > 0:
    chunk = file.read(min(8192, remaining))

    if chunk == b"":
        raise EOFError("file ended before the expected payload length")

    consume(chunk)
    remaining -= len(chunk)

Here, plain EOF is not successful completion. The operation expects a specific payload length, so reaching EOF early is an error.

Trying to force this into iter(callable, b"") would hide an important distinction: “the stream is finished” and “the required record is complete” are not the same condition.

Choose the construct that makes correctness easiest to see, not the construct with the fewest lines.

Common mistakes to avoid

Using the wrong sentinel type

Binary reads return bytes, so their EOF value is b"". Text reads return str, whose empty value is "".

Keep the modes and sentinels consistent:

# Binary
iter(lambda: binary_file.read(4096), b"")

# Text
iter(lambda: text_file.read(4096), "")

Mixing b"" and "" means the equality check will never match the actual empty value. A loop can then keep yielding empty results forever.

Assuming a chunk is always full

read(4096) means “read up to this amount” for the ordinary file interface. Code that indexes into a chunk as though it always contains 4096 bytes is fragile.

Use len(chunk) when size matters, and add an explicit buffering layer when the format requires fixed-size records.

Choosing a sentinel that is valid data

If a callable uses None both as a legitimate item and as a “finished” value, iter(callable, None) cannot distinguish the two meanings.

In that case, change the producer’s interface if you control it, use a richer result type, or write an explicit loop that can distinguish completion from valid values.

When this pattern is a good fit

Use iter(callable, sentinel) when all of these are true:

  • an operation is naturally repeated with the same arguments;
  • it returns a stable, unambiguous value when finished;
  • every other returned value should be processed uniformly;
  • there is one simple termination condition.

For chunked binary file processing, those conditions often hold exactly.

Prefer a regular for loop when the source already provides an iterator. Prefer an explicit while loop when termination depends on several pieces of state, EOF itself is an error, or the producer’s completion signal can collide with valid data.

Conclusion

The two-argument form of iter() is best understood as an adapter. It converts a “call repeatedly until this value appears” API into Python’s iterator protocol.

For ordinary chunked file reads, the mapping is especially clean: a zero-argument callable performs read(size), non-empty chunks are yielded, and b"" ends iteration at EOF.

The pattern is useful because it makes one simple control rule explicit. It is not automatically better than a while loop, and it does not solve parsing, buffering, error recovery, or record framing for you. Use it when the producer truly has one unambiguous sentinel, and keep the loop body focused on processing the values that were successfully produced.

Python Iterators File I/O Standard Library