Temporary data often has an awkward size profile. Most requests may produce a few kilobytes, while an occasional import, report, archive, or upload grows to hundreds of megabytes.

Using io.BytesIO is convenient for the small case, but its contents stay in memory. Using a temporary file avoids keeping the whole payload in memory, but every payload uses file-system-backed storage even when it is tiny.

Python’s tempfile.SpooledTemporaryFile gives you a middle ground. It behaves like a file object while keeping data in memory up to a configured threshold. When the data grows beyond that threshold, it rolls over to a temporary file and continues through the same interface.

The useful mental model is one seekable file-like object with two possible storage phases. Your code should depend on the file interface, not on which phase happens to be active.

Start with a file-like buffer that can spill

Suppose an export function writes bytes in several chunks, and another function expects a file-like object:

from tempfile import SpooledTemporaryFile

def build_export(chunks):
    spool = SpooledTemporaryFile(max_size=1_000_000, mode="w+b")

    for chunk in chunks:
        spool.write(chunk)

    spool.seek(0)
    return spool

The caller can read the result exactly as it would read a normal binary file:

with build_export([b"header\n", b"row-1\n", b"row-2\n"]) as export:
    data = export.read()

print(data)

For data that stays at or below the configured size threshold, the object can remain memory-backed. If its size exceeds max_size, SpooledTemporaryFile writes the contents to a temporary file and continues operating as a file-like object.

The caller does not need separate code paths for the two cases.

The example explicitly uses mode="w+b" because the payload is bytes. For text, use a text mode such as "w+" and specify an encoding when appropriate.

The threshold is a rollover policy, not a payload limit

max_size controls when size-based rollover occurs. It does not reject larger data.

This distinction matters:

with SpooledTemporaryFile(max_size=16, mode="w+b") as spool:
    spool.write(b"0123456789")
    spool.write(b"abcdefghij")

    spool.seek(0)
    payload = spool.read()

assert payload == b"0123456789abcdefghij"

The payload is 20 bytes, so it exceeds the 16-byte threshold. The object remains usable; it simply moves to temporary-file-backed storage.

That makes SpooledTemporaryFile useful when you want to optimize the common small case while still accepting larger inputs.

It is not a defense against unbounded input. A client that can send unlimited data can still consume disk space after rollover. If input size must be constrained, enforce a separate application-level limit while reading.

Rollover preserves the file position and contents

Code that uses a spooled file should not have to restart when storage changes.

Consider a writer that crosses the threshold in the middle of a sequence:

with SpooledTemporaryFile(max_size=8, mode="w+b") as spool:
    spool.write(b"hello")
    spool.write(b" world")

    spool.seek(0)
    assert spool.read() == b"hello world"

The second write makes the total larger than eight bytes. The implementation rolls the data into a temporary file, but the logical file still contains the bytes that were already written.

That continuity is the main reason to prefer the abstraction over manually switching from BytesIO to a temporary file. A manual design has to copy existing data, preserve the current offset, transfer ownership, and ensure cleanup. SpooledTemporaryFile centralizes those transitions.

Seeking still matters when switching from writing to reading

Rollover does not change normal file-position rules.

After writing, the current position is at the end of the data. Calling read() immediately therefore returns an empty byte string:

with SpooledTemporaryFile(max_size=1024, mode="w+b") as spool:
    spool.write(b"report data")

    assert spool.read() == b""

Move the position before reading:

with SpooledTemporaryFile(max_size=1024, mode="w+b") as spool:
    spool.write(b"report data")
    spool.seek(0)

    assert spool.read() == b"report data"

This mistake is easy to misdiagnose as a rollover problem because the same object handles both reading and writing. It is simply ordinary seekable-file behavior.

If multiple stages share the object, document who is responsible for positioning it. A useful convention is for a producer to return the file positioned at the beginning when the next step is expected to read it.

Use the file interface instead of inspecting storage internals

A spooled file exposes the operations normal consumers usually need: read(), write(), seek(), tell(), truncate(), iteration, and context-manager cleanup.

Code should generally not inspect private attributes to determine whether rollover happened:

# Avoid coupling application logic to private implementation state.
if spool._rolled:
    ...

An underscore-prefixed attribute is not part of the public API contract. Logic built around it can also defeat the abstraction: downstream code now cares about storage details that were supposed to be interchangeable.

Prefer capability-based code:

def checksum_input(file_obj):
    file_obj.seek(0)

    total = 0
    while chunk := file_obj.read(64 * 1024):
        total = (total + sum(chunk)) % 2**32

    return total

This function works whether the spool is currently memory-backed or temporary-file-backed. It can also work with other compatible binary file objects, which makes testing and reuse easier.

Some operations force rollover even below the threshold

Size is not the only trigger.

The documented rollover() method explicitly moves the data to a temporary file:

with SpooledTemporaryFile(max_size=1_000_000, mode="w+b") as spool:
    spool.write(b"small payload")
    spool.rollover()

    spool.seek(0)
    assert spool.read() == b"small payload"

Calling fileno() also causes rollover because an in-memory buffer does not itself provide the operating-system file descriptor that fileno() promises:

with SpooledTemporaryFile(max_size=1_000_000, mode="w+b") as spool:
    spool.write(b"small payload")

    fd = spool.fileno()
    print(fd)

This is an important boundary when integrating with lower-level APIs. A library that calls fileno() can turn a small in-memory spool into a real temporary file even though the size threshold was never crossed.

If keeping small payloads memory-backed is important for your workload, check whether downstream components require a file descriptor.

A realistic pattern: stage data before handing it off

Spooling is useful when a producer generates data incrementally but a consumer wants a readable file object.

For example, imagine creating a CSV export before sending it to another layer:

from tempfile import SpooledTemporaryFile

def create_csv(rows):
    spool = SpooledTemporaryFile(
        max_size=8 * 1024 * 1024,
        mode="w+",
        encoding="utf-8",
        newline="",
    )

    spool.write("id,name\n")

    for user_id, name in rows:
        spool.write(f"{user_id},{name}\n")

    spool.seek(0)
    return spool

The eight-megabyte threshold is an application choice, not a universal recommendation. It should reflect expected payload sizes, concurrency, available memory, temporary-storage capacity, and the cost profile of the surrounding system.

If 200 requests can build exports concurrently, an eight-megabyte threshold creates a very different worst-case memory exposure than the same threshold in a single-user command-line tool.

That is why max_size should be chosen from workload constraints rather than copied from an example.

Spooling can reduce memory pressure, but it does not guarantee a memory ceiling

It is tempting to describe a spool threshold as “use at most this much RAM.” That is too strong.

The threshold governs when this file object rolls its stored contents to a temporary file. Your process may still hold other copies of the same data. For example:

chunk = receive_large_chunk()
spool.write(chunk)

chunk continues to exist until your code releases it, regardless of where the spool stores its own contents.

Temporary buffers, parser objects, compressed representations, request bodies, and downstream copies can all contribute additional memory.

For bounded-memory processing, combine spooling with incremental reads and reasonably sized chunks:

def copy_stream(source, destination):
    while chunk := source.read(64 * 1024):
        destination.write(chunk)

Even then, total process memory depends on the rest of the application. Spooling controls one part of the data path; it is not a process-wide memory limiter.

Temporary storage introduces its own failure modes

After rollover, writes depend on the system’s temporary-file facilities.

That means operations can fail because the temporary location is out of space, unavailable, subject to quotas, or otherwise unable to satisfy the write. Treat writes as I/O operations that can raise exceptions rather than assuming rollover always succeeds.

The dir argument can select the directory used for temporary-file storage:

with SpooledTemporaryFile(
    max_size=4 * 1024 * 1024,
    mode="w+b",
    dir="/var/tmp",
) as spool:
    spool.write(b"temporary data")

Only choose a specific directory when deployment guarantees make that location appropriate. Hard-coding a Unix path makes the code less portable, and changing the directory does not remove the need to plan for capacity and cleanup behavior.

For portable libraries, letting tempfile choose the temporary location is usually the better default.

Cleanup should be explicit and ownership should be clear

SpooledTemporaryFile can be used as a context manager, so the simplest ownership model is:

with SpooledTemporaryFile(max_size=1024 * 1024, mode="w+b") as spool:
    spool.write(b"temporary work")
    # Consume the spool here.

The object is closed when the block exits, including when an exception leaves the block.

Returning an open spool from a function is also valid, but ownership then transfers to the caller:

def make_payload():
    spool = SpooledTemporaryFile(max_size=1024 * 1024, mode="w+b")
    spool.write(b"payload")
    spool.seek(0)
    return spool

with make_payload() as payload:
    consume(payload)

Document that responsibility. Leaking temporary-file objects under load can consume file descriptors and temporary storage after rollover.

Choose the simpler alternative when the size profile is predictable

SpooledTemporaryFile is most useful when payload sizes vary enough that both memory-backed and file-backed behavior are valuable.

Use io.BytesIO when data is known to be small and you do not need a real file descriptor. It is a simpler in-memory abstraction with no rollover behavior to reason about.

Use TemporaryFile when the data is expected to be large, when disk-backed storage is acceptable from the start, or when consumers reliably need fileno() anyway.

For data that should never be materialized in full, neither option is ideal. Prefer streaming from producer to consumer so that only a bounded chunk is resident at a time.

Spooling occupies the middle: it is useful when you need seekable file semantics, the common case is small, large cases are legitimate, and switching storage behind the interface is preferable to maintaining two separate paths.

Common mistakes to avoid

The most important mistakes follow directly from the storage model.

Do not treat max_size as an input-size limit; enforce limits separately. Do not assume data remains in memory merely because it is smaller than the threshold, because fileno() or rollover() can force a temporary file. Do not inspect private attributes to make business decisions. Do not forget to seek() when changing from writing to reading. And do not choose a threshold without considering how many spooled objects can exist concurrently.

These are not edge details. They determine whether a spooling design actually improves resource behavior or merely moves pressure from memory to temporary storage.

Conclusion

SpooledTemporaryFile is a useful standard-library tool for temporary data with uncertain size. It gives callers one seekable file-like interface, keeps smaller data memory-backed when possible, and rolls larger data into temporary-file storage without requiring a second code path.

Use it when that transition matches your workload. Set the threshold deliberately, stream data into it in bounded chunks, expect normal I/O failures after rollover, and keep downstream code focused on the public file interface rather than storage internals.

When those conditions do not apply, BytesIO, TemporaryFile, or true streaming may be simpler choices.

Python Standard Library File I/O Performance