Python applications have long had standard-library support for gzip, bzip2, LZMA, and zlib. Python 3.14 adds another important option: Zstandard support through compression.zstd.

Zstandard is useful when a system needs a practical balance of compression ratio and throughput. The new module means many applications can read and write .zst data without adding a third-party Python package. But choosing a compression API is not only about calling compress() and decompress(). Production code also needs to think about streaming, memory limits, frame boundaries, dictionaries, compatibility, and untrusted input.

This article develops those boundaries explicitly.

Start with the Python version boundary

The compression package and compression.zstd were added in Python 3.14. Code that imports them therefore makes Python 3.14 part of its runtime contract.

from compression import zstd

payload = b"event=login user=42\n" * 100
compressed = zstd.compress(payload)
restored = zstd.decompress(compressed)

assert restored == payload

For an application already requiring Python 3.14, this is straightforward. A reusable library supporting older Python releases should not silently introduce the import without either raising its minimum version or providing a deliberate compatibility path.

There is another deployment detail: compression.zstd is an optional CPython module. A Python distributor can ship an interpreter without it. If your application depends on Zstandard, verify that capability in the environments you actually deploy rather than assuming that every Python 3.14 installation contains it.

Use one-shot APIs only when the size is naturally bounded

The module-level functions are convenient for payloads that comfortably fit in memory:

from compression import zstd


def encode_record(data: bytes) -> bytes:
    return zstd.compress(data)


def decode_record(data: bytes) -> bytes:
    return zstd.decompress(data)

compress() returns the complete compressed result as bytes, while decompress() returns the complete expanded result. That shape is excellent for small messages, cache entries, or bounded database values.

It is a poor default for an arbitrarily large upload. A tiny compressed input can expand into much more data, and a one-shot decompression call asks Python to materialize the result in memory.

The important boundary is therefore not merely the compressed byte count. Systems accepting untrusted compressed data need an explicit policy for how much expanded data they are willing to produce.

Stream files instead of materializing them

For files, use zstd.open() or ZstdFile so data can be processed incrementally.

from compression import zstd


def write_events(path, events):
    with zstd.open(path, "wt", encoding="utf-8") as f:
        for event in events:
            f.write(event)
            f.write("\n")


def read_events(path):
    with zstd.open(path, "rt", encoding="utf-8") as f:
        for line in f:
            yield line.rstrip("\n")

Text mode wraps the compressed binary stream in a text layer. For binary formats, use rb and wb instead.

This approach changes the memory model: the program does not need to hold the entire uncompressed file at once. It does not, however, create an automatic application-level size limit. A consumer can still read forever unless your code imposes a bound.

Bound expanded output from untrusted inputs

Incremental decompression gives more control over resource usage. ZstdDecompressor.decompress() accepts max_length, allowing each call to cap the amount of output returned.

A useful design is to keep a separate total output budget:

from compression import zstd


class ExpandedDataTooLarge(ValueError):
    pass


def decompress_bounded(data: bytes, limit: int) -> bytes:
    if limit < 0:
        raise ValueError("limit must be non-negative")

    decoder = zstd.ZstdDecompressor()
    parts = []
    produced = 0
    pending = data

    while True:
        remaining = limit - produced
        chunk = decoder.decompress(pending, max_length=remaining + 1)
        pending = b""

        produced += len(chunk)
        if produced > limit:
            raise ExpandedDataTooLarge(
                f"expanded data exceeds {limit} bytes"
            )
        parts.append(chunk)

        if decoder.eof:
            return b"".join(parts)

        if decoder.needs_input:
            raise ValueError("incomplete Zstandard frame")

The + 1 is intentional: it lets the code distinguish output exactly equal to the budget from output that exceeds it.

This example handles one frame. That distinction matters because the incremental ZstdDecompressor class does not transparently process multiple concatenated frames. The module-level decompress() function and ZstdFile do. If your protocol permits concatenated frames, define that policy explicitly rather than assuming every API treats them identically.

For network streams, the same principle applies, but feed compressed chunks incrementally and maintain both compressed-input and expanded-output budgets.

Understand needs_input, eof, and unused_data

Incremental decompression is a state machine.

After a decompress() call, needs_input tells you whether the decoder requires more compressed bytes before it can produce additional output. If it is False, calling decompress(b"", max_length=...) can drain buffered output.

eof becomes true when the end of the current frame has been reached.

unused_data contains bytes found after that frame. Those bytes may represent another Zstandard frame, another protocol field, or unexpected trailing input. Your application has to decide which interpretation is valid.

Do not discard trailing bytes automatically in a format where exactly one frame is required. Treating ignored data as valid can create parser disagreement between components.

Finish incremental compression correctly

Incremental compression can buffer data internally, so output from each compress() call is only part of the final stream.

from compression import zstd


def compress_chunks(chunks) -> bytes:
    encoder = zstd.ZstdCompressor()
    output = []

    for chunk in chunks:
        output.append(encoder.compress(chunk))

    output.append(encoder.flush())
    return b"".join(output)

Calling flush() finishes the frame and emits buffered data. Forgetting that finalization step can leave an incomplete compressed representation.

Zstandard also distinguishes flushing a block from finishing a frame. A block flush can make current data immediately decompressible while still allowing later blocks to reference earlier data. A frame flush ends that frame; future compressed data belongs to a new frame.

Choose frame boundaries according to the protocol, not merely according to convenient write boundaries.

A checksum is integrity detection, not authentication

Zstandard can include a checksum in a frame. In Python, advanced compression options can enable it:

from compression import zstd

options = {
    zstd.CompressionParameter.checksum_flag: 1,
}

encoded = zstd.compress(b"important payload", options=options)

The checksum helps detect accidental corruption of the uncompressed content. It does not prove who created the data and does not protect against an attacker who can replace both content and checksum.

If a protocol requires authenticity, use an authenticated protocol or cryptographic authentication at the appropriate layer. Compression checksums solve a different problem.

Limit decompression memory as well as output size

Expanded byte count is not the only resource dimension. Zstandard frames can request large history windows. DecompressionParameter.window_log_max can constrain the maximum decompression window.

from compression import zstd

options = {
    zstd.DecompressionParameter.window_log_max: 24,
}

decoder = zstd.ZstdDecompressor(options=options)

The appropriate value depends on the data you need to accept. A stricter window can reject valid inputs produced with larger settings, so this is a protocol and deployment decision rather than a universal constant.

For hostile inputs, think in terms of several budgets together: compressed bytes accepted, expanded bytes produced, decompression window memory, CPU time, and wall-clock lifetime.

No single compression option substitutes for all of them.

Compression levels are policy, not correctness

zstd.compress() accepts a compression level. Higher effort can improve compression ratio but consume more CPU and, at some settings, more memory. Negative levels trade ratio for speed.

Avoid encoding an assumption such as “higher is always better” into application logic. The right level depends on where the bottleneck lives.

For request-path data, latency may dominate. For archival data written once and transferred many times, additional compression work may pay for itself. Benchmark representative payloads on representative hardware.

Also avoid treating compressed size as stable output. Library versions, parameters, dictionaries, and input details can affect representation. Tests should normally assert successful round trips and required format properties rather than one exact compressed byte string unless the protocol specifically requires deterministic bytes.

Dictionaries are part of the wire contract

Zstandard dictionaries can improve compression for many small, structurally similar payloads. Python exposes dictionary training and ZstdDict support.

That optimization introduces state outside the compressed payload. A decoder must have the compatible dictionary used by the encoder.

If dictionaries cross service boundaries, manage them like protocol artifacts:

  • assign stable identifiers or versions;
  • deploy decoders before encoders begin producing data with a new dictionary;
  • retain old dictionaries while old data can still be read;
  • define what happens when a dictionary is unavailable;
  • do not assume a recorded dictionary ID by itself distributes the dictionary.

A dictionary rollout that saves bandwidth but makes stored data unreadable is not an optimization.

Prefer the file interface for ordinary .zst interoperability

When the requirement is simply “read this .zst file” or “produce a .zst file,” the file interface is usually clearer than manually managing compressor state.

from compression import zstd


def copy_to_zstd(source_path, destination_path):
    with open(source_path, "rb") as source:
        with zstd.open(destination_path, "wb") as destination:
            while chunk := source.read(1024 * 1024):
                destination.write(chunk)

This keeps memory bounded to ordinary buffering and delegates frame handling to the file abstraction.

If ZstdFile wraps an already-open file object, closing the ZstdFile does not close that underlying object. Ownership should still be obvious in your code: the layer that opens a resource should normally be responsible for closing it.

Do not confuse compression with an archive format

A .zst stream compresses bytes. It does not by itself define a directory tree, filenames, permissions, or multiple files.

If you need an archive, combine compression with an archive format such as tar, or use an archive API that supports Zstandard. Keep the two responsibilities conceptually separate:

archive format: files + paths + metadata -> byte stream
compression:    byte stream -> smaller byte stream

That separation matters for security too. Path traversal and extraction policy belong to the archive layer; decompression resource limits belong to the compression layer. Applying one policy does not automatically solve the other.

Design the API around explicit ownership and limits

A robust compression boundary can expose policy directly:

from dataclasses import dataclass


@dataclass(frozen=True)
class DecompressionPolicy:
    max_compressed_bytes: int
    max_expanded_bytes: int
    allow_concatenated_frames: bool = False

The exact implementation depends on whether data comes from memory, a file, or a socket, but naming the policy prevents accidental unlimited behavior from becoming the de facto contract.

This also makes testing easier. Boundary tests can exercise a payload just below the expanded limit, exactly at the limit, and one byte above it.

Test behavior, not only happy-path round trips

At minimum, tests around a Zstandard boundary should cover:

  • empty input;
  • small and large round trips;
  • incremental compression finalization;
  • truncated compressed data;
  • corrupted frames;
  • output exactly at and just above configured limits;
  • a frame requiring more window memory than policy allows;
  • trailing bytes after a frame;
  • concatenated frames if the protocol permits them;
  • required and missing dictionaries;
  • text encoding behavior when using text mode;
  • runtime behavior when compression.zstd is unavailable, if that deployment is supported.

Round-trip tests establish basic correctness. Adversarial and boundary tests establish whether the surrounding system keeps its promises under failure.

Keep the abstraction boundary clear

Python 3.14’s compression.zstd removes a dependency for an increasingly common compression format, but the most useful design lesson is broader than Zstandard.

Compression is a transformation with resource consequences. One-shot helpers are appropriate for bounded values. File and incremental interfaces are better for streams. Frame boundaries affect interoperability. Dictionaries become protocol dependencies. Checksums detect corruption but do not authenticate data. Untrusted decompression needs explicit limits.

When those choices are represented in the application API instead of hidden behind a bare decompress() call, compression becomes easier to operate, test, and evolve safely.

Python Zstandard Compression Standard Library Python 3.14