Binary files and network messages often begin with fixed-width fields: a four-byte signature, a one-byte version, a two-byte payload length, or a four-byte identifier. Those fields are easy to describe on paper but surprisingly easy to parse incorrectly in code.
The difficult part is not converting bytes to integers. It is preserving the binary layout contract: exactly which byte belongs to which field, which byte order is used, how wide each value is, and what should happen when the input is incomplete or malformed.
Python’s struct module gives you a compact way to encode that contract. It can pack Python values into bytes and unpack bytes back into values without third-party dependencies.
The useful mental model is: a struct format string is a schema for a fixed binary layout, not a general serialization format. Use it when the layout is already defined and byte positions matter. Keep variable-length data, validation rules, and protocol state outside that fixed-layout schema.
Start with one fixed header
Suppose a small protocol defines this 12-byte header:
| Field | Size | Meaning |
|---|---|---|
| magic | 4 bytes | Constant NLR1 |
| version | 1 byte | Protocol version |
| flags | 1 byte | Bit flags |
| payload length | 2 bytes | Number of payload bytes |
| request ID | 4 bytes | Unsigned identifier |
The corresponding struct format can be written as:
import struct
HEADER = struct.Struct("!4sBBHI")Read the format from left to right:
!selects network byte order, which is big-endian, with standard field sizes and no native alignment padding.4sis one four-byte byte string.- each
Bis one unsigned byte. His one unsigned two-byte integer.Iis one unsigned four-byte integer.
You can ask Python for the exact size instead of repeating 12 throughout the program:
assert HEADER.size == 12That matters because the format string should be the single source of truth for the fixed layout. If the header changes later, code that uses HEADER.size changes with it.
Pack and unpack the smallest useful record
Packing turns Python values into the specified binary representation:
MAGIC = b"NLR1"
header_bytes = HEADER.pack(
MAGIC,
1, # version
0b00000101, # flags
3, # payload length
42, # request ID
)header_bytes now contains exactly 12 bytes.
Unpacking reverses the operation:
magic, version, flags, payload_length, request_id = HEADER.unpack(
header_bytes
)The result is determined by the format. The four-byte s field becomes a bytes object, while the integer fields become Python integers.
One important boundary rule is that unpack() requires the supplied buffer to have exactly the size required by the format. It is therefore a good fit when you already hold exactly one record.
If you have a larger buffer containing a header plus other bytes, use unpack_from() instead.
Parse a header from a larger buffer
A network receive buffer or file block often contains more than one logical field. Slicing out the first 12 bytes works:
fields = HEADER.unpack(data[:HEADER.size])but it creates a new bytes object when data is a bytes object.
unpack_from() reads the fields starting at an offset in an existing buffer:
fields = HEADER.unpack_from(data, 0)This avoids creating that header slice and also makes nonzero offsets explicit.
A parser should still validate the boundary before unpacking so that callers get an error meaningful to the application:
def parse_header(data: bytes):
if len(data) < HEADER.size:
raise ValueError("incomplete header")
magic, version, flags, payload_length, request_id = HEADER.unpack_from(data)
if magic != MAGIC:
raise ValueError("invalid magic")
if version != 1:
raise ValueError(f"unsupported version: {version}")
return flags, payload_length, request_idWithout the length check, struct would raise struct.error when the buffer is too small. That exception is accurate at the binary-decoding layer, but many applications benefit from translating it into a domain-specific failure such as ValueError, a protocol error, or an “await more bytes” state.
Notice what the parser validates separately from the binary conversion. struct knows how to read four bytes as a byte string and two bytes as an integer. It does not know that the magic must be NLR1 or that only version 1 is supported.
Byte order must be part of the protocol contract
Multi-byte integers can be stored with their most significant byte first or last. These conventions are commonly called big-endian and little-endian.
For example:
import struct
assert struct.pack(">H", 0x1234) == b"\x12\x34"
assert struct.pack("<H", 0x1234) == b"\x34\x12"Both byte sequences represent the same integer under different layouts. A parser that chooses the wrong byte order still gets an integer; it simply gets the wrong integer. That makes endianness bugs especially dangerous because they can look like valid data.
For externally defined formats, make the choice explicit:
>means big-endian, standard sizes, no alignment padding.<means little-endian, standard sizes, no alignment padding.!means network byte order, which is big-endian.=uses native byte order but standard sizes and no alignment padding.@, or no prefix, uses native byte order, native sizes, and native alignment.
The default native mode is useful when interoperating with a matching in-memory C layout on the same platform. It is usually the wrong default for files and wire protocols because the representation can depend on the machine and C compiler used to build Python.
A portable format should state its byte order and field sizes rather than inherit them from the host.
Native alignment can add bytes you did not specify
Native mode does more than choose the machine’s byte order. It can also insert padding between fields to match C alignment rules.
For example, the size of this native layout can depend on the platform:
native = struct.Struct("@BI")A protocol that means “one byte followed immediately by four bytes” should instead use an explicit standard mode:
portable = struct.Struct("!BI")
assert portable.size == 5With standard modes such as !, >, <, and =, padding is added only if the format string explicitly includes padding fields such as x.
This distinction is a guarantee of the struct format rules. The exact padding chosen by native mode is platform-dependent, so code should not assume a particular native size unless the platform layout itself is part of the contract.
Validate lengths before trusting them
Fixed headers often contain lengths for variable-size data. Reading that integer is only the first step. Treat it as untrusted input until you validate it.
Consider a complete message consisting of the 12-byte header followed by its payload:
MAX_PAYLOAD = 4096
def parse_message(data: bytes):
if len(data) < HEADER.size:
raise ValueError("incomplete header")
magic, version, flags, payload_length, request_id = HEADER.unpack_from(data)
if magic != MAGIC:
raise ValueError("invalid magic")
if version != 1:
raise ValueError(f"unsupported version: {version}")
if payload_length > MAX_PAYLOAD:
raise ValueError("payload too large")
message_size = HEADER.size + payload_length
if len(data) < message_size:
raise ValueError("incomplete payload")
payload = data[HEADER.size:message_size]
remainder = data[message_size:]
return (flags, request_id, payload), remainderThere are two separate checks for two separate risks.
First, MAX_PAYLOAD prevents a declared length from causing unreasonable buffering or allocation elsewhere in the program. The correct limit is application-specific; 4096 is only an example policy.
Second, the parser verifies that the declared payload actually exists before slicing it as a complete message. A length field describes what the sender claims is present. It does not prove those bytes have arrived or that a file is not truncated.
This is particularly important for streaming transports. One call to recv() is not guaranteed to return one complete application message. The framing layer must keep incomplete data and continue reading until the required number of bytes is available.
Separate binary framing from text decoding
A struct string field such as 4s represents bytes, not text. That is useful because binary layout and character encoding are different concerns.
Suppose a payload is defined as UTF-8 text. Parse the binary frame first, then decode the payload:
message, remainder = parse_message(data)
flags, request_id, payload = message
text = payload.decode("utf-8")Keeping those steps separate gives each boundary a clear failure mode:
- malformed or incomplete framing fails while parsing the binary message;
- invalid UTF-8 fails while decoding the payload.
Do not put an arbitrary text field directly into a fixed-width s format unless the protocol actually defines a fixed-width byte field. 10s, for example, means exactly ten bytes in the binary record; it does not mean ten Unicode characters.
Reuse Struct objects for repeated layouts
The module-level functions are convenient for one-off operations:
struct.pack("!I", 42)
struct.unpack("!I", b"\x00\x00\x00*")When the same layout appears throughout a parser, a Struct object makes the schema reusable:
HEADER = struct.Struct("!4sBBHI")It also keeps pack(), unpack(), pack_into(), unpack_from(), iter_unpack(), and size attached to the same format definition.
This is mainly a clarity and maintainability benefit: code can name the binary record instead of repeating a format string. CPython also caches some module-level format handling internally, so you should not claim that constructing Struct objects is always a meaningful performance optimization without measuring your workload.
If parsing speed matters, benchmark the complete parsing path with realistic record sizes and validation, not just a format-string call in isolation.
Use pack_into when you already own a writable buffer
pack() returns a new bytes object. Sometimes an application already has a bytearray that it is filling.
pack_into() writes directly into a writable buffer at a chosen offset:
buffer = bytearray(HEADER.size)
HEADER.pack_into(
buffer,
0,
MAGIC,
1,
0b00000010,
0,
99,
)This is useful when assembling a larger preallocated frame or repeatedly writing records into an existing buffer.
The trade-off is that buffer management becomes your responsibility. The destination must be writable and large enough for the packed data at the chosen offset. For small or infrequent records, ordinary pack() is often simpler and clearer.
Use iter_unpack for repeated fixed-size records
Some binary formats contain a sequence of identical records with no per-record length field. iter_unpack() can walk such a buffer one record at a time.
Imagine a little-endian record containing a four-byte unsigned sensor ID and a two-byte signed reading:
RECORD = struct.Struct("<Ih")
records = b"".join([
RECORD.pack(1001, 23),
RECORD.pack(1002, -4),
])
for sensor_id, reading in RECORD.iter_unpack(records):
print(sensor_id, reading)The buffer length must be an exact multiple of the record size. If trailing bytes remain, iter_unpack() raises struct.error instead of silently ignoring them.
That behavior is useful for fixed-record files because an unexpected remainder can indicate truncation, corruption, or a mistaken format.
It is not a replacement for a stateful streaming parser. If a read ends in the middle of a record, keep the partial bytes, append later input, and call the fixed-record parser only when enough data is available.
Avoid copies with buffer-aware parsing where it matters
Several struct operations accept objects implementing Python’s buffer protocol, not only bytes. That includes common types such as bytearray and memoryview.
If a larger message is already stored in memory, unpack_from() can read a header at an offset without first creating a copied slice:
packet = memoryview(data)
magic, version, flags, payload_length, request_id = HEADER.unpack_from(packet)A payload can also be represented as a view:
start = HEADER.size
end = start + payload_length
payload_view = packet[start:end]This can reduce temporary byte copies in parsers that process large buffers or many messages.
The benefit is workload-dependent. A memoryview introduces another object and can make ownership and lifetime less obvious. For small messages, straightforward bytes slices are often easier to reason about and plenty fast enough.
Use buffer views when measurements or memory pressure justify the extra complexity, not merely because zero-copy parsing sounds preferable.
Range errors are different from malformed input
Integer format codes have defined ranges. For example, H is an unsigned two-byte integer, so values must fit from 0 through 65535.
Packing an out-of-range value raises struct.error:
HEADER.pack(MAGIC, 1, 0, 70000, 42)This is usually a producer-side bug or missing validation, not a decoding failure.
On the receiving side, every bit pattern of a fixed-width unsigned integer is representable, so unpacking an H field will always produce some integer if the required two bytes are available. Whether that integer is acceptable is a separate protocol rule.
This distinction is useful when designing validation:
structenforces the binary representation;- application code enforces semantic constraints.
Do not expect the binary decoder to know that a request ID must be nonzero, a flag combination is reserved, or a length is too large for your service.
Common mistakes come from mixing layout with meaning
Several bugs recur in binary parsers because two different layers are treated as one.
Using native mode for a portable format
This is risky:
HEADER = struct.Struct("4sBBHI")With no prefix, native byte order, native sizes, and native alignment apply.
If the file or protocol defines its own layout, choose the appropriate explicit prefix:
HEADER = struct.Struct("!4sBBHI")Assuming one read equals one message
struct parses bytes that you already have. It does not provide message framing over a socket or pipe.
If a header says the payload is 500 bytes and only 120 have arrived, the correct action is usually to retain those bytes and read more, not to parse a shorter payload.
Trusting a length before applying a limit
A syntactically valid length can still be operationally unacceptable. Apply protocol and application limits before using an untrusted size to control buffering, allocation, loops, or downstream work.
Treating bytes as text too early
Binary fields are bytes until the protocol says otherwise. Decode text fields with the specified character encoding after the binary boundaries are known.
Guessing sizes instead of asking the format
Use Struct.size or struct.calcsize() rather than duplicating hand-computed constants throughout the code.
When struct is the right tool
struct fits formats with fixed-width numeric fields, byte strings, known padding, and explicit byte order. Common examples include binary file headers, device messages, packet metadata, and small records exchanged with systems written in other languages.
It is less suitable when the data model is primarily nested, self-describing, schema-evolving, or composed of many optional and variable-length fields. Formats such as JSON, CBOR, Protocol Buffers, MessagePack, or a domain-specific parser may provide a better abstraction in those cases.
Even inside a larger protocol, struct can still be useful for the fixed portion. A common design is to use it for a compact header and let another layer handle the variable payload.
Keep the binary contract visible
Reliable binary parsing comes from making the layout explicit and keeping representation separate from validation.
Define one reusable format with an explicit byte order. Derive record sizes from that format. Check that enough bytes exist before unpacking. Validate signatures, versions, ranges, flags, and declared lengths after conversion. Keep text decoding and variable-length framing outside the fixed record definition.
With those boundaries clear, struct becomes more than a shorthand for byte conversion: it becomes an executable description of the fixed part of your binary protocol, while the surrounding code remains responsible for deciding whether the decoded message is actually valid.