PCIe Posted Writes Separate CPU Completion from Device Visibility
An MMIO store can be complete from the CPU’s point of view while the corresponding write is still moving through the I/O path. PCI and PCIe memory writes are normally posted: the requester does not wait for a completion response for each write. Bridges and interconnect logic can accept the transaction and let the CPU continue before the endpoint has consumed it.
That behavior is useful for throughput, but it creates a boundary that device drivers cannot treat like ordinary cached memory. Program order at the CPU, ordering provided by an MMIO accessor, and proof that a device has received a write are related properties, not interchangeable ones.
A posted write has no completion response to wait for
PCIe classifies ordinary Memory Write Requests as posted requests. The requester sends the transaction without receiving a Completion TLP that confirms the endpoint processed it. A Memory Read Request follows a different path: it is non-posted and requires a completion carrying status and, for a successful read, data.
This difference matters when software needs a point at which an earlier control-register write is known to have reached the device. Returning from a CPU instruction or from a platform MMIO write accessor does not, by itself, create that endpoint-visible checkpoint.
The path can contain several stages:
CPU
|
MMIO accessor
|
host bridge / root complex
|
PCIe fabric
|
endpoint registerA posted write may have progressed far enough for an upstream component to accept it while still being pending farther downstream. The exact buffering locations and timing are implementation-specific. Software should rely on the platform and bus ordering contract rather than assume a particular queue inside a bridge.
MMIO ordering and write completion solve different problems
Linux provides readl(), writel(), and related accessors so portable drivers do not express device I/O as ordinary pointer dereferences. These accessors carry architecture-specific semantics for MMIO ordering and interaction with normal memory.
Ordering answers questions such as whether one device access can pass another. Completion answers a stricter question: has an earlier write progressed to the point required by the driver before execution crosses a sensitive boundary?
A barrier intended for normal memory is not a universal substitute for an I/O completion operation. CPU memory ordering can constrain when operations are issued or observed at the processor boundary without forcing a posted PCIe write all the way to its target. The required primitive depends on the architecture, mapping attributes, bus, and driver contract.
This distinction also keeps two common patterns separate. A driver may need ordering between descriptor writes in RAM and an MMIO doorbell. In another path, it may need proof that an interrupt-disable register write reached the device before returning. Both involve ordering, but the second case introduces a device-visible completion requirement.
Readback can establish a device-visible checkpoint
Linux kernel documentation describes a standard technique for cases that require posted writes to reach a PCI device: issue a read from the same device after the write. The read cannot complete until ordering requirements relevant to the preceding write have been satisfied, so the returned read acts as a checkpoint for the earlier posted access.
A simplified sequence looks like this:
writel(mask, regs + IRQ_MASK);
(void)readl(regs + IRQ_MASK);The value may be irrelevant when the purpose of the read is flushing the preceding write. The register chosen for readback still matters. Drivers need a location that is safe to read in the device state being handled.
Reset and removal paths make that constraint visible. A normal device register may no longer respond reliably while hardware is being reset. Linux documentation notes that PCI configuration space can be used in cases where a flush read may encounter a non-responsive device, because the platform can provide defined failure handling for that access.
Readback is therefore not decorative synchronization. It is an I/O transaction selected because its completion creates an observation point that a posted write alone does not provide.
A spinlock does not automatically drain the I/O path
Locks serialize software critical sections, but the lifetime of a posted transaction can extend beyond the critical section unless the platform’s I/O rules provide the required ordering at that boundary.
Consider two CPUs updating the same device register under one lock:
CPU 0: lock
CPU 0: MMIO write A
CPU 0: unlock
CPU 1: lock
CPU 1: MMIO write B
CPU 1: unlockThe source-level lock order is unambiguous. The device-visible order still depends on the MMIO ordering guarantees supplied by the architecture and accessors. Linux has explicit I/O-ordering mechanisms for platforms where posted writes need additional handling around such boundaries.
Current Linux documentation specifies substantial ordering guarantees for the generic readX() and writeX() accessors on default I/O mappings, including ordering between writes protected by the same spinlock. Drivers using relaxed accessors, special mapping attributes, architecture-specific primitives, or older assumptions must follow the contract for those interfaces rather than transfer guarantees from one accessor family to another.
The useful rule is narrow: synchronization that protects software state and synchronization that constrains device I/O must each satisfy their own interface contract.
Doorbells expose the boundary between RAM and MMIO
Queue-based devices often consume descriptors from system memory after software writes a doorbell register. The intended sequence is conceptually simple:
write descriptor fields in RAM
publish descriptor state
write MMIO doorbell
device fetches descriptorCorrectness depends on the DMA and MMIO ordering guarantees that connect those steps. If the device can observe the doorbell before the descriptor contents are visible through its DMA path, it may fetch stale or partially published state.
Linux’s standard MMIO accessors provide defined relationships with prior memory writes for supported default mappings, while the DMA API defines additional rules for coherent and streaming mappings. Drivers cannot replace those contracts with a generic assumption that source-code order reaches every observer unchanged.
Posted-write completion adds another dimension. A driver that merely rings a doorbell often does not need to wait until the endpoint has consumed the doorbell; posting is useful precisely because execution can continue. A path that disables hardware activity and then releases resources may require a stronger checkpoint. The required synchronization follows the operation’s lifetime semantics, not the visual similarity of the two register writes.
Extra readbacks carry a real cost
A posted write allows the CPU-facing path to avoid waiting for a round trip. Following every MMIO write with a read would remove part of that advantage. A read request needs a completion, so the requester must wait for the response before the read itself is complete.
That does not imply one portable latency value. The cost depends on processor architecture, host bridge, topology, device, virtualization layer, and current system state. The defensible property is structural: a dependent readback introduces completion work that a posted write does not require.
Drivers therefore place flushes where correctness needs them rather than after every register update. Control transitions, interrupt masking, reset sequencing, and teardown are common places to inspect because software may cross a lifetime or ownership boundary immediately after the write.
Performance-sensitive paths deserve the same precision. Removing a readback is safe only when the remaining ordering contract still covers the operation. A benchmark improvement cannot establish correctness if the test never exercises the race that the readback prevented.
Device-visible state has its own synchronization boundary
Posted writes separate submission from endpoint observation. That separation is not a defect in PCIe; it is part of the transaction model that lets write traffic proceed without a completion response for every request.
Driver code becomes fragile when it collapses several boundaries into one. CPU execution order does not alone prove endpoint visibility. A memory barrier does not inherently drain a posted I/O transaction. A lock does not replace the MMIO guarantees of the platform. Conversely, a readback is unnecessary when the driver only needs the weaker ordering already supplied by its accessors.
The practical design point is to identify the exact boundary the operation must cross. If software only needs to submit a register update, posting can remain asynchronous. If subsequent code requires the device to have received that update, the driver needs an I/O completion mechanism defined for that device and platform.