PCIe Posted MMIO Writes Can Outlive the CPU Store That Issued Them

A CPU can retire or complete an MMIO store before the corresponding PCIe Memory Write has reached the target device. The gap exists because PCIe Memory Write requests are posted: the requester sends them without waiting for a completion packet from the completer.

That property is useful for throughput, but it creates an important boundary. A software-visible store instruction, an ordering barrier, and device observation of the write are not automatically the same event.

PCIe Memory Write is a posted transaction

PCIe classifies Memory Write requests as posted requests. A posted request has no normal Completion TLP returning success to the requester. Once a host bridge accepts a CPU-originated MMIO write, the transaction can continue through buffering and interconnect stages after the processor has moved on.

A simplified path looks like this:

CPU
 │ MMIO store
 ▼
host bridge / root complex
 │ posted Memory Write TLP
 ▼
PCIe fabric
 │
 ▼
device BAR

The exact buffering structure is implementation-specific. PCIe defines transaction semantics and ordering rules, but it does not require every processor, root complex, or endpoint to expose identical internal queues.

The absence of a completion response is the key architectural property. Software cannot treat instruction completion alone as an acknowledgement from the endpoint.

Ordering and completion are separate properties

A memory barrier can constrain the order in which operations become visible according to the CPU and platform memory model. That does not necessarily convert a posted PCIe write into a transaction with an endpoint acknowledgement.

Consider a driver that programs device state and then starts an operation:

write configuration register
write descriptor address
write command register

The required ordering between those writes depends on the platform accessors, mapping attributes, PCIe rules, and device interface. Correct ordering prevents the command from overtaking state that must precede it.

Completion is a different question:

Has the final posted write actually reached a point at which
the device-side effect is guaranteed to have been observed?

A CPU fence may be required for ordering in some access patterns, but a fence alone is not a universal posted-write drain mechanism for PCIe.

A non-posted read can provide a completion boundary

PCIe Memory Read requests are non-posted. A successful read requires a Completion carrying the requested data. If software performs a suitable MMIO read from the same device after posted writes, the returned completion can provide a stronger boundary than the preceding stores alone.

Conceptually:

CPU                PCIe fabric                 device
 │                      │                         │
 ├─ Memory Write ──────►│────────────────────────►│
 │   posted             │                         │
 │                      │                         │
 ├─ Memory Read ───────►│────────────────────────►│
 │   non-posted         │                         │
 │                      │◄────────────────────────┤
 │◄─ Completion ────────│                         │

For the read completion to serve as a flush, the selected transaction and platform must preserve the relevant ordering relationship with the earlier writes. This is a platform and device-interface rule, not a property of arbitrary reads.

Linux driver code commonly uses a read from a safe register on the same device when posted MMIO writes must be flushed. The register choice matters because reads can have side effects.

Readback is not interchangeable with reading any register

Some device registers clear status bits when read, advance FIFO state, acknowledge interrupts, or return values with other side effects. Such registers are poor flush targets even if the bus transaction itself would create the needed ordering boundary.

A driver therefore needs a register whose read semantics are safe in that state. Device documentation can designate a status, identification, or other stable register for this purpose.

The pattern is conceptually:

writel(value, device_base + CONTROL);

/* Read a documented safe register when a posted-write flush is required. */
readl(device_base + SAFE_STATUS);

This is schematic rather than a universal driver recipe. Linux MMIO accessor semantics vary by architecture, and some subsystems provide dedicated helpers or stronger abstractions. The device specification remains authoritative for register side effects and required sequencing.

Doorbells expose the practical effect

Queue-based devices often use MMIO doorbells. Software prepares descriptors in memory, makes those descriptor updates visible as required by the DMA coherence model, then writes a doorbell that tells the device new work is available.

Those are two distinct ordering domains:

descriptor stores in memory
        │
        ├─ memory-ordering requirement
        ▼
MMIO doorbell write
        │
        ├─ posted PCIe transaction
        ▼
device observes doorbell

A barrier used before the doorbell can ensure that descriptor contents are visible before the device acts on the notification. It does not automatically prove that the doorbell write itself has arrived at the endpoint before software performs some unrelated action.

Most queue submissions do not need a synchronous flush after every doorbell. Adding a readback to every write can add serialization and extra PCIe traffic. A flush belongs only where the device protocol or a subsequent operation requires that stronger completion boundary.

Reset and shutdown paths make completion visible

Posted-write behavior becomes especially important when software is about to remove access to the path carrying the write. Examples include device reset, function disable, power-state transitions, or unmapping resources.

A sequence such as:

write device control
immediately disable transport path

can be incorrect if the first operation is allowed to remain buffered while the second action removes the route needed to deliver it.

The exact safe sequence is device- and platform-specific. Some interfaces require a status poll, some define a reset handshake, and some rely on a readback or another explicit synchronization mechanism. A generic MMIO write cannot substitute for those documented rules.

Posted writes improve throughput by avoiding per-write responses

Requiring a completion packet for every register write would add response traffic and force tighter synchronization between requester and completer. Posted writes avoid that cost and allow multiple writes to progress through the fabric without a return completion for each one.

That advantage also explains the software constraint: low write latency as observed by the CPU is not evidence of endpoint completion. The processor may only have handed the operation to a downstream component.

Performance measurements that time an MMIO store therefore need precise wording. They can measure the software-visible cost of issuing the store under a given platform configuration, but they do not automatically measure the time until the device consumes the write.

Device protocols define the required boundary

PCIe supplies transaction classes and ordering rules; the device interface defines which observations matter to software. A driver needs to combine both layers.

If the protocol only requires ordered delivery, normal MMIO accessors and the platform’s ordering rules may be sufficient. If software must establish that earlier posted writes have progressed before reset, teardown, or another sensitive transition, an explicit completion mechanism may be required.

The central distinction is compact:

CPU store completion
        ≠
PCIe posted-write completion at the device

Treating those events as separate prevents a fast MMIO store from being mistaken for an endpoint acknowledgement that PCIe Memory Write transactions do not provide.