IOMMU IOTLB Invalidation Controls When DMA Remapping Takes Effect
Changing an IOMMU page-table entry does not necessarily change the translation used by the next DMA request. An IOMMU can cache address translations in an I/O translation lookaside buffer, commonly called an IOTLB. Software must invalidate affected cached state when a mapping is removed or replaced, then observe the invalidation semantics required by that IOMMU before treating the old translation as retired.
This creates a boundary that resembles CPU page-table maintenance but belongs to the DMA address space. The page table is authoritative memory state; the IOTLB is hardware state derived from it. Updating one does not, by itself, prove that the other has stopped serving an earlier translation.
IOVA translation adds state between a device and memory
A device normally submits DMA transactions using addresses from its DMA address space. With an IOMMU in the path, an I/O virtual address, or IOVA, can be translated to a physical memory location according to an IOMMU domain and its page tables.
device DMA request
|
v
IOVA
|
v
IOMMU translation
/ \
IOTLB hit page-table walk
\ /
v
physical memoryAn IOTLB hit avoids a page-table walk. The exact cache hierarchy, entry format, lookup rules, and invalidation commands are implementation-specific. The general consequence is portable: a valid translation can survive in hardware after software has changed the page table that originally supplied it.
Linux drivers using the normal DMA API do not usually issue architecture-specific IOTLB invalidation commands themselves. The DMA and IOMMU layers implement those details behind mapping and unmapping operations. Lower-level IOMMU code, virtualization code, and interfaces that expose user-managed translation tables have to handle the boundary explicitly.
Removing a PTE and retiring its translation are distinct events
Suppose IOVA 0x4000 maps to physical page A. Hardware has already cached that translation. Software then removes the page-table entry and plans to reuse the same IOVA for physical page B.
initial state:
IOVA 0x4000 -> page A
IOTLB: 0x4000 -> page A
software changes page table:
IOVA 0x4000 -> page B
IOTLB: 0x4000 -> page A <- stale until invalidatedIf the device can issue another request while the stale entry remains usable, that request can still reach page A. Merely storing the new page-table entry is therefore insufficient for safe reuse.
A correct remapping sequence depends on the IOMMU interface, but its logical stages are:
stop or exclude DMA using the old mapping
|
change or remove page-table entries
|
invalidate affected translation caches
|
wait for required invalidation completion
|
permit IOVA reuse or release backing memoryThe completion step matters because some invalidation mechanisms can be queued or asynchronous. Architectural documentation defines the command ordering and completion mechanism for a particular IOMMU. Software must follow that contract rather than treating submission of an invalidation command as universal proof of completion.
Stale translations turn address reuse into a correctness hazard
IOVA reuse makes invalidation failures especially damaging. Without reuse, a stale entry can still be incorrect, but the old address may remain associated with the same allocation until all DMA has stopped. Reassigning that address to a different page changes the consequence: an old cached translation can direct traffic to memory that software now considers unrelated.
The same lifetime problem appears when backing memory is freed. If a device or translation cache can still direct DMA to a physical page after software returns that page to the allocator, a later owner of the page can receive unintended device writes.
This is a lifetime rule, not just a translation rule. Safe teardown requires both sides of the boundary to be resolved: the device must no longer be issuing relevant DMA, and translation state that could route later requests through the old mapping must be retired according to the platform contract.
Invalidation scope affects work, not the correctness requirement
IOMMUs can provide different invalidation scopes. Depending on the architecture and implementation, software may be able to invalidate one address, a range, a domain, an address-space identifier, or a broader set of cached state.
A narrow invalidation can preserve unrelated cached translations. A broad invalidation can be simpler but may evict useful state and cause later page-table walks. Those are implementation and performance considerations; neither changes the correctness requirement that stale entries covering a changed mapping cannot remain usable when the mapping’s lifetime has ended.
Linux IOMMU infrastructure can also batch invalidation work. Batching can amortize command and synchronization overhead by changing several mappings before issuing or completing a flush. That optimization shifts the point at which an unmapped IOVA is safe to recycle. An implementation that defers invalidation must also defer any reuse or memory release that depends on invalidation completion.
Device-side translation can add another cache boundary
PCIe Address Translation Services, or ATS, allow a capable device to cache translated addresses in a device-side Address Translation Cache. In such a configuration, removing an IOMMU page-table entry can involve more than the IOMMU’s own IOTLB.
The platform and IOMMU architecture define the required coordination for invalidating device translation state. Software cannot assume that flushing an IOMMU-local cache alone is sufficient when a device is permitted to retain translations.
This produces a layered path:
device request
|
device translation cache, when enabled
|
IOMMU translation cache
|
IOMMU page tables
|
physical memoryNot every system uses ATS, and not every device supports it. The extra boundary is conditional, but when enabled it becomes part of translation lifetime management rather than an optional performance detail.
Translation invalidation is separate from CPU cache coherency
IOTLB maintenance and CPU data-cache maintenance solve different problems. IOTLB invalidation controls the address translation used for DMA. CPU cache synchronization controls visibility of buffer contents on systems where CPU and device accesses are not automatically coherent.
A mapping can have a correct, current IOVA translation while the data in the target page still requires cache maintenance. Conversely, the buffer contents can be coherent while an obsolete IOTLB entry still points at the wrong physical page.
Memory ordering is separate again. Barriers can constrain the order in which software publishes descriptors, data, or control state, but a CPU memory barrier is not a generic replacement for an IOMMU invalidation operation. Each mechanism closes a different hardware boundary.
The DMA API keeps most drivers above this boundary
For ordinary Linux device drivers, dma_map_*() and dma_unmap_*() are the portable interface. On an IOMMU-backed system, those operations can allocate IOVA space, install translations, remove translations, and trigger the architecture-specific invalidation work required by the DMA implementation.
That abstraction is significant because the returned dma_addr_t is a device-visible address, not a promise about physical addressing. A driver that bypasses the DMA API’s lifetime rules can accidentally bypass IOMMU bookkeeping as well.
Interfaces for user-managed I/O page tables expose more of the mechanism. Linux IOMMUFD, for example, includes invalidation operations for cases where userspace manages translation structures. Such interfaces define explicit cache-invalidation contracts because the kernel cannot infer every page-table modification made outside the ordinary DMA mapping path.
The practical boundary remains the same at each layer: changing translation tables, retiring cached translations, and proving that an address can be reused are related operations, but they are not the same event.