A service updates an order row and emits an event about that update. If the database commit succeeds but the broker publish fails, durable state says one thing while downstream consumers receive no corresponding message. Reversing the order only moves the gap: a successful publish followed by a failed database transaction exposes an event for state that never committed.
The difficulty is not message syntax or retry configuration. It is atomicity across two systems that do not share a transaction. A transactional outbox changes the boundary. The application writes its domain state and a message record into the same database transaction, then a separate relay publishes committed outbox records to the broker.
That arrangement does not make database and broker operations atomic. Instead, it removes the broker from the application’s commit path and gives the message intent the same atomic commit as the state it describes. The remaining failure modes become visible as relay progress, duplicate publication, ordering, and retention problems.
The dual-write gap has two directions
Consider a state transition from pending to confirmed and an event OrderConfirmed. A direct implementation has two durable effects:
database: pending -> confirmed
broker: publish OrderConfirmedIf these effects are performed independently, there is no application-level ordering that makes every failure point safe.
Database first creates a gap after commit. A process crash, network failure, broker rejection, or timeout can prevent the subsequent publish. Retrying the whole operation may be unsafe if the state transition is not itself repeatable.
Broker first creates the opposite gap. The event may become visible before the database commit. If that commit later aborts, consumers have observed a transition that the source database does not contain.
A distributed transaction protocol can coordinate multiple transactional participants when the involved systems support the required protocol and operational model. The outbox pattern addresses a different constraint: the application relies on its local database transaction and accepts asynchronous publication to the broker.
Message intent becomes transactional state
The central move is small. The transaction that changes application state also inserts an outbox row.
BEGIN;
UPDATE orders
SET status = 'confirmed'
WHERE id = 481 AND status = 'pending';
INSERT INTO outbox (
message_id,
topic,
aggregate_id,
payload,
created_at
) VALUES (
'8d72...',
'orders.confirmed',
'481',
'{"order_id":481}',
CURRENT_TIMESTAMP
);
COMMIT;This sketch assumes the state transition and outbox insert are valid only as a pair. In a real implementation, the application also has to ensure that an outbox record is not inserted when the intended state transition did not occur. For example, a conditional update that affects zero rows may indicate that the precondition was false; the transaction logic must treat that result according to the domain rule.
The useful property comes from the database transaction: either both changes commit or neither does, subject to the guarantees of the selected database and transaction isolation. A reader cannot infer broker publication from the commit. It can infer something narrower and more useful: the committed state includes durable intent to publish.
That distinction keeps the guarantee precise. The outbox establishes atomicity between domain state and the publication record, not between domain state and broker visibility.
The relay creates an at-least-once edge
After commit, a relay reads eligible outbox rows and publishes them. It can run inside the application, in a separate process, or through change-data-capture infrastructure. Those choices alter mechanics, but they share a fundamental acknowledgement problem.
Suppose the relay publishes a message successfully and crashes before marking the outbox row as sent. On restart, the row still appears pending, so the relay may publish it again.
publish succeeds
|
X crash
|
mark sentMarking the row before publication is not a solution. A crash between those operations would permanently suppress a message that was never published. With separate database and broker acknowledgements, a relay that avoids loss by retrying unresolved work must tolerate the possibility of duplicate publication.
The exact delivery semantics depend on the broker, relay design, producer features, and consumer protocol. The outbox itself does not grant exactly-once processing. It gives the system a durable source record from which publication can be retried.
A stable message identifier is therefore part of the data model, not decoration. Consumers that require effect deduplication can record processed identifiers in storage that participates in the same transaction as their own side effects. Other consumers may use naturally idempotent operations or broker-specific facilities. The suitable mechanism depends on what a duplicate message can change.
Ordering is scoped, not automatic
An outbox table gives records a durable existence, but durable existence does not define a universal delivery order.
A single relay that scans rows by a monotonically increasing database key can preserve that scan order under specific assumptions. Parallel relays, partitioned brokers, retries, transaction commit timing, and independent aggregates can all change the order observed by consumers.
The relevant requirement is often narrower than total ordering. Events for one aggregate may need to retain causal order while unrelated aggregates can progress independently. An aggregate identifier and sequence value can make that requirement explicit:
order 481: sequence 17
order 481: sequence 18
order 902: sequence 4A broker partitioning scheme keyed by aggregate can preserve per-key order if the broker documents that property and the producer uses it correctly. The outbox relay must also avoid introducing reorderings that violate the same scope. None of this follows merely from storing rows in a table.
Transaction commit order deserves separate attention. Auto-incremented identifiers or timestamps allocated inside transactions do not always represent commit order under every database and concurrency pattern. If consumers depend on a particular ordering relation, the chosen sequence must actually encode that relation rather than serving as a convenient proxy.
Payloads create a historical contract
An outbox row can store a complete event payload or enough information for the relay to construct one. The choice changes the meaning of the record.
Storing the final payload inside the transaction freezes the message representation alongside the state change. A relay operating later does not need to reconstruct past data from tables that may already have changed. The cost is that serialization and event-version decisions enter the transaction path, and old rows may contain older schema versions.
Storing only a reference keeps the row smaller, but later reconstruction can observe newer state unless the referenced data is immutable or versioned. An event intended to describe a specific transition can then drift away from the transaction that created its publication intent.
This is a temporal boundary. The database transaction occurs at one point; publication can occur much later. Data needed to preserve the event’s intended meaning must survive that interval in a form that does not silently change underneath the relay.
Cleanup is part of the protocol
Published rows cannot accumulate forever in most finite storage systems. Deleting them, however, introduces another state transition that interacts with relay progress and diagnostics.
A common design marks publication status and removes old rows only after a retention interval. Another design treats the table as an append-oriented log and advances relay checkpoints elsewhere. Change-data-capture systems may derive relay position from the database log rather than a sent flag.
Each model needs a clear rule for when a record is no longer required for publication recovery. Deletion based only on creation time can discard an unpublished record if the relay has been stalled longer than the retention window. Deletion based on a confirmed relay state avoids that particular error but requires the confirmation state itself to be trustworthy.
Indexes also matter mechanically. A relay repeatedly searching for pending rows needs an access path that does not degrade into scanning an ever-growing history. The appropriate index depends on the query and database, while aggressive indexing increases write work on the transaction that inserts each outbox record.
The outbox is therefore operational state, not a temporary implementation detail. Its backlog represents committed publication obligations that have not yet crossed the broker boundary.
Atomicity becomes local and recovery becomes explicit
The transactional outbox does not erase distributed failure. It changes its shape.
Without an outbox, the application attempts two independent durable writes during one logical operation and has no local record that can reliably bridge a crash between them. With an outbox, the synchronous atomic boundary is the database transaction. Broker publication becomes recoverable asynchronous work derived from committed records.
That shift is valuable precisely because its limits are inspectable. A committed outbox row can be queried. Relay progress can be measured. Duplicate delivery can be identified by message identity. Ordering requirements can be stated per aggregate or partition. Retention can be tied to confirmed progress.
The pattern is strongest when described in those terms. It is not an exactly-once switch and not a substitute for every form of distributed transaction. It is a way to place state change and publication intent under one transactional authority, then handle the remaining cross-system boundary as an explicit delivery protocol.