Transactional Outbox: Moving Publication Across the Commit Boundary

A service that changes database state and publishes a message has two distinct side effects. A local transaction can make the database change atomic, and a broker can accept the message durably, but those facts do not make the pair atomic.

The awkward interval sits between them. If the database commit succeeds and publication does not, durable state exists without the corresponding message. Reversing the order only reverses the exposure: a message can become visible before the database commit succeeds.

The transactional outbox changes the boundary. Instead of treating broker publication as part of the request’s atomic work, the request stores publication intent in the same database transaction as the business change. A separate relay later transfers that intent to the broker. This removes one dual-write gap, but it does not create exactly-once delivery. The resulting system has a more precise contract: atomic recording, asynchronous transfer, and duplicate-tolerant consumption.

The dual write has no shared commit point

Consider an order service that persists an accepted order and emits OrderAccepted. The direct shape is compact:

begin transaction
insert order
commit transaction

publish OrderAccepted

The database commit is a durable boundary. Once it returns successfully under the database’s documented transaction semantics, the order is committed. The broker call occurs after that boundary and can fail independently.

Putting publication first does not repair the model:

publish OrderAccepted

begin transaction
insert order
commit transaction

Now a consumer can observe OrderAccepted even if the later database transaction rolls back. The two resources still have separate commit points.

A distributed transaction protocol can coordinate resources that support the required protocol, but that is a different architecture with different resource and operational constraints. The outbox pattern takes another route: it narrows the atomic operation to one transactional resource.

Publication intent becomes database state

With an outbox, the business row and an outbox row are written in one transaction:

BEGIN;

INSERT INTO orders (id, status)
VALUES ('o-481', 'accepted');

INSERT INTO outbox (
    id,
    aggregate_id,
    event_type,
    payload
) VALUES (
    'evt-902',
    'o-481',
    'OrderAccepted',
    '{"order_id":"o-481"}'
);

COMMIT;

If the transaction commits, both rows become durable together. If it rolls back, neither row becomes visible as committed state. The guarantee comes from ordinary database transaction semantics; the outbox does not add a second atomicity mechanism.

This is the central shift. The request no longer needs to make the broker acknowledge a message before it can establish durable publication intent. The broker sits outside the database transaction, and the outbox row is the durable handoff point.

That distinction matters for error handling. A broker outage after the commit does not erase publication intent. The relay can inspect committed outbox rows later and attempt transfer again.

The relay introduces an at-least-once edge

A relay reads unpublished outbox records, publishes them, and records completion. A simplified cycle looks like this:

row = next pending outbox record
publish row
mark row as published

There is still no atomic commit spanning the broker acknowledgement and the database update. Suppose publication succeeds, then the relay stops before marking the row as published. On restart, the same row still appears pending and can be published again.

The outbox therefore converts one failure mode into another. It prevents a committed business change from losing its durable publication intent, but a conventional relay can produce duplicate deliveries. That is consistent with at-least-once transfer, not exactly-once processing.

Deleting the row after publication has the same boundary. A stop between broker acknowledgement and deletion leaves a row that can be sent again. Replacing a status update with deletion changes retention behavior, not the atomicity problem.

Stable event identity carries the contract forward

An outbox record needs a stable identifier that survives relay retries. In the example above, evt-902 identifies the logical event. Every publication attempt for that row should carry the same event identity rather than generating a fresh identifier for each attempt.

A consumer can then make duplicate handling explicit. One common design records processed event identifiers in the same transaction as the consumer’s own state change:

BEGIN;

INSERT INTO processed_events (consumer, event_id)
VALUES ('billing', 'evt-902');

UPDATE billing_state
SET status = 'pending'
WHERE order_id = 'o-481';

COMMIT;

A uniqueness constraint on (consumer, event_id) can reject a repeated event before the state transition is applied again. The exact SQL shape depends on the database and application, but the important property is transactional coupling between duplicate detection and the consumer’s state mutation.

This does not make every external effect idempotent. If a consumer also calls another remote service, that call crosses another atomicity boundary. The same analysis must be applied at that boundary rather than hidden behind the word idempotent.

Ordering is a separate property

Atomic recording says that an event intent exists with its business change. It does not, by itself, define the order in which multiple outbox rows reach consumers.

A relay with several workers can publish records concurrently. Broker partitioning can preserve order only within the scope promised by that broker and partitioning scheme. Retries can also cause an older event to arrive after a newer one if the transfer path permits reordering.

Applications that require per-entity order need an explicit ordering model. A monotonically increasing aggregate version is one option:

OrderAccepted   order=o-481 version=7
OrderCancelled  order=o-481 version=8

A consumer can compare versions and reject a stale transition when its domain rules permit that behavior. Another design serializes publication per aggregate key. Neither property follows automatically from having an outbox table.

Global ordering is an even stronger requirement. A database commit sequence, relay query order, and broker delivery order are distinct concepts unless the implementation deliberately connects them.

Polling and log-based relays expose different mechanics

A polling relay queries the outbox table for pending rows. It is easy to express with ordinary database operations, but concurrency control needs attention when several relay instances run at once. Row locking, claim tokens, or database-specific skip-locked behavior can prevent workers from intentionally selecting the same pending row at the same time. Those controls reduce redundant work; they do not remove the publish-then-mark duplicate window.

A log-based relay observes committed database changes through a transaction log or change-data-capture facility. This can avoid repeated polling of the application table and can preserve useful commit metadata, depending on the database and capture system.

The atomicity argument remains the same in both forms. The business state and outbox record share a database commit. Transfer from committed database history to the broker is another stage with its own delivery semantics.

The choice between polling and log-based capture is therefore not a choice between correct and incorrect outbox implementations. It changes the relay mechanism, infrastructure dependencies, latency characteristics, and failure surfaces. The required guarantees must be stated against the actual database, capture system, and broker in use.

Cleanup is part of the data model

An outbox grows whenever the application commits publishable events. A relay status flag alone does not define how long published records remain in the primary table.

Retention can use deletion, archival, or partition expiration. Each option interacts with diagnostics and replay requirements. Removing a published row may be acceptable when the broker or another event store is the durable history. Keeping rows for a fixed interval can support investigation while bounding table growth.

Cleanup must also distinguish a record that is safe to remove from one that is merely old. Age is not evidence of successful transfer. A retention process that deletes pending records after a time threshold can recreate the lost-publication condition that the outbox was meant to avoid.

For high write rates, physical table behavior also matters. Indexes used to find pending records, update patterns, dead tuples or equivalent storage effects, and partition strategy are database-specific concerns. The outbox is application state with a lifecycle, not a temporary in-memory queue that happens to live in SQL.

The useful guarantee is narrower than exactly once

The transactional outbox is strongest when its guarantee is stated without extra promises.

A successful local transaction can atomically persist business state and publication intent. A relay can repeatedly attempt to transfer committed intent. Duplicate publication remains possible around the relay’s broker-acknowledgement boundary, so consumers need semantics that tolerate repeated logical events. Ordering requires a separate design. Remote effects performed by consumers introduce additional boundaries.

That narrower description is also more useful. It identifies the exact state transition protected by the database transaction and leaves the remaining edges visible. Distributed workflows become easier to reason about when each durable boundary has an explicit contract instead of being compressed into a claim of single delivery.