Transactional Outbox Closes the Dual-Write Gap

A service often needs one request to change database state and publish an event. The two operations may look adjacent in application code, but they cross different durability boundaries. A database commit can succeed while a broker publish fails, or the publish can succeed before the database transaction rolls back.

That split creates a dual-write problem. No ordering of two independent writes can make them atomic by itself.

The transactional outbox pattern changes the boundary. The request writes its business data and an outbox record in the same local database transaction. A separate relay later publishes committed outbox records to the broker. Publication becomes asynchronous, but the durable intent to publish is committed with the state that caused it.

Two independent writes leave a failure window

Consider an order service that stores an order and emits OrderPlaced.

A direct implementation might perform:

BEGIN
INSERT order
COMMIT

publish OrderPlaced

If the process stops after COMMIT and before publish, the order exists without its event.

Reversing the order moves the gap rather than removing it:

publish OrderPlaced

BEGIN
INSERT order
COMMIT

Now a crash, constraint failure, or transaction abort can leave consumers with an event for an order that never committed.

Retries do not make the pair atomic. Retrying the database operation can duplicate state unless it is idempotent. Retrying the broker operation can duplicate delivery. More importantly, a retry cannot infer with certainty whether an interrupted remote publish took effect unless the messaging protocol exposes a suitable deduplication or transactional contract.

The core issue is ownership: the database controls one commit decision and the broker controls another.

The outbox joins state and publication intent

With an outbox, the request transaction writes two kinds of rows:

BEGIN;

INSERT INTO orders (id, customer_id, total, status)
VALUES (:id, :customer_id, :total, 'placed');

INSERT INTO outbox (
    event_id,
    aggregate_id,
    event_type,
    payload,
    created_at
)
VALUES (
    :event_id,
    :id,
    'OrderPlaced',
    :payload,
    CURRENT_TIMESTAMP
);

COMMIT;

Both inserts share one database transaction. If the transaction commits, both the order and publication intent are durable. If it aborts, neither becomes visible as committed state.

The broker is deliberately absent from this transaction. The request does not attempt to stretch a local database transaction across an unrelated messaging system.

The outbox row is not merely a log message. It is durable work that remains queryable after the request process exits.

A relay owns the second boundary

A relay scans or receives committed outbox work and publishes it:

request
   |
   v
database transaction
   |-- business row
   `-- outbox row
          |
        commit
          |
          v
        relay
          |
          v
        broker

A simple polling relay can select unpublished rows in batches:

SELECT event_id, event_type, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100;

After a successful broker publish, it marks the row as published or removes it according to the retention design.

This separates request latency from broker availability. A temporary broker outage can leave committed outbox rows pending instead of forcing the original business transaction into an ambiguous cross-system result.

The separation also creates an operational queue inside the database. Its depth, age, throughput, and failure rate need explicit monitoring.

The relay has an unavoidable duplicate window

The outbox closes the missing-event gap between the business write and publication intent. It does not automatically provide exactly-once delivery.

Suppose the relay executes:

1. publish event E
2. broker accepts E
3. relay crashes
4. published marker was not stored

After restart, the relay sees E as pending and publishes it again.

If the relay marks the row first and publishes second, the opposite failure appears: a crash between those steps can permanently suppress the event.

For a relay using an ordinary broker API and a separate database update, at-least-once publication is the practical default. Consumers should therefore tolerate duplicate event delivery, usually through an event identifier and durable deduplication or naturally idempotent state transitions.

event_id = 7f2c...

first delivery  -> apply effect, record event_id
second delivery -> event_id already recorded, skip effect

The exact mechanism depends on the consumer’s storage and side effects. Deduplication held only in process memory disappears on restart and does not protect multiple consumer instances.

Claiming rows requires concurrency control

More than one relay worker may be needed for throughput or availability. They must avoid treating the same pending row as exclusive work without coordination.

On databases that support the required locking semantics, workers can claim batches with row locks and skip rows already locked by peers. Another design assigns a lease or claim token with an expiry. Change-data-capture systems can instead stream committed outbox inserts from the database log.

Each choice changes failure handling.

A lock held only inside a short database transaction cannot remain open across an arbitrarily slow broker call without cost. A lease needs safe expiry and recovery rules. A log-based relay needs durable offsets and a defined relation between log position and broker acknowledgement.

The invariant remains stable: only committed outbox records are eligible for publication, and unfinished publication work must remain recoverable.

Ordering needs an explicit scope

Applications often require events for one aggregate to preserve commit order while allowing unrelated aggregates to progress independently.

A global ORDER BY created_at is not a complete ordering protocol. Timestamps can collide, worker concurrency can reorder publishes, and broker partitioning can introduce another ordering boundary.

If consumers require per-order ordering, the design can carry a monotonic sequence for each order:

aggregate_id = order-42
sequence = 17
event = OrderAddressChanged

aggregate_id = order-42
sequence = 18
event = OrderConfirmed

The relay and broker partitioning strategy then need to preserve that scope, often by routing the same aggregate key to the same ordered partition.

A global total order is much more restrictive and can reduce concurrency sharply. Ordering guarantees should match the business invariant rather than default to the widest possible scope.

Payload design affects coupling

An outbox can store a complete event payload, a reference to business state, or enough fields for the relay to construct a message.

Storing the final payload at transaction time gives the event a stable representation of the committed decision. A relay that later rereads mutable business rows can accidentally publish newer state rather than the state associated with the original event.

For example, an order can move from placed to cancelled before a delayed relay processes its first outbox row. If the relay reconstructs OrderPlaced from the current order row, the message can mix two moments in the lifecycle.

A stored payload avoids that temporal coupling, but it creates schema-management duties. Producers need a versioning policy, consumers need compatibility rules, and sensitive fields should not be copied into the outbox without a retention and access rationale.

Cleanup must not race publication

An outbox grows continuously unless rows are removed or archived. Cleanup therefore belongs to the protocol, not just database housekeeping.

Rows should become eligible for cleanup only after the system has durable evidence that the required publication step completed. A retention delay can provide room for diagnostics and replay, but replay semantics need care: republishing an old row is still another delivery and can trigger effects unless consumers handle it safely.

A useful lifecycle is:

pending -> claimed -> published -> retained -> deleted

Not every implementation needs every state. The important property is that cleanup cannot erase the only durable copy of work that still needs publication.

Indexes also matter. A polling query over published_at IS NULL should not degrade into repeated full-table scans as historical rows accumulate. Partitioning or archival may become appropriate at higher volume.

Backpressure moves into the outbox

Asynchronous publication protects the request path from short broker interruptions, but it does not create infinite capacity.

If producers commit 5,000 outbox rows per second and the relay can publish only 3,000, backlog grows by 2,000 rows per second. Database storage, index maintenance, replication traffic, and eventual recovery time all increase.

Useful signals include:

oldest pending row age
pending row count
publish attempts and failures
relay throughput
claim or lock contention
broker acknowledgement latency

The oldest pending age is often more informative than count alone because it measures how stale downstream state may be.

Capacity policy should define what happens during a prolonged outage. Options can include throttling producers, rejecting selected operations, increasing relay capacity, or accepting a bounded period of downstream lag. The correct choice is a product and consistency decision, not merely a worker tuning parameter.

The pattern narrows the atomic boundary on purpose

A transactional outbox does not make a database and broker participate in one atomic commit. It avoids requiring that distributed atomic commit for a common event-publication workflow.

The local transaction answers one question: did the business state and its publication intent commit together? The relay answers another: has that committed intent reached the messaging system?

That split produces a tractable failure model. Missing publication intent is prevented by the local transaction. Broker outages become recoverable backlog. Duplicate publication remains possible and must be addressed explicitly. Ordering, cleanup, schema evolution, and backpressure remain engineering concerns rather than hidden assumptions.

The value of the pattern is the boundary itself: business state and the obligation to publish cross one durable commit point, while remote delivery proceeds as recoverable asynchronous work.