Transactional Outbox Closes the Database-Broker Commit Gap

A service often needs one request to change database state and publish a message. Those actions may look adjacent in application code, but they cross two independent commit boundaries. If the database and broker do not share a transaction protocol, no ordering of two ordinary writes can make them atomic.

Consider an order service that stores an accepted order and emits OrderCreated. Publishing after the database commit leaves a crash window before the broker call. Publishing first creates the opposite window: consumers can receive an event for state that later fails to commit.

The transactional outbox moves publication intent into the same database transaction as the business mutation. A separate relay handles broker delivery.

A dual write has two failure windows

The direct sequence is compact:

BEGIN
INSERT order
COMMIT

publish OrderCreated

A process failure after COMMIT but before publish leaves durable business state with no corresponding message. Retrying the original request may not repair that gap if request deduplication reports the order as already created.

Reversing the calls does not fix the protocol:

publish OrderCreated

BEGIN
INSERT order
COMMIT

Now the broker can accept the message before the database transaction aborts. Consumers may act on an order that never becomes durable.

The defect is not a missing retry. It is the absence of one atomic boundary covering the two intended outcomes.

The outbox row joins the business transaction

With an outbox, the request transaction writes both the domain state and a durable publication record:

BEGIN;

INSERT INTO orders(id, customer_id, status)
VALUES (:id, :customer_id, 'accepted');

INSERT INTO outbox(
    event_id,
    aggregate_id,
    event_type,
    payload,
    created_at
)
VALUES (
    :event_id,
    :id,
    'OrderCreated',
    :payload,
    CURRENT_TIMESTAMP
);

COMMIT;

If the transaction commits, both rows exist. If it rolls back, neither exists. The broker is deliberately absent from this transaction.

That boundary changes the recovery problem. A crash after commit cannot erase publication intent because the outbox row is already durable. Delivery can resume from stored state after the process restarts.

The relay converts durable intent into broker delivery

A relay repeatedly selects unpublished rows, sends them to the broker, and records delivery progress. It can run as a polling worker or use database change capture, provided the publication record remains the durable source for pending work.

A simple polling shape is:

select pending rows
        |
        v
publish to broker
        |
        v
mark rows delivered

The relay should process bounded batches and avoid holding a database transaction open across slow broker calls unless the design explicitly accepts that coupling. Multiple relay instances also need a claim protocol, such as row locking with skip-locked semantics, leases, or another atomic ownership mechanism.

The exact mechanism varies by database. The invariant is that pending rows remain recoverable and concurrent relays do not treat ownership as an unchecked read-then-write race.

Delivery is normally at least once

The outbox removes the lost-message gap between local commit and publication intent. It does not make broker delivery exactly once.

Suppose the broker accepts an event and the relay crashes before marking the outbox row delivered:

publish event -> broker accepts
crash
mark delivered -> never runs

After restart, the row still appears pending, so the relay sends it again. Avoiding that retry would reintroduce a loss window when the broker outcome is ambiguous.

Consumers therefore need duplicate-safe handling. A stable event_id can support a consumer-side inbox table, unique constraint, conditional write, or naturally idempotent mutation. The deduplication boundary must cover the consumer side effect that needs protection.

Publication state needs a deliberate schema

An outbox table usually carries more than a payload. Operationally useful fields can include a stable event identifier, event type, aggregate identifier, creation time, delivery state, attempt count, and last error.

The payload should represent the contract consumers receive, not an arbitrary serialization of an internal ORM object. Schema evolution rules still apply because queued rows can survive a deployment and be published by newer code.

Large payloads also affect database storage, replication, scans, and cleanup. When events only need identifiers plus selected facts, keeping the envelope compact reduces pressure on the transactional database.

Ordering depends on the required scope

A single outbox table does not automatically create a global event order that every consumer observes. Multiple relays, broker partitions, retries, and consumer concurrency can all affect arrival order.

Many systems only need order per aggregate. An aggregate_id plus a monotonic sequence can make that requirement explicit:

order-42 seq=18
order-42 seq=19
order-77 seq=6

The relay and broker partitioning strategy must preserve the scope the application actually requires. Demanding one total order across unrelated aggregates usually adds coordination without improving domain correctness.

Cleanup must not race with recovery

Delivered rows accumulate unless the system removes or archives them. Cleanup should only target rows whose retention policy makes them safe to discard.

Deleting a row merely because a publish attempt started is unsafe. The relay may have failed before the broker accepted the message. Conversely, retaining every delivered row forever turns the outbox into an unbounded operational table.

A practical design separates delivery state from retention policy and tracks metrics for oldest pending age, pending count, publish latency, retry count, and terminal failures. These signals expose a stuck relay before the outbox backlog becomes a database problem.

The boundary is local, not magical

A transactional outbox gives a precise guarantee: when the local business transaction commits, the intent to publish commits with it. The relay can then retry delivery from durable state.

The pattern does not make the database and broker one atomic system, remove duplicate delivery, define consumer idempotency, or supply ordering beyond the scope the design enforces. Those concerns remain explicit parts of the messaging protocol.

That narrow guarantee is the useful one. It replaces an unrecoverable dual-write gap with durable work that can be retried, observed, and reconciled.