Transactional Outbox Closes the Database-to-Broker Commit Gap

A service often needs one operation to change database state and emit a message. An order may become confirmed while an OrderConfirmed event is sent to a broker. Those actions touch separate systems, so two ordinary writes cannot form one atomic commit unless both systems participate in a distributed transaction.

The dangerous part is the interval between the writes. Commit the database first and the process can fail before publishing. Publish first and the database commit can fail afterward. Reversing the order moves the failure window; it does not remove it.

A transactional outbox changes the boundary. The business mutation and a durable outbox row are committed in the same local database transaction. A separate relay later transfers committed outbox records to the broker.

One local transaction records both facts

The write path keeps the atomic decision inside one database:

BEGIN
  update business state
  insert outbox event
COMMIT

If the transaction rolls back, neither record becomes visible. If it commits, both records become durable together. The request path no longer needs a successful broker call to preserve the intent to publish.

An outbox row commonly carries an event identifier, event type, aggregate identifier, payload, creation time, and publication state. The exact schema depends on the application, but the event must contain enough durable information for the relay to publish without reconstructing transient request state.

outbox
+----------+----------------+--------------+-----------+
| event_id | event_type     | aggregate_id | payload   |
+----------+----------------+--------------+-----------+
| e-1842   | OrderConfirmed | order-731    | {...}     |
+----------+----------------+--------------+-----------+

The outbox is not a second copy of the broker. It is a durable handoff record at the database boundary.

The relay owns delivery to the broker

A relay scans or receives changes for committed outbox rows and publishes them. Polling is simple: claim a bounded batch, publish it, record progress, then continue. Change-data-capture systems can instead stream inserts from the database log.

Both forms separate transaction commit from broker availability. A broker outage can delay delivery without forcing the business transaction to become inconsistent.

request
  |
  v
[database transaction]
  |-- business row
  `-- outbox row
          |
          v
        relay ----> broker ----> consumers

The relay needs backoff and bounded concurrency. When the broker is unavailable, an unbounded publish loop merely converts a broker incident into database pressure and log noise.

At-least-once delivery creates duplicates

The outbox closes one atomicity gap, but it does not create exactly-once delivery by itself. Consider a relay that publishes event e-1842, receives broker acknowledgement, then crashes before marking the outbox row as published. After restart, the same row can be published again.

That ambiguity is fundamental when acknowledgement and database progress are separate commits. A relay can prefer duplicate delivery over silent loss, which normally produces at-least-once behavior.

Consumers therefore need an idempotency strategy. One common design stores processed event identifiers alongside consumer state. A duplicate event can then be recognized before applying the same business effect twice.

receive e-1842
  |
  +-- already processed -> acknowledge, no business mutation
  |
  `-- new -> apply mutation + record e-1842 atomically

Broker deduplication features can reduce duplicates, but application correctness should match the guarantees of the actual broker and retention window rather than assume duplicates are impossible.

Ordering requires a defined scope

A global ordering promise is expensive and often unnecessary. Most applications care about order for one aggregate, account, tenant, or partition key.

Suppose an order emits OrderCreated, OrderConfirmed, and OrderShipped. The relay and broker should preserve the required order for that order if consumers depend on the sequence. A partition key based on order_id can keep related events on the same ordered broker partition when the broker provides ordering within a partition.

Database sequence numbers can also make intended order explicit:

order-731 / version 41 / OrderCreated
order-731 / version 42 / OrderConfirmed
order-731 / version 43 / OrderShipped

Sequence metadata helps consumers detect gaps or stale events. It does not repair ordering automatically; the consumer still needs a policy for delayed or missing versions.

Publication state needs safe concurrency

Multiple relay workers improve throughput, but they must not all claim the same rows as independent work. Databases provide several useful coordination patterns, including row locks with skip-locked semantics, leases, or explicit claim columns.

A claim should be bounded in time or recoverable after worker failure. A permanent processing = true flag can strand events when a worker exits after claiming a batch.

The claim mechanism also must not hold a database transaction open during a slow network publish unless that tradeoff is deliberate. Long transactions retain locks and database resources while the broker controls latency. Many implementations claim work briefly, commit the claim, publish outside the transaction, and rely on retry plus idempotency for ambiguous outcomes.

Cleanup is part of the design

A successful relay eventually leaves historical rows that are no longer required for delivery. Without retention, the outbox grows indefinitely and makes polling, indexes, backups, and maintenance more expensive.

Deletion policy should account for operational diagnosis and any replay requirements. Some systems delete published rows after a short retention period. Others archive them outside the hot table. Partitioned tables can make time-based removal cheaper than deleting large numbers of individual rows.

Cleanup must never race ahead of delivery state. A row should not disappear merely because it is old if the relay has not established the publication condition required by the system.

Metrics expose a stuck handoff

A healthy API does not prove that outbox delivery is healthy. The request can commit successfully while the relay is stalled.

Useful telemetry includes:

  • oldest unpublished event age;
  • unpublished row count;
  • publish attempt, success, and failure rates;
  • relay batch duration and batch size;
  • duplicate observations where measurable;
  • broker acknowledgement latency;
  • cleanup backlog.

The oldest unpublished age is especially useful because it expresses user-visible staleness more directly than queue length. Ten old events can signal a more serious handoff failure than thousands of events created seconds ago during normal traffic.

Alerts should distinguish broker failure, relay failure, and database contention. All three can increase backlog, but their corrective actions differ.

The pattern has a specific boundary

A transactional outbox is useful when one service owns a database transaction and also needs to publish a durable message derived from that transaction. It does not make arbitrary operations across several services atomic, and it does not remove the need for idempotent consumers, delivery retries, retention, or ordering policy.

Its value comes from narrowing the atomic commitment to a system that already provides transactions. Business state and publication intent become one durable decision. Message delivery then becomes asynchronous work with explicit retry semantics instead of a fragile second write attached to the request path.

Software Engineering Distributed Systems Messaging Reliability