Transactional Outbox at the Database-Broker Boundary

A database commit and a broker publish are two separate state transitions. An application can complete either one first, but unless both systems participate in a common transaction protocol, there is an interval in which one side has changed and the other has not.

That interval is the central problem behind the transactional outbox pattern. The pattern does not make a database and broker commit atomically. Instead, it moves the durable publication decision into the same database transaction as the application state change. A separate publisher later converts that recorded intent into a broker message.

The distinction matters. An outbox narrows one consistency gap by changing the boundary of the atomic operation, while introducing a second mechanism whose delivery semantics must still be understood.

Two writes create an ambiguous failure boundary

Suppose an order operation changes a row and emits an event. A direct implementation can write the order and then publish a message:

BEGIN
UPDATE orders ...
COMMIT

PUBLISH order_changed

If the process stops after the commit but before the publish completes, the database contains the new state and no corresponding message has been accepted by the broker. Reversing the order changes the failure mode rather than removing it. Publishing first can expose an event whose related database transaction later rolls back.

A retry cannot always resolve the ambiguity. After a network timeout, the publisher may not know whether the broker accepted a message before the connection failed. Repeating the publish may be correct when the first attempt failed before acceptance, or it may create a duplicate when the first attempt succeeded but its acknowledgement was lost.

These outcomes follow from separate commit points. Local database atomicity says nothing about a remote broker operation, and a successful broker acknowledgement says nothing about a later database commit.

Distributed transaction protocols can coordinate multiple transactional resources when the participating systems support the required protocol and operational model. The outbox addresses a different design point: one database remains the atomic resource, and message publication is derived from durable state stored there.

Publication intent becomes database state

With an outbox, the business mutation and an outbox insert occur in one local transaction:

BEGIN;

UPDATE orders
SET status = 'confirmed'
WHERE id = 481;

INSERT INTO outbox (
    event_id,
    event_type,
    aggregate_id,
    payload,
    created_at
) VALUES (
    'evt_7f2c',
    'order.confirmed',
    '481',
    '{"order_id":481,"status":"confirmed"}',
    CURRENT_TIMESTAMP
);

COMMIT;

The exact schema is application-specific. The important property is transactional placement: the domain change and publication intent either commit together or both roll back.

At commit time, no claim is required that the broker already contains the event. The database records a durable obligation to publish it. A publisher can inspect committed outbox records after the transaction ends and send them independently.

This changes the consistency statement the application can make. Instead of asserting that a database mutation and broker publish happen atomically, it can assert that a committed mutation has a committed publication record. That assertion is supported by the database transaction itself.

The outbox record also creates an inspectable boundary. Pending publication is represented as data rather than as transient control flow between a commit call and a network request.

The publisher is a state machine, not a cleanup loop

A publisher commonly selects unpublished records, sends them, and records completion. That description can hide the most important edge case: broker acceptance and database acknowledgement remain separate operations.

Consider this sequence:

1. read outbox record A
2. publish A to broker
3. broker accepts A
4. publisher stops before marking A as sent

After restart, record A still appears pending. Publishing it again can produce a second broker message. Marking the record as sent before publishing merely creates the opposite gap: a process stop after the mark but before broker acceptance can suppress an event that was never published.

For this reason, a conventional outbox publisher normally has at-least-once publication behavior unless another mechanism supplies stronger guarantees. The database transaction prevents a committed domain change from lacking publication intent, but it does not by itself provide exactly-once message effects.

The publisher therefore has states with observable meaning. A record can be pending, in an active publication attempt, accepted by the broker but not yet acknowledged locally, or marked complete. Some of those states cannot be distinguished after certain failures. A sound design treats that ambiguity as part of the protocol rather than as an exceptional coding detail.

Stable event identity carries information across retries

An outbox record benefits from an event identifier created before publication and retained across attempts. Generating a new identifier for every retry makes repeated publication look like distinct logical events to downstream systems.

A stable identifier does not eliminate duplicates. It gives consumers, broker features, or intermediate processors a value with which repeated representations of the same logical event can be recognized.

For a consumer that requires duplicate suppression, a common model stores processed event identifiers in durable state. If the consumer’s business update and duplicate marker share one local transaction, a repeated event can be detected without applying the same state transition again.

That mechanism has boundaries of its own. Retention policy determines how long an identifier remains useful for deduplication. A consumer that deletes markers after seven days cannot use those markers to identify a duplicate arriving after that interval. The identifier also needs a uniqueness scope that matches the producer’s contract; a value unique only inside one aggregate is not globally unique unless the aggregate identity is part of the key.

Idempotent business operations can reduce the need for explicit duplicate tracking in some cases. Setting a resource to a particular state may tolerate repetition under stated preconditions, while incrementing a counter generally does not have the same property. Duplicate handling therefore belongs to the semantics of the consumer operation, not to the outbox table alone.

Ordering exists only relative to an explicit key

Outbox designs often carry an ordering requirement, but a single phrase such as “preserve order” is incomplete. Ordering can refer to insertion order, commit order, publication order, broker partition order, or consumer processing order. Those are not automatically identical.

Two concurrent transactions can allocate identifiers in one order and commit in another. Multiple publisher workers can read adjacent rows and complete broker sends in a different order. A partitioned broker can preserve order inside one partition while providing no total order across partitions.

If events for one entity must be observed in sequence, the design needs an ordering key and a mechanism that maintains the required relation for that key. An aggregate identifier is often suitable when the invariant is per aggregate. A monotonically increasing aggregate version can make sequence expectations explicit:

order 481, version 17
order 481, version 18
order 902, version 6

The versions describe per-order progression; they do not imply that version 18 for order 481 must be globally ordered against version 6 for order 902.

A publisher with several workers must also account for concurrent claiming. Database row locking, claim columns, leases, or partitioned work ownership can prevent two workers from intentionally processing the same pending row at once. Such coordination reduces concurrent duplication inside the publisher, but it still cannot remove the acknowledgement gap after broker acceptance.

Polling and log-based capture observe different interfaces

One publication model queries the outbox table directly. Another observes database change records through change data capture and converts committed outbox inserts into broker messages.

Polling makes the application table an explicit work queue. Its behavior depends on query cadence, indexing, batch size, claim mechanics, and deletion or archival policy. A short interval can reduce idle delay at the cost of more database activity; a longer interval can increase the time between commit and publication. Those effects depend on workload and implementation, so there is no universal polling interval.

Log-based capture moves detection toward the database’s committed change stream. It can avoid repeated scans for pending rows, but it adds dependence on the database log interface and the capture system’s offset management. A capture process that restarts needs a durable position from which it can resume without silently skipping committed records.

Both models retain the same conceptual outbox boundary. The application transaction writes publication intent into the database. The difference lies in how another component observes that committed intent and advances it toward the broker.

Deletion policy also differs by implementation. Removing a row immediately after local acknowledgement keeps the active table small but discards a direct publication record. Retaining rows provides history but requires a bounded archival or partitioning strategy as data accumulates. Neither choice changes the atomic relationship between the domain mutation and the original outbox insert.

Payload design fixes a temporal contract

An outbox can store a complete event payload or enough information for a publisher to construct one later. These choices have different semantics.

A complete payload captures data as part of the originating transaction. Later changes to source tables cannot alter the already recorded event body. This is useful when the event is intended to represent facts as they existed at commit time.

Storing only an entity identifier and querying current state during publication creates a different contract. The published representation may include changes committed after the transaction that created the outbox record. That can be valid for a notification meaning “resource changed; fetch current state,” but it is not equivalent to an event that records the original transition.

Schema evolution applies to stored payloads as well. An outbox row can remain pending while application code is deployed. If the publisher assumes only the newest payload shape, older pending records may become unreadable. Versioned event schemas or backward-compatible decoders make that temporal boundary explicit.

The same concern applies downstream. Once an event leaves the database, its schema becomes an interface between independently executing components. Renaming a field in application code does not automatically migrate messages already stored in an outbox, retained by a broker, or waiting at a consumer.

Atomicity ends at the outbox commit

The transactional outbox is strongest when described narrowly. It gives one local transaction authority over two pieces of database state: the domain mutation and the durable intent to publish. That removes the direct dual-write gap between those two facts.

Everything after that commit belongs to a message transfer protocol with its own failure states. Publication can be retried. Broker acknowledgement can be lost. Consumers can receive duplicates. Ordering can depend on keys and partitions. Schema versions can coexist while old records remain in flight.

Those properties are not defects hidden behind the pattern. They are the actual boundary the pattern establishes. The outbox replaces an unrecorded interval between database commit and network publication with durable state that can be inspected and retried. Its value comes from making that boundary explicit, while leaving delivery semantics to mechanisms designed to handle them.