Transactional Outbox for Reliable Message Publishing

A common service operation has to do two things: change its own data and tell another part of the system what happened. For example, an order service may mark an order as paid and publish an OrderPaid message.

The awkward part is that the database and message broker usually have separate commit mechanisms. If the service updates the database and then publishes, it can crash between those steps. If it publishes first, the database update can fail afterward. Either order can leave the two systems disagreeing about what happened.

The transactional outbox pattern changes the problem. Instead of trying to update the database and broker atomically, the service stores the business change and the intent to publish in one local database transaction. A separate publisher later sends the stored message to the broker.

This article explains why that works, what guarantee it actually provides, and the duplicate-delivery problem you still need to handle.

The real problem is a dual write

Suppose payment handling looks like this:

function markPaid(orderId):
    database.updateOrder(orderId, status = "paid")
    broker.publish("OrderPaid", orderId)

There are two durable writes here:

1. the order state changes in the database;
2. the message becomes durable in the broker.

Those writes do not become one atomic operation merely because they appear next to each other in the function.

Consider the failure window after the first line succeeds:

database.updateOrder(...)   // succeeds
process crashes
broker.publish(...)         // never runs

The database now says the order is paid, but consumers never receive the message. A fulfillment service waiting for OrderPaid may never start its work.

Reversing the order moves the inconsistency rather than removing it:

broker.publish(...)         // succeeds
database.updateOrder(...)   // fails

Now consumers can react to a payment that the source service did not commit.

The useful mental model is simple: two independent durable systems create a failure gap between their commits.

Put the business change and publication intent in one transaction

The transactional outbox pattern uses the database transaction you already trust for the business change.

Instead of publishing directly, write an outbox record in the same transaction:

begin transaction

update orders
set status = "paid"
where id = "O42"

insert into outbox (
    id,
    topic,
    payload
) values (
    "M9001",
    "OrderPaid",
    { "orderId": "O42" }
)

commit

The exact schema and syntax vary by database. The important property is that both writes participate in the same local transaction.

After commit, one of two states exists:

• neither the order update nor the outbox row is committed;
• both are committed.

That removes the dangerous state where the business change is durable but the system has forgotten that a message must be published.

Notice what the transaction does not guarantee. It does not put the message in the broker. It records durable publication intent. Delivery happens later.

A separate publisher drains the outbox

A background publisher repeatedly finds unpublished outbox records, sends them, and records progress.

A simplified loop looks like this:

for message in outbox.unpublished():
    broker.publish(message.topic, message.payload)
    outbox.markPublished(message.id)

This turns a fragile one-shot action into retryable work. If the process crashes before publishing, the outbox row remains and can be retried. If the broker is temporarily unavailable, the row remains and can be attempted later.

The publisher can run inside the same application process, in a separate worker, or through infrastructure that observes database changes. Those choices affect latency, operations, and scaling, but they do not change the core pattern: the source transaction durably records what needs to be published.

Why the outbox does not give exactly-once delivery

There is still a failure window in the publisher itself.

Imagine this sequence:

broker.publish(message)     // succeeds
process crashes
outbox.markPublished(id)    // never runs

After restart, the outbox record still appears unpublished, so the publisher sends it again.

That is normally the correct recovery behavior. Marking the row as published before sending would create the opposite and more dangerous failure: a crash could cause the message to be permanently skipped.

The consequence is important: a basic transactional outbox usually gives you at-least-once publication, not exactly-once processing across the whole system. A committed outbox message should eventually be publishable under the system’s retry assumptions, but the same logical message may be delivered more than once.

That means consumers need a duplicate strategy when repeating an operation would be harmful.

Give every logical message a stable identity

The outbox row should have an identifier that remains the same across publication retries.

For example:

id:      M9001
topic:   OrderPaid
payload: { "orderId": "O42" }

If publishing M9001 succeeds and the publisher crashes before recording progress, the retry should still carry M9001. Do not create a new message identity merely because this is a second delivery attempt.

A consumer can then remember processed message IDs:

begin transaction

if processed_messages contains "M9001":
    commit
    return

apply fulfillment change
insert "M9001" into processed_messages

commit

The check and the consumer’s business change need appropriate atomicity for the consumer’s storage model. Otherwise two concurrent deliveries can both pass the check before either records completion.

This is one form of an idempotent consumer: repeating the same logical message does not repeat the business effect.

Not every consumer needs a processed-message table. Some operations are naturally idempotent, and some can enforce uniqueness through domain constraints. The design goal is not a particular table. It is making duplicate delivery safe enough for the operation being performed.

Treat outbox state as operational data

Once message delivery depends on the outbox, the table is part of the service’s reliability mechanism. It needs operational attention rather than being treated as temporary implementation detail.

At minimum, you should be able to answer questions such as:

• How many unpublished messages are waiting?
• How old is the oldest unpublished message?
• Are publication attempts failing repeatedly?
• Is the publisher keeping up with the rate at which transactions create outbox rows?

Queue length alone can be misleading. Ten thousand fresh rows may be normal during a burst, while one row that has been stuck for an hour may indicate a poison message or persistent publishing failure. Age is often a useful signal because it reflects how long committed business changes have been waiting to become visible to consumers.

Retention matters too. Successfully published rows should not grow without bound. You can delete or archive them after a period that fits your audit and recovery needs. Cleanup should be designed so it cannot remove rows that still require publication.

Decide how publishers claim work

A single publisher can simply read a small batch and process it. With multiple publisher instances, they need a way to avoid needlessly sending the same row at the same time.

Common designs include database-supported row claiming, leases with an expiry time, or partitioning work by a stable key. The right mechanism depends on the database and throughput requirements.

The key distinction is between coordination and delivery semantics. Preventing two workers from intentionally claiming the same row reduces duplicate work, but it does not remove the crash window after broker publication. Consumers should not assume duplicates are impossible merely because publisher coordination is strong.

Leases also need recovery behavior. If a worker claims a row and dies, another worker must eventually be allowed to take it. A permanent processing = true flag without expiry or recovery can turn a worker crash into a permanently lost message.

Preserve ordering only where the business needs it

Outbox rows often have creation timestamps or increasing database IDs, which makes it tempting to promise a single global message order. That promise becomes difficult once publishers run concurrently, retries occur, or messages travel through broker partitions.

First ask what ordering the business actually requires.

For one order, OrderCreated may need to be observed before OrderCancelled. That does not necessarily mean events for unrelated orders need a total order. Partitioning or sequencing by an aggregate key such as orderId can preserve a useful local order while still allowing independent entities to progress concurrently.

If strict ordering matters, define the scope and design for it explicitly. Do not infer it from the fact that rows happened to be inserted in a particular order.

Keep the source transaction small

The outbox removes broker communication from the business transaction, which is useful for another reason: a slow broker does not have to keep the database transaction open.

The transaction should generally contain the business state change and the small amount of data needed to describe the outgoing message. Large payload generation, network calls, and slow external work are better kept outside it when possible.

This does create a design question: what should the outbox payload contain?

A message can contain the data consumers need, or it can contain an identifier that causes consumers to fetch current state elsewhere. The first approach makes the event more self-contained but duplicates data. The second keeps the message small but adds coupling to another read path and may expose consumers to state that has changed since the event occurred.

Choose based on the contract consumers need, not only on storage size.

Common mistakes that weaken the pattern

The most serious mistake is writing the outbox row outside the business transaction:

commit business change
insert outbox row

That recreates the original dual-write gap, only with a database insert replacing the broker call.

Another mistake is deleting or marking an outbox row as published before the broker confirms acceptance. Doing so favors avoiding duplicates over avoiding loss. For most outbox designs, retries plus duplicate handling are the safer trade-off.

A third mistake is generating a new message ID on every retry. That prevents consumers from recognizing repeated delivery of the same logical event.

Finally, do not assume the pattern makes every downstream workflow transactional. The source service gains an atomic relationship between its own state change and its publication intent. Consumers still run later, can fail independently, and need their own consistency and retry design.

When a transactional outbox is a good fit

Use the pattern when a service must commit local state and reliably publish a message about that committed change, but the database and broker do not share one transaction.

It is especially useful when losing the message would leave downstream systems permanently unaware of a committed state change.

A simpler approach can be better when the message is merely a disposable hint, occasional loss is acceptable, or the system has no separate broker write to coordinate. Likewise, if your platform already provides a well-understood atomic mechanism that covers both operations, adding an outbox may duplicate infrastructure without improving the guarantee.

The outbox also has a cost: another table or durable store, a publisher, retries, cleanup, monitoring, and duplicate-aware consumers. That cost is justified when it closes a real reliability gap, not because every event-driven system must use the pattern.

Design around the guarantee you actually need

The transactional outbox is useful because it narrows an awkward distributed consistency problem into two manageable responsibilities.

The source transaction guarantees that a committed business change carries a durable record of what should be published. The publisher guarantees repeated attempts to deliver that record. Consumers handle the possibility that an attempt is repeated.

When implementing the pattern, start by drawing the failure windows. Make sure the business change and outbox insert commit together, keep a stable message identity across retries, and decide how consumers tolerate duplicates. Those three decisions do more for correctness than adding elaborate messaging infrastructure before the failure model is clear.