Idempotent Consumers: Handle Duplicate Messages Safely

A message broker can deliver the same message more than once. A worker may finish its database update and crash before acknowledging the message. The broker sees no acknowledgement, so it sends the message again. From the broker’s perspective, redelivery is the safe choice. From the application’s perspective, the second delivery can repeat a business effect.

That gap matters whenever an effect must happen once per logical message. Charging an account twice, granting stock twice, incrementing a counter twice, or sending the same fulfillment request twice can turn a routine retry into corrupted state.

An idempotent consumer makes repeated delivery of the same logical message produce the same durable result as one delivery. It does not require the transport to deliver exactly once. Instead, the consumer accepts that duplicates can arrive and makes them harmless.

This article develops that design from the failure sequence upward, then shows how message identity, database constraints, transaction boundaries, retention policy, and observability fit together.

Start with the failure window

Consider an order service consuming PaymentCaptured events. A handler updates an order and then acknowledges the message:

1. receive message M42
2. update order 913 to PAID
3. commit database transaction
4. acknowledge M42

Now place a process crash between steps 3 and 4:

1. receive message M42
2. update order 913 to PAID
3. commit database transaction
4. process stops
5. broker redelivers M42

The database contains the first effect, but the broker has no evidence that processing finished. This is a normal consequence of coordinating two systems without one shared atomic commit.

Acknowledging first merely moves the dangerous window:

1. receive message M42
2. acknowledge M42
3. process stops
4. database update never happens

The system has exchanged duplicate risk for message-loss risk. Neither ordering gives the application a robust once-per-message effect.

The practical answer is to make the database operation recognize repeated message identity.

Give every logical message a stable identity

Deduplication needs an identifier that remains unchanged across delivery attempts. A broker delivery tag is often unsuitable because a new delivery can receive a new transport-level identifier. The useful key belongs to the logical message itself.

For example:

{
  "message_id": "pay_01J8K6M9R2",
  "type": "PaymentCaptured",
  "order_id": "913",
  "amount_cents": 4200
}

If this event is delivered five times, all five copies carry pay_01J8K6M9R2.

A consumer can then store that identity in a table with a uniqueness constraint:

CREATE TABLE processed_messages (
    consumer_name TEXT NOT NULL,
    message_id TEXT NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer_name, message_id)
);

The consumer name is part of the key because two independent consumers may legitimately process the same message. Deduplication should suppress repeats within one consumer’s responsibility, not across unrelated subscribers.

Commit the marker and business effect together

Recording message identity is useful only if the marker and the business update share one transaction.

A safe sequence is:

begin transaction
    insert processing marker
    apply business change
commit transaction
acknowledge message

The insert acts as the gate. A duplicate hits the unique constraint, indicating that this consumer has committed the message before.

A PostgreSQL-oriented sketch looks like this:

BEGIN;

INSERT INTO processed_messages (consumer_name, message_id)
VALUES ('order-payment-handler', 'pay_01J8K6M9R2')
ON CONFLICT DO NOTHING;

-- Continue only when the insert created one row.

UPDATE orders
SET payment_status = 'PAID'
WHERE id = 913;

COMMIT;

Application code must inspect whether the insert actually created a row. If it did not, the handler should treat the message as a duplicate and avoid the business mutation.

The key property is atomicity. If the transaction rolls back, both the marker and the business effect disappear. The next delivery remains eligible for processing. If the transaction commits and the worker stops before acknowledgement, the next delivery finds the marker and skips the effect.

That gives two important outcomes:

The database becomes the authority for whether this consumer has completed a logical message.

Keep the duplicate path successful

A duplicate is not necessarily an application error. In an at-least-once delivery system, it is expected behavior.

If the handler detects a committed message, it can normally acknowledge that delivery and return success:

def handle(message, db, broker):
    with db.transaction() as tx:
        inserted = tx.try_insert_message_marker(
            consumer="order-payment-handler",
            message_id=message.id,
        )

        if inserted:
            tx.mark_order_paid(message.order_id)

    broker.ack(message)

This structure also avoids a subtle mistake: acknowledging inside the database transaction. The transaction should commit before the acknowledgement is sent. If acknowledgement succeeds while the transaction later rolls back, the message may disappear without its intended effect.

Put the marker in the same transactional store

Suppose the business state lives in PostgreSQL but deduplication markers live in Redis. The consumer now has two writes that cannot normally commit as one local transaction:

write marker to Redis
update order in PostgreSQL

A crash between them can leave a marker without the business effect. Reversing the order can leave the business effect without a marker. Either arrangement recreates the coordination gap the design was meant to close.

For database-backed effects, the simplest robust arrangement is usually to keep the deduplication marker in the same database and transaction as the effect.

This does not mean every consumer needs a central global table. A marker can also live on the affected aggregate when that model fits the access pattern. For example, an account row could retain the last applied sequence number from a particular event stream. The essential condition remains the same: duplicate detection and the protected mutation must commit atomically.

A unique constraint is part of the concurrency design

Two workers can receive duplicate copies close together. A read-then-insert check is not enough:

worker A: SELECT marker -> absent
worker B: SELECT marker -> absent
worker A: apply effect
worker B: apply effect

The race exists because both workers can observe absence before either writes the marker.

A database uniqueness constraint turns the decision into an atomic competition. Only one transaction can establish the same (consumer_name, message_id) key. The other must conflict, wait, or observe that the row exists, depending on the database and statement form.

Treat the constraint as correctness machinery, not merely as an optimization. Application-level checks can improve diagnostics, but they should not replace the atomic guard.

Idempotent business operations can reduce marker traffic

Some effects are naturally idempotent. Setting an order status to PAID is often safer under repetition than incrementing a balance:

UPDATE orders
SET payment_status = 'PAID'
WHERE id = 913;

Repeated execution may leave the same final value. By contrast:

UPDATE accounts
SET balance_cents = balance_cents + 4200
WHERE id = 77;

repeats the monetary effect on every execution.

Natural idempotence is valuable, but it does not automatically remove the need for message tracking. A handler may perform several actions, emit another event, update audit data, or trigger an external call. The complete operation must be examined, not just its most visible SQL statement.

A strong design often combines both techniques: shape business mutations to tolerate repetition where practical, and use explicit message identity where a once-per-message boundary matters.

External side effects need another boundary

A local database transaction cannot atomically include an arbitrary HTTP call to another service. Consider this handler:

begin database transaction
insert marker
commit transaction
call shipping service

A crash after the commit but before the HTTP call leaves the message marked as complete even though shipping was never requested. Calling shipping first creates the opposite risk: shipping may accept the request, then the local transaction may fail and the broker may redeliver the event.

A common solution is to convert the external action into durable local intent. In the same transaction that records consumption, insert an outgoing message into an outbox table. A separate publisher then sends that message. The receiving service applies its own duplicate protection.

The chain becomes:

incoming message
      |
      v
local transaction
  - processing marker
  - business update
  - outgoing intent
      |
      v
outbox publisher
      |
      v
next consumer with its own duplicate guard

This composes local atomic guarantees instead of pretending a network call participates in the database transaction.

Retention defines the deduplication horizon

A processed-message table grows continuously unless old markers are removed. Deleting markers is safe only after the system no longer needs to recognize those message identities.

That period is the deduplication horizon. It should cover the longest realistic interval in which an old message can return, including broker retention, dead-letter replay, operational reprocessing, delayed retries, and restoration from backups or archives.

For example, if operators can replay events from the previous 30 days, retaining markers for only seven days allows an older replay to repeat effects.

Retention can be implemented with time-based deletion or partitioning, but the policy is a correctness decision before it is a storage decision. Document the supported replay window and align marker retention with it.

For very long replay histories, another strategy is to make replay a distinct operation with explicit safeguards rather than relying on an indefinitely growing deduplication table.

Sequence numbers solve a related but different problem

A stable message identifier answers: “Has this exact logical message been committed by this consumer?”

A sequence number answers a different question: “Is this event newer than the state already applied?”

For an ordered stream per account, a consumer might store last_applied_sequence and accept only a greater value:

UPDATE account_projection
SET balance_cents = 5100,
    last_applied_sequence = 84
WHERE account_id = 77
  AND last_applied_sequence < 84;

This can suppress duplicates and stale events when the producer guarantees a meaningful monotonic sequence for that entity. It also encodes ordering semantics that a plain message-ID table does not provide.

Do not substitute sequence numbers casually. Global ordering is expensive or unavailable in many systems, and sequences scoped to one entity cannot order unrelated entities. Choose the mechanism that matches the invariant being protected.

Be precise about the guarantee

An idempotent consumer does not make the transport exactly-once. The broker may still deliver duplicates. The handler may execute more than once. Logs may show several delivery attempts.

The useful guarantee is narrower and more concrete:

For a given consumer identity and logical message identity, the protected durable effect is committed at most once, while failed uncommitted attempts remain retryable.

Combined with eventual redelivery after transient failures, this can provide an effectively-once business result for the protected local transaction.

That statement also exposes the boundary. If part of the effect happens outside the transaction, that part needs its own reliability mechanism.

Test the crash points, not only the happy path

A useful test suite exercises the boundaries where duplicates arise. At minimum, cover these cases:

1. Deliver the same message twice and confirm one durable effect.
2. Fail before the transaction commits, redeliver, and confirm the later attempt succeeds.
3. Simulate a stop after commit but before acknowledgement, redeliver, and confirm no repeated effect.
4. Run two duplicate deliveries concurrently and confirm the uniqueness guard admits one effect.
5. Replay a message inside the supported retention window and confirm it remains suppressed.
6. Verify that two distinct consumers can process the same message when both are intended subscribers.

These tests target the protocol, not just handler code. A unit test that invokes a function twice can be useful, but database concurrency and transaction boundaries deserve integration coverage.

Observe duplicates without treating them as incidents

Duplicate rate is operationally useful. A sudden increase can signal broker redelivery, acknowledgement latency, worker instability, network disruption, or a replay operation.

Record enough structured data to distinguish normal duplicate handling from processing failures:

consumer=order-payment-handler
message_id=pay_01J8K6M9R2
outcome=duplicate

Useful counters include total deliveries, newly processed messages, duplicates, transaction failures, acknowledgement failures, and processing latency. Keep message identifiers searchable in logs or traces when privacy and cardinality constraints permit it.

The duplicate counter should not automatically page an operator. The design exists precisely so duplicates can be absorbed safely. Alert on abnormal rates or on failed handling, not on every expected redelivery.

A compact design checklist

Before shipping a consumer that changes durable state, verify these points:

• Each logical message has a stable identity across retries.
• Deduplication is scoped to the consumer that owns the effect.
• A database constraint makes duplicate admission atomic.
• The marker and protected business mutation share one transaction.
• The broker acknowledgement happens after transaction commit.
• Duplicate delivery returns a successful handling outcome.
• External effects are represented as durable intent or protected by their own idempotency mechanism.
• Marker retention covers the supported replay horizon.
• Concurrent duplicate delivery is tested.
• Metrics distinguish new processing, duplicates, and failures.

Closing perspective

Duplicate delivery is not an edge case to eliminate with optimistic assumptions about a broker. It is a normal result of retrying work across process and network failures.

The robust consumer accepts that reality. It gives each logical message a stable identity, uses an atomic database guard, commits that guard with the business effect, and acknowledges only after the transaction is durable. For effects that cross another service boundary, it carries the same discipline forward through durable intent and duplicate protection at the next consumer.

Once those boundaries are explicit, redelivery stops being a correctness threat. It becomes another routine input the system can process safely.