Idempotent Event Consumers for At-Least-Once Delivery

Many queues and event brokers provide at-least-once delivery: a message that has been accepted can be delivered again when acknowledgements are lost, consumers crash, visibility timeouts expire, or the broker retries after uncertain outcomes.

Duplicates are therefore not exceptional. A robust consumer should assume that the same logical event can arrive more than once.

Why duplicates happen

Consider this sequence:

1. a consumer receives an event;
2. it updates the database successfully;
3. the process crashes before acknowledging the message;
4. the broker makes the message visible again;
5. another consumer receives it.

The broker cannot know that the database update happened. Redelivery is the safer choice.

If processing increments a balance, sends a shipment, or creates a record unconditionally, the duplicate can repeat the side effect.

Give each logical event a stable identity

Idempotent processing starts with an identifier that remains the same across redeliveries.

A producer-assigned event ID is ideal:

{
  "event_id": "evt_01JEXAMPLE",
  "type": "order.confirmed",
  "order_id": "ord_123"
}

The ID should identify the logical event, not the delivery attempt.

Do not generate a fresh deduplication ID inside the consumer; every redelivery would receive a different value and bypass the check.

Claim the event and change state atomically

A common pattern stores processed event IDs in the same database transaction as the business update.

Conceptually:

BEGIN;

INSERT INTO processed_events (consumer, event_id)
VALUES (:consumer, :event_id);

-- Apply the business state change.

COMMIT;

The table should enforce uniqueness on the deduplication key, commonly (consumer, event_id).

If the insert conflicts because the key already exists, the consumer knows that this logical event has already been committed for that consumer.

The crucial property is atomicity: the deduplication record and the business change commit together. Recording the event first in a separate transaction can lose work after a crash; recording it after the business change can allow the business change to repeat.

Exact SQL for ignoring or detecting unique conflicts differs by database, so use the database’s documented conflict-handling mechanism.

Make the business operation naturally idempotent when possible

Deduplication tables are useful, but some operations can be expressed idempotently themselves.

Instead of “increment shipment count because event arrived,” an event might carry an authoritative state transition such as “order ord_123 is confirmed at version 7.”

A conditional update can reject stale or repeated versions.

Natural idempotency reduces bookkeeping and can make replay safer, but it requires event semantics designed around state rather than delivery count.

External side effects need their own idempotency strategy

A database transaction cannot atomically include a remote email provider, payment API, or another cloud service.

If event processing both changes database state and calls an external API, there is still a crash window.

Common strategies include:

• an outbox table committed with the business transaction;
• a downstream API that accepts an idempotency key;
• a state machine that records which external action is pending or complete.

If the remote API supports idempotency keys, pass a stable key derived from the logical operation, not from the current attempt.

Acknowledge only after durable success

The consumer should acknowledge or delete the message only after the required durable state has committed.

Acknowledging before commit creates an at-most-once failure window: if the process crashes after the acknowledgement but before persistence, the broker may never redeliver the lost work.

Acknowledging later means duplicates are possible, which is exactly why idempotency is necessary.

Define the deduplication retention period

A processed-event table grows indefinitely unless records are expired or archived.

Before deleting them, understand the broker’s maximum redelivery horizon and any replay workflows your system supports.

If operators can replay events from six months ago but deduplication keys expire after seven days, replay may repeat effects that were assumed to be one-time.

Retention is part of the correctness contract, not only a storage optimization.

Concurrency matters too

Two workers can receive duplicate deliveries at nearly the same time.

A “check then insert” sequence without a uniqueness constraint is racy:

worker A: event not found
worker B: event not found
worker A: process
worker B: process

Use a database uniqueness guarantee or another atomic claim mechanism. Correctness should not depend on workers observing each other’s reads in time.

Decide what an event ID means across consumers

Different consumers may legitimately process the same event once each.

That is why a deduplication key often includes the consumer identity:

(consumer_name, event_id)

If a service contains several independent handlers, decide whether each handler has separate processing state or whether the whole service shares one atomic outcome.

The key should match the scope of the side effect you are protecting.

Common pitfalls

Assuming the queue guarantees exactly-once business effects

Broker delivery semantics and application side effects are different layers.

Deduplicating in an in-memory cache only

A process restart loses the record, and multiple replicas may not share the same view.

Marking an event processed before the business transaction commits

A crash can cause redelivery to be discarded even though the intended update never happened.

Using short retention without considering replay

Old duplicates or intentional replays can cross the retention boundary.

Forgetting remote side effects

Database deduplication does not automatically make a payment or email request idempotent.

Design for redelivery from the start

At-least-once delivery is practical because it favors not losing messages when outcomes are uncertain. The application completes the contract by making repeated delivery safe.

Use stable event identities, atomically claim events with business state changes, acknowledge only after durable success, and give external effects their own idempotency mechanism. When duplicates are treated as normal input rather than an anomaly, event-driven systems become much easier to operate under crashes and retries.