Idempotent Consumers and Durable Duplicate Detection

A consumer commits a database transaction, then loses its connection before acknowledging the message. The broker has no evidence that processing finished, so a delivery protocol that permits redelivery can present the same message again.

The second delivery is not evidence that the first transaction failed. From the consumer’s perspective, the important fact is more precise: message delivery and application commit have separate completion points. If the broker cannot atomically participate in the application’s state transition, an acknowledgement can be lost after the application effect is already durable.

An idempotent consumer addresses that boundary by giving the application durable evidence that a particular message has already contributed its effect. The technique does not eliminate duplicate delivery. It changes duplicate delivery from an ambiguous request to repeat work into a state transition that can be recognized as already complete.

Delivery identity is application state

A broker may expose a message identifier, or a producer may place a stable event identifier in the message envelope. Duplicate detection depends on that identity remaining stable across redelivery. A newly generated identifier on every delivery attempt cannot represent the same logical message.

Consider a consumer that applies account credits. The incoming message carries event_id = evt_8142 and an amount. A processing transaction can record the identifier and apply the balance mutation together:

BEGIN;

INSERT INTO processed_messages (consumer, message_id)
VALUES ('credit-projector', 'evt_8142');

UPDATE accounts
SET balance = balance + 500
WHERE account_id = 73;

COMMIT;

If (consumer, message_id) has a unique constraint, a later transaction attempting the same insert conflicts with the recorded identity. The consumer can interpret that conflict as evidence that this consumer has already committed work for that message, provided the marker and the business mutation were committed in the same transaction.

The consumer name is part of the key in this example because two independent consumers may legitimately process the same event. A global uniqueness constraint on message_id would incorrectly couple their progress.

This model makes duplicate detection persistent. An in-memory set can suppress repeated work only while the process and its memory remain available. It cannot establish that an effect committed before a restart, failover, or reassignment.

Atomic placement carries the guarantee

The marker is useful only when its transaction boundary matches the effect it represents.

If a consumer inserts evt_8142 into processed_messages, commits, and then updates the account in a second transaction, a failure between those commits leaves a marker for work that never happened. A redelivery then appears complete even though the intended mutation is absent.

The reverse order has the complementary gap. If the account update commits before the marker, a failure between the two commits allows redelivery to apply the update again.

Putting both writes in one database transaction removes those two local gaps because the database commits or rolls back the marker and mutation as one unit. This statement depends on both writes being handled by the same transactional resource, or by resources participating in an atomic commit protocol. A marker in one database cannot, by itself, make an unrelated remote side effect atomic.

That limitation is easy to obscure when the consumer performs more than database work. Sending email, calling an HTTP API, or publishing to a separate broker introduces another commit point. Durable duplicate detection can protect the local transaction while leaving the external effect subject to its own delivery and idempotency semantics.

A duplicate is not the same as a repeated command

Idempotent consumer logic is often described as making message handling idempotent, but the exact scope matters. The handler does not need every underlying business operation to be mathematically idempotent. It can instead ensure that one identified message contributes its effect at most once to the protected state.

An increment illustrates the distinction. The operation

balance = balance + 500

is not idempotent when executed repeatedly. With durable message identity, however, evt_8142 can be allowed to execute that increment once and rejected on later deliveries. The combination of identity record and mutation has the desired duplicate-handling property even though the mutation alone does not.

This also separates message identity from business identity. Two distinct credit events for the same account and amount must both apply if they represent separate accepted operations. Deduplicating on (account_id, amount) would collapse legitimate events that happen to contain equal values. The deduplication key must identify the logical operation whose repetition is unwanted.

Retention defines the memory horizon

A processed-message table grows as identities accumulate. Deleting old records is not merely storage maintenance; it changes the period during which the consumer can prove that a message was already handled.

Suppose duplicate markers are retained for seven days. A redelivery after eight days can no longer be distinguished from a first delivery using that table alone. Whether this is acceptable depends on the broker’s retention and redelivery behavior, replay procedures, archival workflows, and any producer contract governing identifier reuse.

The safe retention interval therefore cannot be inferred from table size alone. It is a protocol property spanning the sources that can present old messages and the consumer state used to recognize them.

Some systems embed the last processed position in domain state instead of storing every message identifier. That can be valid when the input has a stable ordering model and the consumer can express progress as a monotonic position. It is not equivalent to arbitrary identifier deduplication. Partitioned logs, concurrent processing, gaps, and out-of-order completion can make a single high-water mark insufficient unless the processing model preserves the assumptions that make the mark meaningful.

Concurrency turns the check into a write conflict

A read-before-write sequence appears straightforward:

if message_id is absent:
    apply effect
    record message_id

On its own, that shape has a race. Two workers can both observe the identifier as absent before either records it. If each then applies the effect independently, the preliminary check has not serialized anything.

A uniqueness constraint moves arbitration into the database write path. Concurrent inserts for the same consumer and message identity cannot both commit when the constraint is enforced. The transaction that fails the uniqueness check must also avoid committing the protected business mutation.

The exact database behavior around unique conflicts, transaction aborts, savepoints, and isolation levels is product-specific. Application code must follow the semantics of its database driver and transaction manager rather than assuming every uniqueness error leaves a transaction usable. The general property comes from atomic constraint enforcement combined with the transaction boundary, not from the earlier existence query.

An existence query can still serve as an optimization when duplicates are common, but correctness should not depend on two concurrent readers seeing each other’s future writes.

Broker acknowledgement remains a separate decision

After the database transaction commits, the consumer can acknowledge the broker message. If acknowledgement succeeds, normal processing ends. If acknowledgement is lost, the broker may redeliver and the durable marker suppresses a second local effect.

Acknowledging before the database commit reverses the risk. A process can acknowledge successfully and then fail before its transaction commits, leaving the broker with no reason to redeliver a message whose application effect is missing. For delivery models that rely on acknowledgement after processing, the ordering between local commit and acknowledgement is therefore part of the consumer protocol.

This does not create exactly-once execution across the broker and database. The handler may execute code more than once, parse the same payload repeatedly, or attempt the duplicate marker insert repeatedly. What becomes singular is the committed effect covered by the transaction and identity record.

That distinction is useful because it describes an observable guarantee without assigning stronger semantics to the transport than it provides.

Identity records form a local receipt

A processed-message record is best understood as a receipt stored beside the state it protects. It says that a named consumer has already incorporated a named logical message into a particular transactional resource.

The receipt cannot certify remote effects it did not commit with. It cannot compensate for unstable message identifiers. It cannot remember duplicates beyond its retention horizon. It also does not require the broker to stop redelivering.

Its value comes from a narrower property: when delivery can repeat, the application retains enough durable identity to decide whether repeating the protected state transition is still valid. That turns redelivery from a transport event with ambiguous application consequences into a decision grounded in committed state.