Make Sensitive Operations Safe to Retry

A client sends a request to create a refund. The server completes the refund, but the response is lost when the connection closes. The client cannot tell whether the operation succeeded, so it retries.

If the server treats the retry as a new operation, one uncertain network failure can become two refunds. The same pattern appears in credit transfers, invitation acceptance, provisioning, job submission, and other state-changing actions where repeating an effect has security or financial consequences.

The practical problem is not simply that clients may send duplicate requests. Distributed systems make retries normal: responses can be lost after work has already committed. A defensive API therefore needs a way to distinguish another attempt at the same operation from a genuinely new operation.

This article develops that mental model using idempotency keys. You will learn how to scope a key, bind it to the intended request, make the check and state change atomic, choose a retention period, and recognize when idempotency is not enough to stop a replay attack.

A retry is not necessarily a new intent

HTTP defines an operation as idempotent when multiple identical requests have the same intended effect as one request. Some operations naturally have that property. Setting a resource to a particular state can often be repeated without creating another logical action.

Many important application operations are different. Consider:

POST /refunds
order = order_4821
amount = 25.00

Two accepted requests might mean two separate refunds. The server cannot safely infer from identical payloads alone that the second request is accidental. A user may legitimately perform two operations with the same amount.

The client knows something the server does not: whether this transmission is a retry of an earlier attempt. An idempotency key carries that identity across attempts.

A simplified request might be:

POST /refunds HTTP/1.1
Idempotency-Key: op_7f3a91c2
Content-Type: application/json

{"order_id":"order_4821","amount":"25.00"}

The exact header syntax is platform-dependent; the important application property is that the client assigns a unique operation identifier and reuses it only when retrying that same logical operation.

Give one logical operation one identity

The server should interpret an idempotency key as an operation identity inside a defined scope, not as permission to execute whatever request happens to carry it.

A useful conceptual record is:

scope:       authenticated account 42
key:         op_7f3a91c2
fingerprint: refund order_4821 amount 25.00
state:       completed
result:      refund rf_9182

On the first request, the server records the operation and performs the effect. On a later request with the same scope and key, it recognizes the retry and returns the previously established outcome instead of creating another refund.

The scope matters. If keys are supplied by clients, two unrelated accounts can accidentally choose the same value. Treating the key as globally authoritative could make one caller collide with another. A common design scopes it to the authenticated principal, tenant, API credential, or another boundary that matches who owns the operation.

The key also needs enough entropy or another uniqueness guarantee to make accidental collisions unlikely within that scope. It does not need to be a secret unless the surrounding protocol gives secrecy a separate purpose.

Bind the key to the request it represents

Remembering only that a key has appeared is incomplete. The server must also detect when the same key is reused for a different operation.

Suppose the first request asks for a refund of 25.00 and a later request presents the same key with an amount of 250.00. Silently treating the second request as the first is confusing. Executing it as a new request defeats duplicate suppression.

Instead, bind the stored key to the security-relevant request semantics. The application can store normalized fields or a fingerprint derived from the canonical representation it actually uses for the operation.

Conceptually:

if key does not exist:
    reserve key for this request
    execute operation
    store outcome
else if request matches reserved operation:
    return established outcome
else:
    reject key reuse

This comparison must use the fields that define the operation. A refund might bind the key to the account, order, currency, amount, and operation type. Omitting a field that changes the effect can allow one operation identity to become ambiguous.

Do not hash raw JSON and assume that solves canonicalization automatically. Whitespace, field ordering, equivalent number representations, or fields added by intermediaries can make bytewise representations differ while the application meaning is the same. Bind to a stable representation of the values the server actually interprets.

Make duplicate detection and execution one state transition

The most important implementation detail is concurrency.

A naive handler can still duplicate an effect:

request A: key not found
request B: key not found
request A: perform effect
request B: perform effect

Both requests passed the check before either recorded the key. This is a time-of-check/time-of-use race.

The reservation of the operation identity therefore needs an atomic uniqueness boundary. Depending on the system, that might be a database unique constraint, a transaction, a conditional insert, or another storage primitive that guarantees only one contender can create the operation record.

A robust flow is:

1. authenticate and authorize the caller
2. validate the requested operation
3. atomically reserve (scope, idempotency_key)
4. if already reserved, verify that the request matches it
5. execute or recover the operation according to its recorded state
6. persist the outcome needed for retries

The exact transaction boundary depends on where the side effect occurs. If the business state and idempotency record live in the same transactional database, they can often be committed together. If execution calls an external system, a local database transaction cannot make that remote side effect atomic. The integration then needs its own operation identifier, idempotent downstream API, durable workflow, or reconciliation mechanism.

This distinction is important: an idempotency table does not magically create a distributed transaction.

Represent in-progress and completed operations deliberately

A duplicate can arrive while the first request is still running. The server needs defined behavior for that state.

For example, an operation record may move through:

reserved -> processing -> completed
                     \-> failed

A concurrent retry that sees processing should not start the work again. It can wait briefly, return a response that tells the client the operation is still in progress, or expose a status resource. The right choice depends on API latency and client behavior.

Failures need similar care. If validation fails before any effect is possible, allowing the client to correct the request under a new key is straightforward. If a failure occurs after the outcome becomes uncertain, immediately deleting the reservation can be dangerous: the retry may repeat an effect that actually succeeded.

For sensitive operations, preserve enough durable state to distinguish known not executed from execution outcome unknown. An unknown outcome should normally trigger recovery or reconciliation rather than blind re-execution.

Keep keys long enough for the retry window

Idempotency records consume storage, so systems usually retain them for a bounded period. Expiry is a security and reliability decision, not merely housekeeping.

If the server forgets a key while a legitimate client can still retry the original request, that old retry becomes indistinguishable from a new operation. The retention period should therefore cover the documented retry behavior plus realistic delays from queues, offline clients, and recovery processes that can resend requests.

Longer retention reduces the window in which a delayed retry can repeat an effect, but increases storage and lookup cost. Shorter retention is simpler operationally but transfers more risk to clients and downstream reconciliation.

There is no universal duration. A synchronous internal API with tightly bounded retries may need much less history than an externally queued financial workflow. Document the retention contract so clients know when a key remains meaningful.

Idempotency and replay protection solve different problems

It is tempting to describe idempotency keys as general replay protection. That is too broad.

Idempotency primarily makes retries of the same logical operation converge on one effect. It helps when duplicates arise from network uncertainty, client retries, queue redelivery, or repeated callbacks.

It does not prove that the request is authentic. If an attacker can create authorized requests, an idempotency key does not remove that authority. If an attacker can steal a bearer credential, the key does not repair the credential compromise. Authentication and authorization still happen independently.

For cryptographically authorized messages, replay resistance may also require the authorization evidence itself to be bound to a unique operation value, freshness window, recipient, and relevant request data. A server must then reject reuse according to that protocol’s rules. Merely accepting a new idempotency key alongside an old valid signature would not make the old authorization single-use.

Similarly, timestamps alone do not provide exactly-once behavior. They can limit how long a message is accepted, but two copies inside the valid window are still two copies unless the receiver tracks an operation identity or nonce.

Use idempotency for duplicate effects. Add protocol-level anti-replay controls when the threat model includes an adversary capturing and resubmitting otherwise valid authorization evidence.

Do not confuse a successful response with a successful operation

A useful idempotency design records the business outcome, not just an HTTP response code.

Imagine the first request creates refund rf_9182, but the process crashes before storing a cached 201 Created response. If the refund record itself carries the operation key, recovery can discover that the effect already exists and reconstruct an appropriate response. If the design only cached responses in volatile memory, the retry may create a second refund.

The durable business state should therefore be the source of truth whenever possible. Response caching is a convenience layered on top of operation identity, not the security boundary by itself.

Also decide which response details are safe to replay. A stored result should not bypass current access control when a later caller asks for it. Scope the lookup to the same security principal or reauthorize access to the resulting resource as appropriate.

Verify the control under failure, not only success

The useful tests are the cases that create ambiguity.

Send two concurrent requests with the same key and confirm that only one business effect exists. Retry after the first request completes and verify that the established result is returned. Reuse the key with changed security-relevant input and confirm that the server rejects the conflict. Simulate a lost response after commit and verify that a retry discovers the committed operation rather than repeating it.

Also test expiry deliberately. Once a key is outside the supported retention window, the client should not assume the server can still recognize the old operation.

For workflows that call external systems, test failures between each durable step. The question is not just whether an exception is handled. It is whether the system can determine, after recovery, which side effects happened and which remain safe to perform.

Choose the control where duplicate effects matter

Not every endpoint needs an idempotency store. A naturally idempotent operation may already have the required retry behavior. A low-impact action whose duplicate effect is harmless may not justify additional state and complexity.

The control becomes valuable when a client can reasonably retry and a duplicate would create a meaningful consequence: charging, refunding, provisioning, redeeming, allocating scarce resources, or triggering an irreversible workflow.

For those operations, the design decision is precise: give each logical operation a stable identity, bind that identity to its intended semantics and caller, reserve it atomically, persist enough outcome state to recover from uncertainty, and retain it for the real retry window.

That does not make every request trustworthy or every distributed workflow exactly once. It does make a common failure mode manageable: when the same intent arrives again, the system can recognize it as the same intent instead of performing the sensitive effect again.