A caller sends a request to create a payment. The server processes it, but the response is lost because the connection closes. The caller now has a difficult choice: retry and risk charging twice, or stop and risk leaving the payment incomplete.

This is not mainly a networking problem. It is an operation-design problem. When a caller cannot tell whether an attempt succeeded, retrying is only safe when the system has a way to recognize that the new attempt represents the same intent.

This article explains the mental model behind idempotent operations, how to make side-effecting operations retryable with an idempotency key, what guarantees that design can and cannot provide, and when a simpler approach is enough.

Start with the uncertainty between attempt and outcome

Consider a simplified payment operation:

charge(customer, 40.00)

Suppose the server successfully charges the customer and then loses the connection before the response reaches the caller. From the caller’s perspective, two realities are possible:

  1. the request never reached the server;
  2. the request succeeded but the response was lost.

A timeout cannot distinguish them.

If the caller repeats the same command and the server treats every request as new, the second attempt may create a second charge. Avoiding retries does not solve the problem either: in the first reality, no charge would ever be created.

The useful design question is therefore not “Did I already send this request?” It is:

Can the receiver tell whether this attempt represents an operation it has already accepted?

That is the problem idempotency addresses.

Idempotency means repeating the operation does not add another effect

An operation is idempotent when applying the same operation more than once has the same intended effect as applying it once.

Setting a user’s preferred language to en is naturally idempotent:

setPreferredLanguage(userId, "en")

Calling it twice still leaves the preference set to en.

Incrementing a balance is different:

addCredit(accountId, 10)

Two successful calls add 20, so repetition changes the result. The operation is not naturally idempotent.

This distinction is about observable effect, not whether code executes more than once. A retry may still reach the application, perform a lookup, write a log entry, or return a cached result. The important property is that repeating the same logical operation does not repeat the protected business effect.

Give one logical operation a stable identity

For operations that are not naturally idempotent, the caller can attach a unique idempotency key to one logical operation.

For example:

idempotency-key: order-814-payment-attempt-1
amount: 40.00
customer: C123

If the caller times out, it retries with the same key and the same operation data. A genuinely new payment uses a new key.

The receiver can then treat the key as the identity of the operation rather than treating each network request as new.

A simplified flow is:

if key has a completed result:
    return stored result

if key is currently being processed:
    return or wait according to the API policy

reserve key
perform business operation
store result for key
return result

The key changes the problem. The server no longer has to infer whether two similar requests are duplicates. The caller explicitly says, “These attempts belong to the same logical operation.”

The check and the side effect must form one reliable protocol

A common mistake is to implement idempotency as an ordinary lookup followed by an ordinary write:

if key not in idempotency_store:
    create_payment()
    idempotency_store[key] = result

Two requests with the same key can race. Both may observe that the key is absent before either stores a result, and both may create a payment.

The receiver needs an atomic way to claim the key, such as a uniqueness constraint or another concurrency-safe reservation mechanism. Only one competing request should be allowed to become the owner of that logical operation.

There is a second failure window. Imagine the server reserves the key, creates the payment, and crashes before recording the result. On recovery, it knows the key exists but may not know whether the side effect completed.

This is why an idempotency table by itself does not magically provide exactly-once execution. The business side effect and the idempotency record need a failure strategy appropriate to where those writes occur.

If both can be committed in one transactional boundary, they can often succeed or fail together. If the side effect is performed by an external system, a single local transaction cannot make both systems atomic. The design then needs another mechanism, such as giving the downstream operation its own stable identifier, reconciling uncertain outcomes, or using a durable workflow that can resume safely.

The practical lesson is to inspect every point where the process could stop. For each point, ask what a retry would observe and whether it could repeat the protected effect.

Store enough information to recognize incompatible reuse

An idempotency key should normally identify one specific intent. If a caller accidentally reuses the same key with different input, silently returning the old result can hide a serious bug.

Suppose the first request is:

key = "checkout-91"
amount = 40.00

and a later request says:

key = "checkout-91"
amount = 75.00

The receiver should not interpret both as the same payment merely because the key matches.

One approach is to store the fields that define the operation, or a deterministic fingerprint of those fields, alongside the key. A retry with matching input can receive the existing outcome. A request that reuses the key for different input can be rejected as a conflict.

Be deliberate about what belongs in that comparison. Transport details such as a trace identifier may legitimately change between attempts, while business inputs such as account, amount, currency, and operation type may define the intent.

Decide what result a retry receives

Preventing duplicate effects is only half of a useful retry contract. The caller also needs predictable behavior after the first attempt finishes.

A common policy is to store the completed outcome and return it for later attempts with the same key. For a successful creation, that may include the created resource identifier and the response status needed by the caller.

Failures require more thought. A validation error such as an invalid amount is usually deterministic for the same input, so retaining that outcome can be reasonable. A transient infrastructure failure may be retryable, so permanently binding the key to that failure could defeat the purpose of retrying.

There is no universal rule for every failure category. Define the policy as part of the operation’s contract:

  • which outcomes complete the idempotent operation;
  • which failures allow another attempt to continue processing;
  • what happens while another request with the same key is still in progress;
  • how long completed keys remain recognizable.

The important point is that these are correctness decisions, not cache settings.

Key lifetime creates a boundary on the guarantee

Keeping every idempotency record forever is often unnecessary, but deleting records changes the guarantee.

If a completed key is retained for 24 hours, a retry during that period can be recognized. After the record expires, the same key may look new unless another durable business identifier still prevents duplication.

Choose retention based on the realistic retry window and the cost of duplicate effects. A background job that retries for ten minutes has different needs from an offline client that may reconnect several days later.

Document the boundary. “Retry-safe” without a stated lifetime can lead callers to assume a stronger guarantee than the service actually provides.

Do not confuse idempotency with deduplicating similar requests

Two requests can contain identical data and still represent two valid operations. A customer may intentionally buy the same product twice or make two payments for the same amount.

Comparing payloads and suppressing requests that look similar therefore changes business semantics.

An idempotency key is different because it carries identity supplied for one logical operation. The caller chooses when attempts are the same intent and when they are new intents.

Likewise, a request identifier used only for tracing is not automatically an idempotency key. A tracing system may generate a new identifier for each retry, while idempotency requires the operation identity to remain stable across those retries.

Keep the scope of the key explicit

A key rarely needs to be globally unique across every operation in a system. Its namespace can be scoped to a customer, endpoint, operation type, or another well-defined boundary.

For example, the stored identity might effectively be:

(customer_id, operation_type, idempotency_key)

The exact scope depends on the application, but it must be unambiguous. If two unrelated operations can accidentally collide in the same namespace, one may incorrectly receive the other’s result.

The caller also needs a clear rule for generating keys. Random identifiers can work well when the caller can persist them with the operation being attempted. Domain identifiers can be even more useful when the business already has a natural unique identity, such as an order ID for “capture payment for this order.”

Prefer natural idempotency when the model already supports it

An idempotency-key mechanism adds storage, concurrency handling, expiry policy, and operational cleanup. Do not add it when the operation can be expressed more simply.

For example, instead of “create another subscription row” on every retry, an operation may be modeled as “ensure customer C123 has subscription S456” with a uniqueness rule on that business identity. The domain constraint itself can prevent duplicate creation.

Similarly, replacing “increment status counter” with “set status to approved” may remove retry ambiguity if setting the state accurately represents the business action.

Do not distort the domain merely to make an operation idempotent, though. Two legitimate purchases should remain two purchases. When repetition has real business meaning, preserve that meaning and give each logical operation a stable identity.

Test the failure windows, not only the happy path

A basic test that sends the same key twice is useful but incomplete. The difficult bugs occur around partial progress and concurrency.

Useful tests include:

  • two concurrent requests arrive with the same key;
  • the first attempt completes but its response is lost, then the caller retries;
  • processing stops after the key is reserved but before the business effect;
  • processing stops after the business effect but before the result is recorded;
  • the same key is reused with different business input;
  • a retry arrives after the documented retention period.

The expected behavior depends on the chosen protocol, but each test should establish whether the protected effect can happen twice and what the caller observes.

These tests force the design to account for the places where uncertainty actually appears.

Use idempotency where retries and duplicate effects are both realistic

Idempotency is especially valuable when callers are expected to retry side-effecting operations after timeouts, connection failures, worker restarts, or duplicate message delivery.

It is less useful for read-only operations or operations that are already naturally idempotent. It may also be unnecessary in a tightly controlled synchronous path where the caller never retries and duplicate delivery is not a credible failure mode, although that assumption should be made deliberately.

The central mental model is simple: a network attempt and a business operation are not the same thing. One business operation may require several attempts. Give that operation a stable identity, make competing attempts converge on one protected effect, and define what later attempts observe.

With that contract in place, a timeout no longer forces the caller to guess whether retrying will repeat the work.

Software Engineering Reliability API Design Error Handling