Make Security-Sensitive State Changes Atomic

Security checks can be individually correct and still fail when two requests run at the same time. A request checks that a recovery code is unused, a withdrawal is within a limit, or an approval is still pending. Before it records the state change, another request performs the same check against the same old state. Both requests then proceed even though the rule was meant to allow only one.

This is a race condition: correctness depends on the relative timing of concurrent operations. A common form is a time-of-check to time-of-use problem, often shortened to TOCTOU, where the fact established by a check can become false before the protected action uses it.

The defensive goal is not to make requests arrive in a convenient order. It is to make the security rule survive any allowed order of concurrent requests. This article explains how to identify security-sensitive check-then-act sequences, make the decisive state change atomic, and test the resulting guarantee.

Start with the invariant, not the code path

An invariant is a condition the system intends to keep true despite normal operations and concurrency. Security-sensitive invariants often sound simple:

a recovery code can be consumed at most once
an approval can be completed at most once
a quota cannot be spent beyond its remaining amount

These statements describe the guarantee that matters. They are more useful than saying that the application “checks first,” because a check by itself does not preserve anything.

Consider a one-time recovery code. A simplified implementation might do this:

record = load_recovery_code(code)

if record.used:
    reject

complete_recovery()
record.used = true
save(record)

With one request, this looks reasonable. With two concurrent requests, both can load used = false before either saves used = true:

request A: read unused
request B: read unused
request A: complete recovery
request B: complete recovery
request A: mark used
request B: mark used

The final database value says the code is used, but the important invariant has already failed: the one-time action happened twice.

The threat does not require extremely precise timing if an attacker can submit concurrent requests deliberately. The same defect can also appear accidentally through retries, duplicated messages, multiple application workers, or impatient users clicking twice.

The dangerous gap is between decision and commitment

A useful mental model is:

read state -> decide -> change state

If another actor can change or consume the relevant state between those steps, there is a race window.

The defense is to move the condition and the decisive state change into one operation whose result other concurrent operations cannot interleave with in a way that violates the invariant:

atomic(condition + state change) -> success or failure

“Atomic” here means that, for the invariant being protected, competing operations cannot both observe themselves as the winner. The exact mechanism depends on the storage system and architecture. It may be a conditional update, a database transaction with suitable locking or isolation, a compare-and-swap operation, or another primitive that provides the required guarantee.

The important point is not the name of the primitive. The important point is what it guarantees under concurrency.

Let the state change decide who wins

For the recovery-code example, a stronger design makes consumption itself conditional on the code still being unused. In SQL-like pseudocode:

UPDATE recovery_codes
SET used = TRUE
WHERE id = :id
  AND used = FALSE;

The application then checks how many rows were changed:

1 row changed -> this request consumed the code
0 rows changed -> the code was already used or unavailable

This example is intentionally simplified. Production syntax and concurrency guarantees depend on the database and transaction configuration. The security idea is that the application does not separately read used = false and later assume that fact remains true. The conditional state transition is the decision.

If two requests attempt the same transition, the storage layer must serialize or otherwise coordinate them so that only one can successfully change the record from unused to used. The losing request observes failure and must not perform the protected operation.

This pattern is often easier to reason about than a read followed by an unconditional write because the success signal directly represents the invariant.

Put irreversible effects on the correct side of the boundary

Atomic database state does not automatically make every external effect atomic.

Suppose an operation both consumes a one-time authorization and sends a request to an external payment service. A local database transaction cannot usually roll back a payment that a remote service has already accepted. This creates a broader coordination problem.

A useful distinction is:

local state transition: can often be protected by one database boundary
external side effect: needs its own duplicate and retry strategy

For a purely local operation, a transaction or conditional update may be sufficient. When a workflow crosses process or service boundaries, also design for retries and ambiguous outcomes. Depending on the system, that may involve an idempotency key, a durable outbox, a unique operation identifier, or a state machine that records what has and has not completed.

Do not hold a database lock across a slow network call merely to make the design appear atomic. Long-held locks can reduce availability and still cannot force the remote system to participate in the same transaction unless a specific distributed transaction protocol is actually in use.

The defensive question is: which system owns the invariant, and what mechanism makes duplicate execution observable and controllable at each boundary?

Authorization can become stale too

Race conditions are not limited to counters and one-time tokens. An authorization decision can depend on mutable state.

Imagine a workflow that checks whether a user is still a project administrator, performs several steps, and then makes a sensitive change. If administrator membership can be revoked concurrently, the application must decide what guarantee it intends to provide.

For some low-risk actions, checking authorization at request entry may be an acceptable policy. For a sensitive state transition, the application may need to evaluate the relevant authorization state as part of the protected transaction or immediately before commitment. Otherwise, a permission that was valid earlier in the request may no longer be valid when the change becomes durable.

This does not mean every permission check must lock every related record. It means the consistency requirement should be explicit. Decide whether authorization is based on state at request start, at commit time, or under another documented rule, then use storage and application primitives that can actually provide that rule.

Choose the smallest mechanism that preserves the invariant

Different invariants need different forms of coordination.

A conditional update is often a good fit when one row contains all state needed to decide a transition. A uniqueness constraint can protect invariants such as “only one active record with this unique identity” when the data model supports that rule. A transaction with row locking may be appropriate when a decision depends on several values that must remain consistent while they are changed. Stronger transaction isolation may be necessary when the invariant spans a set of rows or depends on queries whose result can change concurrently.

Application-level mutexes can be useful inside one process, but they do not coordinate independent processes unless the architecture guarantees there is only one relevant process. In a horizontally scaled service, each worker can hold its own lock while still racing with another worker.

Distributed locks add operational complexity and have failure modes involving leases, pauses, network partitions, and ownership. Use them when the invariant genuinely spans systems and the lock implementation provides the semantics you require, not as the default fix for a database operation that the database can protect directly.

Do not confuse idempotency with atomicity

The two ideas are related but solve different problems.

An idempotent operation is designed so that repeating the same logical request does not produce additional unintended effects. Atomicity controls whether competing operations can observe and modify shared state in an unsafe interleaving.

An idempotency key can help a service recognize retries of the same operation. It does not automatically protect a shared balance, quota, or one-time token against two different operations racing with each other. Conversely, an atomic balance update does not tell an API that a retried client request represents the same logical payment as an earlier request.

Use each control for the failure it addresses. Sensitive workflows often need both.

Test the guarantee, not just the happy path

A sequential test cannot demonstrate that a concurrency invariant holds. Add tests that create contention around the protected state.

For a one-time recovery code, a useful test starts several operations against the same unused code as close together as the test environment allows. The assertion is not that a particular request wins. The assertion is that exactly one operation obtains the right to continue and every other operation fails without performing the protected effect.

Also test failure boundaries:

What happens if the process stops after claiming the state?
Can a retry safely determine what happened?
Can two application instances enforce the same rule?
Does rollback restore the state when the protected local operation fails?

Concurrency tests can expose defects, but passing them does not prove the absence of races. Review the guarantees of the database, queue, cache, lock, or other coordination primitive being used. A test environment may not reproduce every timing or failure condition that production can encounter.

Operational monitoring should also look for invariant violations rather than only application errors. A duplicate one-time action, an impossible negative quota, or repeated completion of the same workflow can indicate that coordination failed even when every individual request returned a normal status.

Know what atomic state changes do not solve

Atomicity reduces risks caused by unsafe interleaving. It does not establish that the business rule itself is correct.

A perfectly atomic operation can still authorize the wrong user, trust client-controlled values, use an overly broad permission, or perform a dangerous transition by design. Input validation, authentication, authorization, least privilege, and secure workflow design remain separate controls.

Atomicity also does not guarantee availability. Stronger coordination can increase lock contention, retries, latency, or transaction aborts under load. Those are real engineering trade-offs. The right design preserves the security invariant while keeping the coordination scope as small and short-lived as practical.

Make concurrency part of the security model

When a security decision depends on mutable shared state, do not assume that state will remain unchanged between a check and an action. Write down the invariant, identify the operation that commits it, and use a coordination mechanism whose documented guarantees match that requirement.

For simple local state, a conditional update, constraint, or well-scoped transaction is often enough. For workflows that cross service boundaries, add explicit duplicate handling and recovery for partial completion.

The practical test is simple: if two valid-looking requests arrive at the same moment, the system should still enforce the rule you intended. Security logic is complete only when that rule survives concurrency.