Fencing Tokens: Stop Stale Lock Holders from Writing

A distributed lock can tell a client that it owns a resource for a limited period. That does not guarantee the client stops acting when the period ends.

A process can pause for garbage collection, lose network access, become descheduled, or stall on an overloaded machine. During that pause, its lease can expire and another client can acquire the same lock. When the first process resumes, it may still believe it is entitled to write.

This is the stale-holder problem. Fencing tokens address it by making the protected resource participate in concurrency control. Each successful lock acquisition receives a token greater than every earlier token. The resource remembers the greatest token it has accepted and rejects operations carrying an older one.

The key shift is simple: a lock grant is not enough. The final write must prove that its grant is newer than competing grants.

A lease alone leaves a gap

Suppose two workers update the same report.

1. Worker A acquires a lease.
2. A pauses for long enough that the lease expires.
3. Worker B acquires the lease and writes a fresh report.
4. A resumes and writes its older result.

From the lock service’s perspective, nothing unusual happened. A’s lease expired before B received ownership. The problem occurs at the report store, which cannot distinguish a current owner from a stale one.

Checking the lease immediately before writing does not fully close this gap. A client can pass the check and then pause before the write reaches storage. In a distributed system, there is always another interval between a check and the action it is intended to authorize.

A robust design moves the decisive check to the resource that accepts the mutation.

Turn lock generations into authority

A fencing service assigns an increasing integer to each successful acquisition:

Worker A acquires lock -> token 41
Worker A pauses

lease 41 expires

Worker B acquires lock -> token 42
Worker B writes with 42 -> accepted

Worker A resumes
Worker A writes with 41 -> rejected

The resource does not need to know whether a lease is still active. It only needs one rule:

accept token T only when T >= greatest accepted token

For operations that must be unique per generation, use a stricter rule appropriate to the resource. The essential property is that an operation from an older generation cannot overwrite state established by a newer generation.

The token therefore acts as an ordering proof, not as a secret credential.

The storage boundary must enforce the rule

Generating tokens without checking them at the mutation boundary provides no protection.

Consider a database row that stores both application state and the latest accepted fencing token:

UPDATE account_projection
SET
    balance_cents = :balance,
    fence_token = :token
WHERE account_id = :account_id
  AND fence_token < :token;

If token 42 has already updated the row, a later request carrying token 41 affects zero rows. The stale worker can detect that result and stop.

The comparison and mutation must be atomic from the resource’s perspective. Reading the current token in one operation and writing in another recreates a check-then-act race.

Some resources can enforce this directly with conditional updates, compare-and-set operations, transaction predicates, object versions, or a small gateway that serializes access. If the final resource cannot reject stale generations, the fencing design is incomplete.

Tokens need a strict generation order

Fencing depends on acquisition generations being ordered consistently.

A counter in a transactional database is a straightforward option:

UPDATE lock_sequence
SET value = value + 1
WHERE lock_name = :name
RETURNING value;

A consensus-backed lock service can also expose a revision, transaction index, or equivalent sequence value with suitable ordering guarantees.

Wall-clock timestamps are a poor substitute. Clocks can differ between machines, move after synchronization adjustments, and have insufficient resolution for concurrent acquisitions. Random identifiers are unique but provide no useful age ordering.

The requirement is not merely uniqueness. A later successful acquisition must carry a token that compares greater than an earlier one.

Fencing and leases solve different parts

Leases remain useful. They let the lock service grant ownership to another client after the current holder disappears.

Fencing handles a separate concern: an expired holder may continue executing.

Together, the responsibilities look like this:

This separation is important. Extending lease duration can reduce the chance of overlap, but it cannot prove that an old process has stopped. Heartbeats can detect many failures, but detection itself can be delayed or partitioned.

The resource-side token check is the part that prevents an older generation from committing after a newer one.

Carry the token through every relevant write path

A common implementation error is to fence the primary write but leave secondary effects unfenced.

Imagine a worker that updates a database and then uploads a generated file. The database may reject stale tokens correctly while the object store accepts an old upload. The system is still vulnerable through the second path.

Map every side effect protected by the lock:

acquire token
   |
   +--> database mutation
   |
   +--> object replacement
   |
   +--> external command

Each effect needs an enforcement strategy. Sometimes that means passing the token directly. Sometimes it means routing mutations through a service that stores the latest generation. In other cases, a resource offers its own conditional-write primitive that can serve the same purpose.

If one effect cannot be fenced, design the workflow so a stale effect cannot become authoritative. For example, write immutable objects under generation-specific names and update a fenced pointer to select the current object.

Make rejection an expected outcome

A stale-token rejection is not an infrastructure accident. It is evidence that another ownership generation has superseded the caller.

Client code should model that explicitly:

err := store.Apply(ctx, accountID, projection, fenceToken)

switch {
case errors.Is(err, ErrStaleFence):
    return ErrOwnershipLost
case err != nil:
    return err
default:
    return nil
}

Blindly retrying the same stale token is incorrect. The token will remain stale.

If the operation still needs to run, the worker must return to the acquisition protocol, obtain a new generation, and usually recompute any state that depended on the earlier snapshot. Reusing old computed output under a new token can preserve the same logical race under a fresh number.

Keep token scope aligned with the protected resource

A fencing sequence needs a clear scope.

If a lock protects one customer account, a per-account sequence can be enough. If one lock protects a whole shard, the generation applies to that shard. A single global counter can work technically, but it may create unnecessary contention and operational coupling.

The lock key, token sequence, and resource-side comparison should describe the same ownership domain.

For example:

lock key:       invoice:7831
token sequence: invoice:7831
resource check: invoice 7831 latest token

Mismatched scopes can produce subtle bugs. A token that is newer globally may have no meaningful relationship to ownership of an unrelated resource.

Watch for token reset

Monotonicity must survive the lifetime in which old clients can return.

If a lock service restarts and resets its counter from 9000 to 1, a resource that has already accepted 9000 will reject every new write. Worse, clearing the resource-side token at the same time can allow an ancient client with token 8999 to become authoritative again.

Persist the sequence or derive it from a durable ordering mechanism. Treat sequence resets as protocol changes, not routine cleanup.

Integer exhaustion is rarely practical with a sufficiently wide counter, but wraparound must not silently reverse ordering.

A concrete stale-worker example

Consider a thumbnail pipeline. Only one worker should publish the current thumbnail for an image.

Worker A gets token 17, downloads the source, and starts rendering. Its machine freezes for two minutes. The lease expires.

Worker B gets token 18, renders from a newer source revision, and publishes successfully.

A then resumes. Without fencing, its older thumbnail can replace B’s output.

A safer publication model stores immutable render outputs and fences only the pointer:

renders/image-52/17.jpg
renders/image-52/18.jpg

current pointer:
image-52 -> renders/image-52/18.jpg
fence    -> 18

A may still upload 17.jpg; that file is harmless because it is not authoritative. Its attempt to move the current pointer with token 17 is rejected.

This pattern is useful when the underlying blob store does not support a custom fencing predicate on object replacement but an associated metadata store does.

Fencing is not a general transaction protocol

Fencing prevents stale ownership generations from committing protected mutations. It does not make a multi-resource workflow atomic.

If an operation writes to three independent systems, token checks can stop old holders at each boundary, but partial completion is still possible. Compensation, idempotency, transactional messaging, or another coordination strategy may still be required.

Fencing also does not repair application-level conflicts that occur inside one valid ownership generation. It answers a narrow question: is this caller’s ownership generation at least as current as the generation already accepted here?

Keeping that scope precise prevents the mechanism from becoming a vague promise of distributed safety.

Testing the failure that matters

A useful test deliberately pauses an old holder across lease expiry.

The scenario should establish this order:

A gets token 7
A pauses before commit
A lease expires
B gets token 8
B commits
A resumes
A commit is rejected
final state is B's state

Do not test only that two ordinary workers avoid simultaneous execution. The important case is a worker that resumes after its authority has expired.

Also test token persistence across service restart, atomic resource-side comparison, and every protected write path. These tests target the protocol boundaries where stale ownership can escape.

Operational signals

Fencing rejections deserve metrics and structured logs. A small number can be normal during pauses, failover, or aggressive lease settings. A sustained increase can indicate overloaded workers, long stop-the-world pauses, network instability, or leases shorter than normal operation latency.

Useful fields include the resource key, rejected token, latest accepted token, acquisition generation, and operation type. Avoid treating the rejection as an opaque storage error; it carries direct concurrency information.

The signal can also expose incorrect client behavior. Repeated submissions with the same rejected token often indicate a retry loop that should instead reacquire ownership.

When to use fencing tokens

Fencing is a strong fit when work is protected by a lease or distributed lock, a stale process can resume, and the authoritative resource can compare ownership generations atomically.

It is especially valuable for long-running jobs, schedulers, controllers, shared-file updates, leader-elected writers, and systems in which process pauses are realistic.

For a single database transaction, native row locking or optimistic concurrency is often simpler. For immutable writes with no shared authoritative pointer, stale overwrites may not exist. Do not add a distributed lock and fencing protocol when a local concurrency primitive already gives the required guarantee.

The design rule

A distributed lock cannot force a paused process to stop.

Design as if an expired holder can return at any moment. Give each ownership generation a monotonically increasing token, carry that token to the protected mutation, and make the resource reject older generations atomically.

That turns ownership from a claim held only in client memory into an ordering rule enforced at the place where stale work could otherwise become real.