Fencing Tokens: Block Stale Lease Holders

A distributed lease can grant one process temporary permission to act, but expiration alone cannot stop that process from acting after its lease has ended. A long pause, network delay, overloaded runtime, or suspended virtual machine can leave an old holder unaware that another process has already acquired the lease.

This creates a subtle safety gap. Two processes can both believe they are entitled to modify the same resource, even when the lease service itself grants ownership correctly.

A fencing token closes that gap at the resource boundary. Each successful lease acquisition receives a strictly increasing number. Every protected operation carries that number. The resource remembers the greatest token it has accepted and rejects operations carrying an older token.

The key shift is simple: do not depend on an expired holder to stop itself. Make the protected resource reject stale authority.

A lease is permission with a deadline

Suppose two workers coordinate access to a report file. Worker A acquires a 30-second lease:

time 00s: A acquires lease, token 41
time 05s: A starts preparing an update
time 10s: A pauses for 35 seconds
time 30s: A's lease expires
time 31s: B acquires lease, token 42
time 35s: B writes the report
time 45s: A resumes and writes its older result

The lease service behaved correctly. It did not issue token 42 until the first lease expired. Yet A still wrote after B because A could not observe expiration while paused.

Checking the clock before the write is not sufficient. A process can pause immediately after the check. Renewing more aggressively also does not eliminate the gap; pauses and delayed messages can occur around any renewal boundary.

The resource needs evidence that can order competing holders.

Add a monotonic token to each acquisition

The lease service maintains a counter. Each acquisition increments it and returns the new value:

acquire("report") -> token 41
acquire("report") -> token 42
acquire("report") -> token 43

The number does not need to represent wall-clock time. It only needs a strict order for successful acquisitions of the protected lease.

A worker includes its token with every mutation:

write_report(token=42, content=...)

The storage layer tracks the highest accepted token:

highest_token = 42

A request with token 41 is stale and must fail. A request with token 42 or a greater valid token can proceed according to the resource’s rules.

This converts an uncertain timing question into an ordering check.

Put the check beside the mutation

The token comparison must be enforced by the system that applies the protected state change, or by a component with an atomic path to that change. A check in application code followed by a separate write can recreate a race.

A relational table can keep the token beside the protected row:

CREATE TABLE report_state (
    report_id      bigint PRIMARY KEY,
    body           text NOT NULL,
    fencing_token  bigint NOT NULL
);

A worker can update only when its token is not older:

UPDATE report_state
SET body = $1,
    fencing_token = $2
WHERE report_id = $3
  AND fencing_token <= $2;

The worker must inspect the affected-row count. Zero rows means the operation did not acquire permission to change that row, assuming the row exists and the token predicate caused the rejection.

For stricter semantics, a stored procedure or transaction can distinguish a missing row from a stale token and return a dedicated error.

The important property is atomicity: comparison and mutation occur as one protected database operation.

Walk through the stale-holder case

Return to the paused workers:

A acquires token 41
A pauses
lease 41 expires
B acquires token 42
B writes with token 42
resource records highest token 42
A resumes
A writes with token 41
resource rejects token 41

A does not need to detect that it is stale before sending the request. The resource detects it.

This matters because process-local state cannot reliably prove current authority after an arbitrary pause. The resource has the final information needed to preserve ordering.

Fencing and mutual exclusion solve different parts

A lease attempts to limit concurrent ownership. A fencing token prevents an older holder from overwriting work performed under a newer acquisition.

These mechanisms complement each other:

A fencing token does not make every operation safe automatically. The protected resource must understand the token, persist enough ordering state, and reject stale values.

Likewise, a lease without fencing can still be useful for best-effort coordination, but it should not be treated as sufficient protection for a critical mutation when stale holders can reach the resource.

Token generation needs one total order

A fencing scheme depends on tokens that increase across successful acquisitions for the same protected domain.

A single transactional counter is conceptually direct:

UPDATE lease_counter
SET value = value + 1
WHERE lease_name = $1
RETURNING value;

Other coordination systems can expose revision numbers, sequence values, or transaction indexes with suitable ordering properties. The exact mechanism can vary, but a token must not be reused after a later token has been issued.

Random identifiers are poor fencing tokens. A UUID can identify an acquisition, but ordinary UUID comparison does not establish acquisition order. The resource needs an order that lets it classify one holder as older than another.

Wall-clock timestamps also require care. Clock skew, limited timestamp resolution, and clock adjustments can undermine the ordering guarantee. A logical monotonic sequence avoids those clock assumptions.

Define the fencing scope explicitly

A token is meaningful only within a defined scope.

If each customer has an independent lease, each customer can have an independent sequence:

customer 17: 8, 9, 10
customer 24: 3, 4, 5

The resource must compare tokens inside the same scope. Comparing customer 17 token 10 with customer 24 token 5 has no useful meaning.

A practical schema often stores both scope and token:

CREATE TABLE customer_export (
    customer_id    bigint PRIMARY KEY,
    payload        jsonb NOT NULL,
    fencing_token  bigint NOT NULL
);

This keeps the ordering rule aligned with the resource being protected.

Treat rejection as a normal concurrency outcome

A stale-token rejection is not necessarily an infrastructure failure. It often means another holder legitimately acquired newer authority.

Code should represent that outcome clearly:

err := store.ReplaceReport(ctx, reportID, token, body)
switch {
case errors.Is(err, ErrStaleToken):
    return nil
case err != nil:
    return fmt.Errorf("replace report: %w", err)
default:
    return nil
}

Whether the worker returns success, abandons the job, or reloads state depends on the business operation. Blindly retrying the same stale token is usually pointless because it can never become current again.

If the work still needs to happen, the process should obtain a new lease and a new token, then recompute any result that depended on old state.

Do not fence only the first write

A holder may perform several mutations while it owns a lease. Every operation that relies on that ownership must carry the token.

Consider a maintenance worker that updates a database record, writes an object, and then publishes a pointer. Fencing only the database update leaves the object or pointer open to stale writes.

Map the full protected boundary:

lease acquisition
      |
      v
 token 57
   /   |   \
  v    v    v
 DB  object pointer

Each destination either needs direct fencing support or must be reached through a component that can enforce equivalent ordering.

This is often the hardest engineering part. Some external systems offer conditional writes or version checks; others do not. If a target cannot reject stale authority, place a fencing-aware service in front of it or redesign the operation so stale writes cannot cause harmful state.

Separate token order from operation order

A higher token establishes newer lease authority, not the desired order of every business event.

Suppose holder 51 sends two valid requests:

token 51, operation A
token 51, operation B

The token alone does not tell the resource whether A or B should be applied first. If ordering within one lease matters, add a separate sequence number or enforce serialization inside that holder’s workflow.

A useful mental model is:

fencing token -> acquisition generation
operation sequence -> order inside that generation

Do not overload one number with guarantees it was not designed to provide.

Plan for token exhaustion and storage types

Use a counter type with ample range. A signed 64-bit integer provides a very large sequence space for most systems, but the design should still define behavior near its maximum.

Never wrap the counter back to zero. Reusing low values after high values destroys the stale-operation test.

Also avoid lossy conversions. If a token passes through JSON, JavaScript, database drivers, or message schemas, confirm that every layer preserves the integer exactly. A decimal string can be safer when a transport cannot represent the full integer range without precision loss.

Observe stale attempts

Fencing failures are valuable operational signals. Track at least:

• stale-token rejection count;
• lease acquisition count;
• lease renewal failures;
• holder identity associated with rejected operations;
• pause or latency indicators around affected workers.

A sudden rise in stale rejections can expose long runtime pauses, overloaded hosts, network disruption, or lease durations that are too short for normal work.

Do not log sensitive payloads merely to diagnose fencing. The token, resource scope, holder identity, and timing data are often enough.

Test the failure sequence, not only the happy path

A useful test deliberately creates a stale holder:

1. acquire token N
2. suspend holder N
3. allow its lease to expire
4. acquire token N+1
5. apply a write with N+1
6. resume holder N
7. attempt a write with N
8. assert that the resource rejects it
9. assert that state from N+1 remains intact

Integration tests should exercise the real atomic comparison used by production storage. A mock that simply returns ErrStaleToken can test caller behavior, but it cannot prove that the storage boundary closes the race.

Also test equal-token behavior, first-write behavior, missing resources, transaction rollback, and large token values.

A compact implementation checklist

Before relying on fencing for a critical resource, verify these properties:

• each successful acquisition receives a strictly increasing token;
• token values are never reused;
• every ownership-dependent mutation carries the token;
• the protected resource compares token and applies mutation atomically;
• the resource persists the greatest accepted token at the correct scope;
• stale-token rejection has an explicit application-level outcome;
• retries do not keep submitting a permanently stale token;
• tests suspend an old holder across lease expiration;
• metrics expose stale attempts and acquisition churn.

If any protected write bypasses the token check, the safety argument has a hole.

Closing perspective

Distributed leases are built on time, and process timing can become uncertain during pauses and network disruption. Fencing tokens add an ordered generation number that survives that uncertainty.

The strongest design gives the protected resource the final decision. A holder presents its token, the resource compares it with accepted authority, and an older holder cannot overwrite state created under a newer acquisition.

That small protocol turns lease ownership from a process belief into an enforceable resource rule.