Fencing Tokens Stop Stale Lease Holders

A distributed lease gives one worker temporary permission to act as an owner. The lease eventually expires so another worker can take over after a crash or network failure. That solves availability, but expiry alone does not guarantee that the old worker has stopped.

A process can pause long enough for its lease to expire, then resume with stale local state. A long garbage-collection pause, scheduler stall, suspended virtual machine, or delayed network path can create this condition. If the old worker writes after a replacement has taken ownership, two workers can affect the same resource even though the lease service never considered both leases valid at the same instant.

A fencing token closes that gap by making ownership order visible to the resource receiving writes.

Lease expiry cannot revoke a paused process

Consider worker A holding a lease:

time ---->

A: acquire lease 41 ---- pause ---------------------- resume -> write
                         |
                         | lease expires
                         v
B:                    acquire lease 42 -> write

Worker A does not receive a magical stop signal when lease 41 expires. During a pause it may not run code, receive messages, or check a deadline. When it resumes, its memory can still say that it owns the job.

Checking the lease immediately before a write narrows the race but does not remove it. The process can pause after the check and before the protected operation reaches storage.

The correctness decision therefore cannot live only in the worker.

Each ownership grant gets a larger token

A fencing token is a monotonically increasing number associated with a successful ownership grant. A later owner receives a token greater than every earlier owner for the same protected scope.

worker A -> lease granted, token 41
worker B -> later lease granted, token 42
worker C -> later lease granted, token 43

Every write carries the token. The protected resource remembers the highest token it has accepted and rejects an operation carrying an older one.

WRITE resource=X token=42  -> accept
WRITE resource=X token=41  -> reject as stale

The resource does not need to know whether worker A is alive. It only needs to compare ownership epochs.

The token must come from a source that can provide the required ordering. A random UUID is useful as a unique identifier but does not provide a greater-than relation for fencing.

Enforcement belongs at the protected resource

Generating tokens without checking them at the write boundary provides no fencing. The database, storage service, coordinator, or another authoritative component must reject stale tokens.

A relational table can keep the latest accepted epoch:

UPDATE jobs
SET output = :output,
    fence_token = :token
WHERE job_id = :job_id
  AND fence_token < :token;

The application then checks that exactly one row was updated. A stale token affects zero rows.

For a resource that supports conditional writes, the same rule can be expressed through a version field or compare-and-set operation. The important property is atomic comparison and mutation. A separate read followed by an unconditional write recreates a race.

If the final destination cannot enforce token order, another component may need to mediate access. That mediator becomes part of the correctness boundary and must itself provide durable ordering and atomic enforcement.

Tokens and leases solve different parts of the problem

The lease controls liveness: it allows ownership to move when the current holder stops renewing. The fencing token controls safety at the resource: an earlier owner cannot overwrite work from a later owner after takeover.

Using only fencing tokens without a lease does not decide when another worker may take over. Using only a lease does not stop delayed operations from an expired holder.

Together, the flow becomes:

1. acquire lease -> token 57
2. perform work
3. write with token 57
4. renew lease while active

after expiry:
5. replacement acquires lease -> token 58
6. resource accepts token 58
7. delayed token 57 is rejected

A worker should still stop when renewal fails. Fencing is the last safety boundary, not a reason to keep issuing known-stale operations.

Token scope must match ownership scope

A single global counter is simple but can create unnecessary contention. Many systems only need ordering within one resource, shard, tenant, or job.

If workers independently own job-A and job-B, their tokens do not need a meaningful ordering across both jobs. A per-resource sequence can preserve the required property with less coordination.

The resource must compare tokens within the same scope used by the lease service. Mixing scopes can reject valid writes or accept stale ones.

Token persistence also matters across coordinator restart. Reissuing a smaller token after losing counter state can let a stale holder appear newer than a fresh one. The sequence therefore needs durability or another construction that preserves monotonic ownership epochs across failover.

Side effects outside the fenced store remain separate

A fenced database update does not automatically fence an email, payment provider call, filesystem write, or arbitrary external API. The token protects only boundaries that validate it.

Suppose a worker sends a remote command and later records completion in a fenced table. A stale worker may still reach the remote service before its database update is rejected. If that remote effect must also exclude stale owners, the remote service needs compatible idempotency, fencing, or another coordination mechanism.

This limitation should shape the workflow. It is often safer to commit a fenced state transition first and hand external work to a durable mechanism designed for retries than to assume one fenced row protects every downstream effect.

Long work needs renewal without trusting renewal alone

Lease duration must tolerate normal scheduling and network variation while still permitting timely takeover. A very short lease increases false expiry during ordinary pauses. A very long lease delays recovery after genuine failure.

Workers usually renew before the deadline and stop initiating new work after renewal fails. Operations already in flight remain the difficult case, which is precisely where fencing matters.

Clock assumptions also deserve care. A centralized lease service can decide expiry using its own time base rather than trusting client clocks. The fencing sequence then provides ownership order independent of wall-clock timestamps.

Metrics should reveal stale activity

Useful telemetry includes lease acquisition rate, renewal failures, takeover count, token allocation, stale-write rejection count, and the age of active leases.

A stale-write rejection is not merely noise. It proves that an old owner attempted an operation after a newer epoch existed. Occasional rejections may be an expected consequence of failover, while a sustained rate can indicate process pauses, network delays, overloaded workers, or an overly aggressive lease duration.

Logs should include resource scope, presented token, accepted token, and operation identity without exposing sensitive payloads. Those fields make ownership transitions traceable during incident analysis.

Fencing moves authority to the write boundary

The core failure is not that two leases overlap in the coordinator. It is that an old holder can continue acting after its authority has expired. No lease service can erase stale state from a paused process.

Fencing tokens make later ownership dominate earlier ownership at the place where state changes. A lease can then handle takeover for liveness while the protected resource rejects delayed work for safety. The combination turns an advisory notion of current ownership into an enforceable ordering rule.