A distributed lease can transfer ownership without stopping the process that previously held it. A worker may pause long enough for its lease to expire, then resume after another worker has acquired the same lease. At that point both processes can execute code that was written under the assumption of exclusive ownership.
Lease expiry settles ownership in the coordination service. It does not revoke CPU time, cancel an in-flight network request, or erase buffered I/O on the former holder. Fencing tokens address that gap by carrying an ordering value from the ownership decision to the resource being protected.
Lease expiry does not terminate the old holder
Consider a worker that acquires a lease, reads an object, performs a long computation, and writes a replacement. Its lease has a finite duration so another worker can take over after a crash.
The same takeover can occur when the first worker has not crashed. A long runtime pause, overloaded scheduler, network partition, or delayed lease-renewal request can make the coordination service consider the lease expired. A second worker then acquires it legitimately.
The first worker can later resume. Local state still says that it acquired the lease earlier, and code already past the acquisition check may continue toward the protected resource. Rechecking the lease before every write narrows some timing windows but cannot make a check and a remote write atomic across independent systems.
This is the stale-holder problem: exclusivity has changed at the coordinator, while an earlier holder still has the ability to issue operations.
A token turns ownership order into resource state
A fencing scheme assigns a monotonically increasing token to each successful acquisition. If worker A receives token 41 and a later owner receives token 42, the ordering is explicit.
Every operation that requires lease ownership carries the token to the protected resource. The resource remembers the greatest token it has accepted for that ownership domain and rejects an operation carrying a lower token.
A simplified rule is:
if request.token < resource.highest_token:
reject request
apply request
resource.highest_token = max(resource.highest_token, request.token)Suppose worker A pauses after receiving token 41. Its lease expires, worker B acquires the lease with token 42, and B writes successfully. When A resumes and sends its delayed write with token 41, the resource rejects it because 41 is older than 42.
The safety property comes from enforcement at the resource. A token generated by the coordinator but ignored by the storage system is only metadata.
Monotonicity matters more than wall-clock time
A fencing token represents acquisition order, not elapsed time. Sequence numbers are therefore a better fit than timestamps when the coordinator can allocate them consistently.
Wall clocks can differ across hosts and can move relative to one another after synchronization adjustments. Even tightly synchronized clocks do not by themselves make two lease acquisitions part of one serialized sequence. The required property is simpler: a later successful acquisition must receive a token greater than every earlier successful acquisition in the same fencing domain.
The token does not need to increase by exactly one. Gaps are harmless if ordering remains strict and values are not reused. A database sequence, consensus-backed revision number, or another serialized counter can provide the required ordering when its documented semantics fit the lease design.
Token wraparound and reuse must be excluded for the operational lifetime of the domain. A finite integer representation is acceptable only when the allocation rate and reset rules make reuse impossible while older operations can still arrive.
The protected resource must compare tokens atomically with mutation
Checking a token in an application process and then issuing an unconditional write to storage leaves another race. A newer owner can write between that check and the older owner’s mutation.
The comparison belongs at the point that serializes the protected state. For a database row, the token can be stored beside the data and included in the conditional mutation:
UPDATE account_snapshot
SET payload = :payload,
fence_token = :token
WHERE id = :id
AND fence_token < :token;The caller must inspect the affected-row count. Zero rows can mean that a newer token has already reached the row, subject to any other predicates in the statement.
For an object store, queue, device controller, or external API, equivalent protection requires an operation that the resource itself can order against the token. If the interface offers no conditional mutation or token-aware command, an application cannot manufacture atomic fencing around an unconditional remote side effect.
That boundary is significant. Fencing is not solely a lock-service feature; it is a contract spanning token allocation and resource enforcement.
Token scope has to match the ownership scope
A single global counter can fence many resources, but global allocation is not required. What matters is that operations that can conflict are comparable.
If leases are independent per customer, shard, or object, each domain can have its own monotonic sequence provided an old token can never be mistaken for a current token within that domain. The resource needs enough identity in the request to associate the token with the correct sequence.
Using tokens from unrelated domains as though they shared one ordering can produce false rejection or false acceptance. The tuple is conceptually closer to (ownership_domain, token) than to a bare integer.
The same issue appears after destructive resets. Recreating a lock record with its counter reset to zero can make a new owner appear older than delayed traffic from the previous generation. A generation identifier or a counter whose state survives recreation prevents that ambiguity.
Fencing does not make every side effect reversible
A resource can reject a stale operation only if the operation reaches an enforcement point before the irreversible effect occurs. This works naturally for state mutations that support conditional ordering.
Some side effects do not expose such a boundary. An email already handed to an external delivery service cannot be recalled by a later fencing token. A payment request accepted by a third-party API cannot be retroactively fenced unless that API participates in the protocol or provides an equivalent idempotency and conditional-state contract.
Systems with mixed effects often separate the fenced state transition from downstream work. For example, a fenced database transaction can record an outbox entry, after which message delivery follows its own duplicate-control and idempotency rules. The fence protects ownership of the state transition; it does not silently extend across systems that never inspect the token.
A lease and a fence solve different failure windows
The lease still has operational value. It limits how many actors normally perform the work, provides takeover after failure, and can reduce redundant execution. The fencing token protects the resource when that coordination assumption is temporarily false.
This distinction also changes failure handling. Losing a lease should normally stop new work from being started by that holder, but correctness does not depend on instantaneous self-termination if every protected mutation is fenced. Conversely, perfect lease-renewal logic cannot replace resource-side fencing when stale operations can survive beyond ownership.
The combined contract is precise: the coordinator serializes acquisitions and emits increasing tokens; holders attach those tokens to protected operations; the resource atomically rejects operations older than the greatest accepted token for the same domain. Each part covers a separate boundary, and omitting the final enforcement step leaves stale writers with a path to mutate state.