Request Coalescing Stops Cache Misses from Multiplying Backend Work

A cache miss is usually cheap when one caller causes one backend lookup. The same miss can become expensive when many callers arrive for the same key at nearly the same time. Each caller observes the key as absent, each starts identical work, and the backend receives a burst precisely when the cache is providing no protection for that key.

Request coalescing changes that concurrency pattern. The first caller becomes the leader for a key. Later callers join the same in-flight operation and wait for its result rather than starting equivalent work. Once the fill completes, the result can populate the cache and be returned to the waiting callers.

A cache miss can amplify load

Suppose a popular entry expires while 200 requests for that key arrive within a short interval. Without coordination, all 200 requests may query the database or remote service.

without coalescing

request A --miss--> backend
request B --miss--> backend
request C --miss--> backend
request D --miss--> backend

The cache has turned one expired entry into many concurrent fills. This pattern is often called a cache stampede. It can raise backend concurrency, consume connection pools, and increase latency for unrelated traffic that shares the same dependency.

The important dimension is not total cache miss rate alone. A modest miss rate concentrated on one hot key can create more damaging concurrency than a larger miss rate spread across many independent keys.

Coalescing is keyed synchronization

A coalescer keeps track of work already running for a key. A new caller checks that in-flight registry before starting a fill.

request A --miss--+--> fill(key) --> backend
request B --miss--|
request C --miss--+--> wait for same fill
request D --miss--|

A minimal state machine has only a few transitions:

absent -> in-flight -> completed -> absent
                    \-> failed  -> absent

The completed value normally belongs in the cache, not in the in-flight registry. The registry coordinates concurrent work; the cache controls reuse after that work has finished.

Implementations must make registration atomic. Two callers that both observe no in-flight entry and both install a leader have already lost the property the mechanism is meant to provide. A mutex, concurrent map primitive, actor, or equivalent serialization point can protect that transition.

Coalescing does not make one result permanently shared

Followers share a particular execution, not an unlimited lifetime of data. Once the leader finishes and the in-flight entry is removed, a later cache miss may start a new fill.

That distinction keeps freshness policy separate from concurrency control. TTL, explicit invalidation, version checks, and stale-serving rules still determine whether a cached value may be reused. Coalescing only determines whether simultaneous callers should duplicate the same fill.

The key used for coalescing therefore needs to represent the operation accurately. If authorization scope, locale, query parameters, tenant, representation format, or another input changes the result, that input may need to participate in the key. Combining semantically different requests can return data to the wrong caller.

Failure needs an explicit fan-out policy

If the leader fails, every follower waiting on that execution needs a defined outcome. The simplest policy propagates the same error to all waiters and removes the in-flight entry so a later request can try again.

Immediate independent retries from every follower defeat the coordination. A failed fill followed by hundreds of simultaneous retries simply moves the stampede to the failure path.

A service can combine coalescing with bounded retries, backoff, jitter, stale data, or a short negative cache where those choices match the data contract. These mechanisms solve different parts of the problem: coalescing limits duplicate concurrent work, while retry and caching policy govern subsequent attempts.

Caller cancellation and shared work are separate decisions

One follower may lose interest before the shared fill completes. Its HTTP connection may close or its deadline may expire. Cancelling the underlying operation immediately can be wrong because other callers still depend on it.

A coalescer therefore needs a cancellation policy. Common choices include keeping the leader alive until its own deadline, tracking active waiters and cancelling only when none remain, or using a fill context independent from individual caller contexts.

caller A cancelled ----X
caller B waiting -------+--> shared fill continues
caller C waiting -------+

The right policy depends on the cost and semantics of the backend operation. What matters is that one caller’s lifecycle does not accidentally control unrelated callers that happen to share the same fill.

A slow leader can hold many followers

Coalescing reduces backend work, but it also creates a temporary dependency on one execution. If that execution stalls, every follower for the key waits behind it.

The fill needs a bounded deadline. Backend timeouts still apply, and telemetry should expose both leader duration and follower wait time. A large waiter count attached to a slow fill is a useful saturation signal even if backend request count remains low.

Some systems permit a second fill after a threshold rather than allowing all callers to remain behind one unusually slow leader. That is a deliberate trade: limited duplicate work can reduce tail latency, but too aggressive a threshold recreates the original amplification. Any such policy needs a strict concurrency bound per key.

Process-local coordination has a process-local boundary

An in-memory coalescer only merges requests that reach the same process. With 20 application instances, a hot miss can still produce as many as 20 backend fills even when each instance performs perfect local coalescing.

That may be acceptable. Reducing thousands of fills to tens can remove most of the pressure without adding distributed coordination to the request path.

Cross-process coalescing requires another coordination mechanism, such as a shared lock or a cache feature with suitable atomic operations. That adds failure modes around ownership, expiry, network partitions, and stale holders. A distributed lock should not be introduced merely to obtain a theoretical single fill if a bounded number of per-instance fills is operationally safe.

Cache refresh can avoid the miss path entirely

Coalescing is especially useful for unpredictable misses, but hot entries can sometimes be refreshed before expiry. Refresh-ahead starts new work while the existing value remains usable, reducing the chance that callers encounter an empty cache at the same instant.

Stale-while-revalidate follows a related pattern: one request refreshes an expired or aging entry while other callers temporarily receive an allowed stale value. This can remove follower waiting as well as duplicate backend work, provided the product contract permits stale data.

These policies require bounds. Refreshing every rarely used key wastes capacity, and serving stale values indefinitely can hide backend failure. Popularity thresholds, maximum stale age, and refresh deadlines keep the behavior finite.

Metrics need to show suppression as well as misses

A high cache hit ratio can coexist with severe stampedes if the remaining misses are concentrated on a few keys. Useful telemetry includes fills started, followers joined, waiter count, fill duration, follower wait duration, fill failures, and the ratio of suppressed fills to actual backend fills.

Raw cache keys may contain sensitive or high-cardinality values, so metrics should avoid using arbitrary keys as labels. Aggregation by route, cache namespace, operation, or bounded key class is usually safer. Detailed key diagnostics can go to sampled traces or controlled logs when appropriate.

A coalescer that reports only backend calls hides the amount of demand it absorbed. Suppression metrics show whether the mechanism is reducing duplicate work or merely adding synchronization around traffic that was already independent.

One in-flight fill can absorb a burst

Caches reduce repeated work across time. Request coalescing reduces repeated work across concurrency. The distinction matters when a popular key disappears, expires, or becomes cold on a newly started process.

A sound implementation atomically elects one leader per key, gives followers a bounded wait, separates caller cancellation from shared execution, removes failed or completed in-flight state, and keeps the coalescing key aligned with result semantics. Distributed coordination is optional rather than automatic; process-local suppression can already provide a large reduction in duplicate work.

The mechanism does not add backend capacity. It prevents simultaneous callers from spending that capacity on the same fill when one execution can serve them all.