Request Coalescing Collapses Concurrent Cache Misses into One Fill

A cache can reduce steady-state backend traffic yet amplify work at the instant a popular entry expires. If one hundred requests observe the same missing key before any replacement value is stored, a conventional lookup path can send one hundred equivalent reads to the origin. The cache is functioning according to its lookup rules; the amplification comes from concurrency around the empty interval.

Request coalescing changes that interval. The first caller for a key starts the fill, while later callers for the same key attach to that in-flight operation instead of starting equivalent work. When the operation completes, its result is distributed to the waiting callers and, when appropriate, stored in the cache.

The mechanism controls duplicate work. It does not extend the lifetime of cached data, define freshness, or make a failed origin request succeed.

The coordination key defines the sharing boundary

Coalescing requires a key that identifies operations safe to share. For a simple object cache, that key may be identical to the cache key. For an HTTP response cache, it may need to include representation dimensions such as locale, encoding, authorization scope, or selected request headers.

Two requests should share an in-flight fill only when one computed result is valid for both. A coordination key that is broader than the data identity can return a result produced for the wrong variant. A key that is unnecessarily narrow preserves correctness but loses deduplication opportunities.

This makes request coalescing a data-identity problem as much as a concurrency problem. The same canonicalization rules used for cache identity are often a sound starting point, but the equality relation must match the actual operation being shared.

The in-flight entry has a shorter lifetime than the cached value

A coalescer typically maintains a map from coordination key to an in-flight record. That record can contain a completion signal plus either a result or an error. Its lifetime begins when the first caller claims the key and ends after the shared operation completes and waiters can observe its outcome.

Conceptually, the critical section is small:

lock inflight map
if key exists:
    join existing operation
else:
    create operation for key
    mark current caller as owner
unlock inflight map

The expensive backend call must not run while the global map lock is held. Otherwise unrelated keys become serialized behind one slow fill. Coordination protects ownership of the in-flight record; backend execution occurs outside that lock.

After completion, the owner publishes the outcome, wakes waiters, and removes the in-flight entry. Removal must be ordered so a new caller cannot accidentally join an operation whose result is no longer publishable. Concrete implementations achieve this with mutexes, futures, promises, channels, or library-specific single-flight primitives.

Coalescing and caching solve different state transitions

A cache answers whether a previously produced value can be reused. A coalescer answers whether a computation already in progress can be shared. Those are separate state machines.

A request can encounter several combinations:

The last row is a different policy from blocking coalescing. A stale-while-revalidate design can let callers continue using an older acceptable value while one actor refreshes it. Pure miss coalescing instead makes callers wait when no reusable value exists. Both can suppress duplicate origin work, but their latency and freshness behavior differ.

Failure is shared unless the design says otherwise

If the single fill fails, every caller attached to it can receive the same failure. Coalescing therefore changes the correlation of failures: many callers no longer make independent origin attempts during the same overlap window.

That property is usually desirable when immediate retries would only duplicate pressure, but it needs explicit treatment. A coalescer is not a retry policy. If callers automatically retry the shared error without backoff or another admission rule, a new burst can form immediately after the failed in-flight entry disappears.

Negative caching is also separate. Keeping a not-found result or an error for a configured interval changes future cache behavior. Coalescing alone retains the outcome only long enough to deliver it to callers already attached to the operation.

Cancellation exposes an ownership question

Shared work complicates cancellation. If the first caller disconnects, cancelling the backend operation immediately can also fail callers that joined later and still need the result. Treating the first caller’s context as permanent ownership of the shared operation therefore couples unrelated request lifetimes.

One design gives the in-flight operation an independent context and tracks waiter interest. Another keeps the fill alive once started, subject to its own deadline. A more elaborate implementation may cancel backend work only after every waiter has left.

There is no universal cancellation rule. The correct contract depends on the cost of continuing the fill, backend cancellation support, and whether a completed value remains useful for the cache after the initiating request disappears. What matters is that cancellation semantics belong to the shared operation, not accidentally to whichever caller happened to arrive first.

Per-key coordination limits the blast radius

A single global lock around all cache misses prevents duplicate fills, but it also couples unrelated keys. A slow fill for one key can delay a miss for another even when the backend could process both independently.

Per-key in-flight records avoid that serialization. The global structure is touched briefly to find or install a record, while each key has its own completion state. This keeps the coordination scope aligned with the unit of duplicated work.

The number of in-flight records still needs a resource bound in systems exposed to high-cardinality or attacker-controlled keys. Coalescing a million distinct misses provides almost no deduplication and can itself consume substantial memory. Admission control, key validation, concurrency limits, or bounded metadata may be required around the coalescer.

The observable metric is concurrent duplication

Cache hit ratio alone does not show whether coalescing is effective. A system can have a high hit ratio and still generate sharp backend bursts when a few hot entries expire together.

Useful observations include the number of fills started, the number of callers joined to existing fills, waiter counts per key, fill duration, shared failures, and backend concurrency during expiry events. These measurements separate ordinary cache misses from misses that would otherwise have become duplicate work.

The distinction also matters when tuning timeouts. A long fill with many waiters may indicate a slow origin rather than a cache defect. Coalescing prevents those waiters from multiplying backend requests, but it cannot reduce the latency of the one operation they all depend on.

Duplicate suppression is bounded by overlap

Request coalescing only merges work that exists at the same time. Once an in-flight record has completed and disappeared, a later miss starts another fill unless the cache now contains a reusable value. The mechanism therefore provides no durable deduplication across separate time windows.

That boundary keeps the abstraction narrow. Cache policy decides whether a completed result remains reusable. Retry policy decides whether failed work should be attempted again. Admission control limits how much work may enter the system. Request coalescing handles one specific concurrency condition: several callers currently want the same absent result, and one execution can satisfy them all.