Request Coalescing Collapses Cache-Miss Bursts

A cache miss is usually cheap when one caller triggers one backend read. The same miss can become expensive when hundreds of callers arrive for the same key before the first fill completes. Each caller sees an empty cache and starts equivalent work, multiplying load precisely when the cached value is unavailable.

Request coalescing places a small concurrency boundary around that fill. The first caller starts the backend operation. Later callers for the same key join the in-flight operation instead of starting another one. When it completes, the result can populate the cache and be delivered to the waiting callers.

without coalescing

A -- miss --> backend
B -- miss --> backend
C -- miss --> backend

with coalescing

A -- miss --+
B -- miss --+--> one fill --> backend
C -- miss --+

The mechanism is often called single-flight suppression. Its useful scope is narrow: duplicate concurrent work for the same logical key.

The in-flight table is separate from the cache

A cache stores completed values. A coalescer tracks work that has started but has not completed. Combining those roles conceptually can produce awkward lifetime rules.

A simple implementation keeps an in-flight map keyed by the same identity used for the fill:

inflight[key] = promise for current fill

On a miss, a caller checks the map. If an entry exists, it waits on that operation. Otherwise it installs a new entry and becomes the caller responsible for the fill. Completion removes the in-flight entry, regardless of success or failure.

Removal matters. A failed promise left in the map can turn a transient backend error into a persistent local failure. Conversely, deleting the entry before completion can reopen the duplication window.

The map also needs atomic check-and-install semantics. Two callers that both observe absence before either publishes its operation can still start duplicate fills. A mutex, per-key synchronization primitive, or library single-flight facility can provide the required serialization.

Key identity defines which work may be shared

Coalescing is safe only when callers grouped under one key are asking for equivalent work.

A URL alone may be insufficient if the response also depends on tenant, authorization scope, locale, feature flags, query parameters, or a version selector. If those inputs affect the backend result, they belong in the coalescing identity unless another layer has already normalized them into an equivalent request.

bad key:
  /profile

possible key:
  tenant + user_id + representation_version

This is the same class of concern as cache-key design, but the consequence can appear before anything is cached: one caller may receive the result initiated for another caller.

The safest design derives both cache lookup and in-flight identity from a shared canonical key function. That reduces the chance that cache equivalence and coalescing equivalence drift apart.

Cancellation needs caller-level semantics

Several callers can share one backend operation while having different deadlines. Cancelling the shared operation when any one waiter leaves is usually too aggressive. A short-deadline caller could abort useful work for every other waiter.

A common policy lets each waiter stop waiting independently while the fill continues as long as useful demand remains. More elaborate implementations can count active waiters and cancel the backend operation after the last waiter departs.

Neither policy is universal. A backend operation may be cheap enough to finish even with no waiters, especially if its result will populate a cache. An expensive operation may justify cancellation once no caller can use its result.

The important separation is between a caller’s waiting lifetime and the shared fill’s lifetime. Sharing execution does not require sharing every caller’s deadline.

Errors should not become accidental long-lived cache entries

If the backend fill fails, all current waiters may receive the same error. That is a natural consequence of sharing one attempt. It does not mean the error should remain available to later callers for an arbitrary period.

After a failed fill, removing the in-flight entry allows a later request to make a fresh attempt. Systems that deliberately cache negative results or errors need an explicit policy with its own TTL and error classification.

Immediate fresh attempts can still create pressure during a backend outage. Coalescing limits concurrency per key while an attempt is active, but it does not by itself limit the rate of sequential failed attempts. Retry backoff, admission control, circuit breaking, or short negative caching may be needed for that separate problem.

Coalescing changes latency distribution for followers

The first caller pays the full fill latency. Followers arriving during that interval wait for the same completion, so their observed latency depends on when they join.

A follower arriving near the end of the fill may return quickly. A follower arriving just after the fill starts may wait almost as long as the leader. Coalescing therefore reduces duplicated backend work; it does not make a slow fill fast.

Metrics should distinguish leaders from followers:

fill_started_total
coalesced_waiter_total
inflight_keys
waiters_per_key
fill_duration
follower_wait_duration
fill_error_total

A high waiter count on one key can reveal a hot-key event even when backend request volume looks modest because suppression is working.

Hot keys can still consume local resources

Collapsing 10,000 backend calls into one is a large reduction in backend load, but 10,000 local waiters still exist. They occupy tasks, futures, request state, memory, sockets, or other runtime resources.

For that reason, coalescing complements rather than replaces bounded admission. A service may cap total concurrent requests, limit waiters per key, or reject followers whose deadlines are too short to justify joining an existing fill.

The in-flight map also needs bounded lifecycle behavior. Keys must disappear after operations finish, and fill operations need deadlines or another termination policy so a backend call that never completes does not pin an entry indefinitely.

Cache expiry can be softened before a stampede forms

Coalescing is most visible after a value has become unavailable, but cache policy can reduce the size of the burst that reaches it.

TTL jitter prevents many unrelated keys from expiring at the same instant. Refresh-ahead can update a popular value before hard expiry. Stale-while-revalidate can serve an acceptable stale value while one caller refreshes it in the background.

Those policies have different freshness semantics. Coalescing does not grant permission to serve stale data, and refresh-ahead does not make every value safe to reuse. Each technique controls a different part of the load and freshness tradeoff.

Even with those policies, a per-key in-flight guard remains useful when a fill can still be triggered concurrently after eviction, invalidation, process restart, or a cold deployment.

The cache remains an optimization boundary

Request coalescing should not be used to create correctness that the backend lacks. If two writes must be serialized, or a read must observe a specific transaction order, a cache-fill coalescer is the wrong coordination primitive.

Its contract is simpler: callers that are eligible to perform equivalent read-like work at the same time may share one execution. The authoritative system still defines the value and its consistency semantics.

That narrow contract keeps the mechanism practical. A cache miss stops being an invitation for every concurrent caller to repeat the same expensive operation. One fill proceeds, compatible followers wait on it, and the in-flight state disappears when the attempt ends.