Request Coalescing at Hot Cache Misses
A cache entry expires at 22:00:00.000. Ten milliseconds later, two hundred requests ask for the same key.
A cache-aside implementation sees two hundred misses. If every caller independently reads the backing service, a single expiration event becomes two hundred concurrent backend operations. Nothing is wrong with the cache lookup itself. The amplification comes from treating identical in-flight work as unrelated.
Request coalescing changes that boundary. Callers that need the same absent key share one active load, while requests for other keys continue independently. The mechanism is small, but its semantics reach beyond a mutex: it defines which operations may share a result, how failures fan out, what cancellation means, and when another load may begin.
A miss is not always independent work
Consider a cache keyed by product identifier. The cache contains product A until its time-to-live expires. A burst of requests arrives immediately after expiry.
Without coordination, the path is:
request 1 -> miss -> database read A
request 2 -> miss -> database read A
request 3 -> miss -> database read A
request 4 -> miss -> database read AIf those reads observe the same source state and have no caller-specific behavior, most of the concurrent work is redundant. The cache has already grouped stored values by key; coalescing extends that grouping to work that has started but has not finished.
The corresponding shape becomes:
request 1 -> miss -> create load A -> database read A
request 2 -> miss -> join load A -----------|
request 3 -> miss -> join load A -----------|-> same result
request 4 -> miss -> join load A -----------|This does not make the backend read cheaper. It changes the number of reads created by one cluster of overlapping misses.
The effect is bounded by overlap. If requests arrive after the first load completes and before its value is cached, an implementation detail can still permit another load. If the cache write happens before the in-flight entry is released, that interval can be kept narrow. If the loaded value is intentionally not cached, a later request is free to start a new operation.
The coordination unit is part of the data model
A coalescer needs an identity for shared work. In the simplest case, that identity is exactly the cache key.
That equivalence can be unsafe when the backend result depends on more than the visible key. A response for account:42 may also depend on authorization scope, locale, representation version, tenant, or feature state. Two callers can share an operation only when the operation is semantically interchangeable for both.
A useful model is:
work identity = all inputs that can change the observable resultThis is the same pressure that exists in cache-key design, but an in-flight result can expose a mistake immediately. A stored cache entry may have a short lifetime or a later invalidation. A wrongly shared active request can hand one caller a result computed under another caller’s context before any persistent cache entry exists.
Coalescing therefore belongs close to the operation whose identity is well specified. A global wrapper around arbitrary reads can hide distinctions that the underlying API considers meaningful.
Key granularity also controls contention. Coalescing all reads behind one lock serializes unrelated keys and changes a duplicate-suppression mechanism into a global bottleneck. Per-key state permits A and B to load concurrently while combining only callers for the same identity.
The in-flight entry is a temporary state machine
A practical coalescer usually maintains a map from key to an in-flight record. That record can hold a completion signal plus either a value or an error.
Its state is conceptually small:
absent -> running -> completed -> removedThe transition from absent to running must be atomic with respect to competing callers. Otherwise two callers can both observe absence and both become the loader.
A language-neutral sketch looks like this:
load(key):
lock map
if key has active operation:
op = active[key]
unlock map
wait for op
return op.result
op = new operation
active[key] = op
unlock map
result = backend.read(key)
lock map
op.result = result
mark op complete
remove active[key]
unlock map
return resultReal implementations need care around notification ordering and cleanup. Waiters must not miss completion. The operation must eventually leave the map on both success and failure. The map lock should protect coordination state, not remain held across the backend call; holding it during I/O would serialize loads for unrelated keys.
Some runtimes provide futures, promises, condition variables, or dedicated single-execution primitives that make the completion edge easier to express. The underlying contract remains the same: exactly one participant owns a given in-flight operation at a time, and joiners observe its terminal result.
Coalescing suppresses duplication, not demand
It is tempting to read request coalescing as a general load-control mechanism. It is narrower.
Suppose ten thousand requests arrive for ten thousand distinct uncached keys. Per-key coalescing combines none of them. The backend can still receive ten thousand operations. A concurrency limiter, queue, admission policy, or backpressure mechanism addresses that dimension.
Even for one hot key, the number of waiting callers can grow while a slow backend operation is active. The backend sees one read, but the application still holds request state for every waiter. Memory, connection slots, deadlines, and response capacity remain finite.
The distinction matters:
- coalescing limits duplicate concurrent work for the same identity;
- concurrency limits bound the total number of active operations;
- rate limits bound admitted work over time;
- caches avoid backend work when a reusable result already exists.
These controls can coexist because they constrain different quantities.
Failure is shared state too
If one backend operation serves fifty waiting callers and that operation fails, the default coalescing semantics distribute the same failure to all fifty.
That is internally consistent: the callers elected to share one operation, and an error is one possible result of that operation. It also creates a visible consequence. A transient failure that would have affected one independent read can now affect every caller attached to the same in-flight record.
Starting fifty immediate replacement reads is usually the wrong reaction because it recreates the amplification that coalescing was meant to suppress. A new operation can begin after the failed record is removed, but retry policy still needs its own limits, timing, and deadline rules.
Some systems briefly retain a failure result, sometimes called negative caching when the retained result represents absence or an error condition. That is separate from coalescing. Coalescing shares an operation while it is active; negative caching deliberately reuses a terminal result for some interval. The acceptable interval depends on the meaning of the failure. A permanent “not found” result and a transport timeout do not carry the same semantics.
Cancellation exposes ownership
Cancellation becomes subtle as soon as several callers share one backend operation.
If caller A starts a load and caller B joins it, A’s client disconnecting does not necessarily mean the backend operation should stop. B still needs the result. Binding the shared operation directly to A’s cancellation token gives the first caller accidental ownership over work now serving multiple callers.
One model gives the shared operation an independent lifetime. Individual callers may stop waiting when their own deadlines expire, while the load continues for remaining waiters and potentially fills the cache.
Another model tracks active waiters. The backend operation is cancelled only when no caller remains interested. This can avoid useless work, but it requires reference accounting and a backend API that responds to cancellation.
Neither model makes cancellation free. Continuing a load after all callers leave can consume backend capacity. Cancelling too aggressively can discard work just before another caller arrives. The correct policy depends on whether completing the operation has value beyond the current waiter set, such as populating a reusable cache entry.
The important boundary is explicit ownership. A caller’s deadline and a shared operation’s lifetime are related, but they are not automatically the same object.
Cache publication and coalescing solve different races
A coalesced loader can still publish stale data.
Imagine a load for key A reads database version 7. Before it writes the cache, another request commits version 8 and invalidates the entry. The old loader then stores version 7. Coalescing ensured that only one version-7 load was active; it did not establish that version 7 remained eligible for publication.
That is a separate ordering problem. Version checks, generation tokens, or another conditional publication rule can reject a fill that has been superseded.
The two mechanisms compose cleanly when their responsibilities stay distinct:
coalescing: who performs this in-flight load?
publication guard: may this loaded value enter the cache now?Combining them conceptually can produce false confidence. Fewer backend reads do not imply fresher cache state.
Process-local coordination has a visible boundary
An in-memory map coordinates only callers that reach the same process.
With eight application instances, a simultaneous miss can create as many as eight backend loads even when each instance perfectly coalesces its local callers. Whether that is acceptable depends on backend capacity, request distribution, and the cost of distributed coordination.
A distributed lock can reduce duplication across processes, but it changes the failure model. Lease expiry, delayed holders, network partitions, and lock-service availability become part of the path. A design that uses leases for correctness-sensitive writes also needs protection against stale holders, commonly through an ordering token enforced by the receiving resource.
For cache-miss suppression, bounded duplicate work is often acceptable. In that case, process-local coalescing can offer a useful trade: it removes duplication within each instance without making every miss depend on a coordination service.
The boundary should be stated in operational terms. “Single load” may mean one per process, one per node, one per region, or one across an entire deployment. Those are different guarantees.
Completion ordering closes a small but important gap
The order of cache publication and in-flight cleanup affects the interval in which duplicate work can reappear.
Consider this sequence:
1. backend load completes
2. in-flight entry is removed
3. cache entry is writtenA caller arriving between 2 and 3 sees no active load and no cached value. It can start another backend operation.
Reversing the final actions narrows that gap:
1. backend load completes
2. eligible result is written to cache
3. waiters are completed
4. in-flight entry is removedExact ordering depends on the implementation, especially if cache writes can fail or block. The general property is more important than one fixed sequence: there should not be an avoidable state in which completed reusable work is invisible both as cached data and as active work.
Cleanup also has to survive exceptional paths. A panic, exception, timeout, or failed cache write must not leave a permanently running marker that causes later callers to wait forever. Structured cleanup constructs can make removal unconditional while preserving the terminal result long enough for existing waiters to observe it.
Measuring the mechanism requires two counts
A normal cache hit ratio does not reveal how much work coalescing suppresses. A burst can have a poor hit ratio and still produce only one backend load for a hot key.
Two separate counts are more informative:
logical misses = callers that did not find a cached value
physical loads = backend operations actually startedTheir difference represents concurrent duplicate work that did not reach the backend.
Waiter count per in-flight key is also useful because extreme values expose hot identities and long-running loads. Duration of the shared operation matters for the same reason: longer operations create a larger window in which callers can accumulate.
These measurements describe mechanism behavior without claiming a universal target. A high waiter count can be expected for a legitimately hot key, or it can signal a backend slowdown. Context determines the interpretation.
A shared operation is a consistency boundary
Request coalescing is often presented as a cache optimization, but its deeper effect is semantic. It declares that a set of concurrent callers may be represented by one operation and one terminal result.
That declaration is safe only when the work identity is complete, failure sharing is acceptable, cancellation ownership is defined, and cache publication has its own freshness rule. It also has a scope: process-local coordination does not become deployment-wide coordination by implication.
The useful abstraction is not “one request does the work while the others wait.” It is a temporary consistency boundary around equivalent in-flight demand. Once that boundary is explicit, duplicate suppression becomes a consequence of the model rather than the model itself.