Request Coalescing Collapses Concurrent Cache Misses

A cache miss can become expensive when many requests ask for the same key at nearly the same time. Without coordination, each caller can start an identical database query, remote call, or computation. The cache eventually fills, but the backend absorbs a burst precisely when the cached value is absent.

Request coalescing changes the concurrency boundary. The first caller starts the load and publishes an in-flight entry for that key. Later callers join that entry instead of starting equivalent work. When the load finishes, its result is distributed to the waiting callers and the in-flight entry is removed.

The mechanism reduces duplicate work; it does not make the underlying load cheaper or more reliable.

Coalescing keys define which work may be shared

Only requests with equivalent semantics belong behind the same in-flight entry. A cache key based only on a URL can be unsafe when authorization, locale, tenant, headers, or query parameters change the result.

The coalescing key therefore needs the same distinctions that make the loaded value reusable. Too broad a key can return the wrong result. Too narrow a key preserves correctness but misses opportunities to collapse duplicate work.

This boundary is separate from storage layout. Two cache entries can share a backend source without being semantically interchangeable.

One leader performs the load

The first request for a missing key becomes the leader. It creates the in-flight state before starting backend work so another caller cannot observe the miss and launch a second load in the gap.

Followers attach to that state and wait for completion. The implementation may use a promise, future, condition variable, channel, or runtime-specific primitive. The essential property is atomic publication of one active load per coalescing key.

After completion, cleanup must allow a later miss to elect a new leader. Leaving a completed or failed in-flight entry installed indefinitely can turn temporary state into a stale result or permanent error.

Failures need an explicit fan-out policy

Sharing work also shares its outcome. If the leader receives a timeout or backend error, every follower waiting on that same attempt may receive the failure.

Immediate independent retries by all followers would recreate the burst that coalescing was meant to suppress. Retry policy should remain bounded and coordinated, whether the next attempt is started by one caller or deferred until a later request.

Some systems cache selected negative results briefly. That can reduce repeated work for stable conditions such as a confirmed missing object, but transient failures usually require different treatment.

Caller cancellation must not accidentally cancel everyone

A follower can lose interest while other callers still need the shared result. Its cancellation should normally detach that follower rather than terminate the leader’s backend work.

The leader is more subtle. If the original caller disconnects, the shared operation may still be valuable to followers. Implementations often separate the lifetime of the in-flight load from any single caller and cancel the backend only when no interested waiters remain or when a shared deadline expires.

That policy needs bounded execution. A detached load without a deadline can consume resources after all useful callers have gone.

Hot keys still need admission control

Coalescing can turn a thousand concurrent misses into one backend operation, but it does not remove pressure from the waiting side. A hot key can still accumulate many suspended callers, retained buffers, or response fan-out work.

Per-key waiter limits, global concurrency limits, deadlines, and load shedding remain useful. Coalescing controls duplicate backend execution; it is not a substitute for bounding demand.

The same distinction matters when the backend cannot serve the key at all. Thousands of waiters sharing one doomed operation still occupy local resources until that operation finishes.

Scope determines whether instances can share work

An in-process coalescer only combines requests reaching the same process. Ten service instances can still issue ten simultaneous backend loads for one key.

Cross-instance coalescing is possible through a distributed lock or coordination service, but it adds network dependency, lease handling, ownership recovery, and stale-holder concerns. For many systems, per-instance coalescing already removes enough amplification without placing distributed coordination on the read path.

The useful scope follows the cost being protected. A database that tolerates one load per application instance may not justify global coordination.

Metrics should expose both collapse and contention

Useful telemetry includes leader loads, follower joins, wait duration, followers per key, backend latency, errors, cancellations, and the ratio of logical requests to actual loads.

A high collapse ratio can show that coalescing is protecting a hot key. It can also reveal a cache policy that expires popular entries in synchronized bursts. Metrics should therefore be read with cache hit rate, eviction behavior, and backend saturation.

Request coalescing is most effective when equivalent concurrent work is common and expensive. Precise keys, atomic leader election, bounded waiting, careful cancellation, and explicit failure handling let many callers share one in-flight operation without turning that shared state into a new source of correctness or availability problems.