Hedged Requests Cut Tail Latency at a Controlled Cost
A service can have acceptable median latency and still produce a small set of very slow responses. Queueing, runtime pauses, storage contention, packet loss, or a temporarily busy replica can push individual requests far beyond the common case.
Hedged requests address that tail by sending a second copy after the first request has been outstanding for a chosen delay. The caller accepts the first valid response and cancels the remaining attempt. The technique trades a bounded amount of extra work for a chance to escape an unusually slow execution path.
That trade is useful only when duplicate execution is safe and the additional traffic is controlled.
A hedge is delayed, not immediate duplication
Sending two copies of every request at the same instant can reduce latency, but it also doubles request traffic before the system has evidence that an attempt is slow. A hedge waits long enough for ordinary requests to finish.
t0 t0 + hedge delay
|---------------------|
request A ------------|-------------------->
request B -------->
|
+-- first valid responseThe delay can be tied to a recent latency percentile for the operation. For example, a caller might consider a hedge only after the primary attempt exceeds a high percentile from a suitable observation window. The exact percentile is a policy choice, not a universal constant.
A delay that is too short creates unnecessary duplicate load. A delay that is too long leaves little room for the second attempt to improve the caller’s deadline. The useful range depends on the latency distribution, remaining deadline, and cost of each attempt.
Replica diversity is part of the mechanism
A hedge has little value if both attempts enter the same congested queue or depend on the same impaired resource. The second attempt should, where the topology permits it, target a different healthy replica or execution path.
This does not require random routing without constraints. Normal eligibility rules still apply: locality, shard ownership, consistency requirements, capacity, and health state remain relevant. The hedge merely avoids repeating the exact placement that may be responsible for the slow path.
Correlated failures limit the benefit. If all replicas wait on one saturated database partition, duplicating the frontend request adds work without escaping the bottleneck. Tail mitigation is strongest when a meaningful portion of latency variation is independent across eligible attempts.
Duplicate execution must be semantically safe
Read-only operations are common candidates, but the label “read” is not sufficient by itself. A request may trigger accounting, access logging with side effects, cache fills, lease extension, or another state transition.
For operations that can mutate state, duplicate execution requires an explicit safety mechanism. An idempotency key can let the authoritative service recognize repeated commands, but its scope and retention period must cover both attempts. A caller cannot assume that cancellation prevents the slower request from reaching the server.
attempt A ---- accepted ---- commit
attempt B ---- same key ---- existing resultIf an operation cannot tolerate concurrent duplicates, it should not be hedged merely to improve latency.
Cancellation limits waste but does not erase it
Once one attempt returns an acceptable result, the caller should cancel the other attempt promptly. Cancellation releases client resources and may stop server work when the transport, framework, and application propagate it correctly.
Cancellation is not rollback. The losing attempt may already have consumed CPU, issued storage I/O, acquired a lock, or committed a side effect before the cancellation signal arrives. Capacity planning must therefore count hedged work as real work, even when most losing attempts are cancelled quickly.
The caller also needs a rule for response validity. A fast transport error should not necessarily defeat a slower successful response. Implementations commonly distinguish a usable result from an attempt failure and continue waiting while another eligible attempt can still succeed inside the deadline.
The request deadline remains the outer boundary
Hedging does not justify extending the caller’s deadline. Every attempt belongs to the same end-to-end time budget.
Before launching a hedge, the caller should check that enough time remains for the second attempt to produce a useful response. Starting a duplicate a few milliseconds before the outer deadline may add load with almost no chance of helping.
outer deadline
|----------------------------------------|
primary ---------------------------->
hedge ----------->Per-attempt timeouts can be shorter than the outer deadline, but they must not create a path that outlives it. Cancellation from the upstream caller should terminate both primary and hedged attempts.
This keeps latency policy composable across service boundaries. A downstream optimization remains subordinate to the request budget rather than silently creating a new one.
A budget prevents hedging from becoming overload amplification
Tail latency often rises during overload, exactly when unrestricted duplication is most dangerous. If every slow request launches another request, the extra work can deepen queues and produce still more hedges.
A production policy needs a hard budget. The system can limit hedges by a token bucket, a maximum hedge ratio, a concurrency cap, or a combination of these controls. When the budget is exhausted, the request proceeds without a duplicate.
The policy should also react to capacity signals. A replica pool near saturation is a poor place to spend aggressive hedge traffic. Suppressing hedges during overload can preserve throughput even if individual tail latency temporarily rises.
This creates an important distinction: hedging is a latency optimization, not a substitute for admission control, capacity, or backpressure.
Measurements should separate primary and hedge work
A single latency histogram cannot show whether hedging is helping efficiently. Telemetry should identify primary attempts, hedged attempts, winners, cancellations, and attempts that continued after cancellation.
Useful measurements include the hedge launch rate, fraction of hedges that win, added request volume, cancellation latency, per-attempt latency, end-to-end latency, and server work attributable to losing attempts. These signals expose a policy that improves a percentile only by spending excessive capacity.
The hedge delay also needs periodic recalibration when it is derived from observed latency. A fixed threshold chosen during light traffic can behave very differently after workload, topology, or service-time distributions change.
Tail reduction has to preserve system stability
Hedged requests are effective when a slow attempt is often an isolated event and another eligible path can complete sooner. The same mechanism becomes costly when duplicate work is unsafe, failures are strongly correlated, or the service is already short of capacity.
A sound implementation delays the duplicate, sends it through an eligible alternate path, shares the original deadline, cancels losing work, and enforces a strict hedge budget. With those boundaries in place, tail latency can fall without turning every slow request into uncontrolled load.