Hedged Requests Cut Tail Latency with Controlled Duplication

A service can have a healthy median latency while a small fraction of requests take far longer. Queueing, garbage collection, storage stalls, packet loss, noisy neighbors, or uneven replica load can all stretch the slow end of the distribution. For a request that fans out to several dependencies, one slow branch can dominate the entire response.

Hedged requests reduce that exposure by starting a second copy after a short delay. The copies target independent execution paths when possible, and the first valid response wins.

t=0 ms      send request to replica A
t=25 ms     A still pending; send duplicate to replica B
t=39 ms     B responds
            return B's response
            cancel or discard A

The technique is deliberately selective. Sending two copies of every request doubles offered work before accounting for cancellation. A hedge starts only after the original has consumed a chosen portion of its latency budget.

The hedge delay defines the cost boundary

A fixed hedge delay is simple, but it should reflect the operation’s observed latency distribution. If the delay is too short, ordinary requests produce duplicates. If it is too long, the duplicate has little time to improve the caller’s deadline.

A percentile-based trigger is often more stable than an arbitrary constant. A service might hedge after the recent p95 latency for the same operation class, subject to minimum and maximum bounds.

hedge_delay = clamp(recent_p95, 20 ms, 80 ms)

This does not require the percentile to be exact for every request. It provides a policy that tracks broad latency shifts without making duplication immediate.

The measurement window matters. A very short window can make the threshold oscillate with transient noise. A very long window can retain a stale threshold after workload or infrastructure changes. Implementations commonly separate latency telemetry from the request path and refresh a compact threshold periodically.

Replica independence determines the benefit

A duplicate helps only when the second execution can avoid the condition delaying the first. Sending both copies through the same saturated queue, process, disk, or downstream dependency can add load without adding useful independence.

Replica selection therefore matters. A hedge can prefer a different host, availability zone, connection, or storage shard replica, depending on the architecture and consistency model.

client
  |
  +---- replica A ---- storage path 1
  |
  +---- replica B ---- storage path 2

Independence is rarely absolute. Two replicas may still share a network link, database primary, or control plane. The useful question is whether the dominant sources of tail latency are sufficiently decorrelated for the second path to have a meaningful chance of finishing sooner.

Cancellation must be treated as an optimization

After one copy returns a valid result, the loser should be cancelled when the protocol and server support cancellation. Cancellation can release queue slots, CPU time, connection capacity, and downstream work.

It cannot be assumed to erase work already performed.

replica A: request accepted -> database query running
replica B: response returned
client:    cancel A

By the time cancellation reaches A, its database query may already have completed or may not support interruption. Capacity planning must therefore account for actual duplicate work, not just duplicate requests that remain visible at the client.

A robust implementation also tolerates late responses. Once the operation has completed at the caller, a late result from another copy is discarded without changing the completed outcome.

Read operations are the safest starting point

Hedging is easiest for side-effect-free reads. Two executions can race without creating duplicate state transitions.

Writes require stricter semantics. Retrying or hedging a non-idempotent write can create duplicate effects if both copies reach the commit point. An idempotency key can make repeated submissions converge on one logical operation, but only if every relevant write path enforces that key atomically.

POST /payments
Idempotency-Key: 7f2c...

replica A ----\
               +--> durable idempotency record --> one logical commit
replica B ----/

The key alone is not sufficient if replicas use isolated deduplication state or if downstream effects bypass the idempotency boundary. For mutation paths, hedging belongs behind a proven duplicate-suppression contract rather than being added as a transport-only optimization.

Consistency rules still apply

Two replicas may return different versions of data. The fastest response is not automatically an acceptable response.

A strongly consistent system may require both candidates to satisfy the same read timestamp, leader epoch, fencing condition, or quorum rule. A bounded-staleness system may reject a fast replica whose version falls outside the permitted window.

The hedge mechanism chooses among responses that already satisfy the operation’s correctness contract. It must not weaken that contract to obtain a smaller latency number.

This distinction is especially important during failover. A newly promoted replica and a lagging replica can have very different response times and data positions. Routing policy needs enough version information to reject a fast but invalid result.

Deadlines cap the useful hedge window

A hedge should inherit the caller’s absolute deadline. Creating a duplicate does not create more time.

Suppose a request has a 100 ms deadline and the hedge delay is 70 ms. The second copy has at most 30 ms to finish, minus routing and cancellation overhead. If that replica normally needs 40 ms, the hedge is unlikely to help.

caller deadline: 100 ms

0 ---------------- 70 ---------------- 100
original running    hedge starts         stop
                    <--- 30 ms --->

A policy can suppress a hedge when the remaining budget is below a configured minimum. That avoids adding work after the opportunity for a useful result has effectively passed.

Deadline propagation also prevents a losing copy from continuing long after the caller has abandoned the operation.

Concurrency limits keep hedging from amplifying overload

Tail latency often rises during overload, exactly when a naive hedging policy would create more duplicates. That feedback loop can turn a latency problem into a capacity problem.

Hedges need their own budget. Useful controls include a maximum ratio of hedged requests, a per-host hedge concurrency limit, a global token bucket, and suppression when queue depth or saturation crosses a threshold.

if original_pending
   and remaining_budget >= min_useful_time
   and hedge_tokens.try_take()
   and target_has_capacity:
       send_hedge()

The hedge budget should be smaller than the normal request budget. The mechanism is a tail-latency control, not an alternate path for unlimited retries.

Load shedding retains priority. When the system is already rejecting work to preserve stability, speculative duplicates should usually be among the first optional operations to stop.

Retries and hedges solve different timing problems

A retry normally follows an explicit failure, timeout, or rejected attempt. A hedge starts while the original attempt is still in flight.

That difference affects both latency and cost. Waiting for a timeout before retrying can consume most of the caller’s deadline. Hedging spends extra capacity earlier in exchange for a chance to complete before that timeout.

The two mechanisms can coexist, but their budgets must be coordinated. Three retries with one hedge each can produce far more executions than an operator expects from either policy in isolation.

A request policy should place one explicit ceiling on total attempts, including originals, hedges, and retries.

Telemetry must separate originals from duplicates

Aggregate latency alone can hide the operational price of hedging. The service needs counters and distributions that expose both the gain and the added work.

Useful measurements include hedge trigger rate, hedge win rate, cancellation success, loser runtime after winner completion, attempts per logical request, p50/p95/p99 latency before and after policy changes, and resource utilization on candidate replicas.

A high trigger rate with a low win rate often signals a threshold that is too aggressive or replicas whose delays are strongly correlated. A high win rate can still be too expensive if loser cancellation is ineffective and backend work is costly.

The key unit is the logical request. Attempt-level metrics remain useful, but they should not obscure how much speculative work was required to complete one caller operation.

Tail latency is reduced by policy, not duplication alone

Hedging works when a slow attempt is an outlier and another eligible execution path can finish sooner. Its value falls when slowness is shared across replicas, when operations are too expensive to duplicate, or when correctness rules prevent racing equivalent candidates.

The practical design is therefore a bounded policy: wait long enough to avoid duplicating normal work, select an execution path with useful independence, preserve the same correctness constraints, inherit the original deadline, cancel losers where possible, and stop speculation during saturation.

With those boundaries in place, a hedge is not a blanket retry. It is a controlled use of spare execution diversity to reduce exposure to rare slow paths.