Cache Stampede Control with Early Refresh

A cache can remove enormous amounts of repeated work, yet a popular entry creates a sharp risk at expiry. If ten thousand requests depend on the same key and that key expires, many requests can discover the miss at nearly the same moment. Each request may then start the same database query, computation, or remote call.

This event is commonly called a cache stampede. The cache works well during the entry lifetime, then abruptly stops protecting the dependency exactly when demand is high.

A robust design does more than cache values. It controls regeneration.

This article develops a practical approach based on early refresh, randomized timing, and bounded refresh ownership. The goal is to spread regeneration work across time while keeping request latency predictable.

The synchronization problem

Consider a product page cached for 60 seconds. During normal operation, requests read the cached value:

request -> cache hit -> response

At second 60, the entry disappears. A burst arriving just after expiry sees:

request A -> miss -> database query
request B -> miss -> database query
request C -> miss -> database query
...

The requests are independent, but expiry synchronizes them around the same missing key. A cache intended to protect the database can therefore create a periodic load spike.

The effect becomes more severe when regeneration is slow. If rebuilding the value takes 800 milliseconds, every request arriving during that interval has an opportunity to start another rebuild.

A useful approximation is:

duplicate work ~= request rate x regeneration time

At 2,000 requests per second and 0.8 seconds of regeneration, one expiry can expose the dependency to roughly 1,600 overlapping attempts for the same logical result.

The exact count depends on scheduling, cache behavior, and application flow, but the relationship is the important part: higher traffic and slower regeneration increase the amplification.

Refresh before hard expiry

One response is to separate freshness time from hard expiry time.

Suppose an entry has these timestamps:

created:       12:00:00
refresh after: 12:00:50
hard expiry:   12:01:10

Before 12:00:50, callers simply use the cached value. Between 12:00:50 and 12:01:10, callers may still use it, but the system can start a refresh in the background. After 12:01:10, the value is no longer acceptable.

This creates a refresh window instead of a single expiry instant.

The request path becomes:

fresh entry
    |
    v
return immediately

refresh window
    |
    +--> return current value
    |
    +--> attempt bounded refresh

hard expired
    |
    v
use fallback miss policy

The old value remains useful while a replacement is prepared. Hot traffic now provides many opportunities to refresh before the hard boundary.

This pattern is especially effective for data that can tolerate a short period of bounded staleness.

Do not let every caller refresh

Early refresh alone is insufficient. If every request inside the refresh window starts regeneration, the stampede merely moves earlier.

Refresh ownership must be bounded.

A process-local implementation can use a per-key in-flight registry:

type Entry struct {
    Value        Product
    RefreshAfter time.Time
    ExpiresAt    time.Time
}

func GetProduct(ctx context.Context, id string) (Product, error) {
    entry, ok := cache.Get(id)
    now := clock.Now()

    if ok && now.Before(entry.RefreshAfter) {
        return entry.Value, nil
    }

    if ok && now.Before(entry.ExpiresAt) {
        refreshGroup.DoAsync(id, func() {
            refreshProduct(context.Background(), id)
        })
        return entry.Value, nil
    }

    return loadAndStore(ctx, id)
}

The important property is not the specific library. For a given key, the refresh coordinator admits only a small number of regeneration attempts, commonly one per process.

The callers still receive the current cached value, so refresh work does not sit on their latency path.

In a multi-instance service, process-local coordination may permit one refresh per instance. That can still reduce amplification dramatically. If stronger control is required, use a distributed lease or another shared ownership mechanism, but account for its failure modes and operational cost.

Add jitter to refresh timing

Fixed lifetimes can synchronize many different keys.

Imagine one million entries populated during a deployment or bulk import. If every entry has a 60-minute lifetime, a large fraction can become eligible for refresh together.

Randomized timing breaks that alignment.

Instead of:

refresh_after = created_at + 50 minutes

use a bounded random offset:

refresh_after = created_at + 45 minutes + random(0..10 minutes)

The refresh work is spread across a ten-minute interval.

Jitter is useful for both refresh thresholds and hard expiry, provided the permitted freshness policy can tolerate the resulting range.

Keep the random range intentional. Too little jitter preserves synchronization. Too much jitter can refresh entries earlier than needed or retain them beyond an acceptable age.

Probabilistic early refresh

A fixed refresh window still has a small coordination edge: the first requests after the threshold are more likely to trigger refresh work.

A probabilistic policy can distribute refresh attempts more smoothly.

As an entry approaches expiry, increase the chance that a request attempts refresh. Far from expiry, the chance is near zero. Close to expiry, the chance rises.

For example:

def should_refresh(now, refresh_start, expires_at, random_value):
    if now < refresh_start:
        return False

    if now >= expires_at:
        return True

    elapsed = (now - refresh_start).total_seconds()
    window = (expires_at - refresh_start).total_seconds()
    probability = elapsed / window

    return random_value < probability

This simple linear rule is only one possible policy. Production systems can use curves tuned to traffic and regeneration cost.

Probability is not a replacement for refresh ownership. It decides when a caller may try; coordination decides how many refreshes may run. Combining both gives smoother timing and bounded work.

Size the refresh window from real latency

A refresh window should give regeneration enough time to finish before hard expiry.

If the normal rebuild takes 100 milliseconds but the 99th percentile is 4 seconds, a 1-second refresh window is fragile. A slow regeneration can cross the hard boundary and force requests onto the miss path.

Useful inputs include:

• regeneration latency percentiles;
• request rate for hot keys;
• downstream capacity;
• acceptable staleness;
• timeout and retry policy;
• refresh failure frequency.

For a high-value entry, a window of several high-percentile regeneration durations can provide room for transient slowness. The correct margin depends on the system’s freshness contract and failure budget.

Do not derive the window from average latency alone. Stampedes are most dangerous during degraded periods, when regeneration is often slower than normal.

Handle refresh failure without deleting useful data

A refresh failure does not necessarily make the current value useless.

Suppose a cached catalog entry is 52 seconds old, its refresh threshold is 50 seconds, and its hard expiry is 70 seconds. A refresh attempt fails because the database briefly times out.

Deleting the entry immediately converts a recoverable refresh error into a cache miss storm.

A safer policy can keep serving the existing value until its hard boundary while recording the failed refresh. Another request can attempt regeneration later, subject to backoff and ownership limits.

The state can be viewed as:

fresh
  |
  v
refresh eligible
  |
  +--> refresh succeeds --> fresh replacement
  |
  +--> refresh fails ----> serve current value, delay next attempt

Backoff matters. Without it, a failing dependency can receive a new refresh attempt on every request.

For example, store a next_refresh_attempt_at timestamp and move it forward after failures. A small exponential backoff with jitter is often sufficient.

Separate hot-key protection from global capacity

Per-key coordination prevents duplicate work for one key, but thousands of different keys can still refresh at once.

Add a global concurrency bound around regeneration:

func refreshProduct(ctx context.Context, id string) {
    if !refreshSlots.TryAcquire() {
        return
    }
    defer refreshSlots.Release()

    product, err := repository.LoadProduct(ctx, id)
    if err != nil {
        recordRefreshFailure(id, err)
        return
    }

    cache.Put(id, newEntry(product))
}

Now two controls work together:

per-key control    -> suppress duplicate regeneration
global control     -> cap total regeneration pressure

This distinction is important. A system can have perfect single-key coordination and still overload a dependency through many simultaneous distinct keys.

Decide what happens after hard expiry

Eventually, an entry can become too old to serve. At that point the application needs an explicit policy.

Common choices include synchronous regeneration, a controlled error, a degraded response, or a separately defined stale-if-error period.

The correct choice depends on the data.

A stale currency conversion rate can be unacceptable for settlement. A slightly old public profile image may be fine. A product description may tolerate more staleness than inventory availability.

Treat hard expiry as a domain decision, not merely a cache setting.

A useful design question is:

At what age does this value become more harmful than an unavailable value?

That boundary should drive the fallback behavior.

Observe regeneration as its own workload

Cache hit rate alone cannot show whether regeneration is healthy.

Track metrics such as:

cache_requests_total
cache_hits_total
cache_refresh_attempts_total
cache_refresh_success_total
cache_refresh_failures_total
cache_refresh_suppressed_total
cache_hard_misses_total
cache_refresh_duration_seconds
cache_entry_age_seconds

The suppressed count is particularly useful. It shows how much duplicate regeneration the coordinator prevented.

Also inspect distributions by key class rather than exporting every raw cache key as a metric label. Unbounded labels can create a separate observability problem.

Logs or traces can carry individual keys when detailed diagnosis is required.

Test the concurrency behavior

Ordinary unit tests can confirm expiry calculations, but stampede control also needs concurrent tests.

A focused test can create many callers for the same refresh-eligible key and assert that only one regeneration runs:

func TestRefreshIsCoalescedPerKey(t *testing.T) {
    repo := newBlockingRepository()
    service := newService(repo)

    service.cache.Put("p-42", refreshEligibleEntry())

    var wg sync.WaitGroup
    for i := 0; i < 100; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            _, _ = service.GetProduct(context.Background(), "p-42")
        }()
    }

    wg.Wait()

    if got := repo.LoadCount(); got != 1 {
        t.Fatalf("expected one refresh, got %d", got)
    }
}

Also test failure and boundary cases:

• refresh fails while the current value remains usable;
• hard expiry arrives during a slow refresh;
• the refresh owner crashes or times out;
• many different keys become eligible together;
• the global regeneration limit is exhausted;
• jitter stays inside the permitted range.

Concurrency tests are strongest when the test controls synchronization explicitly instead of depending on sleeps.

Common mistakes

Treating TTL as the whole policy

A single TTL collapses freshness and availability into one timestamp. Separate refresh eligibility from the point at which data must no longer be served.

Refreshing on every eligible request

A refresh window without ownership control still creates duplicate work. Bound regeneration per key.

Coordinating per key but not globally

Ten thousand distinct keys can overwhelm a database even if each key has only one refresh. Bound total regeneration concurrency as well.

Retrying refresh immediately

A failing dependency can turn refresh traffic into sustained pressure. Back off failed attempts and add jitter.

Using the same lifetime for every entry

Uniform timing can synchronize unrelated keys. Randomize thresholds inside a domain-approved range.

Serving stale data without a contract

Staleness must have a defined limit. Make the hard boundary explicit and connect it to the meaning of the data.

A practical rollout sequence

For an existing cache, introduce stampede control in small steps.

First, measure regeneration latency and identify hot keys or key classes. Next, split the current TTL into a refresh threshold and a hard expiry. Add per-key refresh coordination, then add a global regeneration limit. Introduce jitter after the basic flow is observable. Finally, tune windows and backoff from production measurements.

This sequence keeps each change understandable and makes capacity effects visible.

The central idea is simple: do not wait for a popular cached value to disappear before preparing its replacement.

Early refresh spreads regeneration across time. Jitter reduces synchronized timing. Per-key coordination suppresses duplicate work. Global limits protect shared dependencies. Explicit hard expiry preserves the data contract.

Together, these mechanisms turn cache regeneration from an accidental traffic spike into a controlled workload.