Circuit Breakers Limit Cascading Failure

A slow or failing dependency can consume more than its own capacity. Callers wait, retry, hold sockets, occupy worker slots, and retain memory while requests remain unresolved. As pressure spreads upstream, a local fault can become a service-wide saturation event.

A circuit breaker places a stateful gate around calls to that dependency. It observes outcomes, opens when the configured failure policy is met, rejects calls for a period, then permits a small number of probes. Successful probes can return the breaker to normal traffic; failed probes send it back to the open state.

The mechanism does not repair the dependency. Its job is to bound the amount of caller capacity spent on a dependency that is currently unlikely to serve useful work.

Closed state carries normal traffic

A breaker normally starts closed. Requests pass through to the dependency, and the breaker records the signals used by its policy.

request -> breaker(closed) -> dependency
                         <- success/failure

The signal may be consecutive failures, an error ratio over a rolling window, latency above a threshold, or a combination. The choice matters because not every unsuccessful application result indicates dependency failure.

For example, an HTTP 404 produced promptly by a healthy service is usually different from a connection timeout. Counting both as equivalent failures can open a breaker while the dependency is operating correctly.

A useful policy therefore classifies outcomes before updating breaker state.

Opening needs a bounded observation rule

A raw failure count without a window can preserve old incidents indefinitely. A ratio without a minimum sample count can react to one failure out of one request. Production policies commonly combine a bounded observation window with a minimum volume.

Suppose a breaker evaluates the last 20 eligible calls and opens when at least 10 have completed and more than 60 percent are classified as dependency failures. The exact values depend on traffic shape and failure cost; they are policy inputs rather than universal defaults.

The key property is that the decision is based on recent, relevant evidence. Old failures eventually leave the observation set, while tiny samples do not dominate the state transition.

Open state rejects work before the dependency

Once open, the breaker stops ordinary calls from reaching the dependency.

client -> breaker(open) -X-> dependency
          |
          +-> fail fast

Fail-fast behavior releases caller resources sooner than waiting for a timeout on every request. It also reduces traffic against a dependency that may already be overloaded.

The returned error should remain distinguishable from an error actually produced by the dependency. That distinction helps retry policy, metrics, and incident analysis avoid treating locally rejected calls as fresh evidence from the remote system.

Opening a breaker is not a substitute for a timeout. Calls made while closed still need bounded connect, request, and response times. Without those bounds, enough in-flight calls can exhaust caller resources before the breaker receives outcomes to evaluate.

Half-open state controls recovery traffic

An open breaker cannot remain open forever if automatic recovery is expected. After a configured interval, it enters a probing phase often called half-open.

The half-open state should limit concurrency. If every waiting request becomes a probe at once, the first recovery attempt can recreate the same load spike that contributed to the failure.

open
  |
  | recovery interval elapsed
  v
half-open -> probe 1
          -> probe 2
          -X excess calls

A small probe budget gives the dependency a controlled path back into service. The breaker can close after its success criterion is met, or reopen as soon as a probe demonstrates that the failure condition persists.

Probe semantics must match the dependency. A lightweight health endpoint may not exercise the same path as real work. In many systems, limited real requests provide a more representative recovery signal, provided their side effects and retry behavior are safe.

Breaker scope determines the blast radius

A breaker shared too broadly can turn one failing target into an outage for healthy targets. A breaker scoped too narrowly may provide little protection because each caller sees too few samples or continues sending substantial aggregate traffic.

Useful scope often follows the failure domain: a specific upstream service, host pool, shard, tenant-dependent endpoint, or operation class. The right boundary depends on which failures are correlated.

Consider a client that talks to three independent shards. One global breaker can block all three when only one shard is unhealthy. Per-shard breakers preserve traffic to the healthy shards, while still protecting callers from the failing one.

The same principle applies to credentials, regions, and endpoints when they have independent failure modes.

Retries and breakers need one budget

Retries can work against breaker protection if they are configured independently. A request that fails may be retried several times before the breaker records enough completed failures to open. Across many callers, that amplification can be large.

Retry policy should account for the same deadline and failure classification used around the dependency. Locally rejected calls from an open breaker normally should not trigger immediate retry loops against the same breaker. A retry after a meaningful delay, a fallback path, or propagation of the failure may be more appropriate.

Jitter remains useful when many clients may resume traffic near the same time. It spreads retry and recovery pressure instead of aligning it on a single timer boundary.

Distributed breakers do not need identical state

In a fleet, each process can keep its own breaker state. This avoids adding a coordination dependency to the protection mechanism, but different instances may open and close at different times.

That divergence is often acceptable. Each instance limits its own resource exposure, and aggregate traffic falls as more instances observe the fault. A centrally shared breaker can coordinate a fleet more tightly, but it also introduces shared state, network latency, and another failure mode.

The design choice follows the protection goal. Local breakers protect each caller instance. Shared admission control is a different mechanism when the requirement is a strict fleet-wide traffic ceiling.

Metrics must expose state transitions

A breaker that silently rejects traffic can obscure the original fault. Operational telemetry should separate at least remote attempts, remote failures, local open-state rejections, probe outcomes, and state transitions.

Useful fields include dependency identity, breaker scope, previous state, next state, transition reason, observation count, failure ratio, and probe count. Transition logs are especially valuable because a high rejection count can be a consequence of one earlier opening decision rather than thousands of new remote failures.

Metrics also reveal pathological oscillation. Frequent closed-to-open-to-half-open cycles can indicate a recovery interval that is too short, a threshold that is too sensitive, or a dependency operating near its capacity boundary.

Protection belongs beside bounded calls

Circuit breakers work best as one layer in a broader resource-control path. Timeouts bound individual calls. Retry budgets limit repeated attempts. Concurrency limits cap simultaneous work. Circuit breakers temporarily suppress calls when recent evidence says a dependency is unhealthy.

These controls solve different parts of the same failure path. A breaker adds memory across requests: after enough evidence accumulates, callers stop spending full remote-call cost until controlled probes show that normal traffic can resume.