Circuit Breakers Bound Calls to an Unhealthy Dependency

A remote dependency can fail in a way that is both persistent and expensive. Connections time out, worker slots remain occupied, request queues grow, and retries add more traffic to a service that is already unable to respond. A caller that keeps issuing the same class of request can turn one dependency failure into pressure across its own process.

A circuit breaker places a stateful admission decision in front of those calls. While the dependency behaves within policy, calls pass through. After enough qualifying failures, the breaker opens and rejects new calls locally for a period. Recovery is tested with limited traffic rather than a full return to normal load.

The mechanism is deliberately narrower than a general resilience layer. It does not repair the dependency, make retries safe, or define a fallback response. Its job is to bound futile remote work while retaining a controlled path back to service.

Closed state records outcomes

A breaker normally begins closed. Calls reach the dependency and their outcomes feed a failure policy.

request
   |
   v
[ closed breaker ] ---> dependency
         |
         +--- record outcome

The policy may use consecutive failures, a failure ratio over a rolling window, or another bounded measurement. A ratio usually needs a minimum sample count; opening after one failure in a window containing one request would otherwise make the ratio technically high but operationally noisy.

Not every unsuccessful application result belongs in the breaker signal. A valid 404 for a missing object may say nothing about dependency health. Connection refusal, transport timeout, or selected 5xx responses may be stronger candidates. The classification has to match the contract of the dependency.

Slow calls can also matter even when they eventually succeed. Some implementations track latency or slow-call ratios because a saturated service that completes requests just before timeout can still consume most caller capacity. That policy needs explicit thresholds rather than treating all non-error responses as equivalent.

Open state fails locally

Once the configured threshold is crossed, the breaker moves to open. Calls covered by that breaker no longer attempt the remote operation until the open interval permits a recovery check.

application ---> [ open breaker ] -X-> dependency
                      |
                      +--> local failure

This changes the cost of failure. A local rejection does not spend a connection attempt or wait through the dependency timeout. It can also prevent a large caller fleet from continuously applying traffic to an endpoint that has little chance of serving it.

The open state is not proof that the dependency is globally unavailable. It is a local conclusion based on the outcomes observed by one breaker instance and its configured policy. Different processes can therefore open and recover at different times.

That scope should be intentional. A breaker shared across unrelated endpoints, tenants, or operation classes can let one failing path block healthy work. Conversely, a breaker per individual request key can fragment observations so much that no breaker receives enough samples to react. The useful boundary usually follows a dependency or operation whose failures are expected to correlate.

Half-open state limits recovery probes

After the open interval, immediately releasing all queued or new traffic can recreate the overload that caused the dependency to fail. A half-open state admits only a limited number of trial calls.

open
  |
  | interval elapsed
  v
half-open ---> limited probes ---> dependency
  |                                |
  | success policy met             | failure
  v                                v
closed                            open

If the probes satisfy the recovery policy, the breaker closes and normal admission resumes. A qualifying failure can return it to open. Implementations differ on the exact number of probes and the transition rule, so those details should be treated as policy rather than universal protocol semantics.

Probe concurrency matters. If hundreds of callers independently observe that an interval has elapsed and all become probes, half-open no longer limits load. The state transition needs atomic coordination within the scope of the breaker instance so only the permitted probe budget passes.

Breakers and retries solve different problems

A retry handles the possibility that another attempt may succeed. A circuit breaker decides whether an attempt should reach the dependency at all. Combining them without a clear order can multiply traffic.

Consider three application attempts, each containing three transport retries. One logical operation can produce as many as nine remote calls before higher layers add their own recovery behavior. If the breaker counts each inner attempt, it sees a different failure stream from a breaker that wraps the complete retry operation.

A useful design states the composition explicitly:

request
  |
  v
retry policy
  |
  v
circuit breaker
  |
  v
dependency

or, when the intended semantics call for it:

request
  |
  v
circuit breaker
  |
  v
retry policy
  |
  v
dependency

These arrangements are not interchangeable. The first lets each retry attempt consult breaker state. The second treats the retry sequence as the operation observed by the breaker. Timeout placement, attempt accounting, and metrics also change with the composition.

Retries should remain bounded and appropriate for the operation. A breaker does not make a non-idempotent mutation safe to repeat.

Timeout policy remains necessary

A breaker can only record an outcome after an attempt produces one. Without finite connection and request deadlines, a large number of calls can remain in flight while the breaker has too little completed evidence to change state.

Timeouts therefore remain a separate control. They bound how long one attempt can consume resources. The breaker uses completed outcomes to decide whether subsequent attempts should be admitted. Concurrency limits can add another boundary by restricting the amount of work already in flight.

These controls address different dimensions:

timeout            -> bounds duration of an attempt
concurrency limit  -> bounds simultaneous admitted work
circuit breaker    -> suppresses calls after adverse outcomes
retry              -> permits selected repeated attempts

Treating them as distinct policies makes their interaction visible and reduces accidental amplification.

Fallbacks need their own capacity model

An open breaker often routes execution to a fallback: cached data, a degraded response, a secondary service, or an explicit error. A fallback is not automatically safer than the primary path.

If every failed primary call shifts to the same secondary dependency, that secondary can become the next bottleneck. Stale cache responses may be acceptable for one endpoint and invalid for another. A default value can conceal missing data if callers cannot distinguish it from a real result.

Fallback behavior therefore needs an explicit contract and capacity assumptions. In some paths, returning a fast, visible failure is more correct than manufacturing a degraded success.

Metrics should expose state and cause

Breaker state alone is not enough for operations. A useful telemetry set includes state transitions, rejected-call count, admitted probe count, observed failure classes, slow-call counts when applicable, and the dependency latency seen before opening.

Transitions should carry the breaker identity and policy scope. An alert that says only circuit open is much less actionable than one tied to a particular upstream, operation class, and failure category.

State changes can also flap when thresholds sit close to normal variance. Rolling windows, minimum sample counts, open intervals, and recovery criteria should be selected from the service’s traffic and failure characteristics rather than copied as generic constants.

A breaker is a local traffic boundary

Circuit breakers are most useful when failure has a meaningful correlation boundary and remote attempts carry real cost. They turn repeated evidence of dependency trouble into temporary local admission control, then restore traffic through a limited recovery path.

The guarantee stays modest. An open breaker does not establish the true global health of a service, and a closed breaker does not guarantee the next call will succeed. The value comes from bounding repeated remote work and making recovery admission explicit instead of letting every caller probe a failing dependency at full rate.