Circuit Breakers Limit Repeated Calls to Failing Dependencies

A remote dependency can fail in a way that is both slow and expensive. Requests wait for timeouts, workers remain occupied, retries add more traffic, and a local service can lose capacity even when its own code is healthy.

A circuit breaker places a stateful decision in front of that call path. While the dependency behaves acceptably, calls pass through. After the configured failure condition is reached, the breaker opens and rejects new calls locally for a bounded period. Later, it admits a small number of probes before deciding whether normal traffic can resume.

The useful property is not failure removal. The breaker limits repeated work against a dependency that is already showing evidence of trouble.

Closed state records failure evidence

A breaker normally starts closed. Calls reach the dependency and their outcomes feed a policy. That policy may use consecutive failures, a failure ratio over a rolling window, latency thresholds, or a combination.

The counted outcomes need careful classification. A connection timeout or HTTP 503 can indicate dependency trouble. A caller error such as HTTP 400 usually should not contribute to the same failure threshold. Treating every non-success response as equivalent can open the breaker for conditions that extra waiting would not fix.

The sampling window also changes behavior. A ratio based on two requests is noisy, while a large window reacts slowly. Implementations often require a minimum request volume before evaluating a percentage threshold.

Open state rejects locally

Once the threshold is crossed, the breaker moves to open. New calls fail before consuming a connection, request slot, or remote timeout budget.

That rejection should be explicit. The caller may return an error, use cached data, choose another dependency, or apply a product-specific fallback. A breaker does not make a fallback correct by itself; stale or partial data still needs its own semantic rules.

Local rejection also needs to participate in observability. If metrics only record calls that reached the dependency, an open breaker can make remote traffic look healthy while users are still receiving failures.

Half-open state controls recovery probes

Keeping a breaker open forever would turn a transient outage into a permanent one. After a cooldown, the breaker typically enters a half-open state and allows limited probe traffic.

The limit matters. If every waiting request becomes a probe at once, recovery can produce a new burst against a dependency that has only just regained capacity. A small probe budget gives the system evidence without immediately restoring the previous load.

Successful probes can close the breaker. Failed probes can return it to open and start another cooldown. The exact counts and timing are policy choices, not universal constants.

Breakers need a precise scope

A breaker keyed too broadly can isolate healthy traffic. One global breaker for an entire external platform may open because a single endpoint or region is failing. A breaker keyed too narrowly can create thousands of independent state machines with too little traffic to produce stable signals.

Useful keys often follow the failure boundary: dependency, endpoint class, region, tenant partition, or another routing unit that shares fate.

The same principle applies to connection pools and replicas. If requests can move safely to another replica, breaker state should not automatically suppress every replica unless their failure domain is shared.

Timeouts and retry budgets remain separate controls

A breaker is not a replacement for deadlines. Calls admitted while the breaker is closed still need bounded connection and response time.

It is also not a retry policy. Retrying a failed operation several times before recording one breaker outcome can multiply pressure on the dependency. Systems commonly coordinate retry budgets, deadlines, concurrency limits, and breaker state so each mechanism does not amplify another.

The ordering must be deliberate. Metrics should distinguish original attempts, retries, breaker rejections, probe calls, and final request outcomes.

Distributed instances do not need identical breaker state

Each service instance can maintain its own breaker state. That avoids a coordination dependency in the failure-control path, but instances may open and recover at slightly different times.

This divergence is often acceptable because the breaker is a local protection mechanism. Sharing state can be useful in some architectures, but it adds synchronization, availability, and stale-state concerns. A globally consistent breaker can become another distributed system that must remain available during the outage it is meant to contain.

Recovery policy shapes traffic after an outage

Closing immediately after one successful probe can restore full traffic faster than the dependency can absorb it. Systems with sharp load transitions may combine half-open probing with gradual concurrency growth or rate limits.

The breaker should therefore be evaluated as part of the full admission path. Its state transition answers whether calls may proceed; it does not establish that the downstream system has unlimited recovered capacity.

A well-scoped breaker turns repeated remote failure into bounded local rejection, then tests recovery with controlled traffic. That keeps scarce request capacity available for work that still has a viable path to completion.