Circuit Breakers Stop Repeated Calls to Failing Dependencies
A dependency that is already failing can consume more caller capacity than a healthy one. Requests wait for timeouts, retries add traffic, connection pools remain occupied, and worker slots stay tied to work that has little chance of completing. A circuit breaker places a stateful gate in front of that dependency so the caller can stop issuing calls after failure evidence reaches a configured limit.
The gate is not a replacement for timeouts, retries, or capacity limits. It coordinates repeated calls across a stream of requests. When the dependency appears unhealthy, the breaker fails new calls locally for a period instead of asking the remote system to prove the same failure again.
Closed state records outcomes
A breaker normally begins closed. Calls pass through to the dependency, and their outcomes feed a failure policy.
request
|
v
[ closed breaker ]
|
v
dependency
|
+-- success ------> record success
`-- counted error -> record failureThe policy needs an explicit definition of failure. A connection refusal, deadline expiration, or HTTP 503 may count. A validation error caused by malformed caller input usually should not. Treating every non-success result as dependency failure can open the circuit for faults the dependency cannot correct.
A raw count is sometimes enough, but production policies often use a rolling window or a minimum sample count. For example, opening after 50% failures is noisy when the window contains only two calls. A minimum volume keeps a small sample from producing an oversized reaction.
The breaker should also distinguish slow calls from failed calls if latency is part of the policy. A slow-call threshold can be useful when requests complete but occupy caller resources long enough to threaten capacity. That threshold still needs a clear relation to the request deadline.
Open state rejects locally
When the opening condition is met, the breaker moves to open. New calls covered by that breaker are rejected before reaching the dependency.
client call
|
v
[ open breaker ] --x--> dependency
|
`--> local failureLocal rejection is valuable because it is fast and consumes less downstream capacity. It also gives the dependency room to recover instead of receiving a continuous stream of calls that are likely to fail.
The local result must remain distinguishable from a remote application response. A breaker-open error means the call was not attempted. Logging it as if the dependency returned an error distorts remote error rates and complicates incident analysis.
Callers also need a deliberate policy for the local failure. Some requests can use cached data, a degraded response, another replica, or an asynchronous path. Other requests must fail. The breaker supplies a signal; it does not define the business fallback.
A cooldown alone cannot prove recovery
An open breaker usually stays open for a configured interval. Expiration of that interval does not establish that the dependency is healthy. It only marks a point when testing can resume.
Moving directly from open to full traffic can create a recovery surge. If thousands of callers have queued work or immediately retry, a recovering dependency can be hit with normal load before it has demonstrated usable capacity.
Half-open state limits that transition.
open
|
| cooldown expires
v
half-open
|
+-- admit a small probe set
|
+-- probe policy passes --> closed
`-- probe policy fails --> openOnly a bounded number of probe calls should pass while the breaker is half-open. Other calls are rejected or handled by fallback according to the caller’s policy. Successful probes provide evidence for closing; failed probes send the breaker back to open.
Probe concurrency changes recovery pressure
A single probe is simple but can make recovery sensitive to one unlucky request. Many probes provide a broader sample but can place meaningful load on a dependency that has only just become reachable.
The probe limit therefore belongs to the breaker policy, not to an incidental thread count. A service may permit several probes and require a chosen success ratio before closing. Another may use one probe for an expensive dependency.
Probe calls should retain normal deadlines. A half-open probe without a bounded deadline can leave the breaker stuck while the remote operation hangs.
The policy also needs to define concurrent state transitions. If several requests observe that the cooldown has expired at the same time, they must not all independently become unrestricted probes. Implementations typically coordinate admission so the configured half-open limit remains effective.
Breaker scope determines the blast radius
A breaker needs a key. One global breaker for an entire remote service is easy to operate, but it can reject healthy traffic when only one shard, tenant, endpoint, or replica is failing.
At the opposite extreme, a breaker per request key can create huge state cardinality and too little traffic per breaker to produce useful signals.
Useful scope often follows the failure domain. Examples include:
- one breaker per upstream service when failures are service-wide;
- one per region or cluster when routing domains fail independently;
- one per endpoint when operations have materially different failure behavior;
- one per replica when callers can route around an unhealthy instance.
The scope used for metrics should match the scope used for decisions. A service-wide dashboard can hide one open regional breaker if all regions are aggregated into a healthy average.
Retries and breakers need one policy boundary
Retries can increase the evidence seen by a breaker. If one logical request performs three failed attempts, counting all three as independent demand may open the breaker faster than counting only the final logical outcome.
Neither model is universally correct. The choice depends on what the breaker protects.
A breaker around each physical attempt protects the dependency from repeated attempt traffic. A breaker outside the retry loop observes logical request outcomes but may allow the retry mechanism to keep sending attempts until the outer breaker opens.
breaker outside retry:
breaker -> retry loop -> dependency
breaker around attempts:
retry loop -> breaker -> dependencyThe placement must be intentional. Retry budgets, backoff, and breaker thresholds should be evaluated together because each changes the traffic presented to the others.
An open breaker should not trigger aggressive retries by itself. Retrying a local rejection immediately can turn cheap rejection into caller-side busy work and can produce a burst when the circuit closes.
Timeouts still bound individual calls
A breaker reacts to a sequence of outcomes. It does not place a time bound on a call that has already been admitted.
Every remote attempt still needs a deadline or timeout appropriate to the operation. Without one, a dependency that accepts connections but never completes requests can occupy caller resources while the breaker receives too little failure evidence to open promptly.
Timeouts and breakers therefore act at different layers:
timeout -> bounds one admitted attempt
retry policy -> controls additional attempts
circuit breaker -> gates a stream of attempts
capacity limit -> bounds concurrent resource useCombining them is more robust than asking one mechanism to imitate all four roles.
Metrics should expose state and rejected work
A breaker can reduce remote errors while increasing local rejections, so dependency error rate alone is not enough to evaluate it.
Useful telemetry includes:
- current breaker state and time spent in each state;
- transitions from closed to open, open to half-open, and half-open to closed;
- calls admitted, rejected locally, and handled by fallback;
- counted failures by category;
- half-open probes admitted, succeeded, failed, and timed out;
- rolling sample size and threshold values used for decisions.
State transitions deserve event-level records because a short open interval can disappear in coarse metric aggregation. At the same time, transition logs need rate control if many breaker keys can change state together.
Dashboards should separate remote failures from breaker rejections. During an outage, remote request volume may fall sharply after the breaker opens. That drop is expected protection, not evidence that the dependency recovered.
Recovery needs hysteresis
Opening and closing on the same instantaneous threshold can make a breaker oscillate near the boundary. A rolling window, cooldown, bounded half-open probes, and separate close criteria provide hysteresis.
That hysteresis is operationally important. The breaker should react quickly enough to persistent failure without switching states on every brief fluctuation. Exact values depend on request volume, dependency cost, latency objectives, and the expected duration of transient faults.
A circuit breaker is most useful when its failure classification, scope, timing, probe admission, and interaction with retries are explicit. Its main contribution is not detecting one failed call. It converts repeated failure evidence into temporary local rejection, then restores traffic through a controlled recovery path.