Circuit Breakers Bound Failure Traffic Across Service Calls

A circuit breaker changes the admission decision for an outbound call before the dependency receives it. In the closed state, calls proceed and their outcomes feed a failure policy. Once that policy trips, the breaker enters the open state and rejects subsequent calls locally. After a configured recovery interval, a limited set of probe calls can test whether the dependency is usable again.

That mechanism is distinct from retries. A retry issues another attempt after a failed attempt. A breaker can prevent an attempt from being issued at all. Combining the two without a precise ordering can amplify traffic during an outage or keep a breaker open based on signals that do not represent dependency health.

The breaker state belongs at a call boundary

A useful breaker surrounds one dependency operation whose failures have comparable meaning. The protected boundary might be calls to one remote service, one database endpoint, or one external API operation. A single breaker shared across unrelated destinations can couple independent failure domains: trouble in one destination can suppress calls to another healthy destination.

The state machine normally has three conceptual states:

The exact transition policy is implementation-specific. A breaker can use consecutive failures, a ratio over a rolling window, minimum sample counts, latency thresholds, or selected error classes. Those choices are not interchangeable. A ratio based on two calls has very different statistical meaning from the same ratio over hundreds of calls, so practical policies often require a minimum volume before tripping.

Failure classification defines the signal

Not every unsuccessful application result indicates an unhealthy dependency. An HTTP 404 can be a valid domain response. An HTTP 429 can indicate capacity pressure and may deserve different handling from a malformed request that receives 400. Connection refusal, transport timeout, and selected 5xx responses can carry stronger evidence that another attempt is unlikely to succeed immediately.

The breaker therefore needs an explicit classification function. Conceptually:

result = call_dependency()

if result indicates dependency failure:
    breaker.record_failure()
else:
    breaker.record_success()

The classification must match the protected operation. Treating every non-success status as a breaker failure can turn normal client errors into infrastructure suppression. Ignoring transport failures has the opposite effect: the breaker remains closed while callers continue spending connection and timeout budgets on a dependency that is not responding.

Timeouts also need a defined owner. If a caller abandons a request due to its own deadline while the dependency later completes successfully, the breaker policy must specify which observation counts. Libraries differ, and cancellation propagation can change the result visible at the breaker boundary.

Open state converts remote cost into local rejection

When a breaker is open, rejection happens without waiting for the protected remote operation. This changes both resource consumption and error semantics. No new socket, stream, database query, or remote request needs to be created for a rejected call, assuming those resources are acquired inside the protected boundary.

The local rejection should remain distinguishable from a remote failure. A caller may use that distinction for metrics, fallback selection, or response mapping. Collapsing “breaker open” into the same error as “remote timeout” hides whether work reached the dependency.

Open-state rejection is not a durability or delivery guarantee. It simply says that the breaker declined the call. If the operation represents required work, another mechanism must define persistence, queueing, compensation, or later execution.

Half-open admission is a concurrency contract

Allowing every waiting caller through as soon as the recovery interval expires can recreate the original overload. Half-open state exists to constrain that transition. A breaker can admit one probe, a small fixed number, or another bounded amount of concurrent work while rejecting or holding the rest.

That limit needs atomic coordination when many threads, goroutines, processes, or event-loop tasks share the breaker. A check followed by a separate increment can exceed the intended probe count under races:

if probes_in_flight < probe_limit:
    probes_in_flight += 1
    send_probe()

Two callers can observe the same old value before either increment becomes visible. Implementations commonly protect this state with a lock, an atomic operation, or ownership confined to one serialized execution context.

A process-local breaker coordinates only callers in that process. Ten application instances with a one-probe half-open limit can still generate up to ten concurrent probes. A deployment that requires a cluster-wide cap needs shared coordination or a different admission mechanism. Process-local state must not be described as a distributed limit.

Retries alter the observations seen by the breaker

Breaker placement relative to a retry loop changes what gets counted. With the breaker outside the retry mechanism, one logical request may produce one recorded outcome after several attempts. With the breaker around each attempt, every failed attempt can contribute separately to the breaker policy.

Consider a client configured for three attempts:

breaker(
    retry_up_to_three_times(remote_call)
)

This shape can expose only the final retry result to the breaker. Reversing the composition:

retry_up_to_three_times(
    breaker(remote_call)
)

can expose each admitted attempt to breaker accounting, depending on the library API. Once the breaker opens, later retry iterations may receive local open-state rejection instead of reaching the remote service.

Neither composition is universally correct. The intended signal decides the boundary. Attempt-level protection reacts to repeated remote failures more quickly but couples breaker statistics to retry configuration. Request-level protection treats the retry policy as one operation but can hide repeated load imposed on the dependency.

Recovery intervals are not health checks

A timer that moves a breaker from open toward half-open does not establish that the dependency has recovered. It only creates permission to test again. Closing the breaker before observing successful probes turns elapsed time into an unsupported health assertion.

Probe success also needs a defined threshold. One successful request may be enough for a simple policy, while another system may require several successful probes before restoring full admission. The choice trades recovery speed against the risk of sending normal traffic to a dependency that is only partially available.

A fixed recovery interval can also synchronize many clients. If numerous instances trip at roughly the same time and use the same interval, their probes can align. Jitter or instance-specific timing can reduce that synchronization, but the exact policy remains a deployment choice rather than a property of the breaker pattern itself.

Metrics need state and outcome dimensions

A breaker changes traffic before remote telemetry can observe it. During an open interval, the dependency may show fewer failed requests simply because callers stopped sending them. Service-side error rates alone therefore cannot describe caller-side availability.

Useful observations include breaker state transitions, local rejection counts, admitted call outcomes, probe outcomes, and time spent in each state. These signals should retain the identity of the protected dependency boundary without creating unbounded metric cardinality.

A high local rejection count paired with low remote traffic can be expected during an open interval. A breaker that repeatedly alternates between half-open and open indicates that probes continue to fail or that the closing criteria are too permissive for the dependency’s current condition. The metric interpretation follows directly from the state machine.

Breaker scope determines the failure domain

The most consequential breaker setting is often not a numeric threshold but the scope of shared state. A breaker per request has almost no memory across calls. A breaker shared across every outbound dependency can suppress unrelated traffic. A breaker per endpoint, operation class, or tenant can isolate failures more precisely but creates more state and operational dimensions.

The scope should match the resource or dependency whose availability the recorded outcomes actually represent. Once that boundary is explicit, thresholds, recovery intervals, probe concurrency, retry composition, and telemetry can be evaluated against the same failure domain rather than tuned as independent knobs.