A dependency can fail in a way that is worse than an immediate error. It may accept connections but respond slowly, time out repeatedly, or reject nearly every request while callers continue sending more work.

If your application keeps calling that dependency for every incoming request, the original failure can consume connection pools, worker capacity, and latency budgets in your own service. Retrying every failure can increase the pressure further.

A circuit breaker is a control around an operation that temporarily stops calls after failures indicate that the dependency is unhealthy. After a waiting period, it allows limited probe traffic to discover whether recovery has happened.

This article builds a practical mental model for circuit breakers, explains their states and decisions, and shows when they help—and when a timeout, retry policy, or simpler failure path is enough.

Start with the failure you are trying to contain

Imagine an order service that asks a recommendation service for optional suggestions:

suggestions = recommendationClient.forOrder(order)
return renderOrder(order, suggestions)

Normally the recommendation call takes 40 ms. During an incident, it starts timing out after 2 seconds.

The order service has not crashed, but each request now holds resources while waiting for work that is unlikely to succeed. If traffic continues, many requests can pile up behind the same unhealthy dependency.

A timeout limits how long one call waits. That is necessary, but it does not answer another question:

If recent calls already show that this dependency is failing, should every new request spend time discovering the same failure again?

A circuit breaker answers that question by remembering recent outcomes and changing whether new calls are attempted.

Use three states as the mental model

The common circuit-breaker model has three states: closed, open, and half-open.

The electrical terminology can be confusing at first. A closed electrical circuit allows current to flow, so a closed software circuit breaker allows calls to the dependency.

CLOSED
  calls are allowed
  failures are observed
       |
       | failure policy trips
       v
OPEN
  calls are rejected locally
       |
       | recovery delay passes
       v
HALF-OPEN
  limited probes are allowed
       |
       +-- probes succeed --> CLOSED
       |
       +-- probes fail ----> OPEN

The breaker does not repair the dependency. It changes how the caller behaves while the dependency is unhealthy.

Closed: call normally and measure outcomes

In the closed state, requests pass through:

result = breaker.call(() -> dependency.fetch())

Conceptually, the breaker does something like this:

if state == CLOSED:
    try:
        result = dependency.fetch()
        recordSuccess()
        return result
    catch failure:
        recordFailure(failure)
        if shouldTrip():
            state = OPEN
        raise failure

This is simplified pseudocode. Production implementations must coordinate concurrent callers and define exactly which outcomes count as failures.

That last point matters. A dependency returning “customer not found” may be a successful technical response even though the business result is negative. Counting every non-successful business outcome as infrastructure failure can open the breaker when the dependency is healthy.

The breaker should observe failures that indicate the protected operation is unlikely to work: for example, connection failures, relevant timeouts, or server-side failures according to the contract of that dependency.

Open: fail locally instead of repeating known-bad work

When the trip policy says the dependency is unhealthy, the breaker opens.

New calls do not reach the dependency:

if state == OPEN:
    raise CircuitOpen

This local rejection is the main protective effect. It can preserve resources and return control to the caller quickly instead of waiting for another likely timeout.

The caller still needs a policy for CircuitOpen. For optional recommendations, a reasonable response might be to return the order without suggestions:

try:
    suggestions = breaker.call(() -> recommendationClient.forOrder(order))
catch CircuitOpen:
    suggestions = []

return renderOrder(order, suggestions)

That fallback is valid only because recommendations are optional in this example.

If the protected dependency performs a required payment authorization, pretending success would be incorrect. The caller may instead return an explicit unavailable response, queue work only if the business process permits delayed completion, or use another deliberately designed path.

A circuit breaker does not make fallback behavior safe. It only creates a distinct signal that the protected call was intentionally not attempted.

Half-open: test recovery without releasing all traffic

An open breaker cannot stay open forever. Dependencies recover.

After a configured recovery delay, the breaker enters a half-open state and allows a limited number of probe calls. The purpose is to gather fresh evidence without immediately restoring full traffic.

If the probes succeed according to the recovery policy, the breaker closes and normal traffic resumes. If they fail, the breaker opens again and waits before another probe period.

Why limit probes?

Suppose 2,000 requests are waiting when the recovery delay expires. If all 2,000 are released simultaneously, a dependency that has only just recovered may be overwhelmed again. Half-open probing makes recovery gradual rather than turning the timer into a traffic gate.

The exact probe count and success rule depend on the system. There is no universal threshold that is correct for every dependency.

Choose a trip policy that represents evidence, not superstition

A breaker needs a rule for deciding when recent failures justify opening.

The simplest rule is a consecutive-failure threshold:

open after 5 consecutive qualifying failures

This is easy to understand, but it can behave poorly under mixed traffic. Four failures followed by one success reset the sequence even if the dependency is failing most requests.

Another approach evaluates a recent sample or time window:

open when:
- at least 20 calls have been observed, and
- at least 50% of them are qualifying failures

The minimum sample prevents a single failure from producing a misleading 100% failure rate. The ratio makes the decision reflect a broader pattern.

These numbers are examples, not recommendations. A low-volume internal tool, a high-volume checkout path, and a batch integration have different traffic shapes and consequences.

Choose the policy from the question: what amount of recent evidence is enough to justify temporarily refusing new attempts? Then test that policy against realistic traffic and failure patterns.

Keep timeouts even when you have a breaker

A circuit breaker and a timeout solve different problems.

A timeout limits one attempt:

one call -> stop waiting after its deadline

A circuit breaker limits repeated attempts across calls:

recent failures -> temporarily stop making new attempts

Without a timeout, calls in the closed or half-open state may still wait indefinitely or much longer than the caller can tolerate. The breaker cannot learn promptly from a call that never finishes.

In practice, the protected operation should usually have a bounded completion time appropriate to the caller’s latency budget, even when a circuit breaker surrounds it.

Be careful when combining breakers with retries

Retries can help when failures are brief and another attempt has a reasonable chance of succeeding. They can also multiply load on a struggling dependency.

Consider 100 incoming requests, each configured for three total attempts. A persistent failure can turn those 100 logical operations into as many as 300 dependency calls before other controls intervene.

A breaker can reduce repeated attempts after enough evidence accumulates, but it does not make aggressive retrying harmless. Retry count, backoff, jitter, timeout, and breaker behavior should be designed as one failure policy rather than independent defaults.

Also decide what the breaker observes. If a retry layer makes three attempts and reports one final failure to the breaker, the breaker sees logical-operation outcomes. If the breaker wraps each individual attempt, it sees every failed attempt. Those arrangements can trip at different rates.

Neither arrangement is automatically correct. Document the layer being measured so the breaker threshold has a clear meaning.

Scope the breaker around the failure domain

A breaker is useful only if its state corresponds to something meaningful.

Suppose one client talks to two independent endpoints:

catalogClient.fetchProducts()
catalogClient.fetchCategories()

If the product endpoint can fail independently of the category endpoint, one shared breaker may block healthy category calls because product calls are failing.

At the other extreme, creating a breaker per request stores no useful history across requests.

A practical scope often follows the unit that tends to fail and recover together: a remote service, endpoint group, tenant-specific dependency, or another operational boundary. The correct scope depends on how failures are isolated in the real system.

Ask:

If this operation is failing now, which other calls have enough shared fate that the same evidence should affect them?

That question is more useful than mechanically creating one breaker per client class.

Treat breaker state as operational information

An open breaker changes application behavior. Operators should be able to see that change.

Useful observations include:

  • transitions between closed, open, and half-open;
  • how often calls are rejected because the breaker is open;
  • the failures that contributed to opening;
  • probe outcomes;
  • how long the breaker remains open.

Avoid logging every rejected call at a high severity if that would create a flood during an incident. Metrics and sampled or transition-based logs often communicate breaker behavior more clearly.

The breaker should also preserve the distinction between two events:

dependency call attempted and failed

and:

dependency call skipped because circuit is open

Those events have different causes and require different debugging decisions.

Understand what a circuit breaker does not guarantee

Circuit breakers are sometimes described too broadly as a way to “make a service resilient.” Their guarantee is narrower.

A breaker can reduce calls to an operation after its failure policy detects trouble. It cannot guarantee that:

  • the dependency will recover;
  • fallback data is correct or fresh;
  • an operation that timed out did not complete remotely;
  • retries are safe;
  • partial side effects are rolled back;
  • your own service has enough capacity;
  • the chosen thresholds detect every harmful failure mode.

The timeout ambiguity is especially important for side-effecting operations. If a payment request times out, the caller may not know whether the remote system processed it. Opening a breaker afterward prevents some new calls, but it does not resolve the uncertain outcome of the original request. That problem requires operation-specific mechanisms such as idempotency or reconciliation.

Avoid breakers where a simpler mechanism is enough

Not every dependency needs a circuit breaker.

A breaker adds mutable state, thresholds, timers, concurrency concerns, metrics, and another failure mode to test. That cost is justified when repeated calls to an unhealthy dependency can materially harm the caller or the dependency.

A simpler timeout and clear error path may be enough when traffic is low, failures are cheap, calls are already strongly isolated, or the dependency rejects failures quickly without consuming significant resources.

Similarly, bounded concurrency or load shedding may address the real problem when the danger is too much work in general rather than one unhealthy dependency.

Use a circuit breaker because remembered failure history changes a useful decision—not because resilience designs are expected to contain one.

Test state transitions, not just happy-path calls

A breaker is a small state machine, so its tests should exercise transitions explicitly.

At minimum, verify scenarios such as:

closed + successes -> remains closed
closed + trip-level failures -> opens
open + new call -> dependency is not called
open + recovery delay -> permits limited probe
half-open + successful recovery evidence -> closes
half-open + failure -> opens again

Time-dependent tests are easier to keep deterministic when the implementation can use a controllable clock rather than sleeping in real time.

Also test classification. A business-level negative result that should not count as dependency failure is just as important as a timeout that should.

For concurrent implementations, verify that the chosen library or design limits half-open probes correctly and performs state transitions safely under simultaneous calls. Reimplementing those details casually can create subtle races, so a mature platform library is often preferable when one is available and its semantics fit the system.

A practical decision sequence

When a dependency causes cascading latency or repeated wasted calls, start with the failure path rather than the pattern name.

First, give individual calls appropriate timeouts. Then decide whether retries are justified and safe. If repeated new requests still keep spending resources on a dependency that recent evidence says is unhealthy, consider a circuit breaker.

Define which outcomes count as dependency failures, how much evidence opens the breaker, how long to wait before probing, how much probe traffic is allowed, and what callers do when calls are rejected locally.

Finally, make state transitions observable and test the failure behavior. A breaker whose thresholds and fallback semantics nobody understands can hide incidents rather than contain them.

Conclusion

A circuit breaker is remembered failure evidence turned into a temporary traffic decision.

While closed, it lets calls through and observes outcomes. When failures cross a deliberate threshold, it opens and rejects new calls locally. After a delay, half-open probes test whether the dependency is ready for normal traffic again.

The useful mental model is not “add a breaker for reliability.” It is: stop paying repeatedly for a failure you already have enough evidence to expect, then restore traffic carefully when new evidence shows recovery.

That makes circuit breakers most valuable when a failing dependency can consume meaningful caller resources or amplify an incident. Where repeated failure is cheap, a timeout and explicit error path may remain the clearer design.

Software Engineering Reliability Fault Tolerance Resilience