When a downstream service is failing, continuing to send every request can waste threads, sockets, and latency budget while increasing load on the unhealthy dependency. A circuit breaker temporarily fails calls fast after failures cross a threshold.
Understand the three states
A typical breaker is closed while calls flow normally. It moves open after a configured failure policy is exceeded. After a recovery interval, it becomes half-open and allows a limited number of probes. Successful probes close the circuit; failures open it again.
The breaker should protect one meaningful dependency boundary. Combining unrelated endpoints into one global breaker can make a failure in one capability disable healthy traffic elsewhere.
Coordinate with timeouts and retries
Circuit breakers do not replace timeouts. A call still needs a deadline so a slow dependency cannot consume resources indefinitely.
Retries also need coordination. Retrying several times inside a breaker can amplify load before the breaker sees one final failure. Decide deliberately which layer owns retries and ensure the total retry budget fits inside the caller’s deadline.
Count the right failures
Not every error means the dependency is unhealthy. A validation error or authorization denial usually should not trip a breaker. Connection failures, timeouts, and selected server errors are stronger health signals.
Define classification close to the client adapter so transport details do not leak throughout business logic.
Make half-open probes scarce
If thousands of instances all probe at once, recovery can create another traffic spike. Limit concurrent probes and add jitter to recovery timing where the implementation supports it.
A breaker should help a recovering service receive a small, controlled sample rather than an immediate return to peak load.
Choose fallbacks carefully
Useful fallbacks include cached read data, a reduced feature set, asynchronous acceptance, or a clear temporary error. A fallback must preserve business correctness.
Do not return stale security policy, inventory, account balance, or other correctness-sensitive data merely to improve availability.
Observe breaker behavior
Record state transitions, rejected calls, probe outcomes, and the dependency involved. Alerting on every open circuit can be noisy; sustained or widespread open states are usually more actionable.
Avoid high-cardinality labels such as raw URLs, user IDs, or request IDs in breaker metrics.
Common pitfalls
One breaker for every error
Application-level failures can open a circuit even when the dependency is healthy. Classify errors first.
Very short recovery intervals
Rapid probing can hammer a dependency that needs time to recover. Base timing on operational behavior and test it under failure.
Hidden fallback success
If fallback responses look identical to fresh responses, operators and callers may miss degraded behavior. Expose degradation through metrics and, where appropriate, response metadata.
Breakers without load testing
Thresholds that look sensible on paper can behave badly at low traffic or during bursts. Test failure ratios, minimum sample sizes, concurrency, and recovery behavior.
Treat breakers as load-control mechanisms
A circuit breaker is not primarily an error-handling convenience. It limits wasted work and gives dependencies room to recover. Combined with bounded timeouts, restrained retries, and observable fallbacks, it can prevent one failing service from turning into a wider cloud outage.