A service can fail even when most of its dependencies are healthy. One slow dependency may occupy every worker, connection, or concurrency slot until unrelated requests can no longer make progress.

This is a resource-isolation problem. The dependency failure matters, but the larger outage happens because the system lets one workload consume capacity that other workloads also need.

A bulkhead limits that sharing. It gives a workload a bounded resource budget so trouble in one area is less able to exhaust resources needed elsewhere. This article explains the mental model, shows where bulkheads help, and covers the trade-offs that make isolation useful rather than arbitrary.

Start with the shared-resource failure

Imagine an application with 20 worker slots. It handles two operations:

create_report -> reporting service
view_profile  -> profile service

Under normal conditions, both downstream calls finish quickly. Sharing all 20 workers seems efficient because idle capacity can serve whichever operation needs it.

Now the reporting service becomes very slow. Report requests arrive faster than they finish. Each request holds a worker while it waits.

20 shared workers

report report report report report ...
  |      |      |      |      |
waiting waiting waiting waiting waiting

Eventually all 20 workers can be occupied by report requests. A profile request then waits even though the profile service is healthy.

The cause-and-effect chain is important:

reporting slows
    -> report requests hold workers longer
    -> more workers become occupied
    -> shared capacity is exhausted
    -> profile requests cannot get workers

The reporting failure has crossed a boundary. It has become a profile outage too.

Treat capacity as a failure boundary

A bulkhead creates separate capacity limits for workloads that should not be able to exhaust one another.

For the teaching example, suppose the application reserves at most 8 concurrent workers for reporting and 12 for profile traffic:

reporting pool       profile pool
8 slots              12 slots

[........]           [............]

If reporting becomes slow, it can fill its eight slots. Additional report work must wait in a bounded queue, fail fast, or be rejected according to the system’s policy. It cannot occupy the 12 slots assigned to profile traffic.

The guarantee is deliberately narrow: the reporting workload cannot consume more than its configured share of that isolated resource.

A bulkhead does not guarantee that profile requests succeed. The process can still run out of memory, the network can fail, or both downstream systems can become unavailable. Isolation only protects against failure propagation through the resource being partitioned.

That limited guarantee is what makes the pattern useful. It turns an unbounded blast radius into a boundary engineers can reason about.

Isolate the resource that can actually be exhausted

“Use a bulkhead” does not necessarily mean “create another thread pool.” The relevant resource depends on how the application performs work.

Possible isolation points include:

  • worker or thread pools;
  • concurrent-request permits;
  • connection pools;
  • queue capacity;
  • process or service instances;
  • memory or CPU quotas provided by the runtime or platform.

The right question is: which shared resource lets this workload block unrelated work?

Suppose an asynchronous service does not dedicate a thread to each network request. Creating two thread pools may not address its real limit. If the service can have only 100 expensive downstream operations in flight before memory and connection pressure become unsafe, separate concurrency limits may be the useful boundary:

reporting: at most 20 in flight
profile:   at most 80 in flight

The mechanism changed, but the mental model did not. Each workload has a bounded claim on a resource whose exhaustion could spread failure.

Choose boundaries from failure relationships

Partitioning every endpoint independently usually creates more configuration than value. Bulkhead boundaries should reflect workloads with meaningfully different failure behavior or importance.

Consider an application that calls three external systems:

checkout -> payment provider
checkout -> recommendation service
admin    -> audit search service

Payment is required to complete checkout. Recommendations are optional. Audit searches are used by internal operators and can be expensive.

Putting all three behind one concurrency pool couples their failure modes. A burst of slow audit searches could consume capacity needed for payments. A slow recommendation provider could do the same even though recommendations are not required for a purchase.

Reasonable boundaries might therefore isolate payment, recommendations, and audit search from one another. The exact limits depend on traffic, resource cost, and service objectives; the important decision is based on which failures should remain independent.

This also shows why business priority alone is not enough. Two operations may both be important but still deserve separate limits if one can become slow in a way that should not consume the other’s capacity.

Decide what happens when a bulkhead is full

Isolation moves the problem from “one workload can consume everything” to a more explicit question: what should happen when that workload reaches its limit?

There are several valid policies.

For interactive requests, waiting briefly may be acceptable if the queue is bounded and the caller still has enough time budget. If waiting would make the response useless, rejecting quickly can be better because it releases pressure and gives the caller a clear failure.

For background work, a durable queue may absorb temporary excess demand when delayed processing is acceptable. That queue still needs a capacity and retention strategy; an unbounded backlog simply moves resource exhaustion somewhere else.

Optional work may have a fallback. If recommendation capacity is full, a checkout page might omit personalized suggestions rather than delaying the purchase. That is a product decision as much as a technical one.

The key is to make saturation behavior intentional. A bulkhead without a defined full-capacity policy can replace one hidden bottleneck with another.

Do not confuse isolation with time limits

Bulkheads and time limits solve related but different problems.

A timeout or deadline limits how long an operation may continue. A bulkhead limits how much shared capacity a class of operations may occupy at once.

If a downstream call can hang for a long time, a bulkhead prevents it from consuming every isolated slot, but its own slots may remain occupied indefinitely. Time limits help release those slots.

Conversely, a timeout on every request does not guarantee isolation. If thousands of slow requests can run concurrently for two seconds, they may exhaust connections or memory before their timeouts expire.

In many systems the controls therefore complement each other:

bulkhead: no more than 20 concurrent calls
local timeout: each call may wait at most 300 ms

Those numbers are examples, not recommended defaults. Limits should come from the system’s capacity, latency expectations, downstream behavior, and acceptable failure policy.

Account for queues when reasoning about capacity

A common mistake is to limit active work while allowing an effectively unlimited waiting queue.

Suppose reporting has eight execution slots but accepts 50,000 queued requests. The execution limit protects workers, yet the queue can still consume memory and make callers wait far beyond any useful response time.

A more complete bulkhead defines both active and waiting capacity:

active report operations: 8
queued report operations:  16
when both are full:         reject

Again, the values are illustrative. The important point is that waiting work consumes resources and time too.

Queueing also changes what users observe. A system may look healthy at the worker level while latency grows because requests spend most of their lifetime waiting for a slot. Monitor queue depth, rejection rate, wait time, and active capacity together when those signals are available.

Avoid making partitions too rigid

Isolation has a cost: reserved capacity can sit idle.

Imagine reporting receives no traffic while profile requests are overloaded. With strictly separate fixed pools, unused reporting capacity cannot help profiles. A fully shared pool would use resources more efficiently in that moment.

This is the central trade-off:

more sharing   -> better opportunistic utilization,
                  larger failure coupling

more isolation -> smaller blast radius,
                  potentially more idle capacity

Some systems use limits that provide isolation without permanently reserving physical resources. For example, workloads may share an underlying executor while separate concurrency permits cap how much each workload can consume. Whether that is sufficient depends on what resource can actually become exhausted.

Do not add a bulkhead merely because the pattern exists. If workloads are cheap, bounded, and fail together anyway, a shared limit may be simpler and easier to operate.

Watch for hidden shared resources

A visible bulkhead can give false confidence when important resources remain shared underneath it.

Two separate worker pools may still use:

  • the same small database connection pool;
  • the same memory heap;
  • the same downstream rate limit;
  • the same process CPU;
  • the same bounded socket or file-descriptor capacity.

If both workloads ultimately compete for one exhausted resource, separating workers may not contain the failure.

When designing a bulkhead, trace the path far enough to identify the constrained resource. Then state the protection precisely: “audit search cannot consume more than 10 database connections” is more useful than “audit search is isolated.”

Apply bulkheads where failure containment matters

Bulkheads are most useful when three conditions are present:

  1. workloads share a resource with finite capacity;
  2. one workload can consume enough of it to harm another;
  3. preserving the other workload during that failure is valuable.

That often occurs around slow external dependencies, expensive background jobs, optional features, tenant workloads with very different demand, or operations with different reliability requirements.

A simpler shared pool is often preferable when capacity is naturally bounded elsewhere, the workloads have the same failure fate, or the operational cost of separate limits exceeds the containment benefit.

Start from a concrete failure scenario rather than from the pattern name. Ask what can become exhausted, which workloads should remain available, and what each workload should experience when its allocation is full.

Conclusion

A bulkhead is a resource boundary for failure containment. It prevents one workload from making an unlimited claim on capacity that unrelated work also needs.

The practical sequence is straightforward: identify the shared resource that can be exhausted, group workloads by the failures that should remain independent, set bounded capacity, define what happens at saturation, and observe both active work and waiting work.

The goal is not perfect isolation. It is a smaller, explicit blast radius. When a dependency slows down, the system should lose the work that depends on it before it loses everything that happens to share the same resources.

Software Engineering Reliability Architecture Fault Isolation