Bulkhead Isolation: Contain Failures with Separate Capacity Pools

A service can have enough total capacity and still become unavailable because one dependency consumes all of it.

Imagine an API that calls a payment service and a recommendation service. Both outbound calls use the same worker pool. Recommendations become slow during a traffic spike. Their requests occupy every worker while waiting for responses. Payment requests now have no worker available, even though the payment service itself is healthy.

The failure crossed a boundary that should have existed.

Bulkhead isolation divides a finite resource into separate capacity pools so that pressure in one workload cannot consume every slot needed by another. The name comes from ship compartments: damage in one compartment does not automatically flood the entire hull.

In software, the separated resource might be threads, asynchronous concurrency permits, connection pools, queue capacity, process groups, or even distinct service instances.

The central rule is simple:

A workload may exhaust its own allocation, but it must not be able to exhaust every allocation.

That rule turns one large failure domain into several smaller ones.

Shared capacity creates hidden coupling

Consider a service with 100 concurrent outbound-call slots:

API
 |
 +-- shared pool: 100 slots
       |
       +-- inventory
       +-- recommendations
       +-- payments

Under normal traffic, this arrangement looks efficient. Idle capacity can serve any dependency.

Now suppose the recommendation service slows from 50 milliseconds to 10 seconds. Requests continue arriving. Recommendation calls remain in flight much longer, so they accumulate in the shared pool.

Once all 100 slots are occupied, a payment request must wait for a slot before it can even contact the healthy payment service.

The recommendation dependency has created resource coupling. Its latency now controls payment availability through a resource both paths share.

Timeouts help release slots eventually, but they do not remove the coupling. A five-second timeout still allows slow calls to monopolize capacity for five seconds at a time.

A circuit breaker can stop calls after enough failures are observed, but it also solves a different problem. During the period before the breaker opens, or during partial degradation that does not meet its threshold, shared capacity can still disappear.

Bulkheads address the resource boundary itself.

Split capacity by failure domain

Suppose the 100 outbound slots become three pools:

API
 |
 +-- inventory pool:       35
 +-- recommendation pool:  25
 +-- payment pool:         40

If recommendation calls fill all 25 recommendation slots, the other 75 slots remain unavailable to that workload. Inventory and payment traffic retain their allocations.

The recommendation path may reject requests, serve a fallback, or return a degraded response. That is a local failure. The payment path can continue.

Isolation changes the failure geometry. Instead of asking only whether the process has spare capacity, the design asks which workload is permitted to consume that capacity.

A semaphore is often enough

Bulkhead isolation does not require a separate process for every dependency. In asynchronous applications, a semaphore can provide a useful concurrency boundary.

Here is a Go example using a buffered channel as a fixed set of permits:

package outbound

import (
    "context"
    "errors"
)

var ErrCapacity = errors.New("dependency capacity exhausted")

type Bulkhead struct {
    permits chan struct{}
}

func NewBulkhead(limit int) *Bulkhead {
    return &Bulkhead{
        permits: make(chan struct{}, limit),
    }
}

func (b *Bulkhead) Do(ctx context.Context, fn func(context.Context) error) error {
    select {
    case b.permits <- struct{}{}:
        defer func() { <-b.permits }()
    default:
        return ErrCapacity
    }

    return fn(ctx)
}

A service can create one instance per dependency:

var (
    paymentSlots = outbound.NewBulkhead(40)
    catalogSlots = outbound.NewBulkhead(30)
)

func charge(ctx context.Context) error {
    return paymentSlots.Do(ctx, callPayment)
}

func fetchCatalog(ctx context.Context) error {
    return catalogSlots.Do(ctx, callCatalog)
}

This version rejects immediately when a pool is full. Immediate rejection is often safer than placing unlimited requests behind the semaphore. An unbounded waiting line merely moves overload from active calls into queued goroutines and retained request state.

A bounded queue can be valid when short bursts are expected, but its limit is part of the bulkhead. Both active work and waiting work consume resources.

Size pools from demand and service time

Pool sizes should come from workload behavior, not equal division by convenience.

A useful first approximation comes from Little’s Law:

concurrency = throughput × average time in system

If a dependency normally receives 80 requests per second and each call takes 100 milliseconds on average:

80 × 0.1 = 8 concurrent calls

A limit of exactly 8 leaves no room for normal variation, so production limits usually include measured headroom. Tail latency matters as well. If response times occasionally jump to one second, concurrency demand can rise sharply even when request rate stays constant.

Use production measurements to examine:

• request rate per isolated workload,
• typical and tail latency,
• connection limits imposed by dependencies,
• acceptable rejection rate during bursts,
• memory and CPU cost per in-flight request,
• priority of the user operation.

A critical payment path may deserve reserved capacity even when its average traffic is lower than a noncritical path.

The goal is not maximum utilization of every slot. The goal is predictable containment under stress.

Separate the scarce resource that can spread damage

A logical bulkhead is effective only when it isolates the resource that actually becomes scarce.

Suppose two dependency clients have separate semaphores but share one database connection pool. If either workload can consume every database connection, the semaphore split may not protect the other path.

Likewise, separate connection pools do not fully isolate workloads if both execute CPU-heavy work on the same saturated executor.

Trace the resource chain:

request
  |
concurrency permit
  |
worker or goroutine
  |
connection pool
  |
remote dependency

At each layer, ask whether one workload can occupy all available capacity. The most important boundary is usually near the resource whose exhaustion blocks unrelated work.

Strong isolation may use several layers together: separate concurrency limits, separate connection pools, bounded queues, and distinct process replicas.

Bulkheads and timeouts work together

A bulkhead limits how many operations may be in flight. A timeout limits how long each operation may occupy a slot.

Both dimensions matter.

With 20 permits and no timeout, 20 calls that never finish can permanently exhaust the pool. With a two-second timeout and no bulkhead, a sudden surge can still create thousands of concurrent calls for up to two seconds.

Combined:

maximum occupied slots = 20
maximum intended hold time = 2 seconds

This creates a much tighter bound on resource exposure.

The timeout must also propagate to the remote operation. Releasing a local permit while abandoned work continues consuming a downstream connection or server resource gives a false sense of containment.

Bulkheads and circuit breakers solve different problems

These patterns are often paired, but they should not be treated as substitutes.

A bulkhead protects local capacity from excessive concurrent use. It can reject the 21st call even when the first 20 calls are technically successful but slow.

A circuit breaker uses recent outcomes to decide whether new calls should be attempted. It can stop sending traffic to a dependency that is consistently failing.

A common request path is:

request
  |
bulkhead admission
  |
circuit-breaker decision
  |
timeout-bound remote call

The exact ordering can vary with the client library and desired metrics, but each control needs a precise responsibility. Capacity rejection, dependency health, and call duration are separate signals.

Partition by behavior, not merely by endpoint

A single dependency can contain workloads with very different risk.

For example, a document service might support:

• small metadata reads,
• large document exports,
• background indexing.

Putting all three behind one pool lets slow exports crowd out fast metadata reads. Separate bulkheads can preserve the interactive path.

Useful partition keys include dependency, operation class, tenant tier, traffic source, request priority, and synchronous versus background work.

Too many partitions create another problem: stranded capacity. Ten pools with ten slots each cannot lend idle slots to a busy pool, even when total utilization is low.

This is the main trade-off of strict bulkheads: stronger isolation reduces capacity sharing.

Use the smallest number of partitions that correspond to meaningful failure domains.

Reserved capacity can coexist with shared capacity

Isolation does not always require a completely fixed partition.

A design can reserve a minimum amount for critical traffic while allowing some excess capacity to be shared. For example:

total capacity: 100

reserved for critical traffic: 30
shared capacity:               70

Critical traffic can use its reserved 30 plus available shared slots. Noncritical traffic can use only the shared 70.

This approach improves utilization while preserving a floor for important operations. It is more complex than fixed pools, so the admission rules must remain observable and testable.

Rejection is part of the design

A full bulkhead needs an explicit response policy. Capacity limits without a rejection plan simply move the failure elsewhere.

Possible responses include:

• return a fast overload error,
• serve cached or reduced data,
• omit an optional response section,
• enqueue work in a bounded durable queue,
• retry at a higher layer with a strict attempt budget.

Retries require special care. Immediate retries against a full pool increase arrival rate exactly when capacity is scarce. If retry is appropriate, use backoff, jitter, and a total deadline or attempt budget.

For optional features, graceful omission is often better than waiting. A product page may remain useful without recommendations; a checkout request may not remain useful without payment processing. The fallback should reflect product semantics.

Observe saturation, not only failures

A bulkhead can protect availability while hiding a growing capacity problem unless its state is measured.

Track at least:

in_flight
limit
admission_rejections
queue_depth
queue_wait_time
operation_latency
timeouts

The ratio in_flight / limit shows saturation. Rejection counts show demand that could not enter. Queue wait time shows contention before work begins.

Alerting only on downstream error rate misses a common case: the dependency stays successful, but latency rises enough to keep the bulkhead nearly full. Users then see local rejections even though remote calls mostly return success.

Capacity metrics should be segmented by bulkhead name so operators can see which failure domain is under pressure.

Test containment as a system property

A unit test can confirm that the 21st request is rejected when a limit is 20. That is useful, but the more important property is cross-workload containment.

A resilience test can:

1. make dependency A respond very slowly,
2. drive enough A traffic to fill its allocation,
3. send normal traffic to dependency B,
4. verify that B still meets its latency and success objectives,
5. verify that A traffic is bounded and rejected according to policy.

This test checks the architectural promise: one damaged path cannot consume the capacity reserved for another.

Also test recovery. Once A becomes healthy, permits must return, queues must drain within bounds, and normal admission must resume without a restart.

Common mistakes

One global pool with per-client timeouts

Timeouts bound duration, but every workload still competes for the same slots. A slow path can impair unrelated paths until those timeouts expire.

Unlimited waiting behind a fixed semaphore

The active-call count is bounded, but memory and latency are not. Bound the waiting line or reject immediately.

Pools sized from average traffic only

Average latency and request rate hide bursts and tails. Use distributions, peak periods, and dependency limits.

Isolation at the wrong layer

Separate client objects provide little protection if they share the resource that actually saturates.

Treating rejection as an implementation error

Rejection is the mechanism that preserves the boundary. Handle it as an expected overload outcome with metrics and an intentional product response.

Too many tiny partitions

Excessive partitioning strands capacity and increases configuration burden. Isolate meaningful failure domains rather than every code path.

A practical design sequence

Start with a service map and identify operations that must remain available independently. Then identify the finite resources those operations share.

For each important boundary:

1. choose the resource to isolate,
2. define a concurrency or queue limit,
3. set a maximum operation duration,
4. define the full-pool response,
5. add saturation and rejection metrics,
6. test one partition under sustained degradation,
7. confirm unrelated partitions remain healthy,
8. adjust limits from production evidence.

Do not begin by copying a thread-pool count from another service. Capacity depends on request rate, latency, downstream limits, runtime behavior, and business priority.

The deeper principle

Bulkhead isolation is a form of failure-domain design.

A system is not resilient merely because it has spare capacity. It is resilient when that capacity is arranged so that one fault cannot consume resources required by unrelated work.

Shared pools optimize for flexibility during healthy operation. Isolated pools trade some of that flexibility for containment during abnormal operation.

That trade is valuable when a service has dependencies or workloads with different latency profiles, reliability characteristics, or business importance.

The design question is not only:

How much capacity does this service have?

It is also:

Which failures are allowed to consume which capacity?

A good bulkhead makes that answer explicit, bounded, measurable, and testable.