Concurrency Limits: Bound In-Flight Work
A service can receive traffic at an acceptable average rate and still collapse because too many operations overlap. The issue is not only how many requests arrive per second. It is also how many requests are active at the same time.
A concurrency limit places a cap on active work. When all slots are occupied, additional work must wait, fail fast, or take another explicit path. This simple control can protect database connections, CPU-heavy routines, remote dependencies, worker capacity, and memory that grows with each active operation.
The central idea is:
Bound the work already in progress, not just the work entering the system.
That distinction matters most when operation duration changes under load.
Rate and concurrency control different dimensions
Suppose a service accepts 100 requests per second.
If each request completes in 20 milliseconds, only a small number may overlap. If each request suddenly takes two seconds, roughly 200 requests can be active even though the arrival rate has not changed.
A useful steady-state relationship is:
concurrency ~= throughput * average durationAt 100 requests per second:
20 ms average duration -> about 2 active requests
2 s average duration -> about 200 active requestsThe same request rate can therefore produce radically different pressure on a dependency.
A rate limiter controls admission over time. A concurrency limiter controls simultaneous occupancy. Systems often need both because each addresses a separate failure mode.
Slowdown can create a feedback loop
Imagine an API handler that calls a database. Under normal conditions, each query takes 30 milliseconds. A traffic spike increases contention, and query duration rises to 300 milliseconds.
Longer queries hold database connections for longer. More requests overlap. The connection pool fills. Requests wait for connections, so end-to-end duration rises again. That longer duration increases overlap further.
The sequence can look like this:
dependency slows
|
active work grows
|
resource contention grows
|
operations take longer
|
active work grows againThis is a positive feedback loop. Without a bound, a modest slowdown can become an overload event.
A concurrency limit interrupts the loop by refusing to let active work grow without control.
A semaphore is the basic mechanism
A semaphore with N permits can represent N units of concurrent capacity. An operation must acquire a permit before entering the protected section and release it after completion.
In Go, the core shape can be expressed with a buffered channel:
type Limiter struct {
slots chan struct{}
}
func NewLimiter(limit int) *Limiter {
if limit <= 0 {
panic("limit must be positive")
}
return &Limiter{
slots: make(chan struct{}, limit),
}
}
func (l *Limiter) Acquire(ctx context.Context) error {
select {
case l.slots <- struct{}{}:
return nil
case <-ctx.Done():
return ctx.Err()
}
}
func (l *Limiter) Release() {
<-l.slots
}A protected operation can then use it like this:
func (s *Service) GenerateReport(
ctx context.Context,
id string,
) (Report, error) {
if err := s.reportLimit.Acquire(ctx); err != nil {
return Report{}, err
}
defer s.reportLimit.Release()
return s.generator.Generate(ctx, id)
}The defer is important. Every successful acquisition must have a matching release, including error and cancellation paths.
The mechanism is simple. The engineering decisions around placement, waiting, fairness, sizing, and failure behavior require more care.
Put the limit around the scarce resource
A limit is most useful when it corresponds to the resource you intend to protect.
Consider an endpoint that performs validation, reads cached metadata, runs an expensive renderer, and then stores a small result. If rendering is the scarce step, placing the limit around the entire handler wastes permits on cheap work.
A narrower boundary is usually better:
func (s *Service) Build(
ctx context.Context,
input Input,
) (Output, error) {
validated, err := validate(input)
if err != nil {
return Output{}, err
}
meta, err := s.cache.Get(ctx, validated.Key)
if err != nil {
return Output{}, err
}
if err := s.renderLimit.Acquire(ctx); err != nil {
return Output{}, err
}
defer s.renderLimit.Release()
return s.renderer.Render(ctx, validated, meta)
}This lets inexpensive preparation proceed without consuming renderer capacity.
The opposite error is placing the limit too deep. If several call paths reach the same scarce dependency through different adapters, a limiter in only one adapter leaves the resource exposed through the others. The control boundary should cover every path that competes for the same constrained capacity.
Waiting is not free
A common implementation acquires a permit by waiting indefinitely. That caps active work, but it can move the overload problem into an unbounded queue of waiting requests.
Suppose the limit is 50 and 10,000 requests arrive. Only 50 enter the protected operation, but thousands may remain suspended in memory. They retain request state, deadlines, tracing data, buffers, and client connections. Many may have no realistic chance of completing before their callers give up.
A concurrency limit needs an explicit policy for excess demand.
Common choices are:
- Wait with a deadline. Suitable when short contention is expected and callers can tolerate bounded delay.
- Reject immediately. Suitable when freshness matters more than eventual execution or when upstream retry policy is controlled.
- Use a bounded queue. Suitable when a small amount of smoothing is useful and queue capacity has a clear operational meaning.
- Degrade the operation. Suitable when a cheaper response can preserve useful service.
The key property is boundedness. Active work is bounded, and waiting work should also have a bound.
Queue time consumes the caller’s budget
If a request has a one-second deadline and waits 800 milliseconds for a permit, only 200 milliseconds remain for the protected operation.
Code should preserve that deadline instead of starting a fresh timeout after admission.
func (s *Service) Fetch(
ctx context.Context,
key string,
) (Value, error) {
if err := s.limit.Acquire(ctx); err != nil {
return Value{}, err
}
defer s.limit.Release()
return s.store.Fetch(ctx, key)
}Here the same context covers both permit acquisition and the dependency call. Cancellation during the wait prevents stale work from entering later.
Creating a new full timeout after acquisition can accidentally let queueing extend total request duration beyond the caller’s intended budget.
Limit the correct unit of work
Not every operation costs the same amount.
A semaphore with one permit per request assumes that each request consumes roughly comparable capacity. That is often good enough, but it can be misleading when one request uses 100 times more memory or CPU than another.
For uneven workloads, consider weighted admission:
small export -> 1 permit
medium export -> 3 permits
large export -> 8 permitsA weighted semaphore caps estimated resource use rather than request count.
The estimate does not need perfect precision. It needs enough correlation with actual cost to prevent a few large operations from bypassing the protection that a request-count limit was meant to provide.
If cost cannot be estimated before execution, separate queues or limits for known workload classes can still reduce interference.
Global and per-key limits solve different problems
A global limit protects total capacity:
all report generation: at most 40 activeA per-key limit protects fairness or hot spots:
per tenant: at most 5 activeUsing only a global limit allows one noisy tenant to occupy every slot. Using only per-tenant limits can still overload the service when many tenants are active at once.
The controls can be composed:
request
|
acquire global slot
|
acquire tenant slot
|
perform workBe consistent about acquisition order. If different code paths acquire multiple limiters in different orders, they can create deadlock.
An alternative is to make one admission component own both checks so callers do not coordinate permits themselves.
Choose a limit from capacity, then test it
A concurrency limit should come from the protected system’s useful operating range, not from an arbitrary round number.
For a database-backed operation, useful signals include:
- connection pool capacity,
- database CPU and I/O saturation,
- query latency as concurrency rises,
- lock contention,
- timeout rate,
- throughput at different active-request counts.
For CPU-bound work, start with available CPU parallelism and benchmark the actual workload. More active tasks than CPU cores can be appropriate when tasks block, but excessive runnable work can increase scheduling overhead and tail latency.
For a remote service, the safe limit may be constrained by its published quotas, observed latency curve, or a contract between teams.
Run load tests across a range of concurrency values and observe throughput and latency together. The best limit is often near the point where additional concurrency stops producing useful throughput and starts producing disproportionate latency or errors.
Do not size from peak traffic alone
Peak request rate does not directly tell you the right concurrency cap.
Suppose peak demand is 500 requests per second and the protected operation normally takes 40 milliseconds. That suggests about 20 concurrent operations in steady state. Setting the limit to 500 because the traffic peak is 500 requests per second confuses rate with occupancy.
Also account for duration under stress. If the dependency becomes slower before saturation, the safe concurrency may need to remain below the level that triggers that slowdown.
Capacity tests are more informative than traffic counts alone.
A fixed limit can be safer than a clever adaptive one
Adaptive concurrency control can adjust limits from observed latency or queueing signals. It can be valuable when capacity varies substantially over time, but it introduces another feedback controller into the system.
A fixed limit has useful properties:
- its maximum pressure is easy to state,
- incidents are easier to reason about,
- configuration changes are auditable,
- behavior does not depend on a tuning algorithm.
Start with a fixed limit when the protected capacity is reasonably stable. Move to adaptive control only when measurements show that a static setting leaves significant capacity unused or cannot cope with predictable variation.
A sophisticated controller is not automatically safer than a conservative constant.
Retries can defeat admission control
Suppose a service rejects excess work quickly, but every caller retries immediately. Rejection then creates another burst, which creates more rejection, followed by more retries.
Concurrency control and retry policy must fit together.
Clients should generally use:
- bounded retry counts,
- backoff,
- jitter,
- an overall deadline,
- retry decisions based on operation semantics.
For non-idempotent operations, automatic retries can also duplicate effects unless the API provides an idempotency mechanism.
A limiter protects the local process only if surrounding behavior does not turn rejection into uncontrolled amplification.
Measure occupancy, waiting, and rejection
A limiter that has no telemetry is difficult to tune.
At minimum, record:
configured limit
current active count
permit wait duration
waiting count
rejection count
operation duration after admissionThese signals answer different questions.
High active count near the limit means the protected capacity is fully used. High wait duration means callers are queueing for access. High rejection count means offered demand exceeds the configured admission policy. High operation duration after admission suggests the protected dependency itself is slow even after concurrency is bounded.
Do not rely on utilization alone. A limiter sitting at 100% occupancy may be healthy if operations remain fast and the waiting queue stays small. The same occupancy with rising wait time and timeouts indicates sustained overload.
Preserve the distinction between waiting and service time
End-to-end latency combines at least two components:
total duration = admission wait + operation durationTrack them separately.
If admission wait rises while operation duration stays stable, the limiter is successfully preventing extra pressure but demand exceeds capacity.
If operation duration rises as occupancy approaches the limit, the configured limit may still be too high.
If both stay low but callers see high total latency, the bottleneck is elsewhere.
Separating these measurements makes tuning far more precise than looking at one request-latency percentile.
Avoid holding permits across unrelated waits
A permit should represent active use of the constrained resource.
Consider this sequence:
acquire database-work permit
call unrelated remote API
wait 700 ms
run database query
release permitThe permit is occupied during 700 milliseconds in which the database is not being used. Effective capacity drops even though the protected resource is idle.
Acquire as late as practical and release as early as practical:
call unrelated remote API
acquire database-work permit
run database query
release permitThis is similar to keeping critical sections small around a mutex. The narrower the protected interval, the more accurately permit occupancy represents actual pressure.
Keep limits close to the capacity they protect
A process-local semaphore protects only that process.
If ten service instances each allow 50 concurrent database operations, the deployment can produce up to 500 concurrent operations. That may be correct, or it may overwhelm a shared database.
There are several ways to handle this:
- assign each instance a conservative share of total capacity,
- size instance limits from deployment scale,
- enforce a limit at a shared gateway or worker tier,
- expose a bounded downstream pool that naturally constrains total concurrency.
A distributed semaphore is possible, but it adds coordination cost and failure modes. Often a local limit combined with bounded shared capacity is simpler.
The important step is to state the scope explicitly: per process, per host, per tenant, per cluster, or per downstream resource.
Concurrency limits are not locks
A concurrency limiter controls how much work proceeds. It does not normally protect a data invariant.
A mutex may ensure only one routine modifies a particular in-memory structure at a time. A database transaction may protect consistency across records. A concurrency limiter may allow 20 independent operations to run because the system can support 20 safely.
These mechanisms can coexist, but they answer different questions:
lock: can these operations overlap safely?
concurrency limit: how many safe operations can the system sustain at once?Confusing them can produce either correctness bugs or needless serialization.
Concurrency limits are not connection pools
A connection pool already bounds the number of checked-out connections, so it can act as a form of concurrency control for database access. But relying only on the pool can make every caller queue at the deepest possible point.
An application-level limit can reject or defer work earlier, before it allocates substantial request state or begins other expensive preparation.
The two limits should be coordinated. If the application allows 200 database-bound operations but the pool has 20 connections, most of those operations simply wait at the pool. If the application limit is 15, it may underuse a pool that can safely sustain 20.
Think of the pool as a hard resource boundary and the application limiter as an admission policy around that boundary.
A practical implementation checklist
Before adding a concurrency limit, identify the scarce resource and the exact work that consumes it. Decide whether the limit applies per process or across a broader scope. Set a bounded policy for excess demand. Preserve caller cancellation during permit waits. Release permits on every exit path.
Then validate the setting under realistic load. Measure active work, wait duration, rejection, dependency duration, and throughput. Increase the limit only while additional concurrency produces useful capacity without unacceptable latency or errors.
The final design should make overload behavior explicit. A caller should either enter a bounded amount of active work, wait within a bounded budget, receive a clear rejection, or take a defined degraded path.
Closing perspective
Concurrency limits are a small mechanism with a large operational effect. They turn simultaneous work from an accidental result of traffic and latency into an explicit design parameter.
That matters because systems often fail through overlap rather than raw arrival rate. A dependency slows, active work accumulates, contention rises, and latency expands again. Bounding in-flight work breaks that cycle.
Choose the boundary around the actual scarce resource, keep waiting bounded, preserve deadlines, measure queue time separately from operation time, and size the limit from observed capacity. With those pieces in place, concurrency becomes something the system controls instead of something overload controls for it.