Load Shedding and Bounded Queues for Overload Control

A service can be healthy at 500 requests per second and unusable at 700. The extra load does not merely make every request 40 percent slower. Queues grow, deadlines expire while work is still waiting, memory usage rises, retries create more traffic, and useful requests compete with work that can no longer finish in time.

Overload control keeps that failure mode bounded. Instead of accepting unlimited work, a service limits concurrency and queueing, then rejects or degrades excess work early enough for the remaining requests to succeed.

Capacity is not the same as arrival rate

A service has finite processing capacity. If work arrives faster than it can be completed for a sustained period, unfinished work must accumulate somewhere or be rejected.

A simplified queue relationship is:

arrival rate > completion rate
        -> queue grows
        -> waiting time grows

Short bursts are normal, so some queueing is useful. The problem is unbounded queueing.

A queue does not create capacity. It only moves waiting from the caller into the service.

Bound every place work can wait

A common overload bug is to limit worker concurrency while leaving the input queue unlimited.

Suppose 20 workers each complete one job per second. A burst of 10,000 jobs can still enter an unbounded queue immediately. Even if memory is sufficient, the last jobs will wait many minutes before execution.

Use a bounded queue whose size reflects an intentional waiting budget:

20 active workers
100 queued jobs
additional jobs rejected

The exact numbers depend on workload latency, burst tolerance, memory cost, and service objectives. The important property is that the maximum backlog is explicit.

Also look for hidden queues in HTTP servers, connection pools, thread pools, executors, message consumers, and downstream client libraries. Bounding one queue does not help if another layer can accumulate unlimited work.

Limit concurrency around the scarce resource

Concurrency limits are most useful when they protect the resource that saturates first.

That resource might be CPU, database connections, memory-heavy transformations, calls to a constrained downstream service, or storage throughput.

A global request limit can be too coarse. If image processing consumes most CPU while lightweight metadata reads remain cheap, one shared limit can let expensive work crowd out inexpensive work.

Prefer limits aligned with meaningful resource classes when the system has distinct workloads.

Reject before expensive work begins

Load shedding works best near admission.

If a request is already parsed, authenticated, expanded into several database queries, and sent to two downstream services, rejecting it afterward saves little capacity.

A useful order is:

cheap protocol checks
-> authentication if required for admission
-> overload decision
-> expensive application work

The exact boundary depends on security requirements. For example, an internet-facing service may need enough authentication or routing context to apply per-tenant limits safely.

The principle is to avoid consuming the scarce resource before deciding whether the request is admissible.

Respect deadlines while work is queued

A request with 200 milliseconds left before its deadline should not spend 500 milliseconds waiting for a worker.

Queue entries should retain cancellation or deadline information. Before starting queued work, check whether it is still useful. Removing expired work frees capacity for requests that can still succeed.

A timeout only limits how long the caller waits unless cancellation propagates. The server may otherwise continue doing work after the result has become useless.

Distinguish queue limits from rate limits

Rate limiting and concurrency limiting solve related but different problems.

A rate limit controls how quickly work is admitted over time. A concurrency limit controls how much work is active at once.

For a latency-sensitive dependency, concurrency often maps more directly to resource pressure. If downstream calls become ten times slower, the same request rate creates roughly ten times as many in-flight operations.

Rate limits remain useful for quotas, fairness, abuse control, and known throughput ceilings. Many systems need both.

Choose what to shed deliberately

Not all requests have equal value.

During overload, a service may preserve capacity for interactive requests over background refreshes, writes over optional analytics, health and control-plane operations, or already-admitted transactions over speculative work.

Priority must be designed carefully. A permanently favored class can starve lower-priority work.

Reserve capacity or use separate bounded pools when one workload must remain available independently.

Return failures callers can handle

Early rejection is useful only if callers understand that the request was not processed.

For HTTP services, 503 Service Unavailable is a common response when a service is temporarily unable to handle a request. A Retry-After header can communicate when a later attempt may be appropriate when the server has a meaningful estimate.

429 Too Many Requests is commonly used for rate limiting. Choose between 429 and 503 according to whether the limit represents caller-specific policy or temporary service capacity.

Retries must be bounded and delayed. Immediate retries during overload can amplify the original traffic spike.

Avoid retry storms

Consider a service that can process 1,000 requests per second but temporarily receives 1,500. If all 500 rejected requests retry immediately, the next interval receives the normal traffic plus retries.

This positive feedback can keep the system overloaded after the original burst has ended.

Clients should combine retry limits with backoff and jitter. Servers should avoid suggesting aggressive retry intervals when they do not know when capacity will recover.

Some operations should not be retried automatically unless the operation is idempotent or the protocol provides an idempotency mechanism.

Measure saturation, not just errors

Useful measurements include:

• active work versus concurrency limit;
• queue depth and queue capacity;
• queue waiting-time percentiles;
• rejected work by reason and workload class;
• expired or canceled work removed from queues;
• downstream pool utilization;
• request latency split into queue time and execution time.

A high rejection count is not automatically evidence that the limiter is wrong. During genuine overload, rejection may be what keeps successful-request latency within a usable range.

Set limits with measurements

A concurrency value copied from another service is not a capacity plan.

Load-test the relevant workload and observe the saturation point. Increase offered load gradually while watching latency, throughput, errors, CPU, memory, and downstream constraints.

Healthy throughput often rises with concurrency until a bottleneck saturates. Beyond that point, additional concurrency may increase latency without materially increasing completed work.

Leave operational headroom because production requests and dependencies vary.

Adaptive limits require guardrails

Static limits are easy to reason about but cannot automatically react to changing capacity. Adaptive concurrency algorithms can use observed latency or other feedback to raise and lower admission limits.

They also add a control loop that can oscillate or react to misleading signals.

Before adopting adaptive control, establish minimum and maximum limits, stable measurement windows, cold-start behavior, workload isolation, and observability that explains why the limit changed.

A conservative static limit is often a better first implementation.

Common pitfalls

Treating an unbounded queue as resilience

An unbounded queue converts overload into growing latency and memory pressure. It delays failure rather than controlling it.

Counting only active workers

Queued requests also consume resources and user time. Measure the complete backlog.

Using one limit for unrelated work

A slow bulk operation can consume all permits and block latency-sensitive traffic. Separate resource classes when isolation matters.

Retrying every rejection

Retries consume capacity too. Use bounded retries, backoff, jitter, and operation-specific safety rules.

Ignoring work after caller cancellation

Continuing expensive work for a disconnected or timed-out caller wastes the capacity overload controls are trying to protect.

Setting the queue from memory capacity

The fact that a process can hold a large number of queued objects does not mean users can tolerate the resulting wait. Queue bounds should reflect acceptable delay as well as memory.

Design for graceful overload

A robust overload path has a clear sequence:

1. identify the scarce resource;
2. cap concurrent use of that resource;
3. allow only a bounded amount of waiting;
4. propagate deadlines and cancellation;
5. reject excess work before expensive processing;
6. make retry behavior explicit;
7. measure queueing, saturation, and shedding;
8. test the system above expected capacity.

The goal is not to eliminate overload. Any finite service can receive more work than it can process. The goal is to make overload predictable: preserve useful throughput, keep latency bounded for admitted work, and fail excess requests early instead of letting the entire service collapse.