A service can be healthy at 2,000 requests per second and collapse at 2,400. The extra 400 requests do not merely wait their turn. They may occupy connection slots, queue entries, memory, worker threads, database sessions, and retry budgets while useful throughput falls.
Load shedding places an explicit admission decision before a scarce resource is fully consumed. When the system cannot serve all incoming work within its operating envelope, it rejects selected requests early instead of allowing every request to compete until they all become slow.
The objective is not zero errors during overload. It is controlled errors that preserve useful throughput and keep recovery possible.
Saturation changes the cost of waiting
A lightly loaded service often has spare workers and short queues. Adding one request has little effect on requests already in flight.
Near saturation, the same arrival can have a different cost. A full worker pool pushes work into a queue. Queueing increases latency, which keeps callers waiting longer. Some callers retry, adding more arrivals. Memory retained by queued requests grows, and downstream pools can remain occupied for longer periods.
A system that accepts work faster than it can complete it accumulates debt in the queue. If arrivals continue above service capacity, the debt cannot be paid down until demand falls or capacity rises.
Rejecting some work near the admission boundary prevents that debt from growing without bound.
Admission belongs before the scarce resource
A rejection saves the most capacity when it happens before expensive work begins.
Consider an HTTP service whose database pool is the limiting resource. Rejecting a request after it has parsed a large payload, performed several remote calls, and acquired a database connection protects little. An admission gate placed before those steps can avoid most of that cost.
The relevant boundary is workload-specific. It may sit before a worker pool, database transaction, queue, decompression step, model inference call, or fan-out operation.
An effective gate measures pressure close to the resource it protects. A global CPU threshold may be too indirect when the actual bottleneck is a 50-connection database pool.
A concurrency limit bounds work in flight
One practical admission mechanism is a concurrency limit. If a service permits at most N expensive operations at once, request N + 1 must wait in a bounded queue or receive a rejection.
A minimal model is:
if in_flight >= limit:
reject()
else:
in_flight += 1
execute()
in_flight -= 1The real implementation must release permits on every completion path, including errors and cancellation. It also needs a clear policy for queued work if waiting is allowed.
Concurrency limits differ from rate limits. A rate limit controls arrivals over time. A concurrency limit controls simultaneous occupancy. A slow dependency can make a modest arrival rate dangerous because each request holds capacity for longer.
Both controls can coexist: one bounds arrival intensity, while the other bounds work already admitted.
Bounded queues make overload visible
An unbounded queue postpones rejection rather than eliminating it. Once queue wait exceeds the caller’s useful time budget, accepted requests can become stale before execution starts.
A bounded queue creates a finite waiting budget in units of work. When it fills, the service has a concrete overload signal and can reject new arrivals immediately.
Queue size should reflect the latency contract and service rate, not a desire to avoid visible errors. A larger queue can increase the number of requests that eventually time out while consuming memory and scheduling effort.
Deadline-aware admission can be stricter. If a request cannot plausibly reach execution before its deadline, rejecting it at enqueue time preserves space for work with a viable budget.
Selection policy determines what survives overload
Not all requests carry equal operational value. A service may reserve capacity for health checks, control-plane operations, interactive traffic, or requests required to recover the system.
Priority must be bounded. If high-priority traffic can consume every permit, lower classes can starve indefinitely. Separate pools, reserved permits, or weighted admission can make the allocation explicit.
The policy also needs stable inputs. Client-supplied priority fields are unsafe as the sole authority when clients benefit from marking every request urgent. Priority usually comes from authenticated identity, server-side route classification, or another trusted policy source.
Random shedding can be adequate when requests are equivalent. More selective policies are useful only when the classification represents a real service objective.
Rejections need a retry contract
Early rejection shifts responsibility to the caller, so the response must have defined semantics.
For HTTP, a service may use 429 Too Many Requests for policy or rate-based admission and 503 Service Unavailable for temporary capacity failure, depending on the API contract. Retry-After can provide a retry hint when the server has a meaningful estimate or policy interval.
A retry must not erase the benefit of shedding. Immediate synchronized retries can turn one rejected wave into another. Callers need bounded attempts, backoff, jitter, and the original request deadline.
Non-idempotent operations require extra care. A rejection issued before execution is easier to retry safely than an ambiguous failure after a side effect may have committed. Admission should therefore happen before the operation crosses a durable side-effect boundary whenever the protocol permits it.
Adaptive limits need a stable feedback signal
A fixed concurrency limit is simple and predictable, but safe capacity can change with dependency latency, instance size, cache state, or workload mix.
Adaptive admission can adjust a limit from observed latency or queueing signals. The controller still needs guardrails. A noisy signal can make the limit oscillate, and a rapid increase can push a recovering dependency back into saturation.
The measurement should represent admitted work rather than rejected work alone. Rejection volume says demand exceeds the current gate; it does not prove that the protected resource could safely accept more.
Any adaptive policy needs minimum and maximum bounds, a controlled adjustment rate, and telemetry that exposes both the chosen limit and the signal driving it.
Shedding must compose across service boundaries
Rejecting at one service does not protect every downstream component. A request admitted by the frontend may fan out to several dependencies, each with a different capacity envelope.
Local admission gates should protect local scarce resources. Downstream services still need their own controls because upstream estimates can be stale and other callers may share the same dependency.
The call chain also needs deadline and cancellation propagation. A rejected or expired parent request should stop child work where cancellation is supported. Otherwise, the frontend can appear protected while remote capacity remains occupied by abandoned operations.
Load shedding, bounded concurrency, deadlines, and cancellation address different parts of the same overload path.
Recovery capacity is part of the design
A saturated service needs enough free capacity to process the work that restores normal operation: health probes, cache refreshes, leadership traffic, configuration updates, or requests that drain a backlog.
If ordinary traffic consumes every resource, the service can remain trapped even after peak demand starts falling. Admission policy can reserve a small amount of capacity for recovery-critical operations or isolate those operations in a separate pool.
This reserve is not free throughput. It is capacity intentionally withheld from ordinary work so the system retains a path back to a stable state.
Useful throughput is the target
A good overload policy is visible in more than its rejection count. Operators need admitted request rate, rejected request rate, in-flight work, queue depth, queue wait, dependency latency, timeout rate, retry volume, and resource saturation.
The key distinction is between offered load and useful completed work. During overload, accepting more requests can increase the first metric while reducing the second.
Load shedding makes that trade explicit. Once capacity is exhausted, rejecting selected work early can be the action that keeps the remaining work fast enough to finish, keeps dependencies inside their operating range, and leaves the service with enough headroom to recover.