Load Shedding: Reject Work Before Overload Spreads
A service has finite capacity. When offered work exceeds that capacity for long enough, accepting every request can make the service less useful rather than more useful. Queues grow, deadlines expire, memory pressure rises, dependencies receive more traffic, and successful throughput can fall.
Load shedding is the deliberate rejection of work that the system cannot serve within an acceptable budget. The goal is not to maximize the number of requests admitted. The goal is to preserve useful service during overload.
The central rule is:
Reject excess work before it consumes scarce capacity.
This turns overload from an uncontrolled collapse into an explicit operating mode.
Overload is a capacity mismatch
Suppose an API can sustainably complete 2,000 requests per second. A burst reaches 2,400 requests per second for two seconds. A short queue may absorb the difference.
Now suppose 2,400 requests per second continue for ten minutes. The service cannot complete work as fast as it arrives. A queue that keeps growing does not create capacity. It only delays the point at which excess work fails.
A simple model is:
offered work = 2,400 requests/s
useful capacity = 2,000 requests/s
excess = 400 requests/sIf every excess request waits, the backlog grows by about 24,000 requests each minute. Many of those requests may expire before execution starts.
Rejecting some requests early can preserve low latency and successful completion for the requests that remain.
Admission should happen before expensive work
The placement of the admission decision matters.
Consider this request path:
parse large body
authenticate
allocate buffers
query database
call payment service
check overloadA rejection at the end saves little. The request has already consumed CPU, memory, database capacity, and downstream capacity.
A better path moves the cheap checks forward:
authenticate minimal request data
check admission
parse remaining body
perform business operationAuthentication may still need to precede admission when the policy depends on tenant identity or authorization. The general principle is to reject as early as practical, while retaining enough information to make a correct admission decision.
Load shedding and concurrency limits are complementary
A concurrency limit caps active work. Load shedding defines what happens when capacity is already occupied.
Imagine a renderer with 40 execution slots. The system can wait for a slot, place work in a bounded queue, or reject immediately. The concurrency limit establishes the maximum active pressure. The shedding policy prevents excess demand from becoming an unbounded waiting population.
request
|
admission
|
+--+----------------+
| |
slot available no useful capacity
| |
execute rejectA limit without a bounded excess-demand policy can move saturation into a queue. Shedding without a capacity signal can reject too much or too little. Used together, the two controls provide a clear boundary.
A queue must represent useful waiting
Queues are valuable when waiting can turn temporary mismatch into successful work. They are harmful when they hold requests that are already unlikely to finish.
Suppose an operation usually takes 100 milliseconds and callers have a 500-millisecond deadline. A queue delay of 450 milliseconds leaves almost no execution budget. Admitting that request may waste capacity on work whose result will never reach the caller.
An admission component can consider remaining time:
func hasExecutionBudget(ctx context.Context, reserve time.Duration) bool {
deadline, ok := ctx.Deadline()
if !ok {
return true
}
return time.Until(deadline) >= reserve
}The reserve should come from measured operation behavior, not a guess presented as certainty. Tail latency matters more than a best-case duration when the objective is reliable completion.
Reject before downstream work begins
Overloaded services often amplify pressure by continuing to call dependencies.
Suppose service A is saturated but still accepts every request. Each accepted request calls services B and C. A local overload event now creates additional load in two other systems. If those systems also queue work, the disturbance spreads.
Early shedding changes the propagation path:
without shedding:
client -> A -> B -> C
during overload with shedding:
client -> A -> rejectThe rejected request consumes some edge and application capacity, but it does not occupy the full dependency chain.
This containment property is especially valuable in service architectures where one request fans out to several downstream operations.
Choose an admission signal that tracks capacity
A useful shedding decision needs a signal related to the resource at risk. Common signals include active request count, queue depth, worker availability, memory pressure, dependency saturation, and remaining deadline budget.
A simple active-work gate can be effective:
type Gate struct {
slots chan struct{}
}
func NewGate(capacity int) *Gate {
return &Gate{slots: make(chan struct{}, capacity)}
}
func (g *Gate) TryEnter() bool {
select {
case g.slots <- struct{}{}:
return true
default:
return false
}
}
func (g *Gate) Leave() {
<-g.slots
}A handler can reject when no slot is immediately available:
func (s *Server) Handle(ctx context.Context, req Request) (Response, error) {
if !s.gate.TryEnter() {
return Response{}, ErrOverloaded
}
defer s.gate.Leave()
return s.process(ctx, req)
}This is intentionally simple. It does not predict future capacity. It prevents active work from exceeding a known boundary and gives excess work an explicit outcome.
Do not use CPU percentage as the only signal
CPU utilization is useful telemetry, but it is often a poor single admission signal.
A service can be unhealthy at modest CPU because it is blocked on a database, storage system, lock, or remote dependency. Another service can operate correctly at high CPU if latency and throughput remain stable.
Admission should track the bottleneck that constrains useful work. For a database-heavy endpoint, connection occupancy and query latency may be more informative. For a worker system, runnable job count and worker occupancy may matter more. For memory-heavy transformations, live memory and per-request allocation can be central.
The signal should match the failure mode being controlled.
Protect high-value work with separate capacity
Not all requests have equal operational value.
A system may handle interactive reads, background exports, health probes, administrative actions, and asynchronous maintenance. If all classes share one queue, a flood of low-priority work can block operations needed to keep the service usable.
Capacity can be partitioned:
interactive traffic -> reserved pool
background exports -> separate pool
maintenance -> small reserved poolAnother approach gives critical traffic a reserved portion while allowing it to borrow unused general capacity.
The policy should be simple enough to explain during an incident. Complex priority rules can produce surprising starvation and make capacity planning difficult.
Fairness needs an explicit scope
A global shedding threshold protects the service as a whole, but one tenant can still consume most admitted capacity.
Per-tenant admission can prevent that:
global capacity: 200 active
tenant capacity: 20 activeBoth checks may be required. The global boundary protects total resources. The tenant boundary limits disproportionate occupancy.
Static per-tenant limits are not suitable for every product. Large tenants may legitimately need more capacity than small ones. Weighted quotas or service tiers can encode that difference, but the allocation should remain observable and auditable.
Retrying rejection can recreate overload
Fast rejection is useful only when callers respond appropriately.
If every rejected request retries immediately, the system receives the same excess demand again, often with added synchronization. A thousand clients can form a repeating burst:
reject
|
immediate retry
|
reject
|
immediate retryClients that retry should use a bounded attempt count, backoff, jitter, and an overall deadline. The operation must also be safe to retry. A request that can produce duplicate side effects needs an idempotency mechanism or another explicit safeguard.
Servers can return a response that clearly distinguishes temporary overload from permanent request errors. Protocol-specific retry guidance should match actual server behavior; a server should not invite retries sooner than it can plausibly accept them.
Shedding stale work is often better than executing it
Some workloads lose value with age.
A live dashboard refresh that is already superseded by a newer refresh may not deserve execution. A search request from a client that disconnected has no recipient. A queued task whose deadline passed may only consume resources.
Cancellation and freshness checks can remove such work before expensive execution:
func process(ctx context.Context, job Job) error {
select {
case <-ctx.Done():
return ctx.Err()
default:
}
if time.Now().After(job.ExpiresAt) {
return ErrExpired
}
return run(job)
}This is a form of shedding based on usefulness rather than only system occupancy.
For durable business tasks, expiration must follow domain semantics. A payment capture or ledger update cannot be discarded merely because it waited longer than expected. Shedding policy belongs to the contract of the operation.
Preserve a small control path
During severe overload, operators still need observability and control.
If health endpoints, diagnostic endpoints, configuration refreshes, and administrative actions compete in exactly the same saturated pool as ordinary traffic, the system can become difficult to inspect or recover.
A small reserved control path can preserve essential operations. It must remain narrow. Turning the administrative path into a second unrestricted data path defeats the isolation.
The same idea applies to internal recovery traffic. Capacity needed to drain queues, renew leases, or coordinate failover should not be accidentally consumed by optional work.
Degradation can preserve more value than rejection
Some requests have a cheaper fallback.
A recommendation endpoint might return cached results instead of performing a fresh fan-out. A report endpoint might return a previously generated snapshot. A page might omit an optional personalization component.
The overload path can be:
normal capacity -> full response
reduced capacity -> cheaper response
no safe capacity -> rejectDegradation is useful when the cheaper path actually consumes less of the constrained resource. A fallback that calls the same saturated dependency through another code path is not meaningful protection.
Fallback data also needs clear freshness and correctness rules. Serving stale data may be acceptable for product suggestions and unacceptable for account balances.
Measure admitted and rejected work separately
A service can appear healthy during shedding because latency remains low. That is incomplete if a large fraction of traffic is being rejected.
Track at least:
offered request count
admitted request count
rejected request count
rejection reason
active work
queue depth
queue wait duration
completion latency
dependency latencyBreak rejection metrics down by endpoint, tenant class, region, or workload type when those dimensions affect admission policy.
A useful derived value is the admission ratio:
admission ratio = admitted / offeredIt shows how much demand the service is serving, but it should not replace absolute counts. A 90% admission ratio means something different at 100 requests per second than at one million.
Test overload behavior deliberately
A shedding mechanism that has never faced controlled overload is an assumption.
Load tests should increase offered work beyond sustainable capacity and verify that the system enters the intended mode. Useful assertions include:
- active work remains bounded;
- queue depth remains bounded;
- rejection begins before resource exhaustion;
- successful request latency remains within the target range;
- downstream pressure does not continue rising without bound;
- recovery occurs after offered load falls;
- priority or tenant policies behave as configured.
Test gradual ramps and sudden bursts. A controller that behaves well during a slow increase may respond poorly to an abrupt spike.
Also test dependency slowdown. Capacity can fall even when incoming traffic stays constant.
Recovery needs attention too
Shedding is not complete when rejection starts correctly. The service must also return to normal operation.
If admission uses a dynamic threshold, avoid a controller that rapidly switches between open and closed states. Small measurement noise can cause repeated admission bursts and rejection bursts.
Hysteresis can help:
start shedding at high threshold
stop shedding at lower thresholdFor example, a controller might begin protective action when a queue reaches one boundary and relax only after the queue falls well below it. The exact values require measurement against the actual system.
Fixed concurrency gates naturally recover as active operations finish and release slots. More adaptive policies need explicit recovery behavior.
Common mistakes
Several mistakes reduce the value of load shedding.
Rejecting too late. Expensive work has already consumed the capacity the policy is intended to protect.
Keeping an unbounded queue. The system limits execution but allows waiting work to consume memory and deadline budget without bound.
Treating every request equally. Optional bulk work can crowd out interactive or recovery traffic.
Ignoring caller behavior. Immediate retries can turn clean rejection into repeated overload.
Using one weak signal. A CPU threshold may miss saturation in a remote dependency or storage layer.
Dropping durable work without domain rules. Some operations must be preserved, deduplicated, or moved to durable queues rather than discarded.
Hiding rejection in aggregate success metrics. Low latency among admitted requests does not mean offered demand is being served.
A practical design sequence
Start by identifying the scarce resource and the sustainable operating range. Decide which work can be rejected, delayed, degraded, or moved to a durable path. Put admission before expensive consumption of that resource.
Bound active work and waiting work. Preserve caller deadlines. Define fair capacity allocation where one workload can crowd out another. Make retry behavior part of the design rather than an afterthought.
Then test beyond capacity. Confirm that rejection rises in a controlled way while successful throughput, latency, and downstream stability remain useful. Reduce offered work and confirm that normal admission returns without manual intervention.
Finally, expose enough telemetry to distinguish healthy service, active shedding, and uncontrolled saturation.
Closing perspective
Load shedding accepts a basic constraint: finite systems cannot serve unlimited simultaneous demand.
The important choice is whether excess work fails early and predictably or consumes resources until many requests fail slowly together. Early rejection can protect useful throughput, preserve downstream systems, and keep recovery possible.
A robust design admits work only while the system has a credible path to completion. It bounds queues, respects deadlines, separates critical capacity, coordinates retry behavior, and measures rejection as a first-class outcome.
Under overload, saying no to some work can be the mechanism that keeps the rest of the service available.