Autoscaling is useful, but it is not instantaneous. Traffic can rise faster than new instances start, a dependency can slow down, or a retry storm can multiply work. When demand exceeds safe capacity, accepting every request can make the entire service slower until almost nothing completes.
Load shedding is the deliberate rejection or degradation of work to keep the system inside a recoverable operating range.
Overload is often a queueing problem
Imagine a service that safely handles 200 concurrent requests. A downstream dependency slows from 50 ms to 2 seconds.
Even if incoming traffic stays constant, requests remain in flight much longer. Concurrency rises, memory grows, connection pools fill, and latency increases further.
An unbounded queue does not create capacity. It converts overload into waiting.
A bounded system asks two questions:
- how much work can execute concurrently?
- how much work is worth waiting for?
Once both limits are reached, new work should be rejected or degraded.
Bound concurrency
A concurrency limit prevents a process from starting more expensive work than it can sustain.
Conceptually:
if in_flight >= concurrency_limit:
reject
else:
executeThe correct limit depends on the bottleneck: CPU, memory, database connections, external API quotas, or another scarce resource.
For CPU-bound work, a limit near available execution capacity may be appropriate. For I/O-bound work, a larger limit may be useful, but it should still be finite.
Keep queues bounded
Some waiting can absorb short bursts. Too much waiting only creates stale requests.
If a request has already waited longer than the caller can tolerate, executing it wastes capacity.
Bound queue length and queue time. Reject before the process spends expensive resources on work that is unlikely to be useful.
This matters for asynchronous jobs too. A queue that can grow without limit can turn a five-minute incident into hours of backlog.
Respect deadlines
Callers should send deadlines, and services should stop work when those deadlines expire.
If a request has 100 ms left but the operation normally requires 500 ms, beginning the operation is usually harmful.
Deadline propagation prevents abandoned work from continuing in lower layers after the original request has timed out.
It also improves admission control: work with insufficient remaining budget can be rejected early.
Shed the least valuable work first
Not all requests have equal business or operational value.
A service may prioritize:
- health and control-plane traffic;
- interactive user requests;
- paid or latency-sensitive workloads;
- background refreshes;
- batch or speculative work.
Under pressure, optional work such as cache refreshes, analytics enrichment, image pre-generation, or expensive recommendations can be disabled first.
Priority should be explicit. Otherwise overload policy becomes whichever request arrives first.
Return a clear retry signal
For HTTP services, 503 Service Unavailable is often appropriate when the service is temporarily unable to handle work.
If the server can provide meaningful guidance, Retry-After can help cooperative clients.
Do not encourage immediate retries without backoff. A rejected request that is instantly retried can increase overload.
Clients should use bounded retries with jitter and an overall deadline.
Avoid retry amplification
Suppose one user request crosses three services, and each layer retries several times. One original request can trigger many downstream attempts.
Define where retries belong. Often the layer with enough context to know whether an operation is safe and useful should own the retry policy.
Load shedding and retry budgets must be designed together.
Degrade expensive features
A useful overload mode may still return a partial response:
normal:
profile + recommendations + related content
degraded:
profile onlyThis keeps essential work available while removing optional dependency calls.
Degradation should be tested before incidents. A fallback path that is never exercised can fail exactly when it is needed.
Autoscaling is a partner, not a substitute
Horizontal autoscaling can add capacity after metrics cross a threshold, but it has delay:
- metrics need to detect the increase;
- the scheduler must place instances;
- images may need to pull;
- applications initialize;
- load balancers begin routing.
Load shedding protects the service during that interval.
It also helps when scaling cannot solve the bottleneck, such as a fixed database capacity or an external API quota.
Measure saturation directly
Request rate alone does not show overload.
Useful signals include:
- in-flight requests;
- queue depth and queue age;
- worker-pool utilization;
- database-pool wait time;
- CPU throttling;
- memory pressure;
- rejected or degraded request counts;
- deadline expirations;
- latency percentiles.
A rising rejection rate with stable successful latency can be healthier than zero rejections with rapidly increasing tail latency.
Common mistakes
Using an unbounded queue as a safety buffer
It hides overload until latency and memory are already severe.
Rejecting only after expensive work starts
Admission control should happen before consuming the constrained resource.
Retrying every rejection
Retries need backoff, jitter, safety checks, and a total attempt budget.
Scaling from CPU when the bottleneck is elsewhere
A service can be saturated on database connections while CPU remains low.
Treating all traffic equally
Optional work should not crowd out essential operations during incidents.
A practical overload policy
For each critical service, define:
- the resource that limits safe concurrency;
- a finite concurrency limit;
- a finite waiting queue or maximum queue age;
- request deadlines;
- priority classes;
- which features can degrade;
- a retry contract for callers;
- metrics and alerts for shedding events.
The objective is not to avoid every rejection. The objective is to preserve useful throughput and recovery. A service that says “not now” to some work can remain available for the work it can actually finish.