Designing Stateless Web Services for Horizontal Scaling
Horizontal scaling adds application replicas instead of making one machine larger. The load balancer can send each request to any healthy instance, which only works reliably when instances do not depend on unique local state.
“Stateless” does not mean the application has no state. It means durable or shared state lives outside an individual process so any replica can continue serving the workload.
Identify hidden local state
A service may appear stateless while depending on:
- in-memory login sessions;
- files uploaded to the local disk;
- process-local job queues;
- caches treated as authoritative data;
- generated files needed by a later request;
- scheduled jobs running independently on every replica.
These assumptions often survive on one server and fail as soon as traffic is distributed.
Keep request handling independent of a specific replica
A useful test is: if request one goes to instance A and request two goes to instance B, should the workflow still succeed?
Durable business data belongs in an appropriate shared system such as a database or object store. Temporary request-local values can remain in memory because they do not need to survive after the request.
The distinction is lifetime, not technology.
Design session handling deliberately
Server-side sessions stored only in memory create affinity: the user must keep returning to the same replica.
Sticky sessions can hide the problem, but they complicate failover and unevenly distribute load. If instance A disappears, its in-memory session disappears too.
Common alternatives include:
- a shared session store with explicit expiration;
- signed client-side session data when its size and sensitivity permit;
- token-based authentication with server-side authorization checks.
Do not place sensitive data in client-visible tokens merely to avoid server state. Signing prevents tampering; it does not necessarily provide confidentiality.
Treat local files as ephemeral unless guaranteed otherwise
Cloud instances and containers are commonly replaced during deployment, scaling, or recovery. A file written to one replica may not exist on another.
For user uploads that must persist, write to durable shared storage and keep only temporary processing files locally. Use unique temporary paths and clean them up reliably.
If a platform provides persistent volumes, understand their attachment and concurrency semantics before using them as shared application storage.
Make caches disposable
A cache should improve performance without becoming the only copy of required data.
Process-local caches are fine for immutable or recomputable data, but every replica will have its own contents. That means:
- cache hit rates change as replicas are added;
- invalidation must tolerate multiple copies;
- a newly started instance begins cold;
- cached data should not be required for correctness.
A shared cache solves some consistency problems but is still usually treated as replaceable acceleration rather than the system of record.
Separate asynchronous work from web processes
Putting an in-memory queue inside a web process loses jobs when that process restarts and prevents another replica from continuing the work.
For durable asynchronous tasks, put the queue in a shared broker or database-backed system appropriate to the workload. Workers can then scale separately from request handlers.
Consumers should expect retries. Design jobs to be idempotent or otherwise safe under at-least-once delivery when the queue provides that model.
Coordinate scheduled work
A cron loop embedded in every application replica can execute the same job multiple times after scale-out.
Options include:
- a platform-managed scheduler that enqueues one job;
- leader election;
- a distributed lock with carefully defined failure semantics;
- a separate singleton scheduler process.
Even with coordination, make scheduled tasks safe to retry when possible. Locks prevent some duplication but cannot erase every failure window around external side effects.
Support graceful startup and shutdown
Cloud schedulers routinely add and remove replicas. A service should become ready only after it can safely receive requests and should stop accepting new work before termination.
During shutdown:
- mark the instance unready or remove it from traffic;
- stop accepting new work;
- allow in-flight requests to finish within a deadline;
- close resources cleanly;
- terminate before the platform’s hard deadline.
This makes ordinary scaling events behave like controlled operations instead of small outages.
Do not store identity in the hostname
Application logic should not depend on a specific replica name such as app-03. Instance identifiers are useful for logs and debugging, but requests should not require a particular host for correctness.
When a workflow truly requires ownership, store that ownership in shared state with an explicit lease or coordination mechanism rather than relying on accidental routing.
Common pitfalls
Calling a database-backed service stateless and stopping there
The database is only one state source. Sessions, files, queues, caches, and schedulers still need review.
Using sticky sessions as the architecture
Affinity can be a temporary migration tool, but it weakens load distribution and failure recovery. Remove the underlying local-state dependency where practical.
Assuming replicas make dependencies highly available
Ten stateless web replicas still fail if all depend on one unavailable database. Horizontal scaling of the application tier does not automatically scale or replicate every dependency.
Scaling on CPU alone
Some services saturate database connections, queue lag, external API quotas, memory, or request concurrency before CPU. Choose scaling signals that reflect the actual bottleneck.
Forgetting deployment overlap
Horizontal systems naturally run multiple application versions during rolling releases. Shared data formats and messages should remain compatible across that overlap window.
A practical review
Before adding replicas, simulate replacement. Start two instances, route requests to both, terminate one during active traffic, and confirm that sessions, uploads, jobs, and in-flight work behave as designed.
Stateless architecture is valuable because it makes instances disposable. When no individual process owns irreplaceable state, cloud schedulers can scale, restart, and deploy the service without turning ordinary infrastructure changes into application failures.
