A rate limit expressed only as “100 requests per second” leaves an important policy question open. Can a client send 100 requests at the first instant of each second, or must those requests be spread evenly? A token bucket makes that distinction explicit by separating sustained rate from burst capacity.
The limiter maintains a balance of tokens up to a fixed capacity. Tokens arrive at a configured refill rate. An operation is admitted only when enough tokens are available, and admission deducts its cost from the balance. Idle time accumulates capacity for a later burst, but never beyond the bucket limit.
This model is useful because the long-term rate and short-term burst allowance are independent controls.
Refill rate sets the sustained budget
Let a bucket have capacity (B) tokens and refill at (r) tokens per second. If the stored balance at time (t_0) is (T_0), then before charging a new operation at time (t), the balance can be computed as:
[ T(t) = \min(B,\ T_0 + r(t - t_0)) ]
An operation with cost (c) is admitted when (T(t) \ge c). After admission:
[ T’(t) = T(t) - c ]
Over a long interval, a continuously busy caller cannot consume tokens faster than the refill rate except for tokens that were already stored. That stored balance is the burst allowance, not a second sustained quota.
A bucket therefore does not require a timer that physically inserts one token at every refill tick. An implementation can store the last update time and reconstruct the available balance when traffic arrives.
Capacity sets the maximum stored burst
The capacity (B) limits how much idle credit can accumulate. With (B = 200) and (r = 100) tokens per second, a fully charged bucket can admit a burst costing 200 tokens immediately. Continued traffic then depends on refill at 100 tokens per second.
Increasing (B) changes burst tolerance without changing the eventual sustained rate. Increasing (r) changes the sustained budget and also changes how quickly a depleted bucket recovers.
This separation matters for workloads with legitimate bursts. A user opening a page may trigger several API calls at once even though the user’s average request rate is modest. A bucket can absorb that cluster without granting the same burst repeatedly when the caller remains busy.
Operation cost does not have to be one token
Charging one token per request treats all admitted operations as equal. That is appropriate only when request count is a useful proxy for the protected resource.
A limiter can assign different costs. A small metadata read might cost one token while an expensive export costs more. The bucket arithmetic stays the same; only (c) changes.
Weighted costs need a stable contract. If cost is estimated after substantial work has already started, the limiter no longer protects that work. The cost signal should be available at the admission point and should correspond closely enough to the resource being bounded.
A single bucket also cannot represent every resource dimension at once. CPU time, database concurrency, outbound bytes, and tenant quota can require separate controls when their pressure does not move together.
Rejection and deferral are different policies
When the balance is below the required cost, the limiter must choose what happens to the operation. Immediate rejection keeps queueing outside the protected service and gives the caller a clear overload signal. Deferral waits for future tokens, which turns the limiter into a scheduler with a queue.
Deferral needs explicit bounds. An unbounded wait queue can preserve every request while moving overload into memory consumption and tail latency. A maximum queue length, deadline, or maximum wait time keeps that cost finite.
For HTTP APIs, a rejected request is often represented with 429 Too Many Requests. A Retry-After value can provide a retry hint when the server can compute a meaningful delay. The header is guidance to the client; it does not reserve future capacity.
Timekeeping is part of limiter correctness
Refill arithmetic depends on elapsed time. A process-local limiter should use a monotonic clock for duration measurement so wall-clock adjustments do not create or remove tokens unexpectedly.
Distributed enforcement is harder. Two independent processes that each maintain a full bucket for the same logical identity have multiplied the permitted budget. A shared atomic state store, deterministic partition ownership, or another coordination scheme is needed when the limit must apply globally.
The state transition also has to be atomic at the chosen scope. Concurrent requests must not all observe the same pre-charge balance and each spend it independently.
Clock choice, ownership, and atomicity are therefore part of the rate-limit contract, not implementation trivia.
Hierarchical buckets can enforce more than one boundary
Systems often need several simultaneous limits: per-user, per-tenant, per-endpoint, and service-wide. A request can be admitted only if every applicable bucket can pay its assigned cost.
That composition creates an atomicity question. If the user bucket is charged and the service-wide bucket then rejects the request, the system must decide whether to refund the first charge. A transactional implementation can make the multi-bucket decision atomic; other designs may accept conservative token loss or use reservation logic.
The order of checks also affects contention and cost. A cheap local boundary can reject obvious excess before a more expensive shared limit, provided the resulting semantics match the intended policy.
Metrics should expose both demand and bucket state
Accepted and rejected request counts show the visible outcome but not the shape of pressure. Useful telemetry also includes remaining tokens, refill rate, configured capacity, charged cost, wait time for deferred work, and rejection rate by limiter key.
A bucket that stays near empty indicates sustained demand close to or above refill. A bucket that repeatedly drains from full and then recovers indicates bursty demand. Those patterns call for different capacity decisions even when total request counts look similar.
Limiter metrics should avoid high-cardinality labels for unrestricted user identifiers. Per-tenant debugging can use sampled traces or bounded diagnostic channels rather than turning every identity into a permanent metrics series.
A token bucket makes the admission contract concrete
The refill rate states how quickly permission to do work returns. Capacity states how much of that permission may be saved for a burst. Operation cost states how much permission one request consumes.
Those parameters make a token bucket more precise than a rate number attached to an ambiguous time window. The remaining engineering work sits at the boundaries: atomic charging, monotonic time, bounded waiting, distributed ownership, and telemetry that reveals whether the configured policy matches actual load.