Token Bucket Rate Limiting for Controlled Bursts

A service may handle 100 requests per second comfortably on average while still needing to accept a short burst of 300 requests after a client reconnects. A rigid per-second limit treats those situations as the same problem: once the current window is full, otherwise acceptable work is rejected.

Token bucket rate limiting gives you a more useful control. It separates two decisions: how quickly permission to do work is replenished and how much permission may accumulate for a burst. Once you understand those two numbers, you can reason about the limiter without depending on a particular library or platform.

Think of tokens as permission to start work

A token bucket has three basic rules:

1. The bucket can hold at most a fixed number of tokens.
2. Tokens are added at a fixed rate, up to that capacity.
3. A request must spend one or more tokens before it is admitted.

Suppose a bucket has a capacity of 20 tokens and refills at 5 tokens per second. If the service has been quiet long enough, the bucket contains 20 tokens. Twenty requests can then arrive together and be admitted immediately.

After that burst, the bucket is empty. New requests gain permission at roughly 5 per second because that is the refill rate. If traffic becomes quiet again, unused tokens accumulate until the bucket reaches 20. Extra refill beyond 20 is discarded; the bucket does not build an unlimited credit for future traffic.

That gives the two settings different meanings:

• refill rate controls the sustained admission rate;
• capacity controls how much previously unused allowance can be spent in a burst.

Keeping those meanings separate is the main mental model for token bucket rate limiting.

Work through the smallest useful example

Consider an API with this policy:

capacity = 10 tokens
refill = 2 tokens/second
cost = 1 token/request

Assume the bucket starts full. At time 0, eight requests arrive together. They all pass and leave two tokens.

Half a second later, one token has been replenished, so the bucket contains three tokens. If five requests now arrive together, three can be admitted immediately and two cannot.

The arithmetic is simple:

newTokens = elapsedSeconds * refillRate
available = min(capacity, previousTokens + newTokens)

For each admitted request:

available = available - requestCost

The important detail is that the limiter accounts for elapsed time, not merely for named clock windows such as “the current second.” A correct implementation may update continuously or calculate accumulated tokens lazily when a request arrives. The observable policy is the same if the accounting is consistent.

Why this differs from a fixed window

A fixed-window limiter might allow 100 requests during each wall-clock minute. That rule is easy to explain, but the boundary creates an awkward case: a client can send 100 requests near the end of one minute and another 100 just after the next minute begins. The two bursts are close together in real time even though they belong to different accounting windows.

A token bucket does not reset its allowance at a window boundary. Permission accumulates gradually. A client can spend stored tokens quickly, but after the bucket is depleted it must wait for tokens to return.

This does not make token buckets universally preferable. A fixed window can be perfectly adequate for coarse quotas such as “no more than N exports per day.” Token buckets are especially useful when the engineering question is about traffic rate plus tolerated burstiness.

Choose the refill rate from sustained capacity

Start with the rate the protected operation should be allowed to sustain, not with the burst size.

If a downstream service can safely receive about 50 of these operations per second under the conditions you are designing for, a refill rate near 50 tokens per second expresses that policy when each operation costs one token. The exact value should include whatever safety margin your system needs; a limiter cannot discover downstream capacity for you.

Be careful with units. These settings describe admission, not necessarily concurrency. A refill rate of 50 requests per second does not mean at most 50 requests are running simultaneously. If each request takes two seconds, admitted work can overlap and produce much higher concurrency.

If concurrent work is the actual scarce resource, use a concurrency limit or bounded worker pool as well. Rate and concurrency constrain different dimensions.

Choose bucket capacity from the burst you can absorb

Capacity answers a different question: how much work may arrive faster than the refill rate before requests must wait or fail?

Imagine a service that refills at 20 tokens per second. A capacity of 20 permits roughly one second of accumulated allowance. A capacity of 100 permits a much larger immediate burst even though the long-run refill rate is unchanged.

A larger bucket is useful when bursts are legitimate: clients reconnect, scheduled jobs wake together, or buffered work is released after a short interruption. It also transfers more burst pressure to the protected system.

So avoid choosing capacity merely because a larger number causes fewer rejections in testing. Ask what the downstream path can absorb at once: connection pools, worker queues, memory, external dependencies, and any other constrained resources all matter.

A useful design statement is concrete:

Admit at 40 operations per second over sustained traffic, while allowing up to 80 operations to consume accumulated allowance immediately.

That statement makes both the steady-state policy and burst policy reviewable.

Decide what happens when no token is available

The token bucket only decides whether permission exists. Your application still needs a policy for requests that arrive without enough tokens.

One option is to reject immediately. This is appropriate when callers can retry later, when stale work is not useful, or when queueing would only move overload somewhere less visible.

Another option is to wait until a token becomes available. Waiting can smooth internal workloads, but it creates a queue even if the queue is hidden inside the limiter. That queue needs a bound and a deadline. Otherwise traffic can accumulate faster than tokens refill, turning rate limiting into growing latency and memory use.

For example, if work arrives continuously at 100 requests per second but tokens refill at 50 per second, waiting does not remove the mismatch. The backlog grows by about 50 requests per second until arrivals slow, requests time out, or a queue limit rejects them.

The limiter should therefore make overload behavior explicit: reject, wait within a bounded deadline, or hand work to a separately managed queue. None of those choices is correct for every workload.

Use weighted costs only when they represent real pressure

Not every operation consumes the same amount of capacity. A small metadata read and a large report generation request may have very different costs. A token bucket can represent this by charging different token amounts.

For example:

metadata lookup   = 1 token
report generation = 10 tokens

This can be useful when the weights are a reasonable proxy for the constrained resource. It can also become misleading quickly. If “10” is based on intuition rather than measured or well-understood relative cost, the limiter creates an elaborate policy without a reliable model underneath it.

Start with equal-cost requests unless the workload has a clear reason to distinguish them. Weighted admission is easier to maintain when the cost categories are few, stable, and explainable.

Also check the boundary condition: a request whose cost exceeds the bucket capacity can never be admitted. An implementation should reject such configuration or handle that case deliberately rather than letting the request wait forever.

Put the bucket at the boundary that owns the policy

A limiter’s location determines what it protects.

A per-client bucket at an API boundary can stop one client from consuming its entire allowance at once. A shared bucket around a downstream dependency can protect that dependency from aggregate traffic across many callers. Those are different policies and may both be useful.

This is why “add rate limiting” is underspecified. Before implementing it, name the key used to select a bucket and the resource the bucket protects.

For example:

bucket key: customer account
policy: 20 requests/second, burst capacity 40

is different from:

bucket key: payment provider
policy: 200 requests/second shared by all customers

The first is primarily an allocation policy between customers. The second is a protection policy for a shared dependency.

Distributed limiters need shared accounting or deliberate approximation

A single-process token bucket is straightforward because one component owns the token count. Multiple service instances make the question harder.

If ten instances each enforce a local limit of 100 requests per second, the deployment may admit roughly ten times the rate you intended when traffic is spread across them. Local buckets are valid only if the policy is intentionally per instance or if the configured rate has already been divided appropriately.

A globally shared limit needs coordination: instances must use shared state, route a given key consistently to one owner, or accept an approximation with a documented error bound. Each approach trades implementation complexity, latency, availability, and precision differently.

Do not hide that trade-off behind the word “distributed.” State the guarantee you actually need. A user-facing quota may require tighter accounting than a protective limiter whose purpose is simply to keep load near a safe region.

Avoid common token bucket mistakes

The first common mistake is treating capacity as the rate limit. A bucket with capacity 1,000 and refill 10 per second can still admit 1,000 requests at once after a quiet period. Capacity controls burst allowance, not long-run refill.

The second is using wall-clock jumps directly in token arithmetic. Token replenishment depends on elapsed duration. Implementations should use a time source suitable for measuring elapsed time where the platform provides one, so ordinary clock corrections do not create or remove allowance unexpectedly.

The third is forgetting atomicity. If several workers read the same token count concurrently and all decide a token is available before any deduction becomes visible, the limiter can admit more work than its policy permits. Token checks and deductions need synchronization appropriate to where the state lives.

The fourth is assuming rate limiting is overload control by itself. A token bucket can shape admissions, but accepted work may still pile up because requests become slower, a dependency stalls, or concurrency grows. Pair the limiter with controls for the actual failure mode, such as deadlines, bounded queues, or concurrency limits.

Know when a simpler rule is enough

Use a token bucket when both sustained rate and short bursts matter and you can explain meaningful values for each.

Prefer a simpler counter or fixed-window quota when the policy itself is naturally window-based, such as a small number of expensive administrative actions per hour. Prefer a concurrency limit when the main risk is too many simultaneous operations rather than too many starts per unit time. Prefer a bounded queue when the goal is explicitly to buffer a limited amount of work for later processing.

These controls can be combined, but combining them without a clear reason makes production behavior harder to reason about. Add each control for a named constraint.

Make the policy observable

Once a token bucket is running, observe the decisions it makes. Useful signals include admitted requests, rejected or delayed requests, wait duration when waiting is allowed, and which bucket keys are frequently exhausted.

Those signals answer practical questions. Are ordinary clients regularly hitting the burst limit? Is one shared dependency exhausting its bucket? Did a configuration change turn a protective limit into a bottleneck?

The goal is not to collect every internal token update. It is to make the policy’s consequences visible enough that engineers can distinguish healthy limiting from a misconfigured system.

Start with two explicit numbers

When introducing token bucket rate limiting, write down the policy before choosing a library: the sustained refill rate and the maximum accumulated capacity. Then state what happens when a request cannot obtain tokens and what resource the bucket is intended to protect.

If those decisions are clear, implementation details are much easier to evaluate. If they are vague, a technically correct token bucket can still enforce the wrong system behavior.