Token Buckets Preserve Burst Capacity Without Removing Rate Bounds

A fixed requests-per-second ceiling treats a brief spike and a sustained flood as the same event. That can be too rigid for services whose callers naturally arrive in clusters. A token bucket separates two constraints: the long-run admission rate and the amount of burst traffic the service is willing to absorb.

The model has two parameters. The bucket capacity B is the maximum number of tokens that can accumulate. The refill rate r adds tokens per unit of time, up to B. A request consumes tokens according to its configured cost. If enough tokens are present, the request proceeds; otherwise it is rejected, delayed, or handled by another explicit policy.

capacity B = 100 tokens
refill r   = 20 tokens/s
cost       = 1 token/request

After five idle seconds, the bucket can hold at most 100 tokens. A caller may spend those 100 tokens quickly, but continued traffic can then proceed only as tokens return at 20 per second.

Burst allowance is finite stored capacity

Tokens represent admission credit saved during quieter periods. They do not increase the service’s physical capacity. They permit a caller to consume a predefined amount of future admission flexibility.

For a unit-cost request, the state transition can be written as:

tokens = min(B, tokens + elapsed * r)

if tokens >= 1:
    tokens -= 1
    admit()
else:
    reject()

The cap matters. Without B, an idle client could accumulate unlimited credit and later send an arbitrarily large burst. With a finite bucket, idle time stops increasing burst entitlement once the bucket is full.

A bucket that starts full also permits an immediate burst after creation or reset. Starting empty produces different startup behavior. That choice is part of the traffic contract rather than an implementation detail.

Refill rate controls the sustained envelope

Consider a bucket with B = 60 and r = 10 tokens/s. A client can send 60 unit-cost requests immediately when the bucket is full. If it continues at 10 requests per second, the bucket can remain near empty while requests keep passing as replenishment arrives.

If the client continues at 25 requests per second, demand exceeds refill by 15 tokens per second. The stored balance is consumed, then excess requests encounter the limiter.

full bucket
60 |\
   | \
   |  \   offered rate > refill rate
 0 +---\------------------------> time
        sustained admission ~= r

The burst budget changes transient behavior. It does not change the long-run replenishment rate.

Time accounting needs a monotonic basis

A token bucket depends on elapsed time, so clock handling is part of correctness. Duration measurement should use a monotonic clock when the runtime provides one. Wall-clock adjustments can move civil time forward or backward and should not create or remove admission credit.

Implementations do not need a timer that inserts tokens on every tick. Lazy refill is usually simpler: store the last update point and token balance, then compute replenishment when a request arrives.

elapsed = now_monotonic - last_update
tokens = min(B, tokens + elapsed * r)
last_update = now_monotonic

This avoids background timer work for inactive buckets and makes the state transition local to admission.

Numeric representation also deserves care. Fractional tokens can be represented with fixed-point arithmetic or a sufficiently precise numeric type. Rounding rules must not systematically mint extra credit across repeated updates.

Request cost can represent unequal work

Counting every request as one token is appropriate only when requests have comparable impact on the protected resource. A batch export and a metadata lookup can differ by orders of magnitude in CPU time, database work, or bytes transferred.

Weighted costs let the bucket express a closer approximation of resource demand.

metadata lookup:  1 token
search request:   3 tokens
batch export:    20 tokens

A request costing 20 tokens cannot pass when only 12 remain, even if the request count for the current second is low. This makes the limiter sensitive to configured work classes rather than raw request count alone.

Weights remain an approximation. If actual cost varies widely inside one class, static weights can still admit an expensive mix. Measurement of downstream saturation and completion latency remains necessary.

Scope determines which traffic shares a budget

A token bucket has no useful fairness semantics until its key is defined. One global bucket protects aggregate capacity but allows one busy caller to consume the entire burst budget. A per-user bucket isolates users but may allow aggregate traffic to exceed a shared backend limit.

Common scopes include:

global
tenant:{tenant_id}
user:{user_id}
api_key:{key_id}
route:{route_id}
tenant:{tenant_id}:route:{route_id}

Systems often combine scopes. A request may need credit from both a global bucket and a tenant bucket. The global bucket protects shared infrastructure; the tenant bucket constrains one tenant’s share.

Multi-bucket admission needs atomic semantics for the required set. Consuming one bucket and then failing another can leak credit unless the implementation can roll back safely or perform the decision atomically.

Distributed buckets trade precision for coordination cost

A limiter running on one process can update a local bucket under a lock or atomic operation. A service with many instances has a harder choice. Independent per-instance buckets multiply the effective burst budget and refill rate unless the configured values are partitioned.

A strongly coordinated shared bucket can enforce a tighter global bound, but every admission decision may add network and storage contention. That coordination path can become a bottleneck of its own.

Practical designs choose an explicit precision boundary. Options include partitioning a global budget across instances, leasing chunks of tokens to local limiters, or using a centralized atomic store for traffic where strict global enforcement justifies the coordination cost.

Token leasing can reduce per-request coordination:

global pool
    |
    +-- lease 50 tokens --> instance A
    +-- lease 50 tokens --> instance B

The tradeoff is temporary imprecision. Tokens leased to an idle or failed instance may be unavailable until the lease expires or is reclaimed. Lease size controls the balance between coordination frequency and stranded capacity.

Retry responses must not create a synchronized wave

A rejected request often triggers client retry behavior. Returning an overload response without a retry policy can move pressure from the service into a tight retry loop.

For HTTP APIs, 429 Too Many Requests commonly represents policy-based rate limiting. Retry-After can communicate a delay when the service can provide a meaningful estimate. Clients should still apply bounded retries and jitter so many callers do not return at the same instant.

The limiter’s telemetry should separate original operations from retry attempts. Otherwise a retry wave can appear to be organic traffic growth.

Useful fields include:

limiter=tenant_api
result=rejected
bucket_capacity=100
refill_per_second=20
token_cost=3
tokens_remaining=1.4

Configuration defines both protection and caller experience

A refill rate set below normal legitimate demand produces persistent rejection. A bucket set far above safe transient capacity permits bursts large enough to move the bottleneck downstream. Both parameters therefore need a resource basis.

The protected service may tolerate 500 requests per second in steady state and a brief queue of 200 additional requests. That does not automatically imply r = 500 and B = 200; request cost, concurrent execution, downstream limits, and existing queues all affect a safe configuration. Load tests and production telemetry provide the evidence for those values.

Rate limiting also needs observability at the same scope as enforcement. Aggregate rejection metrics can hide one tenant exhausting its bucket while everyone else remains unaffected.

A token bucket is most useful when its two dimensions remain explicit: replenishment controls sustained admission, and finite stored credit controls burst size. Keeping those roles separate makes overload policy predictable without forcing naturally bursty traffic into an unnecessarily rigid per-second boundary.