Token Bucket Rate Limiting in Go

Rate limiting protects a service from traffic spikes, accidental client loops, and workloads that consume more resources than the system can safely handle. A useful limiter should do more than enforce a fixed request count: it should allow small bursts while keeping the long-term request rate bounded.

The token bucket algorithm provides exactly that behavior. This guide implements a small, concurrency-safe token bucket using only Go’s standard library and then shows how to use it in an HTTP service.

The implementation uses the predeclared min function, which requires Go 1.21 or newer.

How a token bucket works

Imagine a bucket that can hold a fixed number of tokens. Each accepted request consumes one token. Tokens are continuously replenished at a configured rate, but the bucket never grows beyond its capacity.

Two settings control the behavior:

• Capacity controls the maximum burst size.
• Refill rate controls the sustained request rate.

For example, a bucket with capacity 10 and a refill rate of 2 tokens per second can immediately accept up to ten requests after being idle. After that burst, new requests are accepted at roughly two per second as tokens become available.

This is different from a strict fixed-window counter. Traffic near a window boundary cannot suddenly receive two full windows of capacity, and idle time naturally restores burst capacity.

A dependency-free implementation

A token bucket does not need a background goroutine. Instead, it can calculate how many tokens should have accumulated whenever a request arrives.

package main

import (
    "fmt"
    "sync"
    "time"
)

type TokenBucket struct {
    mu       sync.Mutex
    capacity float64
    tokens   float64
    rate     float64
    last     time.Time
}

func NewTokenBucket(capacity int, rate float64) *TokenBucket {
    if capacity <= 0 || rate <= 0 {
        panic("capacity and rate must be positive")
    }

    now := time.Now()
    return &TokenBucket{
        capacity: float64(capacity),
        tokens:   float64(capacity),
        rate:     rate,
        last:     now,
    }
}

func (b *TokenBucket) Allow() bool {
    b.mu.Lock()
    defer b.mu.Unlock()

    now := time.Now()
    elapsed := now.Sub(b.last).Seconds()

    b.tokens = min(b.capacity, b.tokens+elapsed*b.rate)
    b.last = now

    if b.tokens < 1 {
        return false
    }

    b.tokens--
    return true
}

func main() {
    bucket := NewTokenBucket(2, 1)

    fmt.Println(bucket.Allow())
    fmt.Println(bucket.Allow())
    fmt.Println(bucket.Allow())
}

The bucket starts full, so the first two calls are allowed immediately. With no meaningful delay before the third call, the third request is rejected:

true
true
false

The mutex is important. Without it, simultaneous calls could update tokens and last concurrently, allowing more traffic than intended and introducing data races.

Why floating-point tokens are useful

The refill rate does not have to be an integer. A rate of 0.5 means one token every two seconds, while 2.5 means two and a half tokens accumulate each second.

Keeping fractional tokens also avoids losing refill progress between requests. Suppose only 300 milliseconds have passed at a rate of two tokens per second. The bucket has earned 0.6 token. That fraction should remain available so later requests can benefit from the accumulated time.

Using the limiter in HTTP middleware

For a single process, the bucket can sit in front of an HTTP handler:

func limit(bucket *TokenBucket, next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if !bucket.Allow() {
            w.Header().Set("Retry-After", "1")
            http.Error(w, "rate limit exceeded", http.StatusTooManyRequests)
            return
        }

        next.ServeHTTP(w, r)
    })
}

429 Too Many Requests is the standard HTTP response when the client has exceeded an application-defined request limit. Retry-After can help clients decide when to try again, but a hard-coded value is only appropriate when it matches the limiter’s policy. A production limiter may calculate a more precise delay from the refill rate.

Global limits and per-client limits

A single bucket limits all traffic passing through it. That is useful for protecting a scarce shared resource, but it does not provide fairness between clients. One noisy client could consume the entire allowance.

Per-client limiting normally requires a collection of buckets keyed by a stable identity such as an authenticated account or API key. Be careful when choosing that key.

IP addresses are often a poor identity boundary because many users can share one public address, proxies can obscure the original address, and trusting forwarded-address headers without a known proxy configuration can allow spoofing.

A map of per-client buckets also needs lifecycle management. If arbitrary client identifiers create permanent entries, the limiter itself can become an unbounded memory leak. Expire inactive buckets or use a bounded cache when the key space is large.

Rate limiting in multiple application instances

The implementation above is intentionally process-local. If an application runs on five replicas and every replica has a bucket configured for 100 requests per second, the deployment may collectively accept roughly five times the intended global rate.

That can be correct when each replica protects its own local capacity. It is not correct when the requirement is a single quota shared across the entire deployment.

Distributed limits usually need coordination through an API gateway, load balancer, dedicated rate-limit service, or shared data store with atomic operations. Do not describe an in-memory limiter as a global quota when requests can reach multiple independent processes.

Choose capacity and rate separately

A common mistake is to treat burst capacity and sustained throughput as the same number.

Suppose an API should sustain 20 requests per second but normal clients occasionally send ten requests at once. A capacity of 10 with a refill rate of 20 may fit that traffic pattern better than a capacity of 20 simply because the long-term rate is 20.

Start from two operational questions:

1. How much instantaneous work can the service safely absorb?
2. What sustained request rate can it safely process?

Those answers define capacity and refill rate more directly than an arbitrary requests-per-minute value.

Common pitfalls

Refilling with a ticker

A background ticker can work, but it adds a goroutine, synchronization, shutdown behavior, and scheduler timing to a problem that can be solved lazily. Calculating refill from elapsed time keeps the limiter self-contained.

Forgetting concurrency protection

Allow performs a read-modify-write sequence across several fields. Atomic access to one field is not enough; the state transition needs to be protected as a unit.

Using wall-clock timestamps for persistence

time.Time values created by time.Now can carry a monotonic clock reading, which Go uses for elapsed-time calculations inside the process. That is useful here. If limiter state is serialized and restored elsewhere, however, that monotonic component is not preserved. Distributed or persistent rate limiting needs a different design.

Creating unlimited per-client buckets

A limiter keyed by attacker-controlled values can consume memory indefinitely unless old entries are removed. Bound the number of entries and define an expiration policy.

Retrying every 429 immediately

Rate limiting only works if clients respect rejection. Clients should follow a documented Retry-After value when present and otherwise use a bounded retry strategy rather than immediately hammering the endpoint again.

Testing a token bucket

Tests based on real sleeps can become slow and flaky. A production-quality implementation can inject a clock function instead of calling time.Now directly. Tests can then advance a fake clock deterministically and verify exact refill behavior without waiting for real time to pass.

Useful cases include:

• the initial burst consumes capacity correctly;
• requests are rejected when fewer than one token remains;
• elapsed time replenishes fractional tokens;
• tokens never exceed capacity after a long idle period;
• concurrent calls never admit more requests than the available tokens permit.

Running tests with Go’s race detector is especially valuable for changes to limiter synchronization:

go test -race ./...

Keep the limiter close to the resource it protects

Rate limiting is most useful when its scope matches the constrained resource. A public API may need a gateway-level client quota, while an expensive endpoint may also need a tighter application-level limiter around database or upstream calls.

The token bucket is a compact building block, not a complete abuse-prevention system. Used at the right boundary, however, it gives a service predictable burst tolerance and sustained throughput with very little code.