Concurrent Rate Limiters & Bounded Resource Pools

Language-specific mechanics: Concurrency Language Manual →

Two related problems, one underlying skill

"Design a rate limiter" and "design a bounded resource pool (connection pool, thread pool, object pool)" get asked as if they're system-design questions, but the concurrency-correctness core of both is squarely coding-interview material, and it's the same core: some shared, mutable piece of state (a token count, a set of idle connections) is read and updated by multiple threads, and the update has to be atomic end-to-end, not just per individual field. Everything from Topics 2–7 of this roadmap — critical sections, locks, atomics, memory visibility — applies directly here; a rate limiter is, underneath the system-design dressing, a critical-section problem with a clock in it.

Rate-limiting algorithms and what "correct" means for each

There are four standard algorithms, and interviewers expect you to know the trade-offs, not just recite token bucket:

Algorithm	Idea	Burst behavior	Memory per client
Fixed window counter	Reset a counter to 0 every fixed interval	Poor — allows ~2x limit at window boundaries	Tiny (one counter)
Sliding window log	Store a timestamp per request, count how many fall in the trailing window	Exact	High (all timestamps)
Sliding window counter	Weighted blend of current + previous fixed window counts	Good approximation	Tiny (two counters)
Token bucket	A bucket holds up to `capacity` tokens, refilling at `rate` tokens/sec; each request spends one	Controlled bursts up to capacity	Low (two numbers: tokens, last-refill time)
Leaky bucket	Requests queue up and drain at a constant rate; the queue itself is the "bucket"	None — smooths everything to a constant rate	Medium (queue depth)

Token bucket is the default answer for most user-facing APIs (it's what Stripe, GitHub, and AWS use) because it tolerates realistic bursty traffic while still enforcing a long-term average; leaky bucket is the better answer when the downstream system genuinely cannot tolerate any burst (payment processors, video pipelines). Know both, and know why: this is a "pick the right tool" question as much as an implementation one.

But regardless of which algorithm you pick, the concurrency requirement is identical: the read-refill-check-decrement sequence on the bucket's state must be one atomic operation. A token bucket implemented with if (tokens > 0) tokens--; split across two unsynchronized statements has exactly the same race condition as the naive counter++ from Topic 2 — two threads can both read "1 token left," both decide to proceed, and the bucket goes negative. The fix is the same fix as everywhere else in this roadmap: guard the whole read-modify-write with a lock, or make it a single atomic compare-and-swap loop (Topic 7), or — in a distributed deployment — push the whole check into a single atomic operation on a shared store (a Redis EVAL running a Lua script, so the read-refill-decrement happens as one indivisible step on the server, not as several round-trips a second thread could interleave with).

Bounded resource pools: a semaphore gates the count, a collection holds the resources

A connection pool, thread pool, or generic object pool is a different-looking problem with the same synchronization discipline as Topic 4's semaphore material, decomposed into two cleanly separated pieces:

A counting semaphore, sized to the pool's capacity, whose only job is to gate how many callers may hold a resource at once. acquire() blocks a caller when the pool is fully checked out; release() wakes the next waiter the instant a resource is returned. This is exactly Topic 4's "counting semaphore controls concurrent access to N interchangeable resources" pattern — nothing new.
A thread-safe collection (a blocking queue is the standard choice) that actually holds the idle resources. The semaphore controls permission; the queue holds the inventory. Acquiring a resource means: acquire a permit, then pull an item from the queue; releasing means: return the item to the queue, then release a permit — and it's worth noting explicitly that the order matters for correctness (put the resource back before you free up a slot for someone else to look for it, or you can hand out a permit for a resource that isn't actually back in the collection yet).

This two-part decomposition is precisely what LeetCode's "Design Bounded Blocking Queue" is testing, and it's precisely the shape of a real production connection pool (HikariCP, Java's own ArrayBlockingQueue-backed pools): a semaphore (or equivalently, a bounded blocking queue's own internal blocking behavior) for admission control, plus a collection for inventory.

Distributed rate limiting: the same race, one network hop further out

Once a rate limiter has to be enforced consistently across multiple application servers rather than one process, an in-memory Semaphore or AtomicLong stops being enough — every server has its own copy of the state. The standard fix is to move the shared counter into a single external store (Redis is the default choice) and to make sure the check-and-update against that store is itself atomic, exactly the same requirement as the single-process case, just relocated. Redis's INCR is atomic on its own, but a full token-bucket check (read tokens, compute refill, compare, decrement) is several operations — so production implementations wrap the whole sequence in a Lua script, which Redis guarantees runs as one uninterruptible unit server-side. This is the direct bridge into this topic's final subtopic: the primitive changes from an in-process lock to a Lua script or a distributed lock, but the underlying problem — "don't let two concurrent actors both think they got the last token" — is identical.

Pitfalls and interview gotchas

Splitting the check and the decrement into two unsynchronized steps. This is the single most common bug across every implementation in this subtopic, whether it's a token bucket, a semaphore-gated pool, or a distributed counter — always ask "can two threads both pass the check before either applies the update?"
Refilling a token bucket with wall-clock time without protecting the refill calculation itself. tokens = min(capacity, tokens + elapsed * rate) looks like a pure read, but it reads and writes shared mutable state (tokens, lastRefillTime) and needs the same lock/CAS discipline as the decrement.
Returning a resource to the pool before validating it (or after a caller has already been handed a fresh permit for it). Production pools validate a connection isn't stale before handing it out — that's a correctness nuance beyond raw synchronization, but it's the detail that separates a "textbook-correct" answer from a "would survive code review."
Choosing fixed-window counting and not mentioning its boundary-burst flaw. A limit of 100/minute with a fixed window lets a client send 100 requests at 0:59 and another 100 at 1:01 — 200 requests in two seconds, technically within the letter of "100 per window" every window. Naming this is expected; not naming it reads as a gap.
Treating "bounded" as optional. An unbounded thread pool or unbounded work queue isn't a resource-management solution, it's a deferred OutOfMemoryError — Topic 9 covers this in depth, but it's worth remembering here: the entire point of this pattern is the bound.

The bug to avoid isn't the algorithm — it's splitting the refill, the check, and the decrement across multiple unsynchronized statements.

type TokenBucket struct {
    mu           sync.Mutex
    capacity     float64
    refillPerSec float64
    tokens       float64
    lastRefill   time.Time
}

func (b *TokenBucket) TryAcquire() bool {
    b.mu.Lock()
    defer b.mu.Unlock()
    now := time.Now()
    b.tokens = min(b.capacity, b.tokens+now.Sub(b.lastRefill).Seconds()*b.refillPerSec)
    b.lastRefill = now
    if b.tokens < 1 {
        return false
    }
    b.tokens--
    return true
}

sync.Mutex has to cover refill and the check-and-decrement — splitting them lets two goroutines both observe tokens >= 1 and overdraw. atomic.Int64 is the wrong tool here because the refill math isn't a single integer RMW. Same critical-section discipline as the Java version, expressed with defer Unlock().

The semaphore answers "how many callers may hold a resource right now"; the collection answers "which resources are actually free." Returning the item before freeing the permit keeps the two in sync.

type ConnectionPool struct {
    idle chan *Connection // capacity == pool size
}

func NewConnectionPool(conns []*Connection) *ConnectionPool {
    idle := make(chan *Connection, len(conns))
    for _, c := range conns {
        idle <- c
    }
    return &ConnectionPool{idle: idle}
}

func (p *ConnectionPool) Acquire() *Connection { return <-p.idle }
func (p *ConnectionPool) Release(c *Connection) { p.idle <- c }

A buffered channel collapses the semaphore-plus-queue pair into one primitive: capacity is the pool size, receive blocks when empty, send blocks (or select/default rejects) when full. Don't add a separate WaitGroup or Mutex around the inventory — that's how the two counters drift. Closing idle is a shutdown signal, not something you do on Release.

Further Resources (Optional)

Optional Practice (Extra Reps)

For once you've cleared the main set above and want more reps on this pattern. These don't count toward the roadmap's progress stats — solve them purely for your own benefit.

Design Bounded Blocking QueueMediumPremium!!!4/530m
Web Crawler MultithreadedMediumPremium!!!4/535m