Why the queue needs a bound
A thread pool's task queue is where work waits when every worker thread is already busy. It's tempting to make that queue unbounded — it's simple, it never rejects a task, and it "just works" under normal load. That simplicity is exactly the trap: an unbounded queue doesn't fix overload, it hides it.
If tasks arrive even slightly faster, on average, than the pool can process them, an unbounded queue grows without limit. Each queued task holds real memory — the task object itself, any captured state, sometimes a Future a caller is blocked waiting on — and none of it gets freed until a worker actually reaches it. Under sustained overload (a traffic spike, a slow downstream dependency making every task take longer, a bug that stalls workers), the queue can grow for minutes or hours with no external symptom at all, until the process finally runs out of memory and crashes — at which point every queued task is lost anyway, including the ones that could have completed fine under normal load. Production incidents caused by exactly this pattern — an unbounded internal work queue growing silently for days before an out-of-memory crash — are extremely common, precisely because nothing looks wrong until the very end.
A bounded queue converts a silent, delayed failure into an immediate, visible one: once the queue is full, the system must make an explicit decision about what happens next, right now, instead of deferring the decision (and the pain) indefinitely.
Backpressure as a general principle
The bounded-queue-plus-rejection-policy pattern is a specific instance of a much more general idea: a system under sustained overload should push back and shed or slow incoming work, rather than silently accumulating it. This shows up everywhere concurrent systems meet finite resources — TCP's receive-window flow control, HTTP 429/503 responses with Retry-After, reactive-streams' request(n) demand signaling, and message-queue consumer lag are all the same underlying principle applied at a different layer. A thread pool's rejection policy is simply where that principle gets applied at the boundary between "task submitted" and "task queued."
The alternative to backpressure isn't "no overload" — overload happens to every system eventually. The alternative is silent overload, which is strictly worse: it trades a controlled, immediate, debuggable failure for an uncontrolled, delayed, catastrophic one.
Rejection policies: what happens when the queue is full
Once both the queue and the pool (up to its max size) are full, a new task submission has to be handled somehow. The classic policies, and their trade-offs:
| Policy | Behavior | Trade-off |
|---|
| Abort / throw | Reject immediately; raise an exception to the caller. | Loses no information silently — the caller knows it was rejected and can retry, fall back, or surface an error. Requires the caller to actually handle that exception path. |
| Caller-runs | The thread trying to submit the task instead executes it directly, synchronously, itself. | An elegant self-throttling mechanism: no task is dropped, but the submitting thread is now busy running work instead of submitting more of it, which naturally slows the rate of new submissions. The cost: whatever was calling submit (often a request-handling thread) is now blocked for the task's full duration, which can add real latency or, chained enough times, exhaust an unrelated pool of callers. |
| Discard-oldest | Drop the task that's been waiting longest in the queue, then accept the new one. | Prioritizes recent work over stale work — reasonable if old queued tasks are likely to have already missed their useful deadline (e.g., a client that gave up and disconnected). Silently loses work, and if the queue is a priority queue, "oldest" and "least important" aren't the same thing, so this can drop your most important queued task by accident. |
| Discard | Silently drop the new task. | Simplest possible policy, and the most dangerous by default: the caller has no idea its task never ran unless it's independently monitoring rejection counts. Appropriate only when task completion is genuinely optional and unrelated to correctness (e.g., best-effort metrics). |
There's no universally "correct" policy — the right choice depends on whether dropped work, slowed callers, or loud, immediate failures is the least-bad outcome for your specific system. What's not defensible is picking a policy (or a queue bound) by accident rather than by deliberate trade-off analysis.
Caller-runs deserves the spotlight because it's the only one of the four that creates genuine backpressure rather than just discarding or deferring the problem: by tying up the producer, it converts "the system is overloaded" into "the producer slows down," which is exactly the feedback loop a healthy system needs. Its failure mode is equally important to know: if the task is slow and the caller was, say, a thread handling an inbound network connection, caller-runs can end up stalling connection acceptance itself — the backpressure propagates further upstream than you might expect, which is sometimes exactly what you want and sometimes a surprise worth explicitly designing for.
Blocking as an alternative to rejecting
A subtly different option from all four rejection policies above is to make the submitting thread block until space frees up in the queue, rather than rejecting or running inline. No standard rejection policy does this directly, but it's straightforward to build: bound the total number of in-flight-or-queued tasks with a semaphore (acquire a permit before submitting, release it when the task completes), paired with an unbounded queue behind it so the semaphore — not the queue — is the real limiting resource. This gives you the same throttling effect as caller-runs (the producer is slowed down) without forcing the producer itself to execute the task, at the cost of needing to build and reason about that mechanism yourself rather than getting it for free.
Pitfalls and interview gotchas
- Treating rejection as an edge case instead of a designed-for outcome. If you haven't decided what your rejection policy does and you're not monitoring queue depth and rejection rate, you've implicitly chosen "find out during an incident."
- Retry storms after rejection. If every rejected caller immediately retries without backoff, rejection doesn't relieve load — it adds the retries on top of the original load, potentially making the overload worse.
- Discard-oldest on a priority queue. "Oldest" (front of the queue) is not the same as "lowest priority" once the underlying structure is a priority queue — this policy can quietly discard your most important pending work.
- Caller-runs without accounting for who the caller is. If the caller is itself a scarce, latency-sensitive thread (e.g., the one accepting new network connections), tying it up runs a real risk of a different kind of cascading slowdown elsewhere in the system.
- Sizing the queue by intuition instead of by SLA. A queue holding 10,000 tasks at 200 ms average processing time represents roughly half an hour of latency for the last task in line — for many systems that's not "buffering a burst," that's a de facto outage that just doesn't look like one on a queue-depth graph until someone checks.