Thread pools and async

A web server that starts a new OS thread for every request works fine in testing. Then traffic spikes to 20,000 concurrent requests. Each thread reserves about 1 MB of stack, so that’s 20 GB of address space. The OS spends more time switching between threads than running them. The process dies, or crawls.

The fix most servers have used for 25 years is a thread pool: a fixed set of threads that take tasks from a queue. It’s the producer-consumer pattern, with the consumers built in.

Pools raise their own questions. How many threads? How big a queue? What to do when both are full? And for work that mostly waits on I/O, there’s a newer answer that avoids the pool’s limits entirely. One option is async/await on an event loop. Another is very cheap threads (goroutines, Java’s virtual threads) that the runtime schedules itself. This page covers when to use which.

What makes this hard

• A pool bounds how many tasks run at once, and reuses threads. So thread creation and memory stay fixed under load. Its queue must be bounded too, or overload just moves into the queue.
• Size it by the work. Use about one thread per core for CPU-bound work. For I/O-bound work, Little’s law gives the number in flight: throughput × time per task.
• A task that blocks waiting for another task on the same pool can deadlock the pool. Keep pools separate by kind of work, and never block a pool thread on the pool.
• Async/await and virtual threads let a waiting task cost almost nothing. So 10,000 concurrent I/O waits need only a handful of OS threads. They don’t make CPU work faster, and they don’t remove races: every await is a point where other code runs.

Thread pools

A pool has three parts, each a design decision:

1. Workers: N long-lived threads, each looping “take a task, run it”.
2. A queue of submitted tasks waiting for a free worker.
3. A rejection policy for when the queue is full. It can block the submitter, run the task on the submitter’s thread, reject with an error, or drop the task.

Submitting returns a future. It’s a handle the caller can wait on, for the result or the task’s exception. Shutting down means “accept nothing new, finish what’s queued”. Then, after a deadline, it interrupts whatever is still running.

JMain.javaPyexample.pyC#Program.csGomain.goTSexample.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.Future;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

// 100 CPU-bound jobs on a pool sized to the machine, with a bounded queue and a clear policy
// for overload, then a clean shutdown.
public class Main {
    public static void main(String[] args) throws Exception {
        int cores = Runtime.getRuntime().availableProcessors();
        // Built by hand instead of Executors.newFixedThreadPool, whose queue is unbounded:
        // under overload it would hold every submitted task in memory.
        ThreadPoolExecutor pool = new ThreadPoolExecutor(
                cores, cores,                         // CPU-bound work: one thread per core
                0, TimeUnit.SECONDS,
                new ArrayBlockingQueue<>(cores * 4),  // a bounded backlog
                // When the queue is full, the submitting thread runs the task itself. That slows
                // the producer down to the pool's pace: backpressure without dropping work.
                new ThreadPoolExecutor.CallerRunsPolicy());

        List<Future<Long>> results = new ArrayList<>();
        for (int i = 1; i <= 100; i++) {
            final int n = i;
            results.add(pool.submit(() -> sumOfSquares(n * 10_000))); // a Future per job
        }
        long total = 0;
        for (Future<Long> f : results) total += f.get(); // get() waits and rethrows a job's exception

        pool.shutdown(); // stop accepting work; queued jobs still run
        if (!pool.awaitTermination(10, TimeUnit.SECONDS)) pool.shutdownNow(); // then interrupt stragglers
        if (total != expected()) throw new AssertionError("wrong total");
        System.out.println("100 jobs on " + cores + " threads, total = " + total);
    }

    static long sumOfSquares(int n) {
        long s = 0;
        for (long i = 1; i <= n; i++) s += i * i % 7;
        return s;
    }

    static long expected() {
        long t = 0;
        for (int i = 1; i <= 100; i++) t += sumOfSquares(i * 10_000);
        return t;
    }
}

What each version shows:

• Java: ThreadPoolExecutor, built by hand. Executors.newFixedThreadPool(n) looks convenient, but has an unbounded queue. And newCachedThreadPool() has unbounded threads. Under overload, one exhausts memory and the other exhausts the OS. CallerRunsPolicy makes a submitter run the task itself when the pool is full. That slows the producer down to the pool’s pace.
• Python: for CPU-bound work, use a process pool, because the GIL lets only one thread run Python code at a time. Threads still help with I/O, since they release the GIL while waiting. ThreadPoolExecutor’s queue is unbounded, so bound submissions yourself under real load.
• C#: you rarely make a pool, because the runtime has one. Parallel.For caps CPU-bound parallelism. Parallel.ForEachAsync caps async I/O, where a waiting call holds no thread at all.
• Go: goroutines are cheap enough to start one per job. But a fixed set of workers reading a buffered channel bounds CPU and memory, and gives backpressure. Closing the channel is the shutdown.
• TypeScript: one thread runs all JavaScript. So a “pool” for I/O is a cap on promises in flight. CPU-bound work needs worker_threads, or it blocks everything.

How many threads?

CPU-bound work (resizing images, compressing, hashing) keeps a core busy the whole time. More threads than cores just take turns on the same cores, and add context switches. Use about one thread per core (availableProcessors(), NumCPU()). Maybe add one to cover the odd page fault.

I/O-bound work (HTTP calls, database queries) spends most of its time waiting. A waiting thread uses no CPU. There are two ways to size it:

• From the ratio of waiting to computing: threads ≈ cores × (1 + wait time / compute time). Say a request computes for 5 ms and waits 45 ms on the database. On 8 cores, it needs about 8 × (1 + 9) = 80 threads to keep the CPUs busy.
• From the load you must carry, with Little’s law: tasks in flight = arrival rate × time each task takes. At 2,000 requests a second and 50 ms per request, 2,000 × 0.05 = 100 requests are in flight on average. So you need at least 100 threads, plus headroom for bursts.

Then check the resource behind the pool. Say 100 threads all query a database with a 20-connection pool. Then 80 threads are waiting on the connection pool. The smallest limit in the chain sets the real concurrency, so size pools together.

TradeoffBigger pool vs smaller pool

We gain

A bigger pool keeps more I/O in flight. So throughput rises, while the CPU and downstream services have spare capacity.

We give up

Each thread costs memory and scheduling. And a bigger pool passes more load to whatever it calls. When the database slows down, a 500-thread pool sends 500 concurrent queries at it, and makes things worse. A small pool with a bounded queue and fast rejection fails some requests, instead of all of them.

Interviewer

Our service calls a slow downstream API. Should we raise the thread pool from 50 to 500?

What they are testing

Whether you size from numbers (Little’s law: rate times latency), and think about what’s behind the pool. The strong answer asks for the request rate and the downstream latency. It notes that 500 threads also means 500 concurrent calls to a service that’s already slow. And it suggests a timeout, a bulkhead (a separate small pool for that API) and async I/O. The red flag is “Yes, more threads means more throughput”.

Pool starvation: a deadlock without locks

A task on a pool submits a child task to the same pool, and waits for the child’s result. Say every pool thread is a parent doing the same. Then all of them wait for children that sit in the queue, with no thread to run them. Nothing holds a lock, and yet nothing can move. Step through it with a pool of one, then two:

Pool of 1 threadPool of 2 threads

A · Worker 1Step A

1. run parent task
2. pool.submit(child)
3. child.get() // block until done
4. finish parent task

ready

PlayRun 1,000 schedulesReset

A task on a thread pool submits a child task to the same pool and blocks until the child finishes. Step the threads and see whether the child ever runs.

A pool of two only moves the cliff: two parents at once hang it the same way. The fixes:

• Don’t block a pool thread on work queued to the same pool. Compose instead: thenCompose in Java, await in C# and Python. Then the parent gives its thread back while it waits. Java’s ForkJoinPool is built for this. A joining thread runs other queued tasks, instead of sleeping.
• Separate pools for separate kinds of work. This is the bulkhead pattern. One pool serves requests, another the slow downstream call, another background jobs. A flood of one kind can’t use up the threads the others need.

Common mistake

Blocking calls on async code's thread pool

In C#, calling .Result or .Wait() on a task inside a pool thread blocks that thread until the task finishes. And the task may need a pool thread to finish. Under load, the pool grows slowly (about one new thread a second when starved), and requests stall for seconds. In Java, CompletableFuture.get() inside a task on the common pool does the same. Await, compose, or move the blocking call to its own pool.

Go deeperThreadLocal and context leaks in pools2 min

Pool threads live forever, and run thousands of unrelated tasks. Anything stored per thread survives from one task into the next, unless it’s cleared. Examples are Java’s ThreadLocal, a request id in a logging context, or the current user. The classic bugs: a request sees the previous request’s user. Or a ThreadLocal holding a large buffer keeps it alive for the pool’s lifetime.

Clear thread-local state in a finally at the end of each task, or pass context explicitly. Java’s scoped values (standardized in recent releases), C#’s AsyncLocal<T>, Python’s contextvars and Go’s context.Context all scope context to a task rather than a thread. That’s the model to prefer.

Async and event loops

A thread that waits 50 ms for a database holds about 1 MB of stack and an OS thread the whole time, doing nothing. Async/await removes the waiting thread. When a function reaches await on I/O, it returns its thread to the runtime. The runtime resumes it later, when the I/O completes, on whatever thread is free.

In Node and Python’s asyncio, there’s exactly one thread, running an event loop. The loop takes the next ready callback, runs it until it awaits, and repeats.

One thread, many requests

Scroll or pinch to zoom, drag to pan

• Color key
• Database

Here are 10,000 requests that each wait 100 ms, in each language’s lightweight style. A pool of 200 OS threads would need 5 seconds. Each of these finishes in about a tenth of a second:

JMain.javaPyexample.pyC#Program.csGomain.goTSexample.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.time.Duration;
import java.util.concurrent.Executors;

// 10,000 requests that each wait 100 ms on I/O. With a pool of 200 platform threads this takes
// 10,000 / 200 x 100 ms = 5 s. With virtual threads (Java 21) it takes about 100 ms.
public class Main {
    public static void main(String[] args) throws Exception {
        long start = System.nanoTime();
        // One cheap virtual thread per task. When a virtual thread blocks on sleep, a socket or a
        // lock, the JVM parks it and reuses its carrier OS thread for another virtual thread.
        try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
            for (int i = 0; i < 10_000; i++) {
                executor.submit(() -> {
                    Thread.sleep(Duration.ofMillis(100)); // stands in for a database call
                    return null;
                });
            }
        } // close() waits for every task to finish
        long ms = (System.nanoTime() - start) / 1_000_000;
        if (ms > 3_000) throw new AssertionError("took " + ms + " ms");
        System.out.println("10000 waits of 100 ms finished in well under a second of wall time");
    }
}

• Java 21 virtual threads: plain blocking code (Thread.sleep, JDBC, sockets). But each task gets a virtual thread costing a few KB. When it blocks, the JVM parks it, and reuses the OS thread underneath. There’s no async keyword, and no colored functions.
• Python asyncio.gather: one thread, one event loop, and await at every I/O point.
• C# async/await with Task.WhenAll: the awaits hold no thread, and continuations run on the shared pool.
• Go: there is no separate async style. Every goroutine is lightweight. The scheduler parks a blocked goroutine, and runs another on the same OS thread. So blocking code scales like async code.
• TypeScript: Node’s event loop. The second half of the example shows the catch. A 200 ms CPU loop delays a 0 ms timer by the full 200 ms, because nothing else runs until the loop ends. Every request on the server waits behind it.

Async code still races

One thread doesn’t mean no races. Between two awaits, code runs without interruption. But at each await, any other ready task can run. Say a function reads a balance, awaits, then writes based on what it read. That has exactly the check-then-act gap from the race conditions page. Each step below runs a request until its next await. Play 1, 2, 1, 2:

await between check and writeOne conditional update

A · Request 1Step A

1. const b = await db.getBalance()
2. if (b >= 100) await db.setBalance(b - 100)

ready

B · Request 2Step B

1. const b = await db.getBalance()
2. if (b >= 100) await db.setBalance(b - 100)

ready

Play 1, 2, 1, 2Run 1,000 schedulesReset

One Node.js thread serves two withdrawal requests for 100 from a balance of 100. A step runs a request until its next await. Try 1, 2, 1, 2.

The fixes are the same ideas as with threads:

• Keep the read and the write in one synchronous run.
• Push the check-and-write into one atomic operation in the database (UPDATE ... WHERE balance >= 100).
• Or serialize the critical section with an async lock.

TradeoffAsync/await vs a thread pool with blocking calls

We gain

A waiting request costs a few hundred bytes, instead of a thread. So one process holds tens of thousands of concurrent I/O waits. There’s no pool to size for I/O. And on a single-threaded loop, there are no data races between lines of synchronous code.

We give up

“Function coloring”: async spreads up the call stack. And one blocking call stalls every request on that loop. That could be a sync file read, a CPU-heavy loop, or a library that isn’t async. Stack traces and debugging are harder. And it does nothing for CPU-bound work, which still needs threads or processes.

Choosing a model

Work	Java	Python	C#	Go	TypeScript
CPU-bound, parallel	Fixed pool, one thread per core	ProcessPoolExecutor	Parallel.For, the thread pool	Worker goroutines, one per core	worker_threads
Many concurrent I/O waits	Virtual threads	asyncio	async/await	A goroutine per request	async/await (the default)
Limit calls to one dependency	Semaphore, or a small separate pool	asyncio.Semaphore	SemaphoreSlim	Buffered channel	A promise limiter
A few blocking calls in an async app	Not needed with virtual threads	asyncio.to_thread	Task.Run, sparingly	Not needed	worker_threads or a child process

Go deeperVirtual threads: what's still a trap2 min

Virtual threads make blocking cheap, but they still have rules. In Java 21 to 23, a virtual thread that blocks inside a synchronized block is pinned to its carrier OS thread. That carrier then can’t run other virtual threads. With enough of them, the carrier pool is used up. (Java 24 removed most pinning.) Replacing synchronized with ReentrantLock around blocking calls avoided it.

Pooling virtual threads is also a mistake. They’re meant to be created per task and thrown away. To limit concurrency against a database, use a Semaphore around the call, not a pool of virtual threads. And ThreadLocal caches built for a few hundred pool threads become a million copies with a million virtual threads.

Say this out loud

These calls are I/O-bound. At 50 ms each and 2,000 a second, Little’s law gives about 100 in flight. I’d use async I/O or virtual threads, so waiting costs nothing. I’d cap concurrency to the database at its connection limit, with a semaphore. And I’d keep CPU-heavy work on a separate pool sized to the cores.

Mid-level covers

Uses a fixed pool with a bounded queue. Sizes it to cores for CPU work, and shuts it down cleanly. Knows async/await frees threads during I/O, but still races across awaits.

Senior adds

Also sizes I/O concurrency with Little’s law and the downstream limits. Picks a rejection policy on purpose. Recognizes pool starvation, and uses bulkheads. Explains event loop blocking and function coloring. Knows the trade-offs of virtual threads and goroutines against async.

How pools and async fail

Failure	Symptom	Prevention
Unbounded queue	Memory climbs during an outage downstream; latency in minutes	Bounded queue and a rejection policy
Unbounded threads	Thousands of threads, the OS thrashes, out-of-memory errors	A fixed maximum; or virtual threads, which are cheap
Pool starvation	Every worker blocked, CPU idle, queue growing	Never block on the same pool; separate pools
Blocking the event loop	Every request’s latency jumps together	Move CPU work to workers; no sync I/O in async code
Exceptions lost in futures	A failed task leaves no trace	Always get()/await the future, or attach a handler
No shutdown	The process hangs on exit, or drops queued work	shutdown, wait with a deadline, then cancel

Interview questions

Interviewer asksHow do you choose the size of a thread pool?

By the kind of work. For CPU-bound work, use about the number of cores, since more threads only add switching. For I/O-bound work, use the wait-to-compute ratio (cores × (1 + wait / compute)). Or use Little’s law: arrival rate × task duration gives the number in flight. Add headroom. Then check what the pool calls. There’s no point running more concurrent queries than the database pool allows. Finally, measure under load and adjust.

Interviewer asksWhat happens when a ThreadPoolExecutor's queue is full?

First it adds threads, up to maximumPoolSize. (Threads beyond the core size are only created when the queue is full.) Then it applies the rejection policy:

• AbortPolicy throws RejectedExecutionException. This is the default.
• CallerRunsPolicy runs the task on the submitting thread.
• DiscardPolicy drops it silently.
• DiscardOldestPolicy drops the oldest queued task.

Choose on purpose. CallerRunsPolicy gives natural backpressure. AbortPolicy gives fast failure that you can turn into a 503.

Interviewer asksIs async/await faster than threads?

Not for any single request, and not for CPU-bound work. It’s cheaper per waiting request: no thread, no stack, no context switch. So one process can hold far more concurrent I/O waits. That raises throughput when the bottleneck is the number of threads. If the bottleneck is CPU or a downstream service, async changes nothing.

Interviewer asksVirtual threads or async/await for a new Java service?

Virtual threads, in most cases. They give the same scalability for I/O-bound work. And they keep plain blocking code, readable stack traces and existing libraries (JDBC, HTTP clients). Async or reactive styles still make sense for streaming with backpressure, or where a framework is already built on them. Watch for pinning on older JDKs, and don’t pool virtual threads.

Interviewer asksWhy can a Node.js server become slow for every user at once?

Because all JavaScript runs on one thread. Any CPU-heavy synchronous work blocks the event loop. That could be parsing a huge JSON body, a sync crypto or compression call, a slow regex, or a big loop. Every other request, timer and callback waits behind it. To fix it, move CPU work to worker_threads or another service, use async APIs, and monitor event loop lag.

Recap

Idea	Remember
Thread pool	Fixed workers plus a queue: producer-consumer with the consumers built in
Bounded everything	Bounded threads, bounded queue, explicit rejection policy
CPU-bound sizing	About one thread per core
I/O-bound sizing	Little’s law: in flight = rate × duration; respect downstream limits
Pool starvation	Tasks waiting on tasks in the same pool; compose instead, use bulkheads
Async/await	A waiting task holds no thread; one blocking call stalls the loop
Races across await	Every await is an interleaving point; check-then-act still breaks
Virtual threads, goroutines	Blocking code at async scale; no function coloring

Further reading

• Brian Goetz et al., Java Concurrency in Practice, chapters 6 and 8 (task execution and thread pools), including the sizing formula.
• JEP 444: Virtual Threads.
• Don’t block the event loop, from the Node.js docs.
• Back to the concurrency track, or apply all of this to a design with the LLD problems, each of which ends with a concurrency section.