Backpressure and load shedding

Every component has a rate it can sustain. When work arrives faster than that, the difference has to go somewhere. It can wait in a queue. The sender can be made to slow down. Or some of it can be thrown away. Backpressure is the second answer: a signal that travels upstream, and slows the producer. Load shedding is the third: the system refuses work on purpose, choosing what to lose. The queue is the first answer. It only buys time, never capacity.

This page is about the pattern. The mechanisms have their own pages:

• Message queues, for brokers, acks and prefetch.
• Rate limiting, for per-key limits and adaptive concurrency.
• Resilience, for timeouts, deadlines, retries and circuit breakers.

The problem it solves

A checkout service runs 40 workers, each handling a request in about 50 ms, so it completes about 800 requests a second. Normal traffic is 500 a second. A marketing email lands and traffic jumps to 1,200 a second for ten minutes.

Every second, 400 more requests arrive than leave. They wait in the server’s request queue, which nobody ever sized. After one minute, 24,000 are queued, and a request arriving now waits 24,000 ÷ 800 = 30 seconds before a worker looks at it. The client timeout is 5 seconds. The server is at 100% CPU, completing 800 requests a second, and every response goes to a closed connection. Throughput is 800 a second. Useful throughput is zero.

Then the clients retry, doubling arrivals. Memory climbs with the queue, until the process is killed. It restarts, the clients reconnect at once, and it falls over again. The email campaign ended twenty minutes ago. The outage did not.

Three ideas separate this from a system that would have dipped and recovered:

1. A queue does not add capacity. It moves the excess from “refused now” to “served late”. Past the client’s timeout, served late costs the full price, and helps nobody.
2. Throughput is not goodput. The number that matters is completions within the caller’s deadline.
3. The fix lives upstream, or at the door. Tell the producer to slow down (backpressure). Or refuse the excess quickly and cheaply (shedding). Then the work you accept is work someone still wants.

The shape: a producer faster than its consumer

Scroll or pinch to zoom, drag to pan

• Color key
• Database
• Service
• Failure

The same shape appears wherever one stage feeds another: a web tier into a database, a fan-out worker into a push provider.

How it works

Step 1: The queueing numbers

Two results from queueing theory carry most conversations about overload.

Utilization decides waiting time, and not in a straight line. Take one server with service time S, and random (Poisson) arrivals. This is the textbook M/M/1 queue. Utilization ρ is the fraction of time the server is busy. At utilization ρ, the time spent waiting in line is ρ ÷ (1 − ρ) × S. And the average time in the system, waiting plus service, is

time in system = S ÷ (1 − ρ)

Utilization	Time in system	With S = 10 ms	What it feels like
50%	2 × S	20 ms	Nobody notices
80%	5 × S	50 ms	The knee: fine on average, spiky p99
90%	10 × S	100 ms	Latency doubled for 10% more load
95%	20 × S	200 ms	Every small burst becomes a long queue
99%	100 × S	1,000 ms	Effectively an outage

Servers with many workers (M/M/c) stay flat longer, and then turn more sharply. Real traffic is burstier than Poisson, which moves the knee earlier. The tail is worse than the mean. In M/M/1, the time in system is exponentially distributed, so p99 is about 4.6 times the mean (ln 100). That’s why capacity plans aim for 50 to 70% busy at peak.

Little’s law relates the queue to the rate and the wait. For any stable system, whatever the distribution,

items in the system (L) = arrival rate (λ) × time each spends there (W)

At 2,000 requests a second and 50 ms each, 100 requests are in flight. Say a dependency slows to 1 s, at the same arrival rate. Now 2,000 are in flight: twenty times the threads, connections and memory. It also sizes queues. Say a queue should add at most 200 ms of wait, in front of a stage that completes 1,000 a second. Then it holds at most 1,000 × 0.2 = 200 items.

Past 100% utilization neither formula applies, because there is no steady state: the queue grows by arrival rate minus service rate, forever.

Step 2: Bound the queue

The first fix is the one most systems lack: give every queue a maximum size. Java’s Executors.newFixedThreadPool uses an unbounded LinkedBlockingQueue. In-memory channels and async request backlogs are usually unbounded too. An unbounded queue turns overload into memory growth and latency growth. It ends in an out-of-memory kill.

A bound turns the slow failure into a fast decision about what to do when full:

When full	What happens	Wins when	Cost
Block the producer	put waits until there is room	The producer is in-process or can wait: a pipeline stage, a batch job	The producer’s own queue fills next; across a network it ties up connections
Drop the newest	Refuse the arriving item	Every item has similar value, older ones are already promised	Newcomers lose even when old items are about to expire anyway
Drop the oldest	Evict the head, admit the newcomer	Freshness matters more than completeness: telemetry, positions, live prices	Wasted work on what was evicted; senders may not learn of the loss
Reject with 503	Answer at once with 503 and Retry-After; the client decides	Request/response services: the caller can retry, fail over or degrade	Clients must handle it, and retry with jitter and a budget

Blocking is backpressure in its purest form. Think of a bounded channel in Go, or BlockingQueue.put in Java. A Kafka producer whose buffer.memory (32 MB by default) is full blocks in send, for up to max.block.ms (60 seconds by default). The consumer’s pace reaches the producer with no extra code. But only if every stage in between is bounded. One unbounded hop absorbs the signal, and grows instead.

Rejecting is backpressure across a network. An HTTP server cannot block a client’s send. So it answers 503 Service Unavailable at once, which costs microseconds and no memory. Retry-After (seconds, or an HTTP date) tells the client how long to stay away. It is a hint, so clients add jitter, and don’t all return in the same second. Overload is not the caller’s fault, so the status is 503, not 429. A 429 means “you, specifically, are over your limit” (see rate limiting).

A bounded queue deciding what to do with an arrival

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Failure
• Database

TradeoffA bound sized from the latency budget, rejecting when full

We gain

Memory is capped. Worst-case queueing delay is capped (bound ÷ service rate). The excess gets an answer in microseconds, instead of a timeout in seconds. And clients learn to back off.

We give up

Some requests fail that an unbounded queue would have served, during short bursts it could have absorbed. Size the bound too small and normal jitter causes refusals.

Step 3: Let the consumer pull

The cleanest backpressure is a design where the consumer asks for work instead of having it pushed. A consumer that only takes what it can handle can never be overrun.

Mechanism	How the consumer limits the flow
Kafka consumer	Calls poll() when ready, gets up to max.poll.records (500 by default). A slow consumer falls behind and the log keeps the difference on disk.
RabbitMQ, SQS	Prefetch (basic.qos) caps unacknowledged messages per consumer; SQS hands out at most 10 per receive.
Reactive Streams	The subscriber calls request(n); the publisher may send at most n more items. Java’s Flow, Reactor, RxJava and Akka Streams implement it.
TCP	The receiver advertises a window: how many bytes it can buffer. A full buffer advertises zero and the sender stops, probing until it reopens.
HTTP/2 and gRPC	Per-stream and per-connection windows (65,535 bytes initially) refilled by WINDOW_UPDATE frames as the receiver reads.

Credit-based flow control: the consumer grants the right to send

Scroll or pinch to zoom, drag to pan

• Color key
• Background worker
• Service

All five share one idea: credit. The receiver grants the right to send a bounded amount, and grants more only after it has made room. Its memory is bounded by the credit, and the sender learns about a slow receiver without a separate signal.

Pull moves the question, instead of removing it. A Kafka topic absorbs the difference on disk: days of retention, instead of seconds of RAM. But brokers throttle producers over their byte quota, by delaying responses. The producer’s buffer fills, and send blocks. Somewhere upstream, a web request cannot block forever. That is where shedding happens.

Consumer lag is the signal of a pull system. It’s the log-end offset minus the group’s committed offset. The useful unit is time. 10,000 messages behind is a fifth of a second, for a consumer doing 50,000 a second. It’s nearly three hours, for one doing 1 a second. Queues without offsets expose the same thing as the age of the oldest message (ApproximateAgeOfOldestMessage in SQS).

Step 4: Admit only what you can finish

Bounding queues limits waiting. Bounding concurrency limits the work itself: at most N requests in progress, and the rest queued briefly or refused. Little’s law gives N. 800 a second at 50 ms is 40 in flight. With headroom, cap at 60.

A fixed N is wrong as soon as latency changes. Say the database slows from 50 ms to 200 ms. The same 60 slots now complete 300 a second, and the right answer is to admit fewer. Adaptive concurrency limits find N from measured latency, the way TCP finds bandwidth. The rate limiting page covers the algorithm. What matters here is placement. Put it in front of the expensive work, so a refusal costs almost nothing. A virtual waiting room is the same idea, applied to people before they reach the service.

Step 5: Shed the right work

When the system must drop work, the choice of which work decides whether the outage is visible.

Shed by priority. A checkout matters more than a recommendation. Tag every request with a criticality at the edge and shed the lowest first. Google’s RPCs carry one (CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE) and backends refuse from the bottom up as they near their limit. Separate queues or worker pools per priority stop a flood of low-priority work from taking the slots critical work needs.

Shed by deadline. A request whose client has given up is the cheapest thing to drop, and the most wasteful to serve. Say the deadline is propagated with the request. Then check it twice. On arrival, refuse if the estimated wait exceeds the time left. On dequeue, skip anything that cannot finish in time. Under heavy overload, most of the queue is already dead. Skipping it is how the server catches up.

Serve the newest first under overload. FIFO is fair when queues are short. When they are long, FIFO makes every request wait the full queue, so every request misses its deadline. LIFO serves the newest requests, whose clients are still waiting, at once, while the oldest sink and expire. Facebook’s servers use adaptive LIFO: FIFO normally, LIFO when the queue is backed up. It is unfair to the unlucky few by design, and it turns “everyone times out” into “most succeed”.

The snippet combines all three. It keeps one bounded lane per priority, and evicts lower-priority work when full. It checks deadlines on both ends. And it serves newest-first past a depth.

JSheddingQueueDemo.javaPyshedding_queue.pyC#SheddingQueue.csGoshedding_queue.goTSshedding-queue.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.ArrayDeque;
import java.util.Map;
import java.util.TreeMap;

public class SheddingQueueDemo {
    static final int CRITICAL = 0, NORMAL = 1, SHEDDABLE = 2; // lower number = more important
    static final String[] NAMES = {"critical", "normal", "sheddable"};

    // deadline: the time in ms when the client gives up waiting.
    record Request(int id, int priority, double deadline) {}

    // A bounded queue that sheds the least valuable work first. One lane per priority,
    // so finding the cheapest victim is O(1) and a burst of sheddable work can never
    // push out a critical request.
    static class SheddingQueue {
        private final int capacity;
        private final int lifoAbove; // past this depth, serve newest first
        private final ArrayDeque<Request>[] lanes;
        private int size;
        final Map<String, Integer> stats = new TreeMap<>(Map.of("rejected", 0, "evicted", 0, "expired", 0));

        @SuppressWarnings("unchecked")
        SheddingQueue(int capacity, int lifoAbove) {
            this.capacity = capacity;
            this.lifoAbove = lifoAbove;
            lanes = new ArrayDeque[] {new ArrayDeque<>(), new ArrayDeque<>(), new ArrayDeque<>()};
        }

        String offer(Request req, double now) {
            if (req.deadline() <= now) {
                // The client already gave up: queueing it is pure waste.
                stats.merge("expired", 1, Integer::sum);
                return "expired";
            }
            if (size < capacity) {
                lanes[req.priority()].addLast(req);
                size++;
                return "accepted";
            }
            // Full. Evict from a strictly less important lane, least important first.
            for (int p = SHEDDABLE; p > req.priority(); p--) {
                if (!lanes[p].isEmpty()) {
                    // The oldest in that lane has the least time left: the one most
                    // likely to miss its deadline anyway.
                    lanes[p].pollFirst();
                    stats.merge("evicted", 1, Integer::sum);
                    lanes[req.priority()].addLast(req);
                    return "accepted";
                }
            }
            // Nothing less important to drop: refuse now, fast and cheap.
            // The caller turns this into 503 with Retry-After.
            stats.merge("rejected", 1, Integer::sum);
            return "rejected";
        }

        Request poll(double now, double serviceMs) {
            for (ArrayDeque<Request> lane : lanes) { // most important lane first
                while (!lane.isEmpty()) {
                    // Adaptive LIFO: under a deep queue the oldest requests are the ones
                    // whose clients are about to give up, so serve the freshest.
                    boolean newestFirst = size > lifoAbove;
                    Request req = newestFirst ? lane.pollLast() : lane.pollFirst();
                    size--;
                    if (req.deadline() < now + serviceMs) {
                        // It would finish after the client stopped waiting: skip it
                        // instead of spending capacity on a response nobody reads.
                        stats.merge("expired", 1, Integer::sum);
                        continue;
                    }
                    return req;
                }
            }
            return null;
        }
    }

    public static void main(String[] args) {
        // One worker at 10 ms a request: 100 a second. Offered load is 150 a second for
        // 2 seconds, then 50 a second. Every client waits at most 300 ms.
        SheddingQueue q = new SheddingQueue(40, 20);
        int[] served = new int[3], offered = new int[3];
        int[] mix = {CRITICAL, NORMAL, NORMAL, NORMAL, NORMAL, NORMAL, NORMAL, SHEDDABLE, SHEDDABLE, SHEDDABLE};
        double busyUntil = 0, credit = 0;
        int nextId = 0;
        for (int now = 0; now < 4000; now++) { // one step per millisecond
            credit += now < 2000 ? 0.15 : 0.05;
            while (credit >= 1) {
                credit -= 1;
                int priority = mix[nextId % 10];
                offered[priority]++;
                q.offer(new Request(nextId, priority, now + 300), now);
                nextId++;
            }
            if (busyUntil <= now) {
                Request req = q.poll(now, 10);
                if (req != null) {
                    served[req.priority()]++;
                    busyUntil = now + 10;
                }
            }
        }
        for (int p = 0; p < 3; p++) {
            System.out.printf("%-10s served %d of %d%n", NAMES[p], served[p], offered[p]);
        }
        System.out.println(q.stats);
    }
}

In the run, offered load is 150 a second against a capacity of 100 for two seconds, then 50 a second for two more. All 40 critical requests are served, 232 of 240 normal ones and 36 of 120 sheddable ones. The queue refused 18 arrivals, evicted 42 queued requests for more important ones, and skipped 32 that could no longer finish in time. Every request it served reached a client still waiting.

Step 6: Manage the queue by time, not length

A length bound is a proxy. What clients feel is time in queue. The right length depends on how fast the queue drains, and that changes.

CoDel (controlled delay) was designed for network routers. It tracks the minimum time packets spent in the queue. Say that minimum stays above a 5 ms target for a whole 100 ms interval. Then the queue is standing, not a burst. CoDel starts dropping packets at dequeue, more often the longer it lasts. (The gap between drops shrinks as interval ÷ √count.) Short bursts pass untouched, because some packet in each interval got through quickly.

Facebook applied the idea to server request queues. If the queue has not been empty in the last 100 ms, requests may wait only 5 ms before being dropped. If it has, they may wait up to 100 ms. A burst gets the generous timeout. A standing overload gets the strict one.

Go deeperWhy minimum sojourn time, and not average1 min

A queue that fills in a burst, and drains within the interval, has a large average delay. But its minimum is near zero, because some request arrived just after it drained. A standing queue never drains. Every request waits, so the minimum is high too.

So the minimum over an interval is the cheapest reliable way to tell the two apart. A burst is what a queue exists to absorb. Overload is what a queue only hides. Queue length cannot make that distinction, without knowing the drain rate. Delay measures it directly.

Step 7: Autoscaling is backpressure on a timer

Adding capacity is the other way to make arrival and service rates meet. Autoscaling watches a signal (CPU, requests per instance, consumer lag, queue age), and adds workers. It takes minutes: metric averaging, a stabilization window, boot, and warm-up. A spike that doubles traffic in 10 seconds has done its damage before the first new instance serves a request. So autoscaling handles the slow part of the curve: the daily peak, or a backlog that will take an hour to drain. Shedding handles the fast part.

Go deeperWhere the minutes go1 min

Step	Typical time
Metric collected and averaged	15 s to 1 min (HPA syncs every 15 s by default)
Scaling decision, with stabilization	1 to 5 min to avoid flapping
New VM boots and passes health checks	1 to 5 min
New container on an existing node	Seconds to a minute (image pull, JVM warm-up)
New capacity warm (caches, JIT, pools)	Another minute or more

Queue-driven consumers scale well on lag in seconds. KEDA, for example, scales Kubernetes workloads on Kafka consumer lag. Scaling on CPU fails for I/O-bound workers, which are slow while looking idle. And autoscaling has a ceiling. Consumers beyond the partition count sit idle. App servers beyond the database’s connection limit fail to connect. And a hot partition does not get faster with more shards.

Try it

One server with a capacity of 100 requests a second, a queue in front, and clients that give up after 0.4 to 1.6 seconds.

Queue policy

UnboundedBounded, rejectDrop oldestDeadline-awareLIFO

Arrival rate 70 a second, 70%of the capacity of 100

Spike: +200/s for 3 sClients retryPauseReset

Requests a second, last 30 sOfferedThroughputGoodputCapacity

300

Queue length, last 30 s

100

Queue now

0

p50 time to serve

-

p99 time to serve

-

Throughput / goodput

-

Clients gave up waiting

-

Refused fast (503)

-

Rates are per second over the last 5 s. Goodput counts responses that reached a client still waiting.

Things to try:

• Unbounded, rate 70, press Spike. 600 extra requests arrive in 3 seconds. Goodput (green) falls to zero while throughput (blue) stays at 100: the server is serving people who left. It recovers after about 20 seconds, because 70 is below capacity.
• Now turn on Clients retry and spike again. Each timeout spawns a new request at once. After the spike ends, offered load stays above 200 a second, the queue keeps growing and goodput stays at zero. Drag the rate down to 40: it still does not recover. That is a metastable failure: the trigger is gone, and the system holds itself down.
• While it is stuck, switch to Deadline-aware. The backlog is trimmed, dead work is skipped and hopeless arrivals are refused. Goodput returns within a few seconds.
• Set the rate to 150 and compare policies. Unbounded serves 100 a second with a goodput of zero. Bounded-reject keeps goodput near capacity but still serves some requests that time out in line. Deadline-aware keeps goodput at capacity by refusing the hopeless ones up front. LIFO has the best p50 of all, with a steady stream of the oldest requests timing out at the bottom.
• Rate 85, no spike. Nothing is overloaded, yet p99 is several times p50: the 80% knee from the table above.

Step 8: Push the pressure all the way back

A real request crosses several stages, and each needs a bound and a policy. When the deepest stage slows, every stage above it should fill its small bound in turn, and start refusing. Then the refusal happens at the edge, early and cheaply. Not in the middle, after the work is half done.

Bounded at every hop, so pressure reaches the edge

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

Two places may stretch. A durable log between the request path and async work is a deliberate shock absorber. The request returns once the event is stored, and workers drain at their own pace. That suits work that can be late: fan-out, analytics, notifications. A client-side buffer (an outbox on a phone, a disk spool) holds work locally while the server pushes back. Everywhere else, buffers should be small.

Options and tradeoffs

Slow the producer, buffer, or shed

Strategy	Wins when	Fails when
Backpressure (block/pull)	Producer is a program that can wait: pipelines, stream jobs, batch import	The producer is a human or an external client that cannot be blocked
Durable buffer (log)	Work can be late; bursts are short relative to retention	The caller needs the result now; the backlog outgrows the time it stays useful
Shed (reject/drop)	Request/response traffic; caller can retry, degrade or fail over	Every item must be processed (payments, orders) and there is no buffer behind
Scale out	Load grows over minutes; the bottleneck can scale	The spike lasts seconds; the bottleneck is a database, a partition or a vendor

Most designs use all four at different layers.

TradeoffShed at the edge of the request path, buffer behind it

We gain

Interactive requests get a fast answer either way, and work that can wait is never lost: the log holds it until workers catch up.

We give up

Two behaviours to explain to clients: an immediate refusal, versus “accepted, done later”. A lag to monitor on the async side. And a plan for when the backlog itself becomes too old to be worth draining.

Queue discipline under overload

Discipline	Behaviour under overload	Wins when
FIFO, unbounded	Every request waits the whole backlog; goodput falls to zero	Never, for requests with deadlines
FIFO, bounded, reject	Wait capped; excess refused fast	Simple services with similar deadlines
Deadline-aware FIFO	Hopeless arrivals refused; dead work skipped at dequeue	Deadlines vary, and are propagated with the request
Adaptive LIFO	Most requests fast; the oldest starve and expire	Interactive traffic where a fast answer for most beats fairness
Priority lanes	Low priority absorbs the overload; critical keeps flowing	Mixed traffic with a clear order of importance
CoDel (delay-based)	Bursts pass, standing queues shed early	Drain rate varies; a length bound would be wrong half the time

TradeoffPriority lanes with deadline checks, LIFO only past a depth

We gain

Critical work survives any overload that sheddable work can absorb, nothing is served after its client left, and normal operation stays plain FIFO.

We give up

Every caller must send a priority and a deadline, someone must own the priority table, and low priority can starve completely during a long overload.

Go deeperWhere to shed, and what a refusal costs1 min

Where	Cost of a refusal	Knows	Pitfall
Client (adaptive throttling)	Zero: never sent	Its own recent refusal rate	Needs clients you control
Edge or gateway	Microseconds, before any app work	Priority tag, route, tenant	Does not see the backend’s real load
Service, at admission	Microseconds, before the expensive work	In-flight count, queue delay, its own latency	Must refuse before taking a connection or a lock
Inside the work	Already paid for parsing, auth, maybe a DB call	Everything	The most expensive place to give up

The cost of a refusal decides how much overload a server survives. If refusing costs 1% of serving, a server can refuse about 100 times its capacity. If it costs 50% (a refusal after authentication and a database lookup), it collapses at twice its capacity.

How it fails

Unbounded queues and the out-of-memory kill. Memory grows with the queue. Garbage collection pauses lengthen, which lowers the service rate further. The process dies. The restart drops every queued request, and the clients reconnect at once. Fix: a bound on every queue, including the ones a framework hides. And on every hop, because one unbounded channel between bounded stages absorbs the signal, and grows.

Retry storms refill the queue. Every timeout and every 503 becomes another request. Three layers that each try three times turn one failure into 27 attempts at the bottom. The fixes live on the resilience page: backoff with jitter, a retry budget, one retrying layer, and honoring Retry-After.

Metastable failure. The overload outlives its trigger. A spike pushes the system past capacity. A loop keeps it there, and removing the trigger does not help. Bronson and colleagues named the pattern in “Metastable Failures in Distributed Systems” (HotOS 2021). The resilience page covers the retry loop. The queueing side adds a second loop. The server spends its capacity on requests whose clients left. So goodput stays near zero, while it looks busy. The demo shows both.

The loop that keeps a metastable failure going

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Failure

The way out is to break the loop, not to wait. Shed hard at the edge until goodput recovers. Drop the backlog of dead work. And let traffic back in gradually. Systems that are bounded, deadline-aware and on retry budgets rarely enter the loop.

Shedding the wrong work. A server that sheds at random drops checkouts and health checks as readily as recommendations. Dropping health checks is especially bad. The load balancer marks the overloaded server dead, and moves its traffic to the others. Then they fall over in turn. Fix: assign priorities at the edge, and exempt health checks and control traffic.

Starvation. Priority lanes and LIFO are unfair on purpose: during a long overload the lowest lane may get nothing for an hour. For a background job that must eventually run (billing, compaction), that becomes a data problem days later. Give low lanes a small guaranteed share (weighted fair queuing), or bound how long an item may wait before it is promoted or dead-lettered.

Failure	Blast radius	Detect	Fix
Unbounded queue, OOM	The process, then its restarts	Queue depth and heap rising, throughput flat	Bound every queue; size from the latency budget
Retry storm	The service and its dependencies	Attempts per request above 1.1, retries in traces	Backoff with jitter, retry budgets, one retrying layer
Metastable failure	Outlives the trigger, often hours	Goodput low after the trigger ends	Shed hard, drop dead backlog, readmit gradually
Shedding the wrong work	Critical paths, then the fleet	Shed rate by priority; health check failures	Priorities at the edge, exempt control traffic
Starvation	Low-priority work, silently	Age of the oldest item per lane	Weighted share, max wait, promotion or dead letter
Unbounded hop	Memory of one stage	Per-stage memory under 2× load test	Bound it
Lag measured in messages	Late or missing alerts	Lag in seconds vs target	Alert on time, per consumer group

At scale

Go deeperNumbers to remember1 min

What	Figure
Latency at 80 / 90 / 99% busy (M/M/1)	5×, 10×, 100× the service time
p99 vs mean in M/M/1	About 4.6×
Little’s law	In flight = rate × latency: 2,000/s × 50 ms = 100
CoDel defaults	5 ms target, 100 ms interval
Facebook’s queue timeouts	5 ms when the queue has not emptied in 100 ms, else 100 ms
Kafka producer buffer	32 MB buffer.memory, blocks up to 60 s max.block.ms
Kafka consumer	500 max.poll.records, evicted after 5 min max.poll.interval.ms
HTTP/2 initial flow-control window	65,535 bytes per stream and per connection
Tomcat defaults	200 worker threads, 100 queued connections
Autoscaling reaction	Minutes for VMs, tens of seconds for containers on warm nodes

At 10x, one service’s overload policy becomes a fleet’s. Load balancers must see shedding: a server that answers 503 quickly looks healthy and fast to a least-requests balancer, which sends it more traffic. Use outlier ejection on 503 rates, or have servers report their load. Priority and deadline are assigned once at the edge and propagated, or deep services keep working on requests the edge already abandoned.

At 100x, overload becomes local: one cell, one partition, one tenant. Per-tenant queues and concurrency limits keep one tenant’s spike in that tenant’s lane. Async pipelines need a plan for a backlog too old to be useful: skip to the latest offset for telemetry, drain everything for billing. Load tests at two to three times peak become a release gate, because overload behaviour is the part nobody exercises by accident.

What to monitor:

• Goodput next to throughput: completions within deadline per second.
• Time in queue (p50 and p99) per stage, not only depth.
• Consumer lag in seconds per consumer group, or age of the oldest message.
• Shed and reject rates by priority and reason (queue full, deadline, concurrency limit).
• Retries as a fraction of requests, and the adaptive concurrency limit: a falling limit warns of a slow dependency.
• Utilization of the bottleneck against the 70 to 80% knee, not against 100%.

In the interview

Interviewer asksYour service sits behind a queue. Traffic doubles for ten minutes. What happens, and what would you change?

If the queue is unbounded, it grows at the difference between arrivals and capacity. Once the wait exceeds the client timeout, every response is wasted. Throughput stays at capacity, while goodput falls to zero. Retries then add load, and memory runs out. So I would bound the queue at capacity times acceptable wait, and answer the excess with 503 and Retry-After. I would propagate deadlines, so the server refuses hopeless requests, and skips ones whose client is gone. And I would tag priorities, so checkout survives while recommendations are shed. Autoscaling covers the slow part. A ten minute doubling is mostly over before new capacity is warm.

Interviewer asksWhat is backpressure, and how does Kafka provide it?

Backpressure is a signal from a slow consumer that makes the producer slow down. Without it, the difference piles up in memory. Kafka consumers pull. They poll when ready, and take a bounded batch. So a slow consumer is never overrun, and the backlog sits on disk as lag. On the produce side, brokers throttle clients over their quota by delaying responses. The producer’s buffer fills, and send blocks. The pressure lands on whatever called send. For a web request, that means shedding. I would scale consumers on lag in seconds, up to the partition count.

Interviewer asksThe spike ended an hour ago and the service is still down. Why?

A metastable failure: something keeps it overloaded after the trigger. Usually, retries keep offered load above capacity, while the server spends its time on requests nobody is waiting for. A cold cache after restarts can do the same. Waiting does not fix it. I would shed hard at the edge until goodput recovers, drop the dead backlog, and readmit traffic gradually. To prevent it: bounded queues, deadline checks, retry budgets with jittered backoff, and retries at one layer.

Common mistake

"Add a queue to absorb the load"

A queue absorbs bursts that are short, compared to how long items stay useful. For sustained overload, it only converts the excess into latency, then memory, then an outage. Every queue needs a bound, a policy for when it is full, and a reason its maximum wait is acceptable. A durable log for async work is the exception that proves the rule. It holds the backlog on disk, because the work is still worth doing hours later.

Common mistake

"Autoscaling handles spikes"

Autoscaling reacts in minutes: metric averaging, a decision window, boot time, and warm-up. A spike that arrives in seconds does its damage first. And autoscaling cannot scale a bottleneck that is a database, a hot partition or a vendor’s rate limit. Shedding covers the seconds. Autoscaling covers the minutes.

Mid-level covers

Knows queues need a bound, and returns 503 with Retry-After when full. Uses a message queue to decouple producers from consumers. Mentions consumer lag, and autoscaling on it.

Senior adds

Explains the latency knee and Little’s law with numbers. Separates throughput from goodput. Chooses between blocking, pull, buffering and shedding by layer. Sheds by priority and deadline, and knows when LIFO helps. Names metastable failure and retry storms, with the fix for each. Says why autoscaling cannot save a spike.

Seen in

• Distributed message queue: pull consumers as free backpressure, producer quotas, and lag in time.
• Job scheduler: workers pull only what they have slots for, and a fixed shedding order keeps dispatch alive.
• Notification system: a lane per priority, bulk lanes that pause when the critical lane’s oldest message passes 2 seconds, and marketing shed first.
• Ad click aggregator: click servers spool to local disk when the log is slow instead of blocking the redirect.
• Chat: typing indicators, presence, receipts and push previews are shed in that order, so sends survive. Fan-out lag pages above 5 seconds.
• News feed: fan-out workers autoscale on consumer lag, and degradation sheds recommended posts and ranking before the core timeline read.
• Ticketmaster: a waiting room that admits a fixed rate, admission control applied to people.
• Stock exchange: a per-session throttle rejects excess orders instead of queueing them, because a queue would only add latency.
• Rate limiter: per-client limits cannot protect a backend when every client is within its plan, so services still shed and answer 503.
• Metrics and monitoring: a fixed shedding order under stress, with the query frontend throttled first.
• Video streaming: view counts shed first, then uploads, then play API calls, and video segments never.
• Web crawler: the crawler as the well-behaved client, backing off a host on 429 and 503 and honoring Retry-After.

Recap

The shape. Arrivals faster than service. A queue only moves the excess from refused now to served late, and past the client’s timeout late is worthless.

The numbers. Time in system is S ÷ (1 − ρ): 5× at 80% busy, 10× at 90%, 100× at 99%. Little’s law: in flight = rate × latency. Size queues from the latency budget.

Bound everything. When full, block (in-process), drop newest, drop oldest (fresh data), or reject with 503 and Retry-After (request/response).

Pull and credit. Kafka poll, prefetch, Reactive Streams request(n), TCP and HTTP/2 windows: the consumer grants what it can take. Lag in seconds is the signal.

Admit and shed. Concurrency limits, ideally adaptive. Shed by priority, by deadline (skip work whose client left), and newest-first under overload. Manage queues by delay (CoDel), not length.

Autoscaling follows minutes, not seconds, and cannot scale a database or a hot partition.

Failures. Unbounded queues, retry storms, metastable loops, shedding the wrong work, starvation. Watch goodput, not throughput.

Further reading

• On this site: message queues, for prefetch, acks and logs. Rate limiting, for adaptive concurrency and 429 versus 503. Resilience, for timeouts, retry budgets and breakers. Observability, for percentiles and burn-rate alerts. And hot keys, for overload on one partition.
• Using load shedding to avoid overload, Amazon Builders’ Library: goodput, the cost of a refusal, and prioritizing what to shed.
• Fail at Scale, ACM Queue, 2015: Facebook’s controlled-delay queues and adaptive LIFO in production.
• Handling Overload, Google SRE book: request criticality, client-side throttling and per-customer limits.