Timeouts, retries and circuit breakers

Every service that calls another service takes on that service’s failures. A service that is down is the easy case. The connection is refused in a millisecond, and the caller moves on.

A slow service is more dangerous than a dead one. While a call waits, it holds a thread, a connection and memory. So a few seconds of slowness in one optional service can use up every thread the caller has.

This page covers the patterns that stop that chain reaction:

• Timeouts and deadlines: stop waiting after a set time.
• Retries with budgets: try again, but only a little.
• Circuit breakers: stop calling a service that is clearly broken.
• Bulkheads: give each dependency its own small share of threads.
• Fallbacks: answer with something useful when a call fails.
• Hedged requests: send a second copy when the first is slow.

The networking page covers one single call: connect timeouts, and backoff with jitter, with code. This page builds on it.

The problem it solves

A checkout service handles 1,000 requests a second. It has a pool of 200 worker threads. Its traffic is split evenly over three endpoints, and each endpoint calls one other service (a dependency): pricing, inventory or recommendations. Each dependency answers in about 20 ms.

How many threads does that take? Little’s law answers it. Requests in flight = arrival rate × time each request stays. Each request takes about 22 ms: 20 ms waiting for the dependency, plus 2 ms of checkout’s own work. So 1,000 a second × 0.022 s = 22 threads are busy. The pool of 200 is only 11% used.

Then recommendations gets slow. Maybe a bad deploy, a long garbage collection pause, or a lock in its database. Its calls now take 2 seconds. Here is Little’s law for that one endpoint:

Recommendations latency	Threads its 333 requests a second hold	Pool of 200
20 ms	7	11% used in total
200 ms	67	41% used
550 ms	184	Full: everything queues
2 s	666	Needs 3.3 times the pool
No reply	All of them, until connections die	Every thread waits on one dependency

The pool fills in about half a second. About 180 threads are free, and slow calls arrive at 333 a second. None of them leave.

From then on, a request for pricing has to wait for a thread too, even though pricing is healthy. Users give up, but the queued work does not know that. Next, the gateway sees checkout as slow, and the gateway’s own pool fills. One optional service has taken down the whole checkout page.

One slow dependency drains the caller's pool, then the caller's callers

Scroll or pinch to zoom, drag to pan

• Color key
• Slow or out of threads
• Healthy

Three facts drive everything on this page:

• Slow is worse than down. Most of these patterns turn “slow” into “fails fast”.
• Resources are shared by default. A thread pool, a connection pool, an event loop’s memory: if one dependency can fill it, it fills it for everyone.
• Queues hide overload until it is too late. In a simple queue, the wait is utilization ÷ (1 − utilization) times the normal service time. At 90% busy, that is 9 times. At 99% busy, it is 99 times. So a pool of 200 threads shows almost no wait until it is nearly full. Then it falls off a cliff.

How it works

Step 1: Put a timeout on every call

A timeout puts a ceiling on Little’s law. Say recommendations has a 250 ms timeout. Then the most threads it can ever hold is 333 a second × 0.25 s = 83, however slow it gets. The pool survives. The recommendation calls fail, and the page is served without them.

Many HTTP clients have no timeout at all until you set one, or a very long one. A call waits in several places, and each wait needs its own limit:

• Connect: 100 to 500 ms. This catches a dead host. A healthy host in the same region accepts a connection in about a millisecond.
• Pool acquisition: tens of milliseconds. This is the wait for a free connection from the client’s pool. Waiting 30 s here just moves the queue into the client.
• Request: the limit for one attempt.
• Overall deadline: the limit for the whole request, across all attempts. Step 2 covers it.

Go deeperDefault timeouts in common clients1 min

Client	Default
Java java.net.http.HttpClient	No request timeout
Python requests	No timeout: waits forever
Go http.Client{}	No timeout (Timeout zero)
.NET HttpClient	100 seconds
Node fetch (undici)	300 seconds for headers, 300 seconds between body chunks
HikariCP, waiting for a DB connection	30 seconds

Pick the timeout from real latency numbers, not a round number. First decide how many healthy calls you accept losing to the timeout. Say 0.1%, one call in a thousand. Then use the latency that 99.9% of calls beat. That number is called the p99.9. Measure it at the caller, because that is where the timeout fires.

Say pricing’s p99.9 is 110 ms. A timeout of 150 to 250 ms then fails at most one healthy call in a thousand. Amazon’s Builders’ Library describes the same method.

A timeout that is too long caps nothing. For example, 30 s on a 20 ms call allows 1,500 times the normal thread use. A timeout that is too short turns normal slow calls into errors and retries.

TradeoffTimeout at about the dependency's p99.9, measured where the call is made

We gain

About one healthy call in a thousand times out. A slow dependency can hold at most rate × timeout threads.

We give up

You have to measure the percentile and keep it current. Some dependencies are naturally slow, like a report or a search over a big index. Those need their own longer timeout and their own pool.

Step 2: Propagate a deadline, and split the budget

Timeouts set at each hop do not add up to a promise to the user. Here is how. The gateway waits 1 s. Checkout calls pricing, then inventory, with 300 ms each, and retries each one once. That can take 1,200 ms. So the services below keep working after the gateway has already given up and returned an error.

A deadline fixes this. The edge sets one time budget for the whole request. Every call then passes the time remaining down with it.

• gRPC sends it in the grpc-timeout header, and cancels the server’s work when it runs out.
• In Go, it travels inside context.Context.

Send a duration (“400 ms left”), not a clock time (“until 12:00:01.400”). Clocks on different machines drift apart, and a clock time would stretch or shrink with them.

Each hop then follows three rules:

1. Per-call timeout = the smaller of the call’s own timeout and the time remaining, minus a small reserve. The reserve is for the work this hop still does after the call returns.
2. Do not start work that cannot finish. If 40 ms remain and the query takes 80 ms, fail now. A request picked from a queue after its deadline is dropped without running.
3. Split the budget along the calls. Calls made one after another share the remaining time. Calls made in parallel each get all of it.

A 1 s deadline passed down three hops

Scroll or pinch to zoom, drag to pan

• Color key
• Database
• Service

TradeoffEnd-to-end deadlines, not independent per-hop timeouts

We gain

The latency promise holds across any number of hops. Work for requests nobody waits for any more stops at every layer. That frees capacity exactly when the system is struggling.

We give up

Every service and client library must read the deadline, respect it and pass it on, even through queues and async hops. One hop that ignores it breaks the chain below it.

Step 3: Retry at one layer, within a budget

The networking page shows retries with exponential backoff and full jitter in five languages. Backoff means each retry waits longer than the last. Jitter means the wait is random, so callers don’t all retry at the same moment. Three decisions around that code matter more than the exact backoff curve.

Retry only what is safe. A timeout does not mean the call failed. Maybe the card was charged and only the reply was lost. A retry would charge it again.

• Reads are safe to retry.
• Idempotent writes are safe too. An idempotent write has the same effect no matter how many times it runs, like PUT of a full value or DELETE by ID.
• Any other write needs an idempotency key. The server sees the key again and returns the first result instead of running twice. See the idempotency page.

Also, retry only failures that may go away on their own. Never retry one that will repeat.

Go deeperWhich failures to retry1 min

Failure	Retry?
Connection refused, DNS failure	Yes: the request never reached a server
Timeout, connection reset mid-request	Only if idempotent or keyed: it may have run
503, 429	Yes, after Retry-After if present, and within the budget
500	Rarely: often a bug that fails the same way again
400, 401, 403, 404, 409, 422	No: the same request gets the same answer
“Overloaded, do not retry”	No: a lower layer has already retried or is shedding on purpose

Retry at one layer only. Say each layer in a call chain makes 3 attempts. A request that fails at the bottom then becomes 3 × 3 × 3… calls there, one factor of 3 per layer. And this load lands on the part that is already struggling:

Layers that each make 3 attempts	Calls reaching the failing dependency per user request
1	3
2	9
3	27
5	243

Google’s SRE book gives the rule. Retry only in the layer directly above the one that is failing. Every layer above that fails fast. It passes back a status that means “overloaded, do not retry”.

Limit retries with a budget, not a count. “Up to 3 attempts” limits one request, not the whole fleet. If the dependency is down, it gets 3 times its normal traffic while it is trying to recover.

A retry budget caps retries as a share of all traffic, say 10 to 20% on top of first attempts. When a few calls fail, all of them get retried. When most calls fail, the budget runs out, and the load stays near 1.1 times normal. It works like a token bucket: each success earns a token, and each retry spends one.

Go deeperHow gRPC, Finagle and Envoy budget retries1 min

• gRPC retry throttling (per server name, set in the service config): a token count starts at maxTokens. Each failed call subtracts 1, and each success adds tokenRatio. Retries and hedges stop while the count is at or below half of maxTokens.
• Finagle’s RetryBudget: by default, retries up to 20% of requests over a 10 second window. It adds a floor of 10 retries a second, so a quiet client can still retry.
• Envoy’s retry budget: concurrent retries are capped at a share of active plus pending requests (20% by default), with a floor of 3 concurrent retries.

Step 4: Stop calling: the circuit breaker

When a dependency is clearly down, every call still waits the full timeout. That is 83 threads spent on calls that will fail anyway. And the service that is trying to recover still gets its full load.

A circuit breaker fixes both. It watches the results of real calls to one dependency. When they are bad enough, it stops making those calls for a while. It has three states:

• Closed. Calls go through as normal. Each result (success, failure or slow) goes into a sliding window: the last 100 calls, or the last 10 seconds. The breaker opens when two things are true. The window holds a minimum number of calls, say 20. And the share of failed or slow calls passes a threshold, say 50%.
• Open. Calls fail in microseconds, without touching the network. The caller uses its fallback. After a fixed wait of 5 to 60 seconds, the breaker moves to half-open.
• Half-open. A few trial calls go through. The rest still fail fast. If the trials succeed, the breaker closes with an empty window. If one fails, it opens for another wait.

Circuit breaker states

Scroll or pinch to zoom, drag to pan

• Color key
• Ends well
• State

What counts as a failure? Timeouts, connection errors and 5xx responses. A 4xx response is the caller’s own mistake, so it does not count. Slow calls count too, as a separate rate. A dependency that answers correctly in 3 seconds is still failing, as far as the caller’s threads are concerned.

The breaker below keeps the last 20 results in a ring buffer. It makes a decision only once it has 10 results. It opens at 50% failed or slow, waits 5 seconds, then lets 3 trial calls through. In the test, the dependency stops answering at 5 s and recovers at 15 s. A request arrives every 100 ms.

JMain.javaPycircuit_breaker.pyC#CircuitBreaker.csGocircuit_breaker.goTScircuit-breaker.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.Locale;

public class Main {
    enum State { CLOSED, OPEN, HALF_OPEN }

    /**
     * One breaker per dependency. It watches the outcomes of real calls in a sliding window of the
     * last `window` calls, and stops calling the dependency when too many failed or were slow.
     */
    static final class CircuitBreaker {
        private final int window, minCalls, trialCalls;
        private final double failureRate, slowMs, slowRate, openMs;
        State state = State.CLOSED;
        // A ring buffer of the last `window` outcomes. Counting in a ring keeps memory fixed and
        // lets old failures age out, so a burst an hour ago cannot trip the breaker now.
        private boolean[] failed, slow;
        private int next, count, failures, slows;
        private double openUntil;
        private int trialsStarted, trialsOk;

        CircuitBreaker(int window, int minCalls, double failureRate, double slowMs,
                       double slowRate, double openMs, int trialCalls) {
            this.window = window;
            this.minCalls = minCalls;
            this.failureRate = failureRate;
            this.slowMs = slowMs;
            this.slowRate = slowRate;
            this.openMs = openMs;
            this.trialCalls = trialCalls;
            resetWindow();
        }

        /** Ask before every call. False means: do not call, use the fallback. */
        boolean allow(double nowMs) {
            if (state == State.OPEN) {
                if (nowMs < openUntil) return false; // fail fast: no thread held, no load sent
                // The wait is over: let a few trial calls through to see whether it recovered.
                move(State.HALF_OPEN, nowMs);
                trialsStarted = 0;
                trialsOk = 0;
            }
            if (state == State.HALF_OPEN) {
                // Only trialCalls requests probe. The rest keep failing fast, so a dependency
                // that is still sick sees a trickle, not the full rate at once.
                if (trialsStarted >= trialCalls) return false;
                trialsStarted++;
            }
            return true;
        }

        /** Report every call that allow() let through. A timeout is a failure. */
        void record(double nowMs, boolean ok, double durationMs) {
            boolean isSlow = durationMs >= slowMs;
            if (state == State.HALF_OPEN) {
                if (!ok || isSlow) {
                    trip(nowMs); // one bad trial is enough: back to open for another wait
                } else if (++trialsOk == trialCalls) {
                    resetWindow();
                    move(State.CLOSED, nowMs);
                }
                return;
            }
            // CLOSED: overwrite the oldest outcome and keep running totals, O(1) per call.
            if (count == window) {
                if (failed[next]) failures--;
                if (slow[next]) slows--;
            } else {
                count++;
            }
            failed[next] = !ok;
            slow[next] = isSlow;
            if (!ok) failures++;
            if (isSlow) slows++;
            next = (next + 1) % window;
            // Decide only with enough calls: 1 failure out of 1 call is not a 100% failure rate.
            if (count >= minCalls
                    && ((double) failures / count >= failureRate || (double) slows / count >= slowRate)) {
                trip(nowMs);
            }
        }

        private void trip(double nowMs) {
            openUntil = nowMs + openMs;
            resetWindow(); // start fresh after recovery, not with the outage's failures
            move(State.OPEN, nowMs);
        }

        private void resetWindow() {
            failed = new boolean[window];
            slow = new boolean[window];
            next = count = failures = slows = 0;
        }

        private void move(State to, double nowMs) {
            System.out.printf(Locale.ROOT, "t=%5.2fs  %s -> %s%n", nowMs / 1000, state, to);
            state = to;
        }
    }

    // Healthy until 5 s, then no reply (our 250 ms timeout fires) until 15 s, then healthy.
    // While healthy, 1 call in 20 fails anyway: normal noise must not trip the breaker.
    static boolean dependencyOk(double nowMs, int callNo) {
        if (nowMs >= 5000 && nowMs < 15000) return false;
        return callNo % 20 != 0;
    }

    static double dependencyMs(double nowMs) {
        return nowMs >= 5000 && nowMs < 15000 ? 250.0 : 20.0;
    }

    public static void main(String[] args) {
        CircuitBreaker breaker = new CircuitBreaker(20, 10, 0.5, 500, 0.5, 5000, 3);
        int sent = 0, fallbacks = 0;
        for (int i = 0; i < 300; i++) { // one request every 100 ms for 30 s
            double now = i * 100.0;
            if (!breaker.allow(now)) {
                fallbacks++; // serve the cached or default answer instead
                continue;
            }
            sent++;
            boolean ok = dependencyOk(now, i);
            double ms = dependencyMs(now);
            breaker.record(now + ms, ok, ms);
        }
        System.out.printf("calls sent %d, answered by fallback %d%n", sent, fallbacks);
    }
}

Here is what happens when it runs:

• At 6.05 s, failures reach half the window, and the breaker opens.
• At 11.1 s, the trial call fails, so the breaker stays open for another 5 seconds.
• From 16.4 s, three trials succeed, and the breaker closes.

Over 30 seconds, 104 requests got the fallback in microseconds. Without the breaker, each would have held a thread for 250 ms. And the normal noise before the outage, 1 failure in 20, never tripped it.

Keep one breaker per dependency. Keep one per operation too, when operations fail on their own: a slow search should not stop getById. But do not keep one per host. One bad instance out of twenty is the load balancer’s job. Its outlier detection removes that one host and keeps calling the others.

TradeoffA breaker per dependency, on failure rate and slow-call rate over a window

We gain

A failing dependency costs microseconds per call instead of a full timeout. It gets a quiet period to recover, and the fallback runs at once.

We give up

While the breaker is open, every call fails, even ones that would have worked in a partial outage. A badly tuned breaker flaps: it opens and closes over and over.

Step 5: Contain the damage: bulkheads

Timeouts and breakers react to failures. They need some failures before they help. Until then, the slow dependency still competes for the shared pool.

A bulkhead works from the first second. The name comes from the walls that divide a ship’s hull into compartments. If one compartment floods, the ship still floats. A bulkhead gives each dependency its own limited share of the caller’s resources, so it can only use up its own share.

• A pool or semaphore per dependency. A semaphore is a counter of free slots. Say recommendations may hold at most 60 threads or 60 calls at once. When its bulkhead is full, the next call is rejected at once and goes to the fallback. It does not queue. Size the connection pool for that dependency to match.
• Per tenant or priority. Give big customers their own pools, or split interactive and batch traffic. Then one tenant’s export cannot starve another tenant’s page loads.
• Per deployment. Run separate instances per caller. Shuffle sharding gives each tenant a random small set of instances. With 8 instances and 2 per tenant, there are 28 possible pairs. So if one tenant crashes its two instances, any other given tenant shares both of them only 1 time in 28.

Size a bulkhead with Little’s law: rate × normal p99 latency, plus headroom. Pricing gets 333 calls a second with a 75 ms p99, so it needs 25 slots. A bulkhead of 50 to 70 leaves room for spikes. A bulkhead of 200 in a pool of 200 is no bulkhead at all.

Shared pool versus bulkheads

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Out of threads
• Healthy

Non-blocking code does not hold a thread per call. But each call still holds a connection, a buffer and a pending result. A slow dependency can pile up tens of thousands of them, until memory runs out. In that kind of code, the bulkhead is a concurrency limit: a cap on the calls in flight to each dependency.

TradeoffBulkheads per dependency, sized from rate × p99

We gain

Isolation from the first second, before any breaker has evidence. A slow dependency fills its own compartment and nothing else.

We give up

Capacity is split up, so total use is lower. One dependency’s real spike gets rejected even when other compartments are idle. And every new dependency needs a size.

Step 6: Answer anyway: fallbacks and graceful degradation

An open breaker, a full bulkhead and a timeout all end with the same question. What does the caller return instead? Usually something better than an error. That answer is called the fallback, and choosing it is a product decision.

Fallback	Example	Works when
Cached or stale data	Last known price list, HTTP stale-if-error	Data that changes slowly; staleness is harmless
A static default	Bestsellers instead of personal recommendations	Personalization is a nice-to-have
Leave the feature out	Checkout renders without the “you may also like” row	The feature is optional on this page
A cheaper path	Chronological feed when the ranker times out	A simpler algorithm gives a usable answer
Accept and do it later	Queue the review, show “processing”	A write that need not be synchronous
Fail fast with a clear error	503 with Retry-After, “payments are unavailable”	Correctness depends on the dependency

Some answers must never be faked from a cache, like a payment or a seat booking. So sort every dependency into one of two groups:

• Optional, like recommendations, reviews and badges. These get tight timeouts, breakers, and a static fallback or nothing at all.
• Critical, like payment, or inventory at checkout. These get retries within a budget, a second provider to fail over to where one exists, and an honest error.

Degrade in a planned order. Under overload, drop the least important work first:

1. Prefetching and personalization.
2. Live pages, in favor of cached ones.
3. Batch traffic.
4. The core action, last of all.

Ticketmaster writes its own order down. Dropping low-priority work inside one server is covered under rate limiting and backpressure.

A good fallback follows three rules:

• It is cheaper than the call it replaces.
• It does not share fate with that call. It must not depend on the thing that just failed.
• It is exercised often. A path that runs only during incidents is the one most likely to be broken during one.

Step 7: Cut the tail: hedged requests

Tail latency means the slowest few requests, like the slowest 1%. It gets worse as one request fans out to more backends.

Say a request calls 100 backends in parallel. Each backend is slower than its p99 one time in a hundred. The request has to wait for the slowest one. So it hits at least one slow backend 1 − 0.99¹⁰⁰ = 63% of the time. The service’s median is now worse than its backends’ p99.

A hedged request fixes this. If the first copy has not answered by about the p95, send a second copy to another replica. Use whichever reply comes first. Only about 5% of requests get a second copy.

In “The Tail at Scale” (2013), Dean and Barroso hedged after 10 ms. That cut a Google benchmark’s p99.9 from 1,800 ms to 74 ms, for only 2% more requests. A variant called tied requests sends both copies at once. Each copy names the other server, and whichever server starts first cancels the other.

The cost arrives at the worst time. When a dependency is slow because it is overloaded, every request crosses the p95 and gets hedged. That doubles the load that made it slow. So hedging needs the same guards as retries: idempotent reads only, and a cap of 5 to 10% of requests. Hedging fixes one slow replica among fast ones. It cannot fix a dependency that is slow everywhere.

Step 8: Health checks and breakers answer different questions

Four mechanisms all decide “stop sending traffic there”: health checks, outlier detection, circuit breakers and load shedding. They use different signals, and they act at different scopes.

Go deeperHealth checks, outlier detection, breakers and shedding compared1 min

Mechanism	Who decides	Signal	Scope	Action
Active health check	Load balancer	A probe (GET /healthz) every 5 to 30 s	One instance	Remove the instance from rotation
Outlier detection	Load balancer or client	Errors on real traffic to one host	One instance	Eject the host for a while
Circuit breaker	The calling service	Failure and slow rate of its own calls	A whole dependency or operation	Stop calling it, use the fallback
Load shedding	The called service	Its own in-flight work and queue time	Itself	Refuse some requests with 503

A health check asks “is this instance alive?”. A breaker asks “is this dependency useful to me right now?”. These are not the same. A dependency can pass every health check while every real call times out, because its database is locked. Only the breaker sees that.

Keep health checks shallow: check the instance itself, not its database. A deep check that also tests the database fails on every instance at once when the database blips. The balancer then removes the whole fleet. The load balancing page tells that story.

Try it

This is the same checkout service. Pricing (A) and inventory (B) stay healthy at 20 ms. You set how slow recommendations (C) is, and which protections the caller uses. While slow, C can serve at most 400 calls a second. Each square is one thread.

The model shows the steady state. It uses Little’s law for threads, queueing for the shared pool, and the extra load that retries of C’s failures cause. A request fails if the user waited over 1 s.

Recommendations (dependency C) latency

20 ms, healthy300 ms1 s5 sNo reply

Protections in the checkout service

Timeout 250 msRetries, 3 attemptsRetry budget 10%Circuit breakerBulkheads 70/70/60

Shared pool: 200 threads

A pricingB inventoryC recommendationsIdle

A and B succeed

-

A and B p99

-

Threads busy

-

C answers: real / fallback

-

Calls to C a second

-

Breaker on C

-

Things to try:

• 1 s, no protections. C’s endpoint needs about 334 threads, but the pool has 200. A and B are healthy, yet they succeed only 57% of the time. That is the cascade.
• Turn on the timeout. C can hold at most 83 threads, about 75 in practice. A and B are back to 100%, with a 76 ms p99. C succeeds only 22% of the time. The timeout turned “slow” into “failed”, and that is the point.
• Add retries. Up to three attempts push about 740 calls a second at C, far past its 400. The pool fills again, and A and B drop to 83%. Now add the retry budget. C sees about 367 a second, and A and B recover.
• Turn on the circuit breaker. It opens, and C’s endpoint answers from the fallback. The pool is almost empty. C gets a few trial calls instead of hundreds.
• Reset, then use bulkheads alone at “No reply”. C’s 60 threads are stuck. A and B never notice.
• 300 ms with timeout and retries, then the breaker without the budget. The breaker flaps. Most of C’s failures come from the retries’ own load. So C recovers while the breaker is open, the breaker closes, and the retries push C over again. The budget removes the cause.

Options and tradeoffs

Retries

Policy	Load on a dead dependency	Recovers blips	Wins when
No retries	1×	No	Non-idempotent calls without a key; a layer that is not the retrying one
Fixed attempts (3), every layer	3ⁿ for n layers	Yes	Never in a deep call graph
Fixed attempts (3), one layer	3×	Yes	Small systems with low traffic
One layer, attempts + budget	About 1.1 to 1.2×	Yes, while few calls fail	Almost always

TradeoffOne retrying layer with a 10 to 20% budget

We gain

Retries are generous when failures are rare, and switch themselves off when failures are common. Blips still disappear, and a real outage adds at most a fifth to the load.

We give up

In a partial outage, some requests that a retry would have saved fail instead. Every other layer must fail fast. And small clients see little traffic, so budgets need a floor of a few retries a second, which adds some load back.

Circuit breaker window

Window	Trips when	Wins when	Watch out for
Consecutive failures (say 5)	N failures in a row	Low-traffic dependencies; simple	Intermittent failures (every other call) never trip it
Count-based sliding window	Rate over the last N calls passes a threshold	Steady traffic	At low traffic, the window spans minutes of history
Time-based sliding window	Rate over the last T seconds, with a minimum count	Traffic that varies a lot over the day	Needs a minimum count, or one failure at night trips it

Go deeperWhat the common libraries default to1 min

• Hystrix, Netflix’s original library, has been in maintenance mode since 2018. It opened at 50% errors, with at least 20 requests in a 10 second window. It tried again after 5 seconds, and timed calls out at 1 second.
• Resilience4j is its usual successor on the JVM. It defaults to 50% failures over the last 100 calls, 60 seconds open, and 10 trial calls. Its slow-call rule trips only when every call takes over 60 seconds. So in practice that rule is off until you tune it.
• Polly offers the same breaker in .NET.
• Envoy’s “circuit breakers” are really concurrency limits per upstream cluster: 1,024 connections, pending requests and requests by default. That makes them a bulkhead. Envoy’s outlier detection does the per-host ejection.

TradeoffA rate over a count window, with a minimum count and a slow-call rule

We gain

It ignores normal noise and catches both erroring and hanging dependencies. At normal traffic, it reacts within a second or two.

We give up

Four settings per dependency. And at low traffic, the minimum count means the breaker may never gather enough evidence to open.

Isolation

Semaphores and connection pools are the everyday bulkheads. Per-tenant pools, shuffle shards and cells isolate at larger scopes.

Go deeperBulkhead variants compared1 min

Bulkhead	Isolates	Cost	Wins when
Thread pool per dependency	Threads; a timeout can abandon the thread	A context switch per call, idle threads	Blocking clients that cannot be interrupted
Semaphore / concurrency limit	Concurrent calls	Nearly nothing	Non-blocking clients; most new code
Connection pool per dependency	Connections	Already there; just size it	Always, alongside one of the above
Per-tenant pools or shuffle shards	One tenant from the others	Capacity partitioned by tenant	Multi-tenant services with noisy tenants
Separate deployments (cells)	Everything, including bad deploys	Duplicated infrastructure	Large fleets where blast radius is the priority

TradeoffSemaphores per dependency, connection pools sized to match

We gain

Isolation at almost no runtime cost. The same Little’s law sizing works for both limits.

We give up

A semaphore cannot free a thread that is stuck. So it only works when every call underneath already has a timeout.

How it fails

Retry storms. Every layer retries, so the failing dependency gets 3ⁿ times the traffic. It cannot recover while the storm lasts. You can spot it two ways. Calls to the dependency rise while user traffic stays flat. And the retry ratio (retries ÷ first attempts) climbs past 10%.

Recovery in lockstep, and flapping. A one-second stall makes every request in flight time out and retry at the same instant. In the same way, 500 callers’ breakers open in the same second, and all try again together 30 seconds later. Add randomness to the open time, like 30 s ± 20%, so they spread out.

A breaker flaps (opens and closes over and over) for three reasons. Its window is tiny. Its threshold sits close to the normal error rate. Or its failures come from the caller’s own retry load, which disappears while the breaker is open.

A fallback that hammers another service. Say the cache cluster fails, and the fallback reads the database for every request. The database was sized for a 5% miss rate, and now it gets 100%. A fallback has to be cheaper than the primary and independent of it: a static list, a local copy, or a feature left out. Amazon’s Builders’ Library goes further. It avoids most fallback logic, because fallbacks rarely run and can make an outage worse.

Metastable failures. A trigger causes timeouts: a deploy, a cache flush, a 20 second network blip. The retries keep the load high, so more requests time out and retry. Then the trigger goes away, but the overload stays. The extra work it caused keeps it going.

Bronson and colleagues named this a metastable failure (HotOS 2021). The system is stable when healthy, and also stable when broken. Waiting does not fix it. To get out, cut load below normal: turn off retries, drop work past its deadline, and let in only part of the traffic. Then raise traffic back slowly.

A metastable failure keeps itself going after the trigger is gone

Scroll or pinch to zoom, drag to pan

• Color key
• Part of the loop
• The way out

Failure	Blast radius	Detect	Fix
Pool exhaustion (the cascade)	Every endpoint of the caller, then its callers	Pool usage near 100%, queue wait up, healthy endpoints slow	Timeouts, bulkheads, breakers
Retry storm	The failing dependency, kept down	Calls per user request rising, retry ratio over 10%	One retrying layer, budgets, “do not retry” status
Timeouts out of order	Wasted work at every inner layer	Inner spans finishing after the outer one gave up	Deadline propagation
Synchronized recovery	The recovering dependency	Traffic spikes at fixed intervals after an incident	Jitter backoff and breaker open time; limited trial calls
Breaker flapping	Users alternating between real and fallback	State changes per minute	Minimum count, a gap between open and close thresholds, growing open time, retry budget
Fallback overload	The fallback’s own dependency	Fallback dependency load jumps when a breaker opens	Static or local fallbacks; load-test the fallback path
Metastable failure	The whole service, after the trigger is gone	Overload persisting with normal user traffic	Shed below normal, drop expired work, ramp traffic back

At scale

Numbers to remember.

What	Typical figure
Little’s law	Threads held = calls a second × seconds each: 333 × 2 s = 666
Threads a dependency can hold with a timeout	Rate × timeout: 333 × 0.25 s = 83
Request timeout	About the dependency’s p99.9, measured at the caller
Retry amplification	3ⁿ calls for n layers of 3 attempts: 27 at 3 layers, 243 at 5
Retry budget	10 to 20% of requests, plus a floor of a few retries a second
Breaker	50% failed or slow, 20 to 100 calls minimum, open 5 to 60 s
Fan-out tail	100 backends at 1% slow each: 63% of requests wait for a slow one

At 10 times the size, writing this by hand in each service stops working. One team forgets the timeout, and another retries at every layer. So the patterns move into a shared client library, or into a service mesh like Envoy, Linkerd or Istio. A mesh runs a small proxy next to each service (a sidecar). The sidecars apply timeouts, retry budgets, outlier detection and concurrency limits from configuration.

At 100 times the size, the main worry is blast radius: how much one failure can break. Services split into cells. A cell is an independent copy of the whole stack that serves a slice of customers. Fault injection tests the fallbacks all the time. And adaptive concurrency limits, which adjust themselves, replace hand-set bulkhead sizes.

What to monitor for each dependency (see observability):

• p99 and p99.9 latency as the caller sees it, and the timeout rate compared with the 0.1% the timeout was set for.
• The retry ratio, and how often the retry budget runs out.
• Breaker state changes. More than a few an hour means flapping.
• How full each bulkhead and pool is, and how many calls it rejects.
• The fallback rate. When it rises quietly, the product is degraded, and no error chart shows it.
• Deadline-exceeded counts per hop. Also requests in flight and queue wait inside the service itself. These are the earliest signs of the cascade.

In the interview

Interviewer asksYour service calls five dependencies. One of them gets slow and your whole service goes down, even the endpoints that don't use it. Why, and what do you change?

Every endpoint shares one thread pool, and the slow dependency holds a thread for every call. By Little’s law, 333 calls a second at 2 s each need 666 threads. So a pool of 200 fills in half a second, and the healthy endpoints wait behind it. I would add four things. A timeout near the dependency’s p99.9, which caps it at rate × timeout threads. A bulkhead per dependency. A circuit breaker on failure and slow-call rate, so calls fail in microseconds once it is clearly broken. And a fallback for each optional dependency. Then I would check that only one layer retries, within a budget.

Interviewer asksHow do you choose the timeout for a call?

From the dependency’s latency, measured at the caller. I accept about 0.1% false timeouts, so I use the p99.9 plus some headroom. For a p99.9 of 110 ms, that is 150 to 250 ms. Each call then uses the smaller of that and the request’s remaining deadline, minus a reserve for its own work.

Interviewer asksWhere should retries happen, and how do you stop them from making an outage worse?

At one layer only: the one directly above the failing dependency. And only for idempotent operations or keyed writes. Three attempts at each of five layers is 243 calls at the bottom. Other layers fail fast with a “do not retry” status. The retrying layer has a budget of 10 to 20% of its calls. So a full outage adds a tenth to the load instead of tripling it. And it never retries past the deadline.

Interviewer asksThe circuit breaker for recommendations is open. What does the user see, and what happens to the recommendations service?

The page shows a cached bestseller list, or leaves the row out. Recommendations gets almost no traffic from us for about 30 seconds, with some randomness. Then a few trial calls decide whether the breaker closes. I would alert on the fallback rate, because the page looks fine while it is degraded. For payment there is no fake answer. We fail over to a second provider, or else show a clear error.

Common mistake

"Add retries" as the fix for an unreliable dependency

Retries help when failures are rare and unrelated, like a dropped connection or a restarting instance. But when a dependency fails because it is overloaded, retries multiply its load. Three attempts triple it, at every layer that retries. They also keep it down after the cause is gone. That is how a short blip turns into an hour-long metastable outage. Retries need a budget, one retrying layer, idempotency and a deadline. Without those, they are part of the problem.

Common mistake

Treating the circuit breaker as the first line of defense

A breaker needs a window of evidence before it opens. Until then, every slow call holds a thread. Without timeouts, the window fills slowly. Without bulkheads, the pool is gone before the breaker decides. Timeouts and bulkheads protect from the first second. The breaker makes that protection cheap once the failure is clear. A breaker is not a load balancer either. One bad host is removed by outlier detection, not by cutting off the whole dependency.

Mid-level covers

Puts a timeout on every remote call. Retries idempotent calls with exponential backoff and jitter. Adds a circuit breaker with closed, open and half-open states in front of a flaky dependency, and returns a cached or default answer while it is open.

Senior adds

Explains the cascade with Little’s law. Sets timeouts from the p99.9 under a deadline that is passed down. Retries at one layer, within a budget, and only with idempotency. Sizes bulkheads from rate × p99. Tunes breakers on failure and slow-call rates. Sorts dependencies into critical and optional, with fallbacks that share no fate with the primary. Knows how to shed load to escape a retry storm or a metastable failure.

Seen in

• Distributed cache: a 20 ms deadline per call. A timeout returns a miss, and gets are never retried, so a slow node is not made slower.
• Rate limiter: a 5 ms timeout per Redis call, and a breaker per shard that opens for 2 seconds after 5 failures. While it is open, a local fallback limit applies.
• Payment system: a breaker per payment provider. New payments go to the other provider.
• Notification system: retries with jitter, and a breaker per provider. It fails over when errors pass 20% over 30 seconds.
• News feed: a 100 ms deadline on the ranker. The fallback is a cached ranked page, or the feed in time order.
• Proximity service: a 30 ms timeout on the business service, falling back to the summary cache.
• Ticketmaster, hotel reservation and file sync: a written order of what to drop first, so the core action keeps working longest.
• Web crawler: 30 second fetch timeouts and capped bodies, so a slow or hostile site cannot hold a connection forever.

Recap

The problem. A slow dependency holds rate × latency of the caller’s threads (Little’s law). 333 calls a second at 2 s is 666 threads. So a shared pool of 200 fills in half a second. Healthy endpoints wait behind it, and the caller’s callers fill up next.

Timeouts and deadlines. A timeout near the dependency’s p99.9 caps it at rate × timeout threads. The edge sets a deadline, and each hop passes on the time remaining.

Retries. Only for idempotent or keyed calls. Only at one layer, with backoff and jitter, and a budget of 10 to 20%. Layers that each retry 3 times multiply to 3ⁿ.

Circuit breakers. Open on the failure or slow-call rate over a window, with a minimum count. Fail fast while open. Test recovery with a few trial calls.

Bulkheads. A pool or concurrency limit per dependency, sized from rate × p99, that rejects calls when full. They isolate from the first second.

Fallbacks and hedging. Stale data, a default, a feature left out, or an honest error for critical dependencies. A fallback never shares fate with the primary. Hedging after the p95 cuts the tail for about 5% more load.

Failure modes. Retry storms, recovery in lockstep, flapping breakers, fallbacks that overload something else, and metastable failures. Those last ones only end when load is cut below normal.

Further reading

• On this site: networking for backoff and jitter on one call, idempotency for safe retries, backpressure for overload, rate limiting for adaptive concurrency limits, and load balancing for health checks and outlier detection.
• Timeouts, retries, and backoff with jitter, Amazon Builders’ Library: setting timeouts from latency percentiles and retrying at one layer.
• Addressing Cascading Failures, Google’s Site Reliability Engineering book: resource exhaustion, retries, deadlines and how to recover.
• Metastable Failures in Distributed Systems, Bronson, Aghayev, Charapko and Zhu, HotOS 2021: why some overloads outlive their trigger.