Networking for system design

Networking for system design interviews. Round trip numbers, DNS, TCP and TLS handshakes, HTTP/1.1 to HTTP/3, connection reuse, timeouts and retries.

Foundation33 min core · 37 min full depth

Every request in a design crosses a network. The network sets a floor on latency that no server tuning removes. The unit of that floor is the round trip: one message out, and one back.

A request costs a number of round trips: find the address, open a connection, secure it, then ask. Each round trip costs a time set by distance. Multiply the two, and you have most of a latency budget before your code even runs. This page covers the DNS, TCP, TLS and HTTP you need to do that arithmetic, and to change it.

The problem it solves

An app’s servers run in Virginia, and its API answers in 20 ms of server time. A user in Sydney opens the app and waits about a second for the first byte. Nobody’s code is slow:

Step	Round trips	Sydney to Virginia, about 230 ms each
DNS lookup (resolver cache)	none	10 ms
TCP handshake	1	230 ms
TLS 1.2 handshake	2	460 ms
HTTP request and response	1	230 ms + 20 ms server
Time to first byte	4	about 950 ms

The server’s 20 ms is 2% of the wait. The rest is four crossings of the Pacific and North America. Every fix is a networking choice:

Fewer round trips per request: TLS 1.3, HTTP/3, reused connections.
Shorter round trips: an edge or a region nearer the user.
Not paying setup costs more often than needed: pooling, keepalives, sane retries.

Inside a data center, the same arithmetic runs with smaller numbers. 30 sequential calls to a database 1 ms away spend 30 ms on the network.

How it works

Step 1: Round trips, and what distance costs

Light in optical fiber travels at about 200,000 km a second, two thirds of its speed in vacuum. So every 100 km of fiber adds about 1 ms of round trip (100 km out and back at 0.5 ms each way). Cables do not follow great circles and routers queue packets, so measured round trips are usually 1.5 to 2 times the physical minimum.

Path	Great-circle distance	Physics minimum RTT	Typical RTT
Same availability zone	under 1 km	under 0.01 ms	0.1 to 0.5 ms
Different zones, same region	10 to 100 km	0.1 to 1 ms	0.5 to 2 ms
Nearby cities (Frankfurt to Paris)	480 km	5 ms	8 to 12 ms
US coast to coast (Virginia to Oregon)	3,700 km	37 ms	60 to 75 ms
Across the Atlantic (New York to London)	5,570 km	56 ms	70 to 80 ms
US West to Japan (Oregon to Tokyo)	7,800 km	78 ms	100 to 120 ms
Far side of the world (London to Sydney)	17,000 km	170 ms	250 to 280 ms

Users add the last mile. That’s 5 to 20 ms on broadband or Wi-Fi, 30 to 60 ms on 4G, and 10 to 30 ms on good 5G. Through a geostationary satellite, it’s about 600 ms.

Two consequences shape every design on this site:

Within a region, the network is nearly free. 50 sequential sub-millisecond calls cost less than one ocean crossing. That’s why a cache lookup or a Redis rate limit check fits on the hot path.
Across regions, the network dominates. A synchronous call from Europe to the US costs 80 ms, whatever the server does. So cross-region calls stay off the request path.

Step 2: Find the address: DNS

Before connecting, the client turns api.example.com into an IP address. DNS is a hierarchy of caches in front of a hierarchy of servers:

A DNS lookup, cold at every level

Scroll or pinch to zoom, drag to pan

Color key
Service
Cache

Busy resolvers keep the root and TLD answers for days. So a “cold” lookup is usually one trip from the resolver to your authoritative server. An answer from the device’s own cache costs nothing. From the resolver’s cache, it costs 5 to 30 ms. From the authoritative server, it costs 30 to 150 ms more. A lost UDP packet means a retry, after a timeout of a second or more.

Every answer carries a TTL, the seconds a cache may keep it, and it is the one knob you control. Long TTLs (hours) mean fewer lookups but slow changes. Short TTLs (20 to 60 s) let you move traffic quickly, for failover between regions or DNS-based balancing, at the price of more lookups. A TTL is a request, not a guarantee, and a connection that is already open never looks at DNS again. That causes more incidents than any resolver: see How it fails.

Go deeperDNS details that show up in incidents2 min

Lowering a TTL only works ahead of time. Say a record had a TTL of 86,400 s, and you lower it to 60 s just before a migration. Resolvers that cached it yesterday keep the old answer for up to a day. So lower it at least one old TTL before the change. Then migrate, and raise it again.

Negative answers are cached too. A lookup for a name that does not exist returns NXDOMAIN. Resolvers cache that “no” for the time in the zone’s SOA record (RFC 2308), often 5 minutes to an hour. So querying a hostname before you create it can make it look broken afterwards.

Runtimes cache on their own. The JVM caches successful lookups for 30 seconds by default (networkaddress.cache.ttl), and forever when a security manager is installed. Some client libraries resolve once per pool and never again.

DNS runs over UDP port 53 by default. DNS over HTTPS and over TLS encrypt the lookup. The cost is keeping a connection open to the resolver.

Step 3: Open and secure a connection: TCP and TLS

TCP turns a network that drops and reorders packets into an ordered, reliable byte stream. It opens with a three-way handshake: SYN, SYN-ACK, then ACK with the first data right behind it. One round trip before any data.

TLS then proves the server’s identity with its certificate chain and agrees on keys. The version sets how many round trips that takes:

TLS 1.2: two round trips for a full handshake, one when resuming an earlier session.
TLS 1.3: one round trip (RFC 8446). The client sends its key share in the first message, so the server can finish in its reply.
TLS 1.3 0-RTT. A client that has talked to this server before holds a pre-shared key from a session ticket. It can send the request as early data, in its first flight. There’s no TLS round trip at all, though TCP’s handshake still comes first.

A cold HTTPS request with TLS 1.3 takes three round trips to the first byte

Scroll or pinch to zoom, drag to pan

Color key
Service

0-RTT has a catch: early data can be replayed. Someone who records the first flight can send it again, and the server cannot tell the copy from the original. So it is only safe for requests that do no harm when repeated. Servers commonly accept early data only for idempotent requests, such as GET. They pass an Early-Data: 1 header (RFC 8470) to the application. Then the application can answer 425 Too Early, and make the client resend after the handshake.

Step 4: The whole budget

Put together, a request’s time to first byte (TTFB) is:

TTFB = DNS + (handshake round trips + 1) × RTT + server time

The handshake count is 3 for TCP plus TLS 1.2, and 2 for TCP plus TLS 1.3. It’s 1 for HTTP/3, because QUIC merges the two (Step 6). And it’s 0 on a connection that is already open. A large body then needs more round trips (Step 7).

LatencyBudget.javasetup.pyLatencyBudget.cssetup.gosetup.ts

// How a new connection is set up, and the round trips it costs before the request goes.
enum Setup {
    TCP_TLS12(3), // TCP 1 + TLS 1.2 full handshake 2
    TCP_TLS13(2), // TCP 1 + TLS 1.3 1
    TCP_TLS13_0RTT(1), // TCP 1; the request rides in the resumed TLS hello (early data)
    QUIC(1), // QUIC runs transport and TLS 1.3 in one handshake
    QUIC_0RTT(0); // resumed QUIC: the request goes in the first packet

    final int roundTrips;
    Setup(int roundTrips) { this.roundTrips = roundTrips; }
}

final class LatencyBudget {
    static final int INIT_WINDOW_BYTES = 14_600; // 10 segments of 1,460 bytes, Linux's default since RFC 6928

    // A fresh TCP (or QUIC) connection may send only the initial window, then doubles it each
    // round trip it gets acknowledged. Bandwidth does not change this count, distance makes
    // each round longer.
    static int slowStartRounds(long sizeBytes) {
        int rounds = 1;
        long window = INIT_WINDOW_BYTES, sent = INIT_WINDOW_BYTES;
        while (sent < sizeBytes) {
            window *= 2;
            sent += window;
            rounds++;
        }
        return rounds;
    }

    static double timeToFirstByte(double rttMs, double dnsMs, Setup setup, boolean reused, double serverMs) {
        // A reused connection skips DNS and every handshake: that is the whole case for pooling.
        double before = reused ? 0 : dnsMs + setup.roundTrips * rttMs;
        return before + rttMs + serverMs; // the request goes out, the first byte comes back
    }

    static double responseComplete(double rttMs, double dnsMs, Setup setup, boolean reused, double serverMs,
                                   long sizeBytes, double mbps) {
        // A warm, reused connection has already grown its window; a new one pays slow start.
        int extraRounds = reused ? 0 : slowStartRounds(sizeBytes) - 1;
        double transmitMs = sizeBytes * 8 / (mbps * 1000);
        return timeToFirstByte(rttMs, dnsMs, setup, reused, serverMs) + extraRounds * rttMs + transmitMs;
    }
}

class Main {
    public static void main(String[] args) {
        String[] names = {"same zone", "New York-London", "London-Sydney"};
        double[] rtts = {0.5, 75, 260};
        for (int i = 0; i < names.length; i++) {
            double rtt = rtts[i];
            double tls12 = LatencyBudget.timeToFirstByte(rtt, 10, Setup.TCP_TLS12, false, 20);
            double tls13 = LatencyBudget.timeToFirstByte(rtt, 10, Setup.TCP_TLS13, false, 20);
            double h3 = LatencyBudget.timeToFirstByte(rtt, 10, Setup.QUIC, false, 20);
            double warm = LatencyBudget.timeToFirstByte(rtt, 10, Setup.TCP_TLS13, true, 20);
            double big = LatencyBudget.responseComplete(rtt, 10, Setup.TCP_TLS13, false, 20, 1_000_000, 100);
            System.out.printf("%-16s TLS 1.2 %6.0f | TLS 1.3 %6.0f | HTTP/3 %6.0f | reused %5.0f | 1 MB new %6.0f ms%n",
                    names[i], tls12, tls13, h3, warm, big);
        }
        System.out.println("1 MB needs " + LatencyBudget.slowStartRounds(1_000_000) + " rounds of slow start"); // 7
    }
}

The table it prints is this page in five columns. In one zone the protocol barely matters: 32 ms on a new connection against about 20 ms reused, mostly DNS and server time. From London to Sydney the same choices are the difference between 1.07 seconds and 280 ms. Try it:

Distance to the server

CDN

Connection

DNS

HTTP

TLS

Response

Time to first byte: 0 ms
Round trips before it: 0
Response complete: 0 ms
All 31 requests done: 0 ms

First request

Then 30 more small requests

A model, not a measurement: whole round trips, 20 ms of origin server time, 2 ms at a CDN edge 10 ms from the user, 100 Mbps of bandwidth, and an initial TCP window of 10 packets (about 14.6 KB). A reused connection is assumed warm, its window already grown.

Things to try:

New York to London: TLS 1.2, then 1.3, then HTTP/3. Time to first byte goes 330, 255, then 180 ms. That’s one round trip saved per click.
Click Reused. 95 ms: one round trip plus server time, with no new protocol.
London to Sydney on HTTP/2, TLS 1.2, new connection, then Edge hit. The handshakes now happen 10 ms from the user. The time drops from over a second to about 50 ms. Edge miss adds one long trip to the origin, over the edge’s warm connection, instead of four.
1 MB on a new connection, far away. Seven rounds of slow start take over 1.5 seconds, on a link that could move the bytes in 80 ms.
HTTP/1.1, then HTTP/2, and watch the second row. Thirty follow-up requests take five waves over six connections, or one wave multiplexed on one.

Step 5: Reuse connections

A connection that is already open skips DNS, both handshakes and slow start. HTTP/1.1 keeps connections open by default (keep-alive). HTTP clients, and database and Redis drivers, hold pools of open connections per destination. HTTP/2 and gRPC multiplex many concurrent requests on one connection, so one or a few per backend is enough.

Size a pool with Little’s law: connections in use = request rate × time each request holds one. 2,000 requests a second at 25 ms each keeps 50 connections busy. HTTP/1.1 needs a pool somewhat above that, for bursts. Over HTTP/2, the same load fits in one connection, since servers typically allow 100 or more concurrent streams.

Not reusing costs more than latency:

Cost of each new connection	Number
Round trips before the request	2 (TCP + TLS 1.3) or 3 (TCP + TLS 1.2)
Server CPU for a full TLS handshake	Roughly 1,000 to 2,000 a second per core with RSA 2048, far more with ECDSA P-256
Slow start	The first round trip carries only about 14.6 KB
Port held in TIME_WAIT	60 s on Linux, by the side that closes
Ephemeral ports	28,232 by default: about 470 new connections a second to one `IP:port` before they run out

The last row bites services that open a connection per request to one database or proxy. connect() fails with “cannot assign requested address”, while CPU and network look idle. Pooling fixes it. Widening the port range only delays it.

Step 6: HTTP/1.1, HTTP/2 and HTTP/3

The versions differ in how many requests one connection carries at once, and in what one lost packet holds up. The second is head-of-line blocking: the item at the front of a queue stalls everything behind it.

HTTP/1.1 carries one request at a time per connection. Pipelining exists in the spec. But it’s broken in enough proxies that major browsers leave it off. So they open up to six connections per host.

HTTP/2 (2015) splits messages into frames tagged with a stream ID. So hundreds of requests share one TCP connection. It also compresses repeated headers (HPACK).

The blocking at the application level is gone, but not the transport’s. TCP delivers bytes strictly in order. So one lost packet stalls every stream until it is retransmitted, even streams whose own data arrived. This is called head-of-line blocking. At 1 to 2% loss, a phone on a weak signal can do worse on one HTTP/2 connection than on six HTTP/1.1 connections. With six, a loss stalls only one of them.

HTTP/3 (RFC 9114, 2022) runs over QUIC (RFC 9000), a transport on UDP with streams built in. Each stream is in order, but a lost packet stalls only the streams whose data it carried. QUIC folds TLS 1.3 into its own handshake. So a new connection costs one round trip, and a resumed one costs zero. Connections are named by an ID, rather than by IP and port. So a phone moving from Wi-Fi to cellular keeps its connection.

Where one lost packet blocks

Scroll or pinch to zoom, drag to pan

Color key
The setup
Blocked
Not blocked

QUIC has costs. It runs in user space, and uses more CPU per byte than kernel TCP. Some networks block or throttle UDP, so clients race QUIC against TCP and fall back. And a browser learns that a server speaks HTTP/3 from an Alt-Svc header or an HTTPS DNS record. So a first visit often goes over HTTP/2.

Step 7: Big responses: bandwidth, latency and slow start

Bandwidth is bytes a second. Latency is how long one byte takes to arrive. Most requests in a design are bound by latency. A 10 KB response crosses a 1 Gbps link in 0.08 ms. The round trip to fetch it is 80 ms. Doubling bandwidth saves 0.04 ms. Halving distance saves 40 ms.

TCP widens the gap on fresh connections. A sender does not know the path’s capacity. So slow start begins with a window of 10 packets, about 14.6 KB (RFC 6928), and doubles it each round trip:

Round trip	Window	Total delivered
1	14.6 KB	14.6 KB
2	29 KB	44 KB
3	58 KB	102 KB
4	117 KB	219 KB
5	234 KB	453 KB
6	467 KB	920 KB
7	934 KB	1.85 MB

A 1 MB response on a new connection needs 7 round trips, whatever the link speed. That’s 700 ms at 100 ms each, on a gigabit line that could move it in 8 ms. Three rules follow:

Keep first responses small. What fits in about 14 KB arrives in the first round trip.
Reuse warm connections for big transfers. (Linux does shrink the window again after an idle period, by default.)
Move bytes closer. A CDN edge 10 ms away finishes slow start in 70 ms, instead of 700.

Go deeperBandwidth-delay product and congestion control2 min

A TCP connection can have at most one window of data in flight, so its throughput is at most window ÷ RTT. The window needed to fill a path is the bandwidth-delay product: 1 Gbps × 100 ms = 12.5 MB. Without window scaling (RFC 7323) the window caps at 64 KB, which over 100 ms is about 5 Mbps however fast the link. Modern stacks scale windows automatically, but the product explains why bulk transfers across oceans use several parallel connections or multipart uploads, as in blob storage.

After slow start, the classic algorithms (Reno, and CUBIC, the Linux default) grow the window until a packet is lost. Then they cut it, treating loss as congestion. On paths with random loss (Wi-Fi, long intercontinental links), they back off when they should not. Google’s BBR works differently. It models the bottleneck bandwidth and round trip time, and paces packets to match. That raised throughput on lossy long paths. QUIC implementations pick their own congestion control, which is one reason HTTP/3 performance varies between servers.

Step 8: Timeouts, retries, backoff and jitter

Networks drop packets, servers restart and connections reset. How long a client waits, and whether it tries again, decides whether a small failure stays small.

Timeouts. Every network call needs one. Many libraries default to waiting forever, and a hung call holds a thread, a connection and memory. Use three kinds:

A short connect timeout. A healthy server in the region accepts in about a millisecond.
A request timeout, above the dependency’s p99.9.
An overall deadline, passed down the call chain. (gRPC carries one on every call.) Then a service three hops down stops working on a request whose user has given up.

Retries. A retry turns a transient failure into a success: a dropped packet, a restart, a 503. But it’s only safe when the operation is idempotent. A retried “charge card” after a timeout may charge twice, unless it carries an idempotency key, covered in API design. Retry connection failures, timeouts, 503 and 429 (after any Retry-After). Never retry 400 or 404, which fail the same way again.

Backoff and jitter. Exponential backoff waits longer after each failure: 100, 200, 400 ms, up to a cap. On its own, that is not enough. A thousand clients that failed at the same moment all wait exactly 100 ms, and retry at the same moment again. Jitter randomizes the wait. “Full jitter” picks a random time between zero and the exponential ceiling.

Retrier.javaretrier.pyRetrier.csretrier.goretrier.ts

import java.util.Random;
import java.util.function.IntFunction;

// What one attempt returned. Only some failures are worth a second try.
enum Outcome { OK, UNAVAILABLE, TIMEOUT, BAD_REQUEST }

// A retry budget: retries may add at most ~10% to the load. Each success earns a tenth of a
// retry and each retry spends one, so when most calls fail the budget runs dry and the client
// stops multiplying traffic against a service that is already down.
final class RetryBudget {
    private double tokens = 10; // a small starting allowance, also the cap
    void onSuccess() { tokens = Math.min(10, tokens + 0.1); }
    boolean trySpend() {
        if (tokens < 1) return false;
        tokens -= 1;
        return true;
    }
}

final class Retrier {
    private final double baseMs, capMs;
    private final int maxAttempts;
    private final RetryBudget budget;
    private final Random rnd;
    double nowMs = 0; // a simulated clock, so the example runs instantly

    Retrier(double baseMs, double capMs, int maxAttempts, RetryBudget budget, Random rnd) {
        this.baseMs = baseMs;
        this.capMs = capMs;
        this.maxAttempts = maxAttempts;
        this.budget = budget;
        this.rnd = rnd;
    }

    // Exponential backoff with full jitter: wait a random time between 0 and the exponential
    // ceiling. The ceiling spaces attempts out; the randomness stops a thousand clients that
    // failed together from retrying together.
    double backoffMs(int attempt) {
        double ceiling = Math.min(capMs, baseMs * Math.pow(2, attempt));
        return rnd.nextDouble() * ceiling;
    }

    // The deadline is the caller's whole budget for this call, all attempts included. Retrying
    // past it is wasted work: the caller has already given up.
    Outcome call(IntFunction<Outcome> attempt, double deadlineMs, boolean idempotent) {
        for (int i = 0; ; i++) {
            Outcome r = attempt.apply(i);
            if (r == Outcome.OK) { budget.onSuccess(); return r; }
            // A 400 will fail the same way again, and a non-idempotent call may already have
            // happened: retrying either one is a bug, not resilience.
            boolean retryable = r == Outcome.UNAVAILABLE || r == Outcome.TIMEOUT;
            if (!retryable || !idempotent || i + 1 >= maxAttempts) return r;
            double wait = backoffMs(i);
            if (nowMs + wait >= deadlineMs) return r; // no time left for another try
            if (!budget.trySpend()) return r; // the fleet is failing: do not pile on
            nowMs += wait;
            System.out.printf("  attempt %d failed (%s), retry in %.0f ms%n", i + 1, r, wait);
        }
    }
}

class Main {
    public static void main(String[] args) {
        var retrier = new Retrier(100, 2000, 4, new RetryBudget(), new Random(7));
        // A dependency that is briefly unavailable: two failures, then success.
        Outcome r = retrier.call(i -> i < 2 ? Outcome.UNAVAILABLE : Outcome.OK, 1000, true);
        System.out.println("result " + r + " after " + Math.round(retrier.nowMs) + " ms of backoff");

        // 1,000 clients fail at the same instant. Without jitter all retry at exactly 100 ms;
        // with full jitter their first retries spread over 0 to 100 ms.
        int[] buckets = new int[10];
        var spread = new Retrier(100, 2000, 4, new RetryBudget(), new Random(1));
        for (int c = 0; c < 1000; c++) buckets[(int) (spread.backoffMs(0) / 10)]++;
        int busiest = 0;
        for (int b : buckets) busiest = Math.max(busiest, b);
        System.out.println("busiest 10 ms slice: 1000 without jitter, " + busiest + " with full jitter");
    }
}

Two guards keep retries from becoming the outage:

A retry budget: retries add at most about 10% to normal traffic. So when most calls fail, the client stops retrying.
The deadline: no retry starts if it cannot finish in time.

The last line shows jitter working. 1,000 clients failed together. They retry with about 115 in the busiest 10 ms slice, instead of 1,000 in one instant.

Go deeperHedged requests for tail latency1 min

A hedged request sends a second copy of a slow request to another replica, after a short delay (typically the p95 latency). It takes whichever answer comes first. Google’s “The Tail at Scale” reported that hedging after 10 ms cut a service’s p99.9 latency from 1,800 ms to 74 ms, for about 2% more requests. It is a retry that does not wait for the failure. So the same rules apply: idempotent reads only, and a cap on hedges in flight.

Step 9: Long-lived connections, NATs and keepalives

Chat, push and live tracking keep connections open for minutes or days. Between a phone and your server sit devices that track every connection and forget idle ones:

Middlebox	Typical idle timeout
Mobile carrier NAT	A few minutes, sometimes under a minute
AWS NAT gateway	350 s
AWS Application Load Balancer	60 s by default
Linux TCP keepalive, first probe	7,200 s by default: far too late

A NAT that forgets a mapping tells neither end. Both think the connection is open until one sends something and it vanishes or comes back as a reset. Messages pushed to that client go nowhere.

The fix is an application heartbeat, a small ping more often than the shortest idle timeout on the path. Chat and push systems typically ping every 25 to 60 seconds and treat two or three missed pongs as a dead connection to reopen. Pools follow the same rule: evict idle connections before the balancer or NAT does. The load balancing page shows the classic symptom, random 502 errors from a backend keep-alive shorter than the balancer’s idle timeout.

Go deeperPolling, long polling, SSE and WebSockets at a glance2 min

API design covers the choice. Here is the network view:

Mechanism	Connection	Network cost	Wins when
Short polling	A new request every N seconds	Mostly empty responses; up to N seconds late	Rare updates, minutes of delay are fine
Long polling	The server holds each request until data or timeout	One request per update, plus reconnects	Infrequent updates, plain HTTP everywhere
Server-sent events	One long HTTP response streaming events	One connection, server to client only, built-in reconnect	Feeds, dashboards, notifications in a browser
WebSocket	HTTP upgrade, then a two-way framed stream	One connection, 2 to 14 bytes of framing per message, both ways	Chat, collaboration, games

All four hold a connection or a request open. So all need heartbeats or short timeouts. And all make servers keep state about who is connected where. Heartbeats cost something at scale. 10 million phones pinging every 30 seconds is about 330,000 messages a second, and each ping wakes a radio. That is why mobile platforms route push through one shared system connection (APNs, FCM). HTTP/2 server push is not a live-update channel. It only warmed the browser cache, and Chrome removed it in 2022.

Options and tradeoffs

HTTP version

Version	Requests per connection	Handshake on a new connection	One lost packet blocks	Wins when
HTTP/1.1	One at a time (browsers open 6)	TCP + TLS: 2 or 3 RTT	That one connection	Simple clients, debugging, legacy proxies
HTTP/2	Hundreds, multiplexed	TCP + TLS: 2 or 3 RTT	Every stream on the connection	The default for browsers and service to service (gRPC)
HTTP/3	Hundreds, independent streams	QUIC: 1 RTT, 0 when resuming	Only the streams in that packet	Mobile and distant users, lossy networks, network changes

Where the client’s connection ends

Which edge or region a user reaches (GeoDNS, anycast) is covered in CDN and load balancing. The networking question is where TCP and TLS terminate, because that sets the RTT every handshake pays:

Option	Handshakes travel to	Later requests	Wins when
Origin directly, or an L4 balancer	The origin, full distance	Reused client connection only	Users near the region, internal traffic
L7 proxy or CDN edge near users	The edge, about 10 ms away	Edge forwards over warm pooled connections	Distant or mobile users, uncacheable APIs too

An L4 balancer forwards one connection end to end, so the client’s handshakes cross the whole distance. An L7 proxy or edge ends the client’s TCP and TLS nearby. It pays the long distance once per request, over a connection that is already open. That’s why the demo’s edge miss costs one long trip instead of four.

Retry strategy

Strategy	After a failure	Wins when	Loses
No retry	Return the error	Non-idempotent calls without a key	Transient blips reach users
Immediate retry	Try again at once	One dropped packet on a healthy service	Doubles load on a struggling service
Exponential backoff	Wait 100, 200, 400 ms…	Clients that fail independently	Clients that fail together retry together
Exponential + full jitter + budget	Random wait under the ceiling, retries capped at ~10%	Almost always	A little more code

How it fails

Traffic stuck on an old IP. A database fails over to a new address, and DNS is updated with a 60 s TTL. Ten minutes later, a third of the app servers still write to the old one. Their pools opened connections at startup and kept them. Or their runtime cached the answer. DNS matters only when a client resolves.

Detect it as connections to addresses that should be drained. Fix it with a maximum connection lifetime in every pool (5 to 30 minutes), and re-resolving on reconnect. Put a stable endpoint (a virtual IP or proxy) in front of anything that must move in seconds.

Connection storms. A balancer restarts, or a gateway holding 200,000 WebSockets is deployed. Then every client reconnects in the same second. Handshakes cost far more CPU than messages. So the surviving servers spend their CPU on TLS. They slow down, fail health checks, and drop more clients, who reconnect too.

Detect it as new connections a second far above normal. Fix it with jittered, backed-off reconnect delays spread over tens of seconds, and TLS session resumption. Add a cap on accepted connections a second per server, and drain gradually during deploys.

Retry storms. Each layer retries three times, so up to four attempts. With five retrying layers between the user and the database, one request can become 4⁵ = 1,024 database calls. That happens just when the database is slow, which keeps it slow.

Retries at every layer multiply

Scroll or pinch to zoom, drag to pan

Color key
Service
Database

Detect it as the downstream request rate rising while upstream traffic is flat. Fix it by retrying at one layer only, with a budget, jitter, and a deadline the lower layers respect. Lower layers fail fast. And a circuit breaker stops calls to a dependency that is clearly down.

MTU black holes. Ethernet carries 1,500-byte packets, but tunnels and overlays add headers (VXLAN 50 bytes, WireGuard and IPsec more), so the path carries less. A router that cannot forward a large packet sends back an ICMP “fragmentation needed” message and the sender shrinks its packets. If a firewall drops that ICMP, large packets just disappear:

Small packets pass, large ones vanish

Scroll or pinch to zoom, drag to pan

Color key
Service

The signature: connections open and small requests work, but anything with a large response hangs (a certificate chain, a big JSON body). Detect it with “don’t fragment” pings of increasing size. Fix it in one of three ways. Let ICMP “fragmentation needed” (and IPv6 “Packet Too Big”) through. Clamp the TCP MSS at the tunnel. Or enable MTU probing (tcp_mtu_probing on Linux).

Expired certificates. When a certificate expires, every client refuses the connection at once. An expired certificate in Ericsson software cut O2’s UK mobile data for most of a day in December 2018. An expired authentication certificate locked users out of Microsoft Teams in February 2020.

Lifetimes are shrinking. Under CA/Browser Forum ballot SC-081, public TLS certificates are capped at 200 days from March 2026, 100 days from March 2027, and 47 days from March 2029. So manual renewal stops being practical. Automate renewal (ACME). Alert at 30 and 7 days on what is actually served. And remember that intermediates and roots expire too.

Failure	Blast radius	Detect	Fix
Traffic stuck on an old IP	Clients with old connections or caches	Connections to drained addresses after a change	Max connection age, re-resolve on reconnect, stable VIPs
Connection storm	The whole tier after a restart or deploy	New connections a second; TLS CPU	Jittered reconnect, session resumption, accept rate caps
Retry storm	Every service below the first retrying layer	Downstream rate up while upstream is flat	One retrying layer, budgets, jitter, deadlines, breakers
MTU black hole	Large responses on one path	Small requests fine, large ones hang	Allow ICMP, MSS clamping, MTU probing
Certificate expiry	Every client of the endpoint, instantly	Days to expiry on what is served	ACME automation, alerts at 30 and 7 days
Idle connection cut	Pushed messages and pooled requests	Resets or timeouts on first use after idle	Heartbeats under the shortest idle timeout, idle pool eviction

At scale

What	Figure
Round trip: across an ocean / the world	70 to 120 ms / 250 to 280 ms
Round trips before the request	TCP + TLS 1.2: 3; TCP + TLS 1.3: 2; QUIC: 1; resumed QUIC: 0
TCP initial window	About 14.6 KB; 1 MB takes 7 round trips on a new connection
HTTP/2 concurrent streams per connection	Typically 100 to 250
Ephemeral ports, Linux default	28,232, TIME_WAIT 60 s: about 470 new connections a second to one destination
DNS: resolver cache hit / cold lookup	5 to 30 ms / 30 to 150 ms more

At 10x, connection counts grow faster than request counts, once clients hold connections open. A front door with a million concurrent connections needs raised file descriptor limits, sized connection tracking tables, and memory per socket (tens of KB with TLS buffers). One proxy IP can hold only about 28,000 connections to one backend IP:port. So big proxy tiers use several source IPs, or HTTP/2 to backends. Jittered reconnects and retry budgets become required: every deploy is a reconnect wave.

At 100x, users are everywhere. End TCP and TLS at edges near users. Carry traffic to a few regions over warm connections. Replicate the data every request needs per region, so no request crosses an ocean synchronously. TLS handshake capacity, not request handling, often sizes the edge fleet. That makes resumption and reuse capacity features, not only latency ones.

What to monitor:

DNS, connect, TLS and time to first byte from real clients (browser timing APIs, mobile SDKs), by country and network type. Server-side latency hides the first three.
New connections a second against requests a second. A falling reuse ratio means a broken pool or keep-alive.
TCP retransmissions and resets per path, ports in TIME_WAIT, pool wait time.
Retry rate per layer against its budget, and days until every served certificate expires.

In the interview

Interviewer asksOur API responds in 20 ms, but users in Australia see a second before anything loads. Where does the time go?

Round trips. Sydney to US East is about 230 ms a round trip. A cold HTTPS request with TLS 1.2 takes four: TCP, two for TLS, and one for the request. That’s about 950 ms before our 20 ms counts. I would cut the count and the length. TLS 1.3 removes one round trip, and HTTP/3 another. Reuse removes all setup on later requests. A CDN or edge proxy in Sydney makes the handshakes 10 ms long, and forwards over a warm connection. So even uncacheable requests pay one long round trip, not four. If Australia is a big market, a region there removes the long trip, and brings in replication questions.

Interviewer asksWould you use HTTP/3 between services inside one data center?

Usually not. HTTP/3 wins on handshake round trips, loss and network changes. Inside a region, round trips are sub-millisecond, loss is rare, and connections are pooled for hours. QUIC costs more CPU per byte than kernel TCP. HTTP/2 (gRPC) over pooled connections gets multiplexing cheaply. HTTP/3 belongs at the user-facing edge.

Interviewer asksA request passes through four services to a database. How do you set timeouts and retries?

Start from the user’s deadline, say one second. Pass the remaining time down every hop, so work stops when it runs out. Each call’s timeout comes from that dependency’s p99.9, and connect timeouts are short. Retries happen at one layer, near the edge. They’re only for idempotent operations, or ones with an idempotency key. They use exponential backoff, full jitter and a budget of about 10%. Lower layers fail fast. Otherwise four attempts at each of four layers is 256 database calls per request, exactly when the database is slow. A circuit breaker stops calls to a dependency that is down.

Interviewer asksWe pointed a service's DNS name at a new IP with a 300 second TTL. An hour later some callers still hit the old IP. Why?

DNS is only consulted when a client resolves, and many never resolve again. Pools opened their connections before the change, and keep reusing them. Some runtimes cache answers themselves: the JVM caches forever under a security manager. And if the TTL was long before it was lowered, resolvers kept the old answer for the old TTL. The fixes: a maximum connection age in every pool, lowering TTLs a full old TTL before planned moves, and a stable address in front of anything that must fail over in seconds.

Seen in

Chat: hundreds of millions of WebSockets, heartbeats sized to carrier NAT timeouts, and jittered reconnects so a deploy is not a connection storm.
Collaborative editor: one WebSocket per open document on dedicated gateways, with keystrokes batched to keep the message rate down.
Ride-hailing: one persistent connection per driver’s phone, because a TLS handshake per location ping would cost more than the ping.
Typeahead: a 100 ms budget where the round trip takes 40 to 100 ms. So under 10 ms of server time, served as plain GETs over a reused HTTP/2 connection.
File sync: a long poll per device instead of 830,000 polls a second, and large files moved as retried blocks.
Notification system: a connection to APNs that drops mid-request, when the push may or may not have been accepted, and whether to retry.
Rate limiter: a 2 ms budget that affords one same-zone Redis round trip per request.
Web crawler: DNS as a hidden bottleneck at 10 to 200 ms a lookup, fixed with a partitioned DNS cache.
Stock exchange: the far end of the scale, microsecond budgets, colocation and multicast instead of a TCP stream per reader.

Recap

The unit. Latency is round trips times RTT, plus server time. Fiber costs 1 ms of round trip per 100 km. That’s under 1 ms in a region, 70 to 120 ms across an ocean, and 250 ms or more across the world.

A cold request. DNS, TCP (1 RTT), TLS (2 RTT for 1.2, 1 for 1.3), and the request (1 RTT). HTTP/3 merges transport and TLS into 1 RTT. 0-RTT resumption sends the request at once, for idempotent requests only.

Reuse. A warm connection skips DNS, every handshake and slow start. Pool everything, size pools with Little’s law, cap connection age, evict idle connections before middleboxes do.

HTTP versions. 1.1 is one request at a time per connection. 2 multiplexes on one TCP connection, where a lost packet stalls every stream. 3 runs on QUIC, where it stalls only the affected stream.

Big responses. Slow start begins at about 14.6 KB and doubles per round trip: 1 MB takes 7 on a new connection. Latency, not bandwidth, bounds most requests.

Retries. Timeouts on every call, and a deadline passed down. Retries at one layer, for idempotent calls, with exponential backoff, full jitter and a budget of about 10%.

Failures. Pools that never re-resolve DNS, reconnect and retry storms, NATs dropping idle connections, MTU black holes, certificates expiring all at once.

Networking for system design

The problem it solves

How it works

Step 1: Round trips, and what distance costs

Step 2: Find the address: DNS

Step 3: Open and secure a connection: TCP and TLS

Step 4: The whole budget

Step 5: Reuse connections

Step 6: HTTP/1.1, HTTP/2 and HTTP/3

Step 7: Big responses: bandwidth, latency and slow start

Step 8: Timeouts, retries, backoff and jitter

Step 9: Long-lived connections, NATs and keepalives

Options and tradeoffs

HTTP version

Where the client’s connection ends

Retry strategy

How it fails

At scale

In the interview

Seen in

Recap

Further reading

The designbeyondcode digest, every Saturday

Networking for system design

The problem it solves

How it works

Step 1: Round trips, and what distance costs

Step 2: Find the address: DNS

Step 3: Open and secure a connection: TCP and TLS

Step 4: The whole budget

Step 5: Reuse connections

Step 6: HTTP/1.1, HTTP/2 and HTTP/3

Step 7: Big responses: bandwidth, latency and slow start

Step 8: Timeouts, retries, backoff and jitter

Step 9: Long-lived connections, NATs and keepalives

Options and tradeoffs

HTTP version

Where the client’s connection ends

Retry strategy

How it fails

At scale

In the interview

Seen in

Recap

Further reading

Keep reading

The designbeyondcode digest, every Saturday