Load balancing

How load balancers work, L4 vs L7, round robin to power of two choices, health checks, draining and sticky sessions, for system design interviews.

Foundation34 min core · 38 min full depth

Read first: Networking essentials

A load balancer is one address in front of many servers. It decides which server gets each connection or request. It notices when one is sick. And it lets you add, remove and redeploy servers without users seeing it. Every design with more than one server has one. So the interesting questions are never “do we need one”. They are: where does it sit, what can it see, how does it pick, and how does it fail?

The problem it solves

One app server handles about 2,000 requests a second. Your peak is 10,000. So you run at least five servers, really seven or eight for headroom, and now three problems appear at once:

One address. Clients know api.example.com, not eight IPs. Something has to fan out.
Uneven servers and uneven requests. One server is an older instance type, one request takes 5 ms and the next takes 900 ms. Equal turns do not mean equal load.
Servers die and get replaced. A crash, a deploy every hour, an autoscaler adding four servers at 9 a.m. Traffic must stop going to a dead server within seconds. And a server leaving for a deploy must not cut off the requests it is serving.

A load balancer answers all three. It also becomes the one component that every request passes through, which makes its own failure modes the most important thing on this page.

How it works

Step 1: Spread connections with DNS

The simplest balancer is no balancer: publish several A records for api.example.com and let clients pick one. Resolvers rotate the order, so clients spread out roughly evenly.

It breaks on failure. When a server dies, its IP stays in resolver and client caches until the TTL runs out. Many clients and resolvers keep records longer than the TTL says. With a 60 s TTL, a dead server keeps receiving a share of new connections for a minute or more. And the client has no idea which other IP to try, unless it was written to retry. DNS also knows nothing about load. It is a good tool for picking a region (step 6), and a poor tool for picking a server.

Step 2: One box in the middle, at L4 or L7

Put a proxy at one IP and let it choose a server for each connection. The big decision is how much of the traffic it reads.

L4 (transport layer) sees IP addresses and ports. It picks a backend when the TCP connection opens (or the first UDP packet arrives) and forwards packets from then on. It never decrypts TLS and never sees a URL. It is fast and cheap: a software L4 balancer forwards millions of packets a second per machine.
L7 (application layer) ends the client’s TCP and TLS connection, and parses the HTTP request. Then it opens (or reuses) a second connection to the backend it picks. Because it reads each request, it can do a lot more. It can route /api/* and /ws to different pools, and retry a failed request on another server. It can add headers, rate limit, and balance per request instead of per connection.

L4 forwards one connection, L7 runs two

Scroll or pinch to zoom, drag to pan

Color key
Service

That second connection is the price of everything L7 can do. The proxy holds two sockets per active client, and spends CPU on TLS and HTTP parsing. The backend sees the proxy’s IP as the source. So the real client IP travels in a header for L7 (X-Forwarded-For, or the standard Forwarded). For L4 proxies that do not keep the source address, it travels in the PROXY protocol preamble.

NGINX, HAProxy and Envoy are the common software L7 proxies; AWS ALB and Google Cloud’s HTTP load balancer are managed ones. AWS NLB, Google’s Maglev and Linux IPVS are L4.

Step 3: Pick a server

Once the balancer has a list of healthy servers, an algorithm chooses one. This is a Strategy in the textbook sense: the balancer owns the list and the health, the algorithm only answers “which one”. The options are in the next section. Play with them first.

Algorithm

Total capacity is 90 req/s: each server runs 4 requests at once. Click a server to make it slow, kill it, or restore it.

Served /s: 0
p50: –
p99: –
Errors, last 5 s: 0%

p99, last 30 s (top = 500 ms)error rate

Press Start traffic. Round robin sends a quarter of the traffic to d, which can only take a ninth of it.

Four things to try, at 50 requests a second:

Round robin gives d a quarter of the traffic. But d can serve only a ninth of the fleet’s capacity. So its queue grows, and p99 climbs into seconds, while a, b and c have capacity to spare.
Weighted 4:2:2:1 fixes it, because the weights match capacity. Now click b once to make it six times slower. The weights do not know, and p99 explodes again.
Least connections and two random choices route around the slow b without being told. A slow server holds requests longer, so it has more in flight, so it gets picked less.
Kill a server (click it twice) under least connections. A dead server fails instantly, so it has zero requests in flight and looks like the best choice. It swallows a big share of the traffic as errors until the health check takes it out. Turn on passive ejection and repeat.

Step 4: Health checks, active and passive

The balancer needs to know which servers are alive. Two signals, and production uses both:

Active checks. The balancer probes each server on a schedule, typically GET /healthz every 5 to 30 s. It takes a server out after N failures in a row, and back in after M passes. With AWS ALB’s defaults (30 s interval, out after 2 failures, back after 5 passes), a dead server can take a full minute to be removed, and 2.5 minutes to come back. Tighten it for anything user-facing: 5 s and 2 failures removes a dead server in about 10 s.
Passive checks (outlier detection). The balancer watches the real responses. With outlier detection configured, Envoy’s defaults eject a host after 5 consecutive 5xx responses. The ejection lasts 30 s, multiplied by the number of times it has been ejected. At 1,000 requests a second across 10 servers, 5 errors arrive in about 50 ms. So a dead server is caught almost at once. But passive checks see only what traffic sees, so an idle server is never checked. That’s why they add to active checks, rather than replace them.

What the check tests matters as much as how often. A shallow check is safe: the process is up and can answer. A deep check sounds more thorough: it can reach the database, the cache, two downstream services. But it is the cause of the outage in How it fails. A shared dependency fails, and every server fails its check at once.

Step 5: Drain before you remove

A deploy takes a server out of rotation. If it stops at once, every request in flight on it fails. Connection draining (AWS calls it deregistration delay, default 300 s) separates “stop sending new work” from “stop the process”:

Draining a server during a deploy

Scroll or pinch to zoom, drag to pan

Color key
Service

Three details make it work:

The server must handle SIGTERM by finishing current work, rather than exiting.
The drain timeout should be a little longer than your slowest normal request, not the 300 s default. A 30 s API timeout wants a 35 s drain. Otherwise, deploys of 50 servers in batches take an hour.
A new server should get slow start. A cold JIT, empty caches and a fresh connection pool make the first seconds slow. Without slow start, least-connections would send it a flood, because it has nothing in flight.

Long-lived connections break this model. A WebSocket can stay open for hours, so draining cannot wait for it. The server closes its sockets a few thousand at a time after the drain starts. Clients reconnect with jittered backoff, which lands them on other servers.

Step 6: Remove the balancer as a single point of failure

One balancer is one machine. Three ways to make the tier itself survive a failure, in the order systems usually grow:

Active-passive with VRRP, or active-active with ECMP

Scroll or pinch to zoom, drag to pan

Color key
Service

Active-passive with a floating IP. Two balancers share one virtual IP via VRRP (keepalived is the usual tool). The active one sends an advertisement every second. If the standby hears nothing for about 3 seconds, it claims the IP. It’s simple. But half the hardware idles, capacity is capped at one box, and connections on the dead box are lost.
Active-active with ECMP. Every balancer announces the same IP to the router over BGP. The router spreads flows across them, by hashing each packet’s 5-tuple (equal-cost multi-path). Add a node, and capacity grows. When a node dies, its BGP session drops, and the router stops using it within seconds. This is how Google’s Maglev and most large L4 tiers are built.
A managed balancer. AWS ALB/NLB or a cloud equivalent is already a fleet spread across zones. You still choose zones, health checks and timeouts, and you still need a plan if the provider’s regional control plane has a bad day.

ECMP has one catch. When the set of next hops changes, a plain modulo hash moves many flows to a different balancer. That balancer has never seen those connections. The fix is to make every L4 node pick the backend with the same consistent hash, and track connections it has already seen. Then a flow that lands on a new L4 node still reaches the same backend. Maglev’s paper describes exactly this.

Step 7: Many regions

Above one region, the balancing question is which region. Two tools:

GSLB (global server load balancing) in DNS. The authoritative DNS answers each resolver with the IP of the nearest healthy region. It uses geography, latency measurements and health probes. Failover is bounded by TTLs, and by clients that cache longer.
Anycast. Every region (or edge site) announces the same IP prefix via BGP, and the Internet’s routing delivers each packet to the nearest one. Failover is a route withdrawal, which takes seconds, not a TTL. Cloudflare, Google’s front end and most DNS providers run on anycast. TCP works over it because routes rarely change mid-connection.

The full tier, grown stage by stage. Step through it, push the traffic slider to 100,000 and 1,000,000 requests a second, and trace a request through every hop:

Load balancing tier · architecture

The full design, stage by stage

Color keyServiceDatabaseNew in this stageAsync or occasional

Stage

Requests per second: 10,000

Stage 4 of 4

Many regions

Why this stage

A region is one failure domain, and users in Asia pay 150 ms or more per round trip to a US region.

What we add

GSLB in DNS answers each user with the nearest healthy region; each region runs the same tiers.

An anycast front can replace DNS steering.

What it costs

Failover is only as fast as clients honour DNS TTLs, and data that lives in one region still pays the long trip.

At 10k requests per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One app server handles about 2,000 requests a second.
One L7 proxy handles about 20,000 HTTPS requests a second, with keep-alive and session resumption.
One L4 node handles several hundred thousand requests a second, because it only forwards packets.

Other traffic

About 10% of requests are WebSocket frames.
Instances scale out to stay near 60% busy, never fewer than 2.

Load balancing tier · architecture

Trace a request: An HTTPS API call

Color keyServiceDatabaseNew in this stageAsync or occasionalHop in motionHops doneResponse

An HTTPS API call: step 1 of 6

Users to Edge routers

The client resolves api.example.com to one virtual IP. The edge router has several equal-cost routes to it and hashes the 5-tuple, so every packet of this connection takes the same path.

20 ms of a 50 ms budget

Users → Edge routers20 ms
Edge routers → L4 balancers0.1 ms
L4 balancers → L7 proxies2 ms
L7 proxies → App servers15 ms
App servers → Database3 ms
Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Options and tradeoffs

Where the balancing happens

Option	Sees	Wins when	Loses
DNS / GSLB	The resolver’s location	Picking a region; no box in the data path	Failover bounded by TTLs; no idea of server load
L4 (TCP/UDP)	IPs, ports	Huge throughput, any protocol, TLS stays end to end, edge tier	Balances connections, not requests; no routing by path; no retries
L7 (HTTP/gRPC)	URL, headers, cookies, each request	Per-request balancing, path routing, retries, TLS offload, gRPC	CPU for TLS and parsing, a second connection, more ways to misconfigure
Client-side (in the SDK)	Whatever the client measures	Service-to-service at scale, no extra hop (gRPC, Finagle)	Every client needs the library and service discovery; harder to change

The algorithm

Algorithm	How it picks	Wins when	Fails when
Round robin	Next in turn	Identical servers, uniform cheap requests	Servers differ, requests differ, or one server turns slow
Weighted round robin	Turns in proportion to a static weight	Mixed instance types with known capacity; canary at 5% weight	Capacity changes at runtime; weights are a guess that goes stale
Least connections	Fewest open connections	Long-lived connections (WebSockets, databases)	HTTP/2 multiplexing makes connection count meaningless; fast-failing hosts attract traffic
Least outstanding requests	Fewest requests in flight (L7)	Mixed request costs, gray failures; the best general L7 default	Needs global state if many balancers share one view
Power of two random choices	Sample two, take the less loaded	Many balancers with stale local views; O(1) per pick	Very small pools (2 or 3 servers), where it is close to least-requests
Ring hash / Maglev (consistent hash)	Hash a key (user, session, cache key) to a server	Affinity: per-user caches, shard owners, L4 tiers that must agree	Hot keys overload one server; load ignores capacity unless weighted

How each wins, in one line each:

Round robin is perfect when every request costs the same on every server. That is rarer than it looks.
Weighted lets an 8-core and a 32-core server share a pool, or sends 5% to a canary. The smooth version below spreads a server’s turns out instead of sending them in a burst.
Least connections is right when the unit of work is a connection. It has a known trap. A server that fails fast (connection refused, instant 503) has nothing in flight. So it attracts more traffic, as the demo shows. Pair it with passive ejection.
Least outstanding requests is least connections, done per request at L7. It responds to real load within milliseconds. So it routes around a slow server before any health check fires. AWS ALB offers it as an option. Envoy’s LEAST_REQUEST is the same idea.
Power of two choices solves a problem that only appears with many balancers. Say twenty L7 proxies share a load report, refreshed every second. They all see the same “least loaded” server, and all send to it at once. This herd moves to a new victim with every report. Sampling two servers at random, and taking the better one, spreads them out.

The math is famous. Place n requests on n servers at random, and the busiest server holds about log n / log log n of them. Take the better of two random choices, and it drops to about log log n. That’s an exponential improvement, for one extra sample. Envoy’s least-request balancer does exactly this by default.
Consistent hashing is for affinity, not balance. It sends the same key to the same server. That helps a per-server cache or a shard owner. And when a server leaves, only that server’s keys move. Kill a server under ring hash in the demo. Only its own users move (a fifth to a third of them, depending on which server). Hash mod N moves about three quarters. The full mechanism is on the consistent hashing page.

SmoothWeightedRoundRobin.javasmooth_weighted_round_robin.pySmoothWeightedRoundRobin.cssmooth_weighted_round_robin.gosmooth-weighted-round-robin.ts

import java.util.ArrayList;
import java.util.List;

// One server and its weight. `current` is the running score that smooth weighted round robin
// keeps per server; it is the only state the algorithm needs.
final class Backend {
    final String name;
    final int weight;
    int current;

    Backend(String name, int weight) {
        this.name = name;
        this.weight = weight;
    }
}

// Weighted round robin as nginx does it: a gets 4 of every 9 requests, but never 4 in a row.
// The naive version (a list with a repeated 4 times) sends bursts to one server. Here every
// server gains its weight each turn, the highest score wins, and the winner pays back the total.
// Over one cycle each server wins exactly `weight` times, spread as evenly as possible.
final class SmoothWeightedRoundRobin {
    private final List<Backend> backends;

    SmoothWeightedRoundRobin(List<Backend> backends) {
        this.backends = backends;
    }

    Backend pick() {
        // Health filtering happens before this, in the balancer. A server leaving the list
        // changes `total`, and the scores settle again within one cycle.
        int total = backends.stream().mapToInt(b -> b.weight).sum();
        Backend best = backends.get(0);
        for (Backend b : backends) {
            b.current += b.weight;
            if (b.current > best.current) best = b;
        }
        best.current -= total;
        return best;
    }
}

class Main {
    public static void main(String[] args) {
        // Weights match capacity: a is a new instance with 4 times the throughput of d.
        var wrr = new SmoothWeightedRoundRobin(List.of(
                new Backend("a", 4), new Backend("b", 2), new Backend("c", 2), new Backend("d", 1)));
        List<String> picks = new ArrayList<>();
        for (int i = 0; i < 9; i++) picks.add(wrr.pick().name);
        System.out.println(String.join(" ", picks));  // a b c a d a b c a
    }
}

Sticky sessions

Sticky sessions (session affinity) send every request from one client to the same server. The balancer usually does this with a cookie it sets (AWSALB, for example), or by hashing the source IP. Sticky sessions exist because the app keeps session state in server memory: a login, a shopping cart, an upload in progress.

Option	How	Wins	Loses
Stateless servers	Session in a shared store (Redis) or a signed token	Any server takes any request; deploys and failures are invisible	One store lookup per request (about 1 ms), or a token size limit
Cookie stickiness	Balancer sets a cookie naming the server	No app change	Uneven load, a server death logs out its users, scale-out does not help existing users
Source IP hash	Hash the client IP	Works at L4, no cookies	Thousands of users behind one carrier NAT land on one server; mobile IPs change
Consistent hash on a request key	Hash user ID or document ID	Deliberate affinity for caches or owners	Hot keys; a server change moves keys with no handover

TLS termination

Where TLS ends	Wins	Loses
At the L7 balancer, plain HTTP inside	Certificates in one place, L7 features, backends spend no CPU	Plaintext inside the network; fails most compliance regimes today
At the balancer, re-encrypted to backends	L7 features and encryption in transit everywhere (often mTLS)	Two handshakes’ worth of CPU; certificates on every backend
Passthrough at L4 (route by SNI)	Keys never leave the backend; end to end encryption	No L7 features; each backend does its own TLS

Go deeperWhat TLS termination costs1 min

A full TLS 1.3 handshake costs one round trip, and the server’s key exchange. With ECDSA certificates, one core does several thousand per second. RSA 2048 signing is several times slower. Session resumption skips most of that work. But only if every proxy in the fleet shares the session ticket keys. Without shared keys, a client that reconnects to a different proxy pays a full handshake again.

How it fails

Load balancing tier · architecture

Break it: what fails, who notices, what users see

Color keyServiceDatabaseNew in this stageAsync or occasional

Break

Failure: click any box to break it

The database goes down

What breaks

Every app server fails at once.

If the health check queries the database, every server fails its check.

How we notice

All backends unhealthy at the same moment, which is a shared cause, not 40 bad servers.

What users see

Errors either way.

Without panic mode the balancer also has nobody to send to and returns 503s for requests that did not need the database.

What we do

Keep health checks shallow (is this process alive), and fail open when most of the pool looks unhealthy.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Health checks take everything out. The health check calls the database. The database has a 30-second failover. Every server fails its check within the same 10 seconds. The balancer removes all of them, and returns 503 for every request. That includes the ones that never touch the database: static pages, cached reads, the status page. The outage is bigger than the failure that caused it.

A deep health check turns a dependency blip into a full outage

Scroll or pinch to zoom, drag to pan

Color key
Database
Service

There are three fixes, and you want all three:

Keep active checks shallow. The check proves this process can serve. Dependency health goes to dashboards and alerts instead.
Cap ejection. Never let passive checks remove more than a set fraction of the pool. (Envoy’s default is 10%.)
Fail open. When most of the pool looks unhealthy, the signal is more likely wrong than the servers. So spread traffic over all of them. (Envoy calls this the panic threshold, 50% by default.)

HostPool.javahost_pool.pyHostPool.cshost_pool.gohost-pool.ts

import java.util.ArrayList;
import java.util.List;

// What the balancer knows about one server. `healthy` comes from active checks (a probe every
// few seconds); the other fields come from real responses (passive checks).
final class Host {
    final String name;
    boolean healthy = true;
    int consecutiveErrors;
    double ejectedUntil;
    int timesEjected;

    Host(String name) {
        this.name = name;
    }
}

// Decides which hosts the balancing algorithm may pick from, in the spirit of Envoy. Two guards
// stop the health machinery from taking the whole service down: a cap on how many hosts passive
// checks may eject, and panic mode when too few hosts look healthy.
final class HostPool {
    private final List<Host> hosts;
    private final int consecutiveErrors = 5;
    private final double baseEjectionS = 30;
    private final double maxEjectionPct = 0.1;
    private final double panicPct = 0.5;

    HostPool(List<Host> hosts) {
        this.hosts = hosts;
    }

    // Called after every response, so a dead host is caught in a few requests, not seconds.
    void record(Host host, boolean ok, double now) {
        if (ok) {
            host.consecutiveErrors = 0;
            return;
        }
        host.consecutiveErrors++;
        if (host.consecutiveErrors < consecutiveErrors || host.ejectedUntil > now) return;
        // Ejecting is a guess from errors one balancer saw. If many hosts fail at once the cause
        // is more likely shared (a database, the network, the balancer itself), and ejecting more
        // would only pile their traffic onto the rest. At least one host may always be ejected.
        long ejected = hosts.stream().filter(h -> h.ejectedUntil > now).count();
        if (ejected >= Math.max(1, (int) (hosts.size() * maxEjectionPct))) return;
        // Repeat offenders stay out longer: 30 s, then 60 s, then 90 s.
        host.timesEjected++;
        host.ejectedUntil = now + baseEjectionS * host.timesEjected;
        host.consecutiveErrors = 0;
    }

    List<Host> available(double now) {
        List<Host> live = hosts.stream().filter(h -> h.healthy && h.ejectedUntil <= now).toList();
        // Panic mode: if under half the hosts look usable, the health signal is more likely wrong
        // than half the fleet is dead. Spread over every host and accept some errors, rather than
        // send all traffic to the few survivors and knock them over too.
        if (live.size() < hosts.size() * panicPct) return List.copyOf(hosts);
        return live;
    }
}

class Main {
    public static void main(String[] args) {
        List<Host> hosts = new ArrayList<>();
        for (int i = 0; i < 10; i++) hosts.add(new Host("h" + i));
        var pool = new HostPool(hosts);

        // h3 starts returning 503s. Five in a row and it is out for 30 s.
        for (int i = 0; i < 5; i++) pool.record(hosts.get(3), false, 0);
        System.out.println(pool.available(1).size());  // 9

        // h4 fails too, but the 10% cap (one host of ten) is used up, so h4 stays in.
        for (int i = 0; i < 5; i++) pool.record(hosts.get(4), false, 2);
        System.out.println(pool.available(2).size());  // 9

        // A deep health check that also pings the shared database fails on six hosts at once.
        for (Host h : hosts.subList(0, 6)) h.healthy = false;
        System.out.println(pool.available(3).size());  // 10: panic mode, traffic goes everywhere
    }
}

The death spiral. Five of twenty servers die. The other fifteen now take a third more load. Their latency rises, and their health checks start timing out. The balancer removes some of them, and the survivors take even more. The pool shrinks to zero in minutes.

The roots are no headroom, and health checks that measure load rather than liveness. Plan capacity so the fleet survives losing its largest failure domain (usually one zone, a third of the fleet). Give health checks a generous timeout. Shed load at the server (a fast 503 with Retry-After), rather than letting queues grow. And rely on panic mode as the last line.

Gray failure. A server with a failing disk or a noisy neighbour answers /healthz in 2 ms, and real requests in 2 s. Active checks never see it. Round robin keeps sending it a full share, and your p99 becomes its p99. Detect it by comparing per-backend latency and error rate. Route around it with least-outstanding-requests. Remove it with outlier detection on success rate.

Retry storms. An L7 balancer retries a failed request on another server. That’s right for one bad server, and terrible for an overloaded fleet. Three retries per request quadruple the load on servers that are already failing. Retry only idempotent requests, only once, and under a retry budget: retries at most about 10 to 20% of normal traffic.

Long-lived connections skew. Balancing happens once per WebSocket or gRPC connection. Scale out from 10 gateways to 15, and the 5 new ones get only new connections. The old ones keep every socket they had, possibly for days. A deploy does the reverse. Draining one gateway pushes its 100,000 sockets onto the rest at once. The fixes:

Cap connection age, so clients reconnect every 30 to 60 minutes. That spreads load over time.
Make clients reconnect with jitter.
Balance new connections by least connections.
Have servers refuse new connections above a limit, so the balancer picks another.

Go deeperHow the balancer itself fails1 min

A misconfigured idle timeout is a classic. Say the balancer’s idle timeout is 60 s (as on AWS ALB). If that’s shorter than the backend’s keep-alive timeout, fine. But if the backend closes idle connections first, the balancer occasionally sends a request down a connection the backend just closed. The user sees a random 502. So set the backend’s keep-alive longer than the balancer’s idle timeout.

Other failures: the certificate expires (automate renewal, and alert at 30 days). The connection table fills under a SYN flood (use SYN cookies, and a DDoS front). And a bad config push takes every balancer down at once (roll config out like code, zone by zone).

Failure	Blast radius	Detect	Fix
One server dies	Its in-flight requests	Active check in ~10 s, passive in milliseconds	Remove it; idempotent requests retried once
Gray failure	Every request routed to it	Per-backend p99 and error rate	Least outstanding requests, outlier ejection
Shared dependency fails a deep check	The whole service	All backends unhealthy at the same moment	Shallow checks, ejection cap, panic mode
Balancer node dies	Connections through it	VRRP or BGP session lost	Standby takes the VIP in ~3 s, or ECMP drops the next hop
Region down	That region’s users	GSLB probes from several locations	DNS or anycast moves users, if other regions have headroom

At scale

Numbers to remember.

What	Typical figure
Added latency, L7 proxy in the same zone	Well under 1 ms per hop (a few hundred microseconds)
One L7 proxy (nginx, HAProxy, Envoy), 8 cores	Tens of thousands of HTTPS requests/s with keep-alive
One software L4 node	Millions of packets/s; about a 10 Gb/s NIC per Maglev machine in the 2016 paper
TLS 1.3 full handshakes, ECDSA P-256	Several thousand per core per second; RSA 2048 several times fewer
VRRP failover	About 3 s (three missed 1 s adverts)
AWS ALB health check defaults	30 s interval, 2 failures out, 5 passes in
AWS deregistration delay default	300 s
Envoy outlier defaults	5 consecutive 5xx, 30 s base ejection, at most 10% ejected
Envoy panic threshold	50% of hosts healthy

At 10x (10,000 to 100,000 requests a second), the single active-passive pair stops being enough. One box cannot do all the TLS. Move to a fleet of L7 proxies behind an L4 tier or a managed balancer. Share TLS session ticket keys across the fleet. Switch from round robin to least outstanding requests, because with dozens of proxies and hundreds of backends, you will always have a slow one.

Connection pools from proxies to backends start to matter. 30 proxies × 200 backends × 10 pooled connections is 60,000 connections. So subset the backends: each proxy talks to a random 20 to 50 of them.

At 100x (a million requests a second, and several regions), the balancer is a distributed system. It uses ECMP with consistent hashing at L4, and anycast or GSLB at the edge. Service-to- service calls use client-side balancing, with no proxy hop at all. A control plane (xDS for Envoy) pushes endpoint lists to thousands of proxies. Config changes become the biggest risk. So they roll out gradually, and can be rolled back in seconds.

What to monitor:

Per-backend request rate, p99 latency and error rate, side by side. A balancer that sends equal traffic to unequal servers shows up here first.
Healthy host count per pool, and an alert when it drops by more than one at a time.
Balancer-generated errors (502, 503, 504), separately from backend errors. A 502 from the balancer means it could not talk to the backend. A 503 means it had nobody to send to.
Active connections and new connections per second on the balancer. And connections per backend for WebSocket pools: the skew shows up as one gateway with 3× the others.
TLS handshake rate and resumption ratio: a drop in resumption means ticket keys are not shared.
Certificate expiry dates.

Go deeperDirect server return, connection tracking and subsetting3 min

Direct server return (DSR). Responses are usually much bigger than requests. In DSR, the L4 balancer forwards the inbound packets to a backend that also owns the virtual IP, on a loopback interface. The backend replies straight to the client, bypassing the balancer. So the balancer carries only inbound traffic, often a tenth of the bytes. It works only at L4 (the balancer never sees the response). And it needs network setup on every backend. Maglev and IPVS both support it.

Connection tracking. An L4 balancer must send every packet of a connection to the same backend. A per-node table of 5-tuple to backend handles that. But the table is lost if the node dies, or if ECMP moves the flow to another node. Using the same consistent hash on every node is the fallback. A new node computes the same answer without the table. Maglev’s lookup table is built so that removing one backend changes very few entries. And every backend owns an almost equal share.

Subsetting. With 1,000 clients and 1,000 backends, a full mesh is a million connections. And every backend’s health is checked by every client. So each client talks to a subset of 20 to 50 backends instead. They’re chosen so every backend gets about the same number of clients. (Google’s SRE book describes deterministic subsetting.) That means fewer connections and fewer health checks. And the balancing algorithm runs over a smaller list.

Weighted least request. When weights differ, Envoy scores each host as its weight divided by (active requests + 1). So a host twice as big gets twice the concurrency before it looks busy. That is how mixed instance types and least-request work together.

In the interview

Interviewer asksYour load balancer uses round robin and p99 latency doubled after you added a few older instances. What happened and what do you change?

Round robin gives every server the same number of requests. So the older instances get the same share as the new ones, with less capacity. Their queues grow, and their latency becomes the tail for everyone. There are two fixes. Weighted round robin, with weights proportional to capacity, works while the capacity is known and stable. Least outstanding requests, sampled with two random choices, adapts on its own. A slower server holds requests longer, so it is picked less. It also handles a server that becomes slow later, for a reason weights cannot know. I would pick the second as the default, and keep weights for canaries.

Interviewer asksYour database had a 30 second failover and the whole site returned 503 for two minutes. Why, and how do you prevent it?

The health check probably checked the database. So every app server failed its check at once, and the balancer emptied the pool. With a 30 s check interval and 5 passes required to come back, recovery took minutes after the database was already back. Prevent it three ways. Make the check shallow (the process can serve requests), so dependency failures show as request errors, rather than as removed servers. Cap how many hosts can be ejected. And enable panic mode: when most of the pool looks unhealthy, the balancer sends to all of it. Requests that do not need the database keep working, and recovery is immediate.

Interviewer asksHow do you make the load balancer itself highly available?

Small: two balancers sharing a virtual IP with VRRP. Failover takes about 3 seconds, and connections on the failed node reset. Larger: an active-active fleet behind the router with ECMP. Every node announces the same IP over BGP. Every node uses the same consistent hash with connection tracking, so a flow that moves between nodes still reaches the same backend. In the cloud, a managed balancer is already a multi-zone fleet. Across regions, GSLB in DNS or anycast moves users away from a failed region. And each region needs headroom to absorb the traffic.

Interviewer asksYou run 20 WebSocket gateways and scale to 30. A day later the new 10 are nearly idle. Why?

Balancing happens once per connection, and WebSocket connections last for hours. The old gateways kept every socket they had. The new ones only got connections opened after they joined. Balancing new connections by least connections helps, but does not move existing ones. The fix is to make connections turn over. Use a maximum connection age of 30 to 60 minutes, with jittered client reconnects. Or have gateways close a slice of connections when they are above the fleet average. The same mechanism makes deploys safe, because draining a gateway closes its sockets gradually, instead of all at once.

Seen in

Chat: gateways hold WebSockets behind a plain L4 balancer, and a session registry finds Bob’s gateway instead of hashing users to gateways.
Collaborative editor: rejects sticky L7 hashing by document ID in favour of slot leases, so documents get a handover when servers change.
Rate limiter: 330 gateways behind a round-robin balancer are why per-node buckets fail and the count must be shared.
Ride-hailing: edge gateways drain millions of driver sockets on deploy and survive reconnect storms with jittered backoff.
Unique ID generator: ID servers behind a balancer fail health checks when their clock steps back, and the balancer retries elsewhere.
URL shortener: a balancer in front of stateless app servers is the step that takes it from one server to many.
Pastebin: balancer health checks are how a crashed app server is removed, with only a few failed requests.

Recap

The job. One address for many servers. Pick a server per connection or request, stop sending to sick ones, and let servers come and go without users noticing.

Layers. DNS and GSLB pick a region. L4 forwards connections fast and cheaply. L7 terminates TLS, reads each request, and balances, routes and retries per request. Large systems put L4 in front of an L7 fleet.

Algorithms. Round robin for identical servers and requests. Weighted, for known capacity differences and canaries. Least outstanding requests with two random choices, as the general default. Consistent hashing only when you want affinity.

Health. Active checks find dead idle servers, passive checks catch failures in milliseconds. Keep checks shallow, cap ejection, fail open below 50% healthy. Drain before removing, slow start when adding.

Numbers. Under 1 ms per proxy hop. Tens of thousands of HTTPS requests/s per L7 proxy. VRRP failover in about 3 s. ALB defaults of 30 s checks and a 300 s drain.

Biggest risks. A deep health check emptying the pool. A death spiral from no headroom. Gray failures that pass checks. Retry storms. And long-lived connections that never rebalance.

Load balancing

The problem it solves

How it works

Step 1: Spread connections with DNS

Step 2: One box in the middle, at L4 or L7

Step 3: Pick a server

Step 4: Health checks, active and passive

Step 5: Drain before you remove

Step 6: Remove the balancer as a single point of failure

Step 7: Many regions

The full design, stage by stage

Why this stage

What we add

What it costs

Trace a request: An HTTPS API call

Options and tradeoffs

Where the balancing happens

The algorithm

Sticky sessions

TLS termination

How it fails

Break it: what fails, who notices, what users see

What breaks

How we notice

What users see

What we do

At scale

In the interview

Seen in

Recap

Further reading

The designbeyondcode digest, every Saturday

Load balancing

The problem it solves

How it works

Step 1: Spread connections with DNS

Step 2: One box in the middle, at L4 or L7

Step 3: Pick a server

Step 4: Health checks, active and passive

Step 5: Drain before you remove

Step 6: Remove the balancer as a single point of failure

Step 7: Many regions

The full design, stage by stage

Why this stage

What we add

What it costs

Trace a request: An HTTPS API call

Options and tradeoffs

Where the balancing happens

The algorithm

Sticky sessions

TLS termination

How it fails

Break it: what fails, who notices, what users see

What breaks

How we notice

What users see

What we do

At scale

In the interview

Seen in

Recap

Further reading

Keep reading

The designbeyondcode digest, every Saturday