Replication

The problem it solves

A shop keeps its orders in one Postgres server. It holds 2 TB of data. At the evening peak, it takes 40,000 reads a second and 2,000 writes a second. One machine has four separate problems.

• Durability. The disk or the host dies. Everything since last night’s backup is gone, up to 24 hours of orders. And restoring 2 TB at 200 MB/s takes about 3 hours, before the replay of the day’s log even starts.
• Availability. Every kernel patch, failed power supply and database upgrade is downtime. At 99.99% the budget is 4.3 minutes a month.
• Read throughput. One node serves about 15,000 point reads a second. The peak needs 40,000.
• Latency. A customer in Sydney pays about 200 ms per round trip to a server in Virginia, and a page makes several.

Replication answers all four with one idea: keep a copy of the same data on several machines and ship every change to each copy. Each reason asks something different of the copies:

Reason	What the copy must do	What that implies
Durability	Hold every acknowledged write	The write reaches the copy before the client hears “ok”
Availability	Take over in seconds	Already caught up, plus a way to decide who leads
Read throughput	Serve reads	Readers accept that a copy can be slightly behind
Latency	Sit near the reader	Far from the writer, so the lag is longer

Replication does not scale writes: every copy applies every write. Spreading writes is partitioning, and real systems do both. Nor does it protect against mistakes: a DROP TABLE reaches every copy in milliseconds.

Copying data that never changes is easy. The subject is data that changes. When a write lands on one node, when do the others learn of it? Who may accept writes? And what does a reader see in between?

How it works

Step 1: Ship the log, not the data

A database already writes every change to a write-ahead log (WAL) before it changes its data files, so it can recover after a crash. Replication reuses that log. A second server connects, receives the stream of log records, and applies them in the same order. Two machines that start from the same state and apply the same changes in the same order end in the same state.

Every record has a position in the log, its log sequence number (LSN). A follower’s position says exactly how far behind it is. Say the leader is at LSN 9,400, and the follower has applied 9,350. Then the follower lags by 50 records. Lag in bytes, and lag in seconds since the last applied commit, are the two numbers every replicated database reports.

A new copy starts from a snapshot of the data files (Postgres pg_basebackup, or a disk snapshot), taken at a known LSN. Then it streams the log from there. For 2 TB at 200 MB/s, that takes about 3 hours. And the leader must keep every log record written meanwhile.

Step 2: One leader takes the writes

If any copy could accept writes, two could accept conflicting ones at the same moment. So exactly one node, the leader (primary), accepts writes, decides their order and ships it to the followers (replicas, standbys). Reads can go to any copy.

Single-leader replication

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database
• Cache
• Queue or stream
• Background worker

Postgres, MySQL, SQL Server Always On, MongoDB replica sets, Redis and Kafka partitions all work this way. There is never a conflict to resolve, because one node orders writes. What goes wrong comes from two questions: when is a write safe, and what happens when the leader dies.

Step 3: Decide when a write counts

The leader can tell the client “committed” at three points, and each one decides what a crash can lose.

Where the acknowledgment happens

Scroll or pinch to zoom, drag to pan

• Color key
• Database
• Service

• Asynchronous. Acknowledge after the local flush, ship afterwards. As fast as one node, but if the leader dies, what it had not shipped is lost, though the client was told it succeeded.
• Semi-synchronous. Wait for one follower (or any one of a set) to confirm it has received and logged the record, not applied it. Every acknowledged write is in two places, for one extra round trip, about 1 ms across zones.
• Fully synchronous. Wait for every follower. Every commit pays for the slowest copy, and one dead follower stops all writes.

The amount of acknowledged data a failure can lose is the recovery point objective (RPO). Asynchronous replication has an RPO equal to the lag at the moment of the crash: usually milliseconds, sometimes minutes. Semi-sync gives RPO 0 for the loss of any one machine, as long as the follower keeps confirming (the box below shows when it stops). The time until writes work again is the recovery time objective (RTO), decided by failover in step 4.

Try it. Leave the mode on Async, and set the lag to 2 seconds. Press Write 3, read from node C, then crash node A before the writes arrive. Then do the same in Semi-sync and in Sync. And in Sync, crash node C first.

AsyncSemi-syncSync

Lag 1.0 s

Async: the leader acknowledges as soon as the write is on its own disk.

Node Afollower

zone a

Read ACrash

Node Bfollower

zone b, lag × 1

Read BCrash

Node Cfollower

region 2, lag × 2

Read CCrash

WriteWrite 3Reset

Epoch

1

Acked

0

Stale reads

0

Acked, then lost

0

Write a few times, read from C, then crash the leader.

Lag is slowed down so you can watch it. Real asynchronous lag is usually milliseconds, and seconds or minutes when a follower falls behind.

In async mode, writes the new leader never received are gone, though the client heard “ok” for each. In semi-sync mode nothing acknowledged is lost: every acknowledged write is on a follower, and the election picks the most up-to-date one. In sync mode with a node down, nothing is lost and nothing is written either.

Go deeperHow Postgres and MySQL spell these modes2 min

Setting	Meaning
Postgres synchronous_commit = local	Async: wait for the local flush only
Postgres = remote_write	Wait until the standby has written the WAL to its OS, not yet flushed. Survives a standby process crash, not a standby power loss
Postgres = on (with standbys named)	Wait until the standby has flushed the WAL to disk
Postgres = remote_apply	Wait until the standby has applied it, so a read on the standby sees it
synchronous_standby_names = 'ANY 1 (b, c)'	Quorum commit: whichever of b or c confirms first. One standby can die without blocking writes
MySQL semi-sync AFTER_SYNC	The default since 5.7. The commit becomes visible only after a replica has the binlog event, so no reader sees a write that a failover could lose
MySQL semi-sync AFTER_COMMIT	Older behavior: other sessions can read the commit before it is replicated, and a failover can make it vanish
rpl_semi_sync_source_timeout (default 10 s)	After this long without a replica ack, MySQL falls back to async and keeps writing

That last line matters: MySQL semi-sync is RPO 0 only while a replica keeps up. After a 10 second stall it quietly becomes asynchronous until a replica catches up again.

Step 4: Fail over without two leaders

Failover is the most dangerous operation in replication. It has four parts:

1. Detect. Nobody can tell a dead leader from a slow or unreachable one. So detection is a timeout on heartbeats or a lease, typically 10 to 30 seconds. Shorter timeouts fail over on GC pauses. Longer ones extend every real outage.
2. Choose the follower with the most recent log. With a synchronous standby, that is the standby.
3. Reconfigure. A proxy, DNS name, virtual IP or service registry now points writes at the new leader.
4. Fence the old leader. A 20 second GC pause or a partition looks exactly like a crash from outside. When the old leader wakes up, it still believes it leads.

If two nodes both accept writes, that is split brain: two histories diverge, and merging them is manual and sometimes impossible. The defense makes leadership a numbered, expiring grant that the rest of the system checks.

• Majority. Elect through a majority of an odd-sized group (3 or 5 nodes, or an etcd or ZooKeeper cluster). Two majorities always share a member, so only one leader wins a given term. Two data nodes alone can never do this safely: add a third voter, even a witness with no data.
• Epochs. Each new leader gets a number one higher than the last: a Raft term, a Kafka leader epoch, a Postgres timeline. Every node rejects messages from an epoch lower than the highest it has seen. This is a fencing token.
• Leases. A leader that cannot renew its lease (say 10 seconds) stops taking writes before it runs out. And no successor is elected until it has run out. This needs clocks that run at about the same rate, not clocks that agree.
• STONITH (“shoot the other node in the head”). Before promoting, power the old leader off through a management API, when nothing else can be trusted.

A paused leader wakes up and is fenced by the epoch

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

The check is a few lines at every node that accepts writes:

JLeader.javaPyleader.pyC#Leader.csGoleader.goTSleader.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.ArrayList;
import java.util.List;

// One node of a three-node group. Every node remembers the highest epoch it has seen. That one
// number is the fence: a deposed leader that wakes from a pause still believes it leads, but no
// node that has seen a newer epoch takes its writes.
class Replica {
    final String name;
    private long epoch = 0;
    private final List<String> log = new ArrayList<>();

    Replica(String name) {
        this.name = name;
    }

    // A vote in an election. Once a node has seen epoch 2, epoch 1 is over for it.
    synchronized void observe(long newEpoch) {
        epoch = Math.max(epoch, newEpoch);
    }

    // Returns false to tell the sender that a newer leader exists.
    synchronized boolean append(long senderEpoch, String entry) {
        if (senderEpoch < epoch) return false; // a stale leader: reject, never merge
        epoch = senderEpoch;
        log.add(entry);
        return true;
    }

    synchronized List<String> log() {
        return List.copyOf(log);
    }
}

class Leader {
    private final Replica self;
    private final List<Replica> followers;
    private final long epoch;
    private boolean leading = true;

    Leader(Replica self, List<Replica> followers, long epoch) {
        this.self = self;
        this.followers = followers;
        this.epoch = epoch;
    }

    boolean isLeading() {
        return leading;
    }

    // Acknowledge only when a majority of the group, the leader included, holds the entry.
    boolean write(String entry) {
        if (!leading || !self.append(epoch, entry)) return false;
        int acks = 1;
        for (Replica f : followers) {
            if (f.append(epoch, entry)) {
                acks++;
            } else {
                leading = false; // someone has seen a newer epoch: step down at once
                return false;
            }
        }
        return acks * 2 > followers.size() + 1;
    }
}

class Main {
    public static void main(String[] args) {
        Replica a = new Replica("A"), b = new Replica("B"), c = new Replica("C");

        // Epochs come from the election: a Raft term, or a compare-and-set in etcd or
        // ZooKeeper. Two leaders can never be handed the same number.
        for (Replica r : List.of(a, b, c)) r.observe(1);
        Leader leaderA = new Leader(a, List.of(b, c), 1);
        System.out.println(leaderA.write("w1")); // true

        // A stalls in a 20 second GC pause. Its lease expires and B wins epoch 2 with C's vote.
        b.observe(2);
        c.observe(2);
        Leader leaderB = new Leader(b, List.of(a, c), 2);

        // A wakes up, still sure it leads, and writes.
        System.out.println(leaderA.write("w2")); // false: B and C have seen epoch 2
        System.out.println(leaderA.isLeading()); // false: A stepped down

        System.out.println(leaderB.write("w3")); // true: A takes epoch 2 and follows
        System.out.println(c.log()); // [w1, w3]
        // A still holds w2, which was never acknowledged. On rejoin it must truncate its log
        // back to the last entry it shares with the leader (w1), then copy w3 again.
        System.out.println(a.log()); // [w1, w2, w3]
    }
}

The old leader’s leftover w2 shows the last step of every failover. A rejoining node must truncate its log to the last entry it shares with the new leader. Postgres uses pg_rewind. MongoDB writes such entries to a rollback directory. Raft overwrites them. Under a majority rule, they were never acknowledged. Under async replication, they may have been. That is exactly the data an async failover loses.

Here is the whole setup for one database: read replicas, a synchronous standby in a second zone, and a failover manager that holds the leader’s lease. Step through the stages. Then follow a commit and a read.

Leader-follower replication · architecture

Stage 2: A synchronous standby and automatic failover

Grow itTrace a request

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

12

Reads per second:

40,000

10k40k peak2×10×

Stage 2 of 3

A synchronous standby and automatic failover

Why this stage

The product promises that an acknowledged order is never lost (RPO 0) and that writes recover within 30 seconds (RTO).

What we add

A standby in another zone that must confirm every commit before the client hears “ok”, and a failover manager that holds the leader lease in etcd and bumps an epoch on every promotion.

What it costs

About 1 ms more per commit for the cross-zone round trip, and writes stall if the standby is down and nothing replaces it.

At 40k reads per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One database node serves about 15,000 point reads a second, or applies about 4,000 writes a second.

Why replicas do not help writes

• Every replica applies every write. So replicas scale reads, never writes.
• At 2× traffic, the leader and every replica hit 100% on writes at the same time.

Fresh reads

• About 10% of reads need fresh data, so they go to the leader.

Leader-follower replication · architecture

Trace a request: Commit an order

Commit an orderRead your own write

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Play

PreviousNext

Commit an order: step 1 of 5

Users to App servers

POST /orders with an idempotency key.

2 ms of a 20 ms budget

1. Users → App servers2 ms
2. App servers → Leader2 ms
3. Leader → Sync standby1 ms
4. Leader → Read replicasasync
5. Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Step 5: Live with replication lag

Asynchronous followers are usually milliseconds behind. But some things make them fall seconds or minutes behind: a write burst, a long-running query on the follower, a slow disk, or a cross-region link. Then a reader sees the past. Three anomalies show up in products, each with a standard fix.

Anomaly	What the user sees	Fix
Read-your-writes	Edits their profile, the page reloads from a follower, the old name is back	Remember the LSN of the user’s last write and read only from a copy that has applied it, else the leader
Monotonic reads	Refreshes twice, each from a different follower: a comment appears, then disappears	Pin each user to one replica (hash of user id), or track the highest LSN they have seen and never go below it
Consistent prefix	Sees an answer before the question it replies to	Keep causally related writes in one partition, or carry the causal LSN with each write

The third one needs partitions. Within one log, order is always kept. With many partitions, each is replicated separately. The question lands on partition 3, and the answer on partition 7. And a reader can see partition 7 catch up first.

Read-your-writes broken by a lagging follower, then fixed

Scroll or pinch to zoom, drag to pan

• Color key
• Database
• Service

One number in the session gives both read-your-writes and monotonic reads: the highest log position this user has written or read. Any replica at or past it can answer.

JReadRouter.javaPyread_router.pyC#ReadRouter.csGoread_router.goTSread-router.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.List;

// What the router knows about each replica: how far through the leader's log it has applied.
// Postgres reports it as pg_last_wal_replay_lsn(), MySQL as the executed GTID set.
class ReplicaInfo {
    final String name;
    long appliedLsn;

    ReplicaInfo(String name, long appliedLsn) {
        this.name = name;
        this.appliedLsn = appliedLsn;
    }
}

// Carried with the user, in a cookie or token, so it works across app servers. One number
// gives two guarantees: the highest position this user has written or read.
class Session {
    long minLsn = 0;
}

class ReadRouter {
    private final List<ReplicaInfo> replicas; // nearest first
    private long leaderLsn;

    ReadRouter(List<ReplicaInfo> replicas, long leaderLsn) {
        this.replicas = replicas;
        this.leaderLsn = leaderLsn;
    }

    // A commit on the leader returns its log position. The session remembers it.
    void write(Session s) {
        leaderLsn++;
        s.minLsn = Math.max(s.minLsn, leaderLsn);
    }

    // Any replica that has applied the session's position can serve the read: it holds the
    // user's own writes (read-your-writes) and everything they saw before (monotonic reads).
    String read(Session s) {
        for (ReplicaInfo r : replicas) {
            if (r.appliedLsn >= s.minLsn) {
                s.minLsn = Math.max(s.minLsn, r.appliedLsn);
                return r.name;
            }
        }
        // Nobody is caught up. The leader always is. Waiting a few ms for a replica is the
        // other option, and keeps load off the leader when lag is short.
        s.minLsn = Math.max(s.minLsn, leaderLsn);
        return "leader";
    }
}

class Main {
    public static void main(String[] args) {
        ReplicaInfo near = new ReplicaInfo("replica-1", 100);
        ReplicaInfo far = new ReplicaInfo("replica-2", 90);
        ReadRouter router = new ReadRouter(List.of(near, far), 100);
        Session alice = new Session();

        System.out.println(router.read(alice)); // replica-1: Alice has written nothing yet

        router.write(alice); // her profile update commits at LSN 101
        System.out.println(router.read(alice)); // leader: no replica has 101 yet

        near.appliedLsn = 101; // replica-1 catches up a few ms later
        System.out.println(router.read(alice)); // replica-1

        // replica-1 is drained for maintenance. replica-2 is still at 90, so it would show
        // Alice an older profile than the one she just saw: the router skips it.
        near.appliedLsn = -1;
        System.out.println(router.read(alice)); // leader
    }
}

Simpler rules work for many products. Read a user’s own profile from the leader. Or send a user’s reads to the leader for 10 seconds after a write. Across devices, the LSN must live with the account, not in one browser’s cookie. The guarantees themselves are defined in consistency models.

Step 6: More than one leader

A single leader lives in one region: users elsewhere pay a cross-region round trip on every write, and a region failure stops writes. Multi-leader replication puts a leader in each region, accepting local writes and shipping them to the others asynchronously. An offline-capable app (a calendar, a notes app, a shared document) is the same shape: every device is a leader.

The price is write conflicts. Two regions can change the same row at the same moment, both succeed locally, and the conflict is discovered only when the changes cross. By then both users have been told “saved”.

Two leaders accept conflicting writes

Scroll or pinch to zoom, drag to pan

• Color key
• Database

There are four ways out, compared under options:

• Avoid conflicts, by giving each record a home region.
• Keep the last write, by timestamp.
• Use data types whose merges always converge (CRDTs).
• Keep both versions, and let the application merge them.

Multi-leader also breaks constraints a single leader enforces for free. Two regions can each accept the last seat, the same username, or a withdrawal against the same balance, because each checked its own copy. Anything that must be unique or must not go negative needs one owner.

Step 7: No leader at all

Leaderless replication (Amazon’s Dynamo, Cassandra, ScyllaDB, Riak) has no leader. A coordinator sends each write to all N replicas of the key and waits for W acks. A read waits for R answers and keeps the newest version.

If R + W > N, every set of R replicas shares at least one member with every set of W. So a read reaches at least one replica that has the latest successful write. With N = 3, W = 2, R = 2, one replica can be down, for reads and writes alike. There’s no election, and no failover pause.

N = 3, W = 2, R = 2 with one replica down

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

JQuorumClient.javaPyquorum_client.pyC#QuorumClient.csGoquorum_client.goTSquorum-client.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

// A value with the version that wrote it. Here the version is a client timestamp (last write
// wins); Dynamo-style stores use vector clocks to also detect concurrent writes.
record Versioned(long version, String value) {}

class Node {
    final String name;
    boolean up = true;
    final Map<String, Versioned> data = new HashMap<>();

    Node(String name) {
        this.name = name;
    }

    // A replica keeps the newer version. Writes are idempotent and order does not matter.
    void store(String key, Versioned v) {
        data.merge(key, v, (old, neu) -> neu.version() > old.version() ? neu : old);
    }
}

// Leaderless replication: the client (or a coordinator) talks to all N replicas and waits
// for W acks on a write and R replies on a read. With R + W > N every read quorum
// overlaps every write quorum in at least one node, so the read sees the newest write.
class QuorumClient {
    private final List<Node> nodes;
    private final int w;
    private final int r;

    QuorumClient(List<Node> nodes, int w, int r) {
        this.nodes = nodes;
        this.w = w;
        this.r = r;
    }

    boolean put(String key, String value, long version) {
        int acks = 0;
        for (Node n : nodes) {
            if (n.up) {
                n.store(key, new Versioned(version, value));
                acks++;
            }
        }
        return acks >= w; // fewer than W: the client sees a failure, though some nodes kept it
    }

    String get(String key) {
        List<Node> replied = new ArrayList<>();
        for (Node n : nodes) if (n.up && replied.size() < r) replied.add(n);
        if (replied.size() < r) throw new IllegalStateException("fewer than R replicas up");

        Versioned newest = null;
        for (Node n : replied) {
            Versioned v = n.data.get(key);
            if (v != null && (newest == null || v.version() > newest.version())) newest = v;
        }
        if (newest == null) return null;
        // Read repair: push the newest version to any replica that answered with an older one.
        for (Node n : replied) n.store(key, newest);
        return newest.value();
    }
}

class Main {
    public static void main(String[] args) {
        Node a = new Node("A"), b = new Node("B"), c = new Node("C");
        QuorumClient client = new QuorumClient(List.of(a, b, c), 2, 2); // N=3, W=2, R=2

        client.put("cart:42", "1 book", 1000);

        c.up = false; // C is down for a deploy
        System.out.println(client.put("cart:42", "2 books", 1005)); // true: A and B acked

        c.up = true;
        a.up = false; // C is back but missed v1005, and now A is down
        System.out.println(client.get("cart:42")); // 2 books: B overlaps the write quorum
        System.out.println(c.data.get("cart:42").value()); // 2 books: C was repaired by the read
    }
}

Without a leader, replicas drift apart after every failure, and three mechanisms bring them back:

• Hinted handoff. The coordinator keeps a write for a down replica as a hint. It replays it when the replica returns. (Cassandra keeps hints 3 hours.) A sloppy quorum goes further. A node outside the key’s N replicas takes the write, and counts toward W. So writes succeed during a partition. But R + W > N no longer guarantees overlap.
• Read repair. A read that finds an older version writes the newest one back, as in the diagram. It only fixes keys that get read.
• Anti-entropy. A background job compares replicas with Merkle trees. Each leaf hashes a range of keys. Each parent hashes its children. Replicas compare roots, and descend only where hashes differ. So a billion keys with 1,000 differences cost tens of thousands of hash comparisons, instead of a billion.

Comparing two replicas with Merkle trees

Scroll or pinch to zoom, drag to pan

• Color key
• Database
• Failure

Quorums give availability, not linearizability: last-write-wins timestamps, sloppy quorums and partially failed writes all let a quorum read return something other than the latest value. The full design, with the cases where overlap breaks, is worked through in Design a key-value store.

Options and tradeoffs

Topology: who accepts writes

Option	Writes go to	Conflicts	Failover	When it wins
Single leader	One node	None	Election plus promotion, 10 to 60 s	Almost always: relational databases, anything with transactions or unique constraints
Consensus (Raft, Paxos)	One leader, majority must ack	None	Built in, a few seconds	Leader-follower with automatic, provably safe failover: etcd, ZooKeeper, CockroachDB, Spanner
Multi-leader	A leader per region or device	Yes, resolved after the fact	None needed: other leaders keep going	Writes in several regions, offline clients, data that merges naturally
Leaderless quorums	Any W of N replicas	Yes, versions and repair	None needed: quorums tolerate loss	High write availability, simple key-value access, tolerance of stale reads

Consensus replication is a single leader whose log entries commit only once a majority has them, with the election built into the protocol. It is semi-sync with a majority quorum and a proof attached. Consensus covers how Raft does it.

TradeoffSingle leader with a synchronous standby, by default

We gain

One order of writes, no conflicts, transactions and unique constraints that work, and no lost acknowledged write when any one machine or zone dies.

We give up

Writes are limited to one node and one region. Failover pauses writes for 10 to 30 seconds. And the failover machinery is the riskiest code in the system.

Acknowledgment: who to wait for

Option	Added commit latency	RPO on leader loss	If a follower dies	When it wins
Async	None	The lag: ms to minutes	Nothing	Caches, analytics copies, read replicas, cross-region DR copies
Semi-sync, one standby	1 cross-zone RTT, about 1 ms	0 for one failure	Postgres: writes block. MySQL: async after a 10 s timeout	Most OLTP databases that must not lose orders or payments
Quorum (ANY 1 of 2, or majority)	The faster standby’s RTT	0 for one failure	Nothing, the other answers	The same, where a standby outage must not stop writes
Sync to every copy	The slowest copy’s RTT	0 for N − 1 failures	Writes stop	Rarely: two copies in one data center, where RTT is microseconds

TradeoffA quorum of two standbys, not one sync standby

We gain

RPO 0 for any one failure, and a dead standby no longer blocks every write.

We give up

A third copy of the data, and the failover manager must promote whichever standby actually confirmed the last commits.

What the log contains

Option	Ships	Strengths	Weaknesses	Used by
Statement-based	The SQL text: UPDATE ... SET n = n + 1	Small	NOW(), RAND(), triggers and auto-increment can produce different results on each copy	Old MySQL default; now mostly avoided
Physical (WAL shipping)	Byte changes to data pages	Exact copy, cheap to apply, every table and index included	Same major version and architecture on both sides; whole cluster or nothing; no filtering	Postgres streaming replication, Oracle Data Guard
Logical (row-based)	Row images: “row 42 of orders now has status = shipped”	Works across versions, filters tables, feeds other systems	More CPU, schema changes need care, sequences and DDL are not always included	MySQL binlog ROW (default since 5.7.7), Postgres logical replication, Debezium CDC

Logical replication is also how zero-downtime major upgrades work: replicate from Postgres 15 to Postgres 17, then switch. The same stream, read by a connector, is change data capture: the outbox pattern and search indexes are fed this way.

TradeoffPhysical replication for replicas, logical for everything else

We gain

Replicas that are byte-identical and cheap to apply, plus a row-level stream for upgrades, search and analytics.

We give up

Two mechanisms to monitor, and logical slots that hold back WAL on the leader if a consumer stops reading.

Resolving conflicts

Option	How	What it loses	When it wins
Home region per record	Route each key’s writes to one region	Write latency for users far from the home	Accounts, orders, anything with invariants
Last write wins	Highest timestamp survives	Concurrent writes, silently; skewed clocks pick wrong	Caches, sessions, values where any recent one is fine
CRDT	A merge that is commutative, associative and idempotent: each region counts its own increments and the total is the sum	Only types with such a merge; metadata grows	Counters, sets, collaborative text (Riak, Redis Active-Active, Automerge)
Siblings, app merges	Keep all concurrent versions, merge on read	Application complexity on every read path	Shopping carts, documents, where a merge has a meaning

DynamoDB global tables and Cassandra both default to last write wins. That’s fine for a user’s last-seen location. It’s wrong for an inventory count, where a clock skew of a few milliseconds silently picks which update survives. The collaborative editor problem compares CRDTs with operational transformation, for text.

TradeoffHome region per record over conflict resolution

We gain

No conflicts at all, and the invariants of a single leader, while each region still writes its own users’ data locally.

We give up

A user who travels writes across an ocean, and moving a record’s home is a migration with its own failure modes.

Go deeperMulti-leader topologies, and chain replication2 min

With more than two leaders, changes travel along a topology. All-to-all sends each change to every other leader. It’s the fastest, and the most tolerant of one link failing. But changes can arrive out of order, so an update can reach a region before the insert it modifies.

Circular and star topologies forward through intermediaries. They use fewer links. But one failed node breaks propagation for everyone downstream. And each change must carry the IDs of the nodes it has passed, so it is not forwarded forever.

Chain replication arranges copies in a line. Writes enter at the head, and flow down the chain. The tail acknowledges. Reads go to the tail, which only holds fully replicated data. It gives strong consistency with high throughput. And a failed middle node is simply skipped. Several key-value and object stores use variants of it.

Across regions

Option	Write latency	RPO if a region dies	Monthly cost to note	When it wins
Async replica in a second region	Unchanged	Seconds of lag	Transfer per GB, a second set of nodes	Disaster recovery and local reads
Sync across regions	Plus 60 to 150 ms per commit	0	The same, plus slower everything	Ledgers and exchanges that cannot lose a write
Majority across three regions	Plus the RTT to the nearer region	0	Three full copies	Spanner and CockroachDB style global databases
Home region per record, async elsewhere	Local for home users	Seconds, per region	Copies in every region	Global consumer apps

The transfer bill is easy to forget. A leader writing 10 MB/s of WAL ships 26 TB a month to each cross-region replica. AWS charges $0.02 per GB between most US and European regions (other pairs cost more). So that is about $520 a month per replica, before the second region’s machines. Cross-zone traffic you replicate yourself is billed too, at about $0.01 per GB in each direction. Managed services such as RDS Multi-AZ do not charge for their own replication traffic, though.

How it fails

Failure	What breaks	How you see it	Fix
Async failover loses writes	Acknowledged writes vanish; auto-increment IDs are handed out again	Rows missing after failover; duplicate key errors	Semi-sync for data that matters; never reuse IDs from the old leader
Split brain	Two leaders accept writes, histories diverge	Writes in both logs under different epochs	Majority election, epochs checked everywhere, leases, STONITH
False failover	A GC pause or network blip triggers promotion; writes pause twice	Failovers with no real fault; flapping	Timeouts of 10 to 30 s, require a majority to agree the leader is gone
Replica falls behind	Stale reads, then a replica useless for failover	Lag in seconds climbing, apply rate below the leader’s write rate	Route reads away from it; parallel apply; split the write load
Replication slot holds WAL	A dead replica or CDC consumer makes the leader keep WAL until its disk fills	Leader disk usage climbing, slot “retained WAL” growing	Alert on retained bytes, cap with max_slot_wal_keep_size, drop dead slots
Sync standby dies	Every commit waits	Commits stuck waiting for the standby	Quorum of two standbys; an automated, alerting fallback
Old leader rejoins with extra writes	Its log diverges from the new leader’s	Replication refuses to start	Rewind or truncate to the common point (pg_rewind), save the cut writes for review
Leaderless repair neglected	Deletes come back once tombstones are purged	Deleted rows reappearing	Run anti-entropy more often than the tombstone grace period (10 days in Cassandra)

Two real incidents show the first two rows. In September 2012, GitHub’s automated failover promoted a MySQL follower that was behind. The new leader handed out auto-increment IDs the old one had already used. Those IDs were also keys in Redis. So some private repository data was shown to the wrong users. In October 2018, a 43 second network partition led GitHub’s failover tooling to promote a leader in another region. Writes had landed in both regions. Restoring consistency took about 24 hours of degraded service.

The leader is the component whose failure matters most. Break it, and the others, in the drawing:

Leader-follower replication · architecture

Break it: what fails, who notices, what users see

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional

Break

The leaderThe sync standbyA read replicaThe failover managerThe link to region 2An app server

Failure: click any box to break it

The leader goes down

What breaks

Every write fails.

Reads from replicas keep working, frozen at the last applied LSN.

How we notice

The leader stops renewing its lease in etcd.

The lease runs out after about 10 seconds.

What users see

Writes fail for 10 to 30 seconds and clients retry with the same idempotency key.

What we do

The manager promotes the standby, which has every acknowledged commit, under epoch n+1.

The old leader is fenced: storage and peers reject epoch n, and it rejoins as a follower after rewinding its log.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Go deeperWhy a replica falls behind1 min

Replay of the log is often single-threaded. (Postgres replays WAL in one process.) So a leader writing with 32 cores can outrun a follower applying on one. A long query on a follower can block replay of the rows it is reading. Postgres waits up to max_standby_streaming_delay, 30 seconds by default, before cancelling the query.

One huge transaction, such as a backfill of 50 million rows, arrives as a burst. Every follower must apply it before anything after it. MySQL applies in parallel, with replica_parallel_workers. Postgres cannot, so the fix there is batching big writes.

Common mistake

Replicas are a backup

Replication copies mistakes as faithfully as data. A dropped table, a bad migration, or ransomware reaches every replica in milliseconds. In 2017, GitLab lost about 6 hours of production data. A directory was deleted on the primary, while replication was already broken. And several backup methods turned out not to work. Backups are separate copies at points in time, restored and tested on a schedule. A delayed replica, applying the log an hour late, is a useful middle ground for fast recovery from mistakes.

At scale

Numbers worth knowing:

Quantity	Typical value
Round trip within a zone / across zones / across a continent / across an ocean	0.1 ms / 0.5 to 2 ms / 30 to 70 ms / 70 to 200 ms
Async lag, healthy	Under 100 ms, usually a few ms
Async lag, under stress	Seconds to minutes
Automated failover, Patroni / a Raft group	10 to 30 s / a few seconds
Amazon RDS Multi-AZ failover	1 to 2 minutes; under 35 s for Multi-AZ clusters
Aurora storage	6 copies in 3 zones, writes need 4, reads 3
Aurora Global Database cross-region lag	Typically under 1 s
Rebuilding a 2 TB replica at 200 MB/s	About 3 hours
Kafka durable produce	acks=all, min.insync.replicas=2, replication factor 3

Every copy applies every write, so the write rate sets the ceiling. One node here applies about 4,000 writes a second, so at 2,000 every copy is half busy before serving a single read. Slide the traffic in the drawing: at 2× the leader and every replica reach 100% together, and no number of replicas fixes that.

At 10× (400,000 reads, 20,000 writes a second), reads alone would need about 25 replicas. But writes are now five times what one node applies. So no replica could keep up, and lag would grow without bound. That is a partitioning problem: 16 shards, at 1,250 writes a second each.

Applying writes then takes under a third of each copy. That leaves about 10,000 reads a second per copy. So each shard’s 25,000 reads need its leader, its standby, and 2 or 3 read replicas. Each replica is a separate stream the leader sends. So a shard that needs more than 5 to 10 replicas has them replicate from each other (cascading replication), instead of all from the leader.

At 100×, hundreds of replica groups mean a failover somewhere every week. So none can be manual. Systems this size make each shard a small Raft group. (CockroachDB splits data into ranges of a few hundred MB, each with its own.) And failover becomes routine.

What to monitor:

• Lag, in bytes and seconds, per replica, with an alarm well below what read-your-writes routing tolerates.
• Apply rate versus the leader’s write rate: a replica whose apply rate is below the write rate will never catch up.
• Retained WAL per replication slot, and leader disk headroom.
• Sync standby state: whether commits are actually waiting for a standby, or have silently become asynchronous.
• Failover count and duration, and the number of acknowledged writes lost per failover, ideally zero.
• For multi-leader and leaderless: conflicts resolved per second, hint backlog, and when each token range was last repaired.

In the interview

Interviewer asksYour async replica is 5 seconds behind and the leader dies. What happens, and what do you do?

Promoting it loses up to 5 seconds of acknowledged writes. And its counters may reissue IDs the old leader already used. Either promote now and accept the loss (fine for a feed, wrong for payments). Or wait to recover the old leader’s WAL, extending the outage. The real fix is to never face the choice for important data. A synchronous standby, or a quorum of two, in another zone gives RPO 0. Async copies serve reads only. After an async promotion, keep the old leader’s unreplicated writes for review, and never reuse its ID range.

Interviewer asksHow do you make sure there are never two leaders?

A node can always believe it leads, so make that belief harmless. Elect through a majority of an odd group (etcd, ZooKeeper, or Raft among the replicas), so one leader wins each epoch. Give the leader a lease and have it stop writing when it cannot renew. Have every follower and the storage reject writes from a lower epoch, so a paused leader that wakes up is refused. Where storage cannot check epochs, power the old leader off before promoting.

Interviewer asksA user updates their profile and the refresh shows the old one. Fix it without sending every read to the leader.

The read hit a lagging replica. Return the commit LSN with the write and keep it in the session. Route each read to a replica whose replay LSN is at or past it, else to the leader, or wait a few milliseconds. Storing the highest LSN the user has read in the same field also gives monotonic reads. Across devices, keep the LSN with the account, not in a cookie.

Interviewer asksWhy not make everything synchronous and avoid all of this?

Every commit then waits for the slowest copy. And any one copy failing stops all writes. So availability falls as copies are added. Across regions, it adds 60 to 150 ms per commit. The usual design is one synchronous copy nearby, or a quorum of two, for durability. Asynchronous copies serve reads and distant regions, with lag handled explicitly.

Common mistake

"Semi-sync means no data loss"

Only while a replica keeps acknowledging. MySQL falls back to asynchronous after 10 seconds without an ack and keeps writing. Postgres blocks instead, so operators often script a switch to async, with the same result. RPO 0 holds only if that fallback is forbidden or treated as an incident.

Common mistake

"Add read replicas to handle more writes"

Every replica applies every write, so replicas add read capacity only, and each adds a stream the leader must send. Write-heavy load needs partitioning, batching or a write-optimized store.

Mid-level covers

Puts read replicas behind the database, says they can be stale, and knows failover promotes one.

Senior adds

Picks an acknowledgment mode per data type, by its RPO. Explains fencing against split brain. Fixes read-your-writes with LSN routing. Knows replicas never scale writes. Names the failures that actually happen: async loss on failover, slots filling disks, a sync standby blocking writes.

Seen in

• Key-value store: leaderless N = 3 quorums, hinted handoff, read repair and Merkle tree repair.
• Message queue: a leader per partition with in-sync followers, leader epochs and a high watermark.
• Distributed cache: an async replica per shard, and the delete that async replication can lose.
• Stock exchange: a hot standby replaying the sequenced journal, with an epoch that fences a deposed sequencer.
• Collaborative editor: one owner per document, fenced by writing each revision only if it is absent.
• Dropbox: replicas skipped when behind the client’s cursor, and shard epochs after failover.
• Hotel reservation: a synchronous standby so a confirmed booking survives failover.
• Job scheduler: synchronous replication to a nearby region for RPO 0 on accepted jobs.
• Leaderboard: Redis replicas that take the rank reads.
• Metrics and monitoring: every sample on 3 ingesters, merged at query time.
• Payment system: synchronous replication on every payment, and CDC from the WAL weighed as the outbox relay.
• Proximity service: an index small enough to replicate instead of shard.
• Rate limiter: replicas for failover only, accepting a few ms of lost counts.
• Ride hailing: driver claims in a synchronously replicated database, so failover never double-assigns.
• URL shortener: read replicas behind the caches, and the primary for brand-new codes.
• Chat: a message counts as sent once it is on 2 of 3 replicas.
• Ticket booking: holds in Redis that a failover can lose, in front of a durable sale.

Recap

Leader-follower replication · architecture

The full design, stage by stage

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

123

Reads per second:

40,000

10k40k peak2×10×

Stage 3 of 3

A replica in a second region

Why this stage

A whole-region outage must not take the data with it, and readers in Europe pay 80 ms per round trip to a US region.

What we add

An asynchronous replica in region 2 that serves local reads and can be promoted if region 1 is lost.

What it costs

Cross-region transfer billed per GB, lag of 100 ms or more, and a regional failover that loses whatever had not crossed yet.

At 40k reads per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.The full setup. Step back through the stages and say what forced each one.

The numbers behind the busy bars

What one machine can do

• One database node serves about 15,000 point reads a second, or applies about 4,000 writes a second.

Why replicas do not help writes

• Every replica applies every write. So replicas scale reads, never writes.
• At 2× traffic, the leader and every replica hit 100% on writes at the same time.

Fresh reads

• About 10% of reads need fresh data, so they go to the leader.

Why. Durability, availability, read throughput, latency. Not write throughput, not protection from mistakes.

How. Ship the write-ahead log in order. A follower’s LSN says how far behind it is. New copies start from a snapshot plus the log.

When a write counts. Async: RPO equals the lag. Semi-sync or a quorum of standbys: about 1 ms more, RPO 0 for one failure. Fully sync: one dead copy stops writes.

Failover. Timeout, promote the most up-to-date copy, repoint clients, fence the old leader with a majority, epochs and leases. Rejoining nodes truncate to the common point.

Lag. Read-your-writes, monotonic reads, consistent prefix. A session LSN fixes the first two.

Other topologies. Multi-leader for regions and offline devices, paid for in conflicts: a home region, LWW, CRDTs or app merges. Leaderless N, W, R with R + W > N, repaired by hints, read repair and Merkle trees.

Numbers. Cross-zone RTT about 1 ms, cross-region 60 to 150 ms, failover 10 to 30 s, transfer about $0.02 per GB between regions.

Further reading

• On this site: consistency models, for the guarantees behind the lag anomalies. Consensus, for Raft. Partitioning, for scaling writes. And CAP and PACELC, for the latency and consistency trade every replicated system makes.
• Martin Kleppmann, Designing Data-Intensive Applications, chapter 5 (Replication): the clearest treatment of every topic on this page.
• Dynamo: Amazon’s Highly Available Key-value Store, SOSP 2007: sloppy quorums, hinted handoff, vector clocks and Merkle trees.
• PostgreSQL documentation, High Availability, Load Balancing, and Replication: streaming replication, synchronous commit levels and replication slots.