Multi-region and active-active

Running in more than one region buys three things:

• Users far from your first region get lower latency.
• The service survives losing a whole region.
• Data can stay in the country or bloc where a law or a contract requires it.

The cost is a second or third copy of everything, plus cross-region transfer. It adds routing, failover and drills. And it forces one hard question: where does each write go, and what happens to it when its region dies?

This page covers the pattern: the topologies, the numbers that decide between them, and how failover goes wrong. The mechanisms have their own pages:

• Replication for lag and conflicts.
• CAP and PACELC for what each topology can promise.
• Consensus for majorities.
• Load balancing and CDN for how a request reaches a region.

The problem it solves

A shop runs in us-east, in three availability zones. It promises 99.99% availability, which is 52 minutes of downtime a year. A third of its users are in Europe and a fifth in Asia and Australia. Three things go wrong at once.

• Latency. Sydney is about 200 ms from Virginia per round trip. A checkout page that makes four sequential calls after the TCP and TLS handshakes spends over a second on the network.
• Regional failure. Zones protect against a fire or power loss in one building. They don’t protect against a bad deploy of a regional control plane, or a storage service failing across the region. Outages of several hours have hit a region at every large cloud provider. One four hour outage spends four and a half years of a 99.99% budget.
• Data residency. A German bank customer’s contract says its data stays in the EU. India’s central bank requires payment system data to be stored only in India. A single US region cannot meet either.

Adding regions answers all three, at a price:

Cost	What it looks like
Hardware	1.3 to 2 times the servers: every region carries spare capacity to absorb another’s users (see the headroom math)
Cross-region transfer	About $0.02 per GB between AWS US and EU regions, more for other pairs. 10 MB/s of replication is 26 TB and about $520 a month per destination, before the second copy of any server
Write latency or conflicts	A write that must survive a region loss waits 10 to 230 ms for another region to confirm. One that does not wait can be lost or can conflict
Complexity	Routing by user, failover, fencing, reconciliation, drills, and every dependency (auth, config, secrets) run once per region

Three zones in one region handle machine and building failures at 1 to 2 ms between zones, and many 99.95% services stop there. The real question is how much of the system runs in each region, and how far behind each copy may be.

RPO and RTO

• RPO, recovery point objective: how much recent data you may lose, measured in time. RPO of 1 second at 3,000 writes a second means up to 3,000 acknowledged writes can vanish.
• RTO, recovery time objective: how long users are down, from failure to service restored.

RPO is set by how replication works: asynchronous replication loses its lag, synchronous or quorum replication across regions loses nothing. RTO is set by how failover works: detection, the decision, promotion, warming up capacity and moving traffic. Every topology below is a point on both dials.

How it works

Step 1: One region, with a way back

The cheapest multi-region design serves everything from one region and keeps enough in a second region to rebuild. The AWS disaster recovery guidance names four strategies, a useful vocabulary on any cloud:

Strategy	What runs in the second region	RPO	RTO	Cost over one region
Backup and restore	Nothing. Snapshots and archived logs sit in object storage	Minutes (log archive) to 24 h (nightly)	Hours	A few percent
Pilot light	Databases, replicating. App servers defined but not running	Seconds (replication lag)	Tens of minutes	10 to 20%
Warm standby	A scaled-down copy of everything, live, taking no user traffic	Seconds	Minutes	25 to 50%
Multi-site active-active	A full copy, serving its share of users all the time	Near zero to seconds	Near zero	100% or more

Between the last two sits a hot standby: a full-size copy that takes no traffic until failover (step 2). The restore time in the first row is simple arithmetic. A 2 TB database copied back at 200 MB/s takes about 3 hours. Then the archived log replays on top, and then caches start cold. Backups are still the only tier that protects against a bug or an operator that deletes data. Replication copies a DELETE as faithfully as an INSERT.

Step 2: Active-passive, a hot standby region

Active-passive keeps a full copy of the stack in region B. Region A takes all writes, and usually all traffic. The database ships its log to B asynchronously, so writes stay at in-region latency. When A fails, failover is five steps, and the RTO (recovery time) is their sum:

Failing over from region A to region B

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

Step	Typical time	What sets it
Detect	30 to 90 s	Probe interval times failures required; probes from outside A
Decide	0 automatically; 5 to 20 min with a human	Paging, joining a call, confirming it is really A and not the probes
Fence and promote	10 to 60 s	Revoke A’s right to write, then let B’s database take writes
Warm up	0 for hot standby; 10+ min for warm	Scaling out, filling caches
Move traffic	The DNS TTL (30 to 60 s) plus clients that ignore it	DNS, or seconds with anycast and a global load balancer

The RPO (how much data you lose) is the lag at the moment A died, times the write rate. Lag is a few hundred milliseconds on a quiet day. It grows under the load spikes that come before many outages. Replication covers promotion and fencing. Across regions, the same rules hold with longer timeouts. A 90 ms link that drops for 20 seconds looks exactly like a dead region.

TradeoffActive-passive with asynchronous replication

We gain

Writes at in-region latency, no conflicts (one region writes), and a simple story: a second copy, one switch, minutes of RTO.

We give up

A full second region that serves nobody. Seconds of acknowledged writes lost on failover. Far users paying the round trip to A on every request. And a cold standby path that breaks on first use, unless it is drilled.

Step 3: Get each user to a region

Once more than one region serves traffic, something must pick a region for each request. It must also change that pick when a region fails. Networking covers DNS, and the CDN page covers anycast. The choices are:

Mechanism	How it picks	Failover time	Wins when
GeoDNS / latency-based DNS	The authoritative DNS answers with the region nearest the resolver	Detection plus TTL, with a tail of clients that cache longer	Almost every web API: cheap, works everywhere
Anycast	Every region announces the same IP; BGP delivers to the nearest	Seconds to a minute: BGP withdraws the route	DNS, CDNs, edge proxies; UDP and short TCP flows
Global load balancer	Anycast front ends near users, which proxy to a healthy region over the provider’s backbone	Seconds, with health checks per backend	HTTP APIs where a TTL tail is not acceptable
Client-side choice	The app holds a region list and measures, or the server tells it its home region	As fast as the client notices	Mobile apps and SDKs you control

Nearest is not always right. Writes may belong to a home region (step 4). Residency may require a particular region (step 7). And the nearest survivor may be full. So anycast or GeoDNS reaches a nearby gateway, and the gateway decides which region owns the request.

Step 4: Active-active with a home region per user

Active-active means every region serves users all the time. The question is which region accepts which writes. Most products land on one answer, and the problem pages on this site keep choosing it: every user, tenant or document has one home region. It sits near most of its activity, and only that region accepts its writes. Other regions hold asynchronous copies, for local reads and failover.

Requests routed by home region

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

The gateway needs the home region before it can route. Three ways to know it:

• A directory lookup: user or tenant to home region, replicated to every region and cached in the gateways. Read on every request, written almost never.
• Encoded in the identifier: region bits inside the ID, as the unique ID generator does. No lookup, but moving a user means a new ID or an override table.
• A claim in the session token: signed at login. Moving a user waits for the token to refresh.

Inside one home, it is ordinary single-leader replication: linearizable per user, with no conflicts. Now take a cross-home interaction, like Ana messaging Bob. It writes in Ana’s home, and reaches Bob’s region asynchronously. That’s fine for messages, follows and likes. It’s wrong for anything that must be atomic across both. Those need one owner (both rows in one home), or a distributed transaction across regions.

Failure is contained. When eu-west dies, only EU-homed users lose writes. That lasts until their data is promoted where its copy lives. With three regions, that’s a third of users.

The directory below maps homes to owners, with an epoch. It fails a home over to the next region its residency rules allow. Each copy refuses writes from an older epoch. So an old owner that comes back cannot overwrite the new one:

JRegionRouter.javaPyrouter.pyC#RegionDirectory.csGodirectory.goTSdirectory.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

// Region-aware routing: every user has a home region, writes go to the region that currently owns
// that home, failover moves ownership to the next allowed region with a higher epoch, and copies of
// the data refuse writes stamped with an older epoch.
import java.util.*;

public class RegionRouter {
    static final List<String> REGIONS = List.of("us-east", "us-west", "eu-west", "eu-central");

    // Failover order per home region, nearest first. Planned in advance, not chosen during an outage.
    static final Map<String, List<String>> FAILOVER = Map.of(
            "us-east", List.of("us-west", "eu-west"),
            "us-west", List.of("us-east", "eu-west"),
            "eu-west", List.of("eu-central", "us-east"),
            "eu-central", List.of("eu-west", "us-east"));

    // Residency narrows the failover list: data homed in the EU may only be served from the EU.
    static final Map<String, Set<String>> ALLOWED = Map.of(
            "EU", Set.of("eu-west", "eu-central"),
            "ANY", Set.copyOf(REGIONS));
    static final Map<String, String> RESIDENCY = Map.of(
            "us-east", "ANY", "us-west", "ANY", "eu-west", "EU", "eu-central", "EU");

    // One region's copy of one home region's data. It remembers the highest epoch it has seen and
    // refuses anything older, so a deposed owner cannot overwrite the new owner's data.
    static class Copy {
        int epoch = 0;
        final Map<String, String> rows = new HashMap<>();

        boolean write(int epoch, String key, String value) {
            if (epoch < this.epoch) return false; // fenced: a newer owner exists
            this.epoch = epoch;
            rows.put(key, value);
            return true;
        }
    }

    record Owner(String region, int epoch) {}

    // The control plane: who owns writes for each home region, and at which epoch. In production it
    // lives in a store replicated by consensus across three regions, so a region that is cut off
    // can neither change it nor pretend it changed.
    static class Directory {
        final Set<String> healthy = new HashSet<>(REGIONS);
        final Map<String, Owner> owner = new LinkedHashMap<>(); // insertion order: a stable log
        // Every region that may own a home keeps a copy of its data (replicated asynchronously).
        final Map<String, Map<String, Copy>> copies = new HashMap<>();

        Directory() {
            for (String home : REGIONS) {
                owner.put(home, new Owner(home, 1));
                Map<String, Copy> c = new HashMap<>();
                c.put(home, new Copy());
                for (String r : FAILOVER.get(home)) c.put(r, new Copy());
                copies.put(home, c);
            }
        }

        void failOver(String dead) {
            healthy.remove(dead);
            for (String home : REGIONS) {
                Owner o = owner.get(home);
                if (!o.region().equals(dead)) continue;
                Set<String> allowed = ALLOWED.get(RESIDENCY.get(home));
                Optional<String> target = FAILOVER.get(home).stream()
                        .filter(r -> healthy.contains(r) && allowed.contains(r))
                        .findFirst();
                if (target.isEmpty()) {
                    // Pausing writes keeps the residency promise;
                    // serving EU data from the US would break it.
                    System.out.println("  " + home + ": no allowed region left, writes pause");
                    continue;
                }
                // The new owner claims the copy with a higher epoch before taking any write.
                owner.put(home, new Owner(target.get(), o.epoch() + 1));
                copies.get(home).get(target.get()).epoch = o.epoch() + 1;
                System.out.println("  " + home + ": owner " + dead + " -> " + target.get()
                        + ", epoch " + (o.epoch() + 1));
            }
        }
    }

    // Runs in every region's gateway. Looks up the user's home, then the home's current owner.
    static class Router {
        final Directory dir;
        final Map<String, String> homes; // user -> home region, chosen at sign-up

        Router(Directory dir, Map<String, String> homes) {
            this.dir = dir;
            this.homes = homes;
        }

        String write(String user, String key, String value) {
            String home = homes.get(user);
            Owner o = dir.owner.get(home);
            if (!dir.healthy.contains(o.region()))
                return user + ": unavailable (home " + home + " has no owner)";
            boolean ok = dir.copies.get(home).get(o.region()).write(o.epoch(), key, value);
            return user + ": " + (ok ? "ok" : "fenced") + " via " + o.region() + " at epoch " + o.epoch();
        }
    }

    public static void main(String[] args) {
        Directory d = new Directory();
        Router router = new Router(d, Map.of("ana", "eu-west", "bob", "us-east"));
        System.out.println(router.write("ana", "cart", "book"));
        System.out.println(router.write("bob", "cart", "lamp"));

        System.out.println("eu-west stops answering health checks");
        d.failOver("eu-west");
        System.out.println(router.write("ana", "cart", "book, pen"));

        // eu-west was cut off, not dead. Its old primary still thinks it owns the EU-west users and
        // ships a write it accepted at epoch 1. The new owner's copy has seen epoch 2 and refuses it.
        boolean shipped = d.copies.get("eu-west").get("eu-central").write(1, "cart", "book, mug");
        System.out.println("old eu-west primary replicates at epoch 1: " + (shipped ? "accepted" : "fenced"));

        System.out.println("eu-central fails too");
        d.failOver("eu-central");
        System.out.println(router.write("ana", "cart", "book, pen, ink"));
        System.out.println(router.write("bob", "cart", "lamp, rug"));
    }
}

The failover order is planned ahead, so the decision during an outage is mechanical. The directory bumps the epoch from a consensus store spread over three regions, so a region cut off from the others cannot. When only non-EU regions are left, EU-homed writes pause instead of moving: that is what a residency promise costs in availability.

TradeoffA home region per user or tenant

We gain

Local write latency for most users, no conflicts to merge, single-leader correctness per user, and a regional failure that only touches the users homed there.

We give up

A directory to route by. A cross-region hop for travellers and shared objects. No atomic operations across homes. And seconds of writes lost for the failed region’s users.

Step 5: Every region writes everything

Multi-leader active-active lets any region accept any write, and ships changes to the others asynchronously. Every user writes locally. Failover is a routing change, with no promotion. But conflicts become normal: two regions change one record in the same second, and both say “saved”. Replication covers how to resolve them, and consistency models covers what readers then see. What matters here is which data can live this way:

Data	Multi-leader works?	Because
Counters, likes, view counts	Yes, as CRDT counters	Increments commute; each region owns its slot
Shopping carts, sets of tags, presence	Yes, with set CRDTs or add-wins merges	A merge that keeps both sides is acceptable
Profiles, settings	Usually, with last-writer-wins per field	Losing one of two edits in the same second is tolerable
Usernames, seats, inventory, balances	No	Two regions each check their own copy and both say yes
Anything with an invariant across rows	No	No merge rule can restore it after the fact

Real systems mix the two. Posts and profiles live in a home region. Counters are CRDTs, written everywhere. Usernames have one global owner. Managed stores offer the multi-leader half directly. DynamoDB global tables (in their default mode) and Cassandra across datacenters accept writes in every region, and reconcile by timestamp. The key-value store does this with LOCAL_QUORUM.

When a region dies under multi-leader, writes it accepted but had not shipped are stranded, not gone. They come back with the region and are merged. Some may conflict with what users did elsewhere in the meantime.

Step 6: Globally consistent databases

Spanner, CockroachDB and YugabyteDB put the replicas of each range in several regions. They commit through consensus: a write is durable once a majority of replicas has it. Lose one region, and a majority survives with every acknowledged write. RPO is zero, and failover is a leader election of seconds. Nobody writes conflict-resolution code.

The bill comes on every write: a commit waits at least one round trip, from the range’s leader to the nearest region that completes its majority.

Replica placement	Commit latency from the leader	Survives
3 zones in one region	1 to 3 ms	A zone
3 regions on one continent (Virginia, Ohio, Oregon)	10 to 50 ms	A region
3 regions on three continents (US, EU, Asia)	90 to 160 ms	A region
Leader’s region holds 2 of 5 replicas, 3 regions total	The nearest other region’s RTT	A region

The levers that make it usable are where leaders live and which reads skip them:

• Pin each row’s leader to its users’ region. CockroachDB’s REGIONAL BY ROW tables and Spanner’s leader placement do this. Local reads stay local. If the data must survive a region loss, writes pay one hop to the nearest other region. If surviving a zone loss is enough, they stay local. This is the home-region pattern, with the database doing the failover.
• Global tables for read-mostly data. Every region holds a consistent readable copy, and writes pay hundreds of ms for it. Fits currencies, product catalogs, feature flags.
• Stale reads. Reading the nearest replica at a timestamp a few seconds old needs no leader and no cross-region trip. Reads that cannot accept that go to the leader.

TradeoffA globally consistent database

We gain

RPO zero on a region loss. Automatic failover in seconds. Serializable transactions across regions. And no merge rules, so uniqueness, balances and inventory are safe everywhere.

We give up

Every write pays a cross-region round trip, 10 to 160 ms by placement, and a multi-statement transaction can pay it several times. A second region failure stops writes for ranges that lost their majority.

Go deeperCells, to limit what one failure reaches1 min

A region is a large blast radius for failures that have nothing to do with geography: a bad deploy, a poison message, a runaway tenant. Cell-based architecture splits the fleet into independent copies of the whole stack, each serving a fixed slice of users behind a thin router. Cells share nothing, so with 8 cells a failure reaches an eighth of users, and deploys roll out one cell at a time. Cells and regions compose: the notification system runs four cells in each of two regions. The cell router is the one shared piece, so it stays small, cached everywhere, and able to route with a stale map.

Step 7: Route by law, not only by distance

Residency adds a constraint to every step above: some users’ data may only live in some regions. Know where the rule comes from. GDPR does not require EU data to stay in the EU. It allows transfers out of the EEA, to countries with an adequacy decision, or under safeguards such as standard contractual clauses. Strict localization comes from customer contracts, sector rules and some national laws, like the Indian payment rule above.

• Home region from residency first, distance second. An EU tenant that promised EU-only storage gets an EU home, even if its users sit nearer to us-east.
• Failover inside the boundary. That tenant’s failover list names only EU regions. With one EU region, an EU outage pauses its writes, rather than moving them. The fix is a second EU region.
• The global directory holds no personal data. It maps an opaque ID to a region. Email lookups, search indexes and analytics that span regions must not copy residents’ records out.

Go deeperResidency beyond the database1 min

Backups, logs, traces, support tools, caches and CDNs hold copies of personal data too. Residency leaks most often through observability pipelines that ship every log to one central region. Each of these needs the same per-region rule as the database.

Try it

Three regions serve six cities, with two homed in each, at 500 writes a second per city. Pick a topology, and press Send writes. Watch where each write goes (blue), what it waits for before the acknowledgment (orange), and what is copied afterwards (green). Then kill a region. Read the write latency, the writes lost, the outage timeline, and the load on the regions left.

Topology

Backup and restoreWarm standbyActive-passiveActive-active, home regionsMulti-leaderGlobal consensus

Promotion decided by

AutomationA human

A dead region's users go to

The nearest regionBoth survivors

Async replication lag 1 s

Each region's peak utilization 60%

Kill a region (press again to bring it back)

Kill us-eastKill eu-centralKill ap-southeastSend writes

Write and its acknowledgmentQuorum round trip (before the ack)Async replication (after the ack)Round trips are approximate public figures, rounded.

Write latency, mean / worst user

-

Writes lost or stranded (RPO)

-

Outage (RTO)

-

Busiest surviving region

-

Load on each region after failover

Outage timeline for affected users

User	Home	Writes go to	Write latency

Things to try:

• Active-passive, then Active-active, home regions. With every write going to us-east the mean write latency is 136 ms and Sydney pays 203 ms. Home regions bring the mean to 84 ms and the worst to 118 ms, from the same three regions.
• Kill us-east under Active-passive. 3,000 acknowledged writes are gone: one second of lag at 3,000 writes a second. Every city is down for 2.5 minutes, or 17.5 if a human decides. With automation and Warm standby, the outage is 13 minutes, and eu-central runs at 240% until it scales up. With Backup and restore: 900,000 writes and 3.6 hours.
• Kill us-east under home regions. Only Los Angeles and São Paulo are affected, and 1,000 writes are lost. GeoDNS sends both cities to eu-central, now at 120%. With Both survivors each takes one city and sits at 90%: the 1.5 times of the headroom rule.
• Global consensus. Madrid’s writes cost 123 ms instead of 33, because each waits for us-east. Kill a region: nothing acknowledged is lost and the displaced cities are back in about a minute.
• Multi-leader, kill eu-central. Failover needs no promotion, but 1,000 writes are stranded in eu-central until it returns.

Options and tradeoffs

Topology

Topology	Write latency	RPO on a region loss	RTO	Conflicts	Cost	Wins when
Backup and restore	One region’s, far users pay	Minutes to hours	Hours	None	Lowest	Internal tools; 99.9% targets; protection against deletion
Pilot light or warm standby	One region’s	Seconds	Minutes to tens of minutes	None	+10 to 50%	A region outage can cost half an hour
Active-passive, hot	One region’s	Seconds	1 to 5 min (minutes more with a human)	None	About 2x	Users in one area, 99.99% target, no appetite for conflicts
Active-active, home regions	Local for home users	Seconds, failed region’s users only	1 to 5 min, a third of users	None	N regions + headroom	Global consumer and SaaS products: the usual answer
Active-active, multi-leader	Local for everyone	Seconds, stranded not lost	Routing only, about a minute	On every concurrent write	N regions + headroom	Commutative data: counters, carts, presence, caches
Global consensus database	Plus a cross-region RTT per write	Zero	Seconds (election) plus routing	None	Highest per write	Money, inventory, uniqueness across regions

TradeoffHome regions for most data, a consistent store for the few invariants

We gain

Local latency for nearly every request. A blast radius of one region’s users. And correctness only where it is needed: usernames, balances or stock in a consensus store, counters as CRDTs.

We give up

Three data paths to understand and operate. A directory on every request. And a design that must say, for each table, which path it is on.

Synchronous or asynchronous across regions

Synchronous replication to a region 10 ms away costs 10 ms a commit, and loses nothing. The job scheduler does this. To a region 90 ms away, few user-facing writes accept the cost. The common middle path is synchronous across zones, and asynchronous across regions. Watch the backlog as “data a failover would lose right now”. (Replication has the full table.)

Who decides to fail over

Decision	RTO added	Risk	Wins when
Automatic	Nothing	Fails over on a false alarm or a partition; flaps between regions	Stateless tiers; databases with fencing and RPO zero
Human decides, automation acts	5 to 20 minutes	Slower; depends on the on-call being reachable and calm	Asynchronous databases, where failover loses data
Human runs a manual runbook	30 minutes to hours	Steps skipped or done in the wrong order under stress	Nothing, once a system matters

Most teams split it. Routing away from a dead region is automatic. But promoting an asynchronous database is a human’s one-click decision. Promotion throws away unreplicated writes. And if the old primary is only cut off, not dead, it risks two primaries. Failovers that lose nothing automate both.

TradeoffAutomatic traffic failover, human-approved data promotion

We gain

Stateless traffic moves in about a minute. And the one step that loses data, or can split the brain, gets a person who checks that the old region is really gone.

We give up

The failed region’s users wait 5 to 20 minutes for that person, which a 99.99% budget absorbs about three times a year.

Go deeperFail over through the data plane, not the control plane1 min

DNS providers and cloud load balancers have two halves:

• The data plane answers queries and forwards packets, everywhere, with high availability.
• The control plane is the API that changes records and settings. It often runs in fewer places, sometimes in one region.

A runbook step like “call the API to change the record” depends on the control plane at the worst moment. During a large regional event, everyone else is calling it too. Prefer failover that the data plane does on its own. Use health-checked records that stop answering for a dead region. Or use routing controls set up in advance, flipped with a call designed for exactly this. The same goes for your own dependencies. An auth service, a secrets store or a feature flag service that lives in one region makes every region depend on it.

How it fails

Split brain after a partition

The link between regions A and B drops for two minutes. Probes in B see A as dead, B is promoted, and A, still alive and serving its local users, keeps accepting writes. Two primaries diverge, and the merge afterwards is manual. The defences:

A witness region and an epoch fence the old primary

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

• Decide with a majority across three places. A third region breaks the tie. It can be a witness, or the consensus store behind the directory. Then the side cut off from two of three cannot promote itself. And the primary that loses the majority stops accepting writes when its lease expires.
• Fence with an epoch. Every promotion bumps a number that every copy checks, so writes from the old primary are refused when it returns. The code above does this.
• Probe from several vantage points. “A is unreachable from B” is not “A is down”.

Failover that moves the servers but not the data

Region B’s app servers pass their health checks, so DNS sends everyone there. And every write fails. B’s database is still a read-only replica. Or B’s app servers still point at A’s database endpoint, across an ocean. Health checks must test the path a real request takes, including a write. Promote data before moving traffic. And drain a region whose data is not writable.

Replication lag eats acknowledged writes

Lag is a few hundred milliseconds on average, and tens of seconds in the bad minutes. Those are often the minutes before an outage. A forced failover drops whatever was in flight. The defenses:

• Alert on lag as live RPO.
• Let clients resubmit with idempotency keys.
• Keep the old primary’s disk, so its unreplicated tail can be reconciled.
• Use a synchronous or consensus store where loss is unacceptable.

The payment system rebuilds the lost window from the payment provider’s records, before the survivor takes payments.

The surviving regions need headroom

When one of N equal regions fails, its load spreads over the other N - 1. So each survivor carries N / (N - 1) times its normal load. To survive that, each region’s peak utilization must stay below (N - 1) / N:

Regions	Survivor load after one fails, and hardware vs one region at 100%	Highest safe peak utilization
2	2x	50%
3	1.5x	67%
4	1.33x	75%
5	1.25x	80%

That is the even case, and practice is worse:

• GeoDNS is not even. It sends all of a dead region’s users to the nearest survivor. In the demo, that doubles one region’s load. A planned failover map that splits them restores the 1.5x.
• Survivors start cold. So the databases behind their caches see more than 1.5x for minutes.
• Clients reconnect at once. That is why the chat and collaborative editor add jitter to reconnects.
• Autoscaling is minutes behind. So load shedding protects the region in the meantime.

Size for N / (N - 1) of the busiest hour, not the average.

Failover that was never tested

The standby path runs only during a disaster, so it rots: a missing secret, a firewall rule nobody copied, a lapsed capacity reservation. Game days fail a region on purpose, in production, at a scheduled time: first by draining its traffic, later by cutting it off. Netflix runs this as Chaos Kong, exercises that evacuate all traffic from one AWS region to the others. A drill measures the real RTO and RPO, and the first one nearly always finds something.

Failback is a second failover: the old region catches up, its stranded writes are reconciled, and traffic moves back on purpose, off-peak.

Go deeperEvery failure mode at a glance1 min

Failure	Blast radius	Detect	Fix
Split brain after a partition	Every record written on both sides	Two regions claim the same epoch; write divergence	Majority across 3 places, leases, epochs checked by every copy
Servers fail over without data	Every write after failover	Write success rate per region, synthetic writes	Promote before routing; health checks that write
Lag lost on failover	Writes in the lag window	Cross-region lag in seconds and bytes	Lag alerts, idempotent resubmits, keep the old disk, sync where it matters
Survivors overloaded	Every user, not just the failed region’s	Utilization vs (N - 1) / N; error rate after failover	Headroom, spread failover map, shedding, jittered reconnects
DNS stickiness	Clients that cached or kept connections	Traffic still arriving at the dead region	Short TTLs, re-resolve on reconnect, anycast or global LB
Untested standby path	The whole failover	Days since the last successful drill	Game days, regular region evacuations
A hidden single-region dependency	Every region	Dependency map; a drill that cuts the region	Run auth, config and secrets per region; fail over on the data plane

At scale

What	Typical figure
Round trip, zones in one region	0.5 to 2 ms
Round trip, US coast to coast	60 to 75 ms
Round trip, across the Atlantic	70 to 90 ms
Round trip, US to Asia or Europe to Asia	100 to 230 ms
Async replication lag	100 ms to a few seconds normally; tens of seconds under load
99.99% yearly budget	52 minutes: one human-decided failover can use a third of it

At 10 times, a region stops being one unit. Each region runs several cells, and homes are assigned to cells. Failover moves cells, which can be rehearsed one at a time. Cross-region transfer becomes a line item. Compress replication. Replicate only what another region needs: timelines in the news feed stay home, and are rebuilt on failover. And keep chatty internal calls inside a region.

At 100 times, there are 6 to 12 regions and N / (N - 1) becomes cheap: at 8 regions survivors need only 14% more. Failover maps are computed from live capacity, evacuation is routine, and the home directory and failover controller are the most protected systems in the company.

What to monitor:

• Cross-region replication lag per stream, in seconds and bytes: the data a failover would lose now.
• Write and read success per region from external probes, including a real write.
• Each region’s peak utilization against (N - 1) / N and against the failover map.
• Traffic share per region, to spot clients stuck on the old region after a move.
• Every epoch and ownership change in the directory, with who made it.
• Days since the last successful region drill, and the RTO and RPO it measured.

In the interview

Interviewer asksThe service runs in us-east. Leadership wants it multi-region. What do you do?

First I ask which reason applies: latency, surviving a region, or residency. I also ask what RPO and RTO the business accepts, because those pick the topology. For availability alone, with users in one area, a hot standby region with async replication works. It gives minutes of RTO and seconds of RPO, for about twice the cost. For global users, every region serves, and each user has a home region for writes. Writes are local, there are no conflicts, and a region failure touches only its own users. Usernames or balances go to one owner, or a consensus database. With 3 regions, each runs at most 67% at peak.

Interviewer asksUsers in the US and Europe both edit their own profiles and documents. How do you make writes active-active?

Not by letting every region write every document. Each user and document gets a home region, near most of its editors. Gateways everywhere route writes there, using a cached directory or region bits in the ID. Your own writes are local. A collaborator abroad pays one hop. Reads use the local copy when a second of staleness is fine. View counts and presence are written in every region, as CRDTs. Only a document with live editors on two continents makes me consider a CRDT for the document itself.

Interviewer asksThe home region dies with async replication. What do you lose, and how do you avoid two primaries?

The lag’s worth of writes, for users homed there. At one second and 1,000 writes a second, that’s about 1,000. Clients resubmit with idempotency keys, and we keep the dead primary’s disk to reconcile the rest. Against two primaries: promotion needs a majority across three regions. So a region that is only cut off cannot promote itself. And the old primary stops writing when its lease expires. Each promotion bumps an epoch that every copy checks, so the old primary’s late writes are refused.

Common mistake

"Active-active" as every region writing every record

Active-active only says that every region serves traffic. Letting every region write every record means conflicts on every concurrent update and no way to keep a username unique or a balance non-negative. Most active-active systems give each record a home region and write everywhere only the data that merges.

Common mistake

"DNS failover takes the TTL, so 60 seconds"

The TTL is only the last term in a sum. First comes detection (30 to 90 s), then any human decision (minutes), then database promotion, then the TTL. After that, some resolvers hold answers longer. And open HTTP/2 or WebSocket connections never re-resolve until they break. Quote RTO as that whole sum. Make clients re-resolve on reconnect. Use anycast or a global load balancer when the tail matters. And measure it in a drill.

Mid-level covers

Knows the reasons to go multi-region. Explains active-passive with async replication, and that failover loses the lag. Routes users with GeoDNS. Mentions conflicts if every region writes.

Senior adds

States RPO and RTO, and what sets each. Picks home regions for most data, CRDTs for mergeable data, and a consensus store for invariants. Prices a consistent write as a cross-region round trip. Fences failover with a majority and an epoch. Sizes survivors with N/(N-1) and a planned failover map. Keeps data promotion human-approved, and plans drills.

Seen in

• URL shortener, pastebin and proximity service: the read path in every region, creates or owner edits in one home region.
• News feed: a home region per user for timelines, feeds rebuilt by pull after a failover.
• Chat and collaborative editor: a home region per conversation or document, because order needs one decider.
• Key-value store: multi-leader across two datacenters with LOCAL_QUORUM, each sized to carry both.
• Job scheduler: synchronous replication to a region 10 ms away for RPO zero, with epochs on takeover.
• File sync: an asynchronous second region, gated so metadata never points at missing blocks.
• Rate limiter: a global limit split into regional shares that follow traffic.
• Unique ID generator: region bits in every ID, so regions mint with no coordination.
• Top-k and ad click aggregator: ingest in every region, then a global merge.
• Message queue and notification system: cells, with an asynchronous mirror or standby in a second region.
• Ticketmaster and hotel reservation: one home region per event or hotel, so two regions never sell the same seat or room.
• Payment system: the window lost in a failover rebuilt from the provider’s records.

Recap

Why. Latency for far users (70 to 230 ms a round trip across oceans). Surviving a region (one long outage spends years of a 99.99% budget). And residency. Cost. 1.3 to 2 times the hardware, cross-region transfer, write latency or conflicts, and operational work.

Two numbers. RPO is set by replication (async loses its lag), RTO by failover (detect, decide, promote, warm, move traffic).

Topologies.

• Backup and restore: hours.
• Pilot light: tens of minutes.
• Warm standby: minutes.
• Hot standby: minutes, with seconds of loss.
• Home regions: local writes, and a failure affects one region’s users.
• Multi-leader: only for mergeable data.
• Global consensus: RPO zero, with a cross-region round trip per write.

Routing. GeoDNS, anycast or a global load balancer reach a gateway. The gateway routes by home region. Residency narrows homes and failover lists.

Failure. Split brain without a majority and an epoch. Servers failing over without their data. Lag lost on failover. Survivors without N/(N-1) headroom. Standby paths never drilled. Automate traffic failover, and let a human approve promotions that lose data.

Further reading

• On this site: replication for async, sync and multi-leader mechanics, consistency models, CAP and PACELC, consensus, networking, load balancing and CDN. Scaling patterns that failover leans on: idempotency for resubmitted writes, backpressure for overloaded survivors, and resilience for timeouts and circuit breakers around a failing region.
• Disaster Recovery of Workloads on AWS: the four DR tiers, RPO and RTO, and failing over through the data plane.
• Spanner: Google’s Globally-Distributed Database, OSDI 2012: Paxos across regions, TrueTime and commit wait.
• Managing Critical State, Google SRE book: consensus across regions, replica placement and what it costs in latency.