CAP and PACELC

CAP is the most quoted result in distributed systems, and one of the most misquoted. This page states it the way it was proved. It shows what a system actually does when the network splits. Then it moves to PACELC. That describes the tradeoff a replicated system pays on every request, not only during an outage.

The ladder of guarantees lives on consistency models, and leaders, lag, quorums and conflict resolution on replication. This page covers only what CAP and PACELC add.

The problem it solves

A shop runs in two regions, East and West, about 70 ms apart. Every cart and every order lives in both, so either region can serve its local users and survive the loss of the other.

At 14:02 a fiber cut between the regions drops every packet on the link. Both regions are up. Their load balancers pass health checks, their databases answer, and their users keep clicking. Only the link is gone. The incident lasts 3 minutes.

During those 3 minutes, a user in West presses “Add to cart” and a request lands on West’s replica. That replica has two options, and only two:

1. Answer now, from what West knows. The write is accepted, and East will not see it until the link heals. A user reading the same cart in East gets an older cart.
2. Refuse (or wait) until it can confirm with East. The user sees an error or a spinner.

There is no third option that answers now, and also guarantees every reader sees the newest cart. The information needed for that is on the other side of a cut wire. That is the CAP theorem. It applies to every request that touches data replicated across the cut.

Now take the same shop on a normal day with no partition. Every write in West can either wait 70 ms for East to confirm, or be acknowledged locally and copied to East a moment later. The first option is slower on every request. The second leaves a window in which East reads an older cart. That is PACELC’s “else” half, and it matters more, because it is paid millions of times a day while partitions happen for minutes a year.

How it works

Step 1: The three words, as they were defined

Eric Brewer stated CAP as a conjecture in a keynote in 2000. Seth Gilbert and Nancy Lynch proved a precise version in 2002. Their definitions are narrower than the slogan:

Letter	Gilbert and Lynch’s definition	What it is often confused with
C	Linearizability: every operation appears to happen at one instant, on one copy, in real-time order	“Consistent” in ACID (constraints hold), or “eventually the same”
A	Every request received by a non-failing node gets a response (not an error), eventually	“99.99% uptime” or “fast”
P	The network may lose any number of messages between nodes	A feature you can choose to have

Three details matter:

• C is the strongest single-object model. Causal consistency, read-your-writes and the other session guarantees are weaker, and can stay available during a partition. CAP says nothing against them.
• A is total, and has no deadline. If any live node returns an error, or never answers, the system is not available in CAP’s sense, even if 99% of users are fine. And a reply after 30 seconds still counts, which no product manager would accept.
• P is a property of the network, not of the system. It describes what the environment may do to you. The model is a single read/write register replicated on several nodes, which is why the result reaches far: almost every database contains that register.

Step 2: The proof fits in one picture

Take the smallest replicated system: two nodes, G1 and G2, each holding a copy of x, initially v0. Cut the link between them. A client writes v1 to G1. Then a client reads x from G2.

Why no system can be both C and A during a partition

Scroll or pinch to zoom, drag to pan

• Color key
• Service

Availability forces G1 to accept the write, and G2 to answer the read. Linearizability forces the read to return v1, because the write finished before the read started. G2 has no way to learn v1, since every message is lost. So G2 must break one of the two. The proof is that short. And it holds for any number of nodes. Put the cut anywhere, send the write to one side, and the read to the other.

What the proof does not say is just as important:

• It says nothing about a system while the network is healthy. With no partition, a system can be linearizable and answer every request.
• It says nothing about latency, except that “wait for the other side” can last as long as the cut does.
• It applies only to operations that need data from both sides. A key whose replicas are all on one side of the cut is unaffected.

Step 3: Why “pick two of three” is misleading

The slogan “consistency, availability, partition tolerance: pick two” suggests three equal options. That includes a “CA” system that gives up partition tolerance. That option does not exist for a distributed system, because partition tolerance is not something you give up.

Switches fail. Links get cut. And a node in a 20 second garbage collection pause looks to its peers exactly like a node behind a cut link. A system that “is not partition tolerant” still experiences partitions. It just has no defined behavior when they happen. In practice, that means it becomes inconsistent, unavailable, or both.

So the honest reading is: when a partition happens, each operation that spans it gives up either C or A. When there is no partition, it can have both. Brewer said this himself in 2012, calling the “two of three” framing misleading.

Three more corrections follow from the definitions:

1. “CA” describes a single node, or a cluster that assumes partitions never happen. A single Postgres server is linearizable and available, until the network between it and its clients fails. Then it is neither, from those clients’ view.
2. The choice is per operation, not per product. The same cart can accept “add item” on both sides of a cut, and refuse “place order” on the minority side. DynamoDB offers eventually consistent and strongly consistent reads on the same table.
3. Many real configurations are neither C nor A. A leaderless store at quorum reads and writes refuses on the minority side (not A). And it is still not linearizable on the majority side (not C). The key-value store walks through why.

Common mistake

"Our system is CA because we run in one data center"

One data center still has switches, links and paused processes, and nodes in it still lose contact with each other. Calling a system CA means it has no defined behavior when that happens. Say instead what each operation does when a replica cannot reach the others: refuse, or answer from local state.

Step 4: What a CP operation does in a partition

A CP operation refuses, instead of risking a wrong answer. In practice, that does not mean the whole system goes down. Consensus systems (Raft, Paxos) require a majority to commit a write or confirm a read. And at most one side of any cut can hold a majority. So:

• The majority side keeps working. It elects a leader if it lost one, then commits writes and serves linearizable reads as usual. Users who can reach it see nothing wrong.
• The minority side refuses. Its nodes cannot reach a majority. So writes time out or return an error, and so do reads that need the latest value. A leader stranded on the minority side steps down when its lease expires.
• Clients near the minority see errors unless they can reach the majority some other way, for example through another region’s endpoint.

A leader stranded on the minority side steps down, and the majority carries on

Scroll or pinch to zoom, drag to pan

• Color key
• Database

The cost is bounded. West’s users get errors on CP operations during the cut. And if the leader was on the minority side, everyone waits out a failover (typically 1 to 30 seconds).

One trap: a node on the minority side that serves reads from its own copy is no longer CP for those reads. ZooKeeper does exactly that by default. Its writes go through a leader and a quorum. But any server answers reads locally, so a read can be stale, unless the client calls sync first. etcd serves linearizable reads by default, and offers local “serializable” reads as an opt-in. Know which read path a system uses before calling it CP.

Step 5: What an AP operation does in a partition

An AP operation answers from whatever replica it reaches. Both sides of the cut keep accepting writes, so the copies diverge. When the link heals, the system must reconcile two histories, and how it does that decides whether acknowledged writes survive:

• Last writer wins (LWW) keeps the version with the newer timestamp. Simple, and it silently discards the other side’s writes, even though their clients were told “ok”. Clock skew makes it worse: the side with the fast clock wins regardless of real order.
• Keep both (siblings) stores concurrent versions, detected with version vectors, and hands them to the application to merge on the next read. Nothing is lost, but every reader must know how to merge.
• Merge automatically with a CRDT. That’s a data type whose merge is defined so that any two replicas converge. Examples are a counter that sums per-replica counts, or a set that tracks adds and removes. Nothing is lost, and no application code merges. But only some data fits.

Amazon’s Dynamo paper describes the classic case. A shopping cart must accept “add item”, even during failures. It’s merged by union on reconciliation. The known side effect is that a removed item can reappear. A union cannot tell “never added” from “removed on the other side”, without extra bookkeeping (a tombstone, or an OR-set).

Try both choices below. Pick CP or AP, add items in each region, cut the link, keep adding and reading on both sides, then heal.

CP: refuseAP: serve

Last writer winsUnion merge

East

Add itemRead cart

link upCut link

West

Add itemRead cart

Served

0

Refused

0

Stale reads

0

Acked, undone on heal

0

Reset

Blue items were added in East, purple in West. Click an item to remove it. A dashed border marks an item added during the cut that the other region has not seen yet.

No partition: the price of every writeRound trip 70 ms

EL: ack in the local region, replicate after

1 ms

EC: ack after the other region confirms

71 ms

In CP mode, East holds two of the three replicas, so it never notices the cut. Every West request is refused until the link heals. Nothing needs reconciling, because only one side wrote.

In AP mode, every request is served in 1 ms. Reads on one side miss what the other side acknowledged. And healing with last writer wins drops items whose clients were told “ok”. Union merge keeps every add, because adds never conflict. Now remove an item on one side, during the cut. The union brings it back, which is the Dynamo cart’s anomaly. Fixing it takes a tombstone per removal, as an OR-set keeps.

The choice between the two does not have to be made once for the whole system. The code below gives one replica both kinds of operation. Adding to the cart stays available, and merges by union on heal. Placing an order needs a majority.

JReplica.javaPyreplica.pyC#Replica.csGoreplica.goTSreplica.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;

// One replica of a shopping cart. CAP is decided per operation here, not per system:
// adding an item stays available on both sides of a partition, placing an order does not.
class Replica {
    final String name;
    final int clusterSize;
    int reachable; // replicas this one can talk to right now, itself included
    final Set<String> cart = new LinkedHashSet<>();
    final List<String> unsent = new ArrayList<>(); // accepted while cut off, shipped on heal

    Replica(String name, int clusterSize, int reachable) {
        this.name = name;
        this.clusterSize = clusterSize;
        this.reachable = reachable;
    }

    // A majority can only exist on one side of a cut, so it is the test for "safe to decide".
    boolean hasMajority() {
        return reachable > clusterSize / 2;
    }

    // AP operation: always accept. Adds commute, so a union merge on heal loses nothing.
    void addItem(String item) {
        cart.add(item);
        if (reachable < clusterSize) unsent.add(item);
    }

    // CP operation: an order charges money, so it must never be placed on both sides of a cut.
    // Without a majority this side cannot know what the other side did, so it refuses.
    String placeOrder() {
        if (!hasMajority()) {
            throw new IllegalStateException(name + ": 503, no majority, retry later");
        }
        return name + ": order placed for " + cart;
    }

    // Heal: each side applies what the other accepted alone. Union is the merge for a set
    // that only grows; removals would need tombstones or an OR-set.
    void healWith(Replica other) {
        cart.addAll(other.unsent);
        other.cart.addAll(unsent);
        unsent.clear();
        other.unsent.clear();
        reachable = clusterSize;
        other.reachable = other.clusterSize;
    }
}

public class PerOperation {
    public static void main(String[] args) {
        // Three replicas: two in East, one in West. The link between the regions is cut.
        Replica east = new Replica("east", 3, 2);
        Replica west = new Replica("west", 3, 1);

        east.addItem("book"); // both sides keep taking adds during the cut
        west.addItem("mug");

        try {
            System.out.println(west.placeOrder());
        } catch (IllegalStateException e) {
            System.out.println(e.getMessage()); // west: 503, no majority, retry later
        }

        west.healWith(east);
        System.out.println(west.placeOrder()); // west: order placed for [mug, book]
    }
}

Step 6: Else, latency or consistency (PACELC)

CAP describes a few minutes a year. Daniel Abadi pointed out something in 2010, and in a 2012 paper. The tradeoff a replicated system pays most often happens with no partition at all. PACELC extends CAP, instead of replacing it:

If there is a Partition, choose Availability or Consistency. Else, choose Latency or Consistency.

The “else” half exists because consistency needs coordination, and coordination takes round trips. A linearizable write cannot be acknowledged until enough replicas have it that no later read can miss it. That means waiting for a majority, or for a synchronous replica. If those replicas are in another region, every write waits for a cross-region round trip:

Where the replicas that must confirm are	Round trip	Cost on every consistent write
Same availability zone	Under 0.5 ms	Negligible
Three zones of one region	1 to 2 ms	Small: most systems simply pay it
us-east to us-west	About 65 to 70 ms	A page with 4 sequential writes takes 280 ms
US to Europe	About 75 to 90 ms	Same order as above
US to Southeast Asia	About 200 to 230 ms	Interactive writes start to feel slow

Choosing latency (EL) means acknowledging after the local replica has the write, and shipping it to the others afterwards. Writes take about a millisecond. For at least the one-way delay after each one, other regions read the old value. (That’s half a round trip, more if replication queues.) Choosing consistency (EC) means waiting. The slider at the bottom of the demo above shows both bars.

Reads have the same choice. A read from the nearest replica is fast, and possibly stale. A linearizable read must confirm with the leader or a majority that no newer write exists. That’s a round trip to wherever they are. (See Reads that do not lie on the consensus page.)

PACELC: two questions, asked at different times

Scroll or pinch to zoom, drag to pan

• Color key
• A question
• An answer

A system’s PACELC label is the pair of choices, such as PA/EL or PC/EC. The two halves usually match. The mechanism that makes a system wait during a partition (needing a majority) is the same one that makes it slower when healthy. They can differ, though. A single-leader database with asynchronous read replicas is PC/EL for replica reads. Writes need the leader, but reads take whatever the replica has.

TradeoffSynchronous replication across regions (EC)

We gain

No acknowledged write is lost when a region fails, and every reader everywhere sees it at once.

We give up

Every write pays a cross-region round trip, 65 to 230 ms, on every request. And a write cannot complete at all while the remote replicas are unreachable.

Step 7: Harvest and yield, a finer-grained availability

CAP’s availability is binary: a response or not. Armando Fox and Eric Brewer proposed two finer measures in 1999 that fit real services better:

• Yield: the fraction of requests that get an answer. This is what uptime numbers measure.
• Harvest: the fraction of the data that the answer reflects. A search over 100 shards that hears back from 98 of them has a harvest of 98%.

With a shard unreachable, a search can refuse the query (yield drops) or answer from the rest (harvest drops). For search and feeds, a 98% answer now beats a 100% answer never. For a bank balance, a partial answer is a wrong answer, so the system refuses.

JHarvest.javaPyanswer.pyC#Program.csGoanswer.goTSanswer.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.ArrayList;
import java.util.List;

// Harvest and yield (Fox and Brewer): instead of all-or-nothing, a fan-out query answers with
// the shards that replied in time and says how much of the data that covers.
record Shard(String name, boolean up, int latencyMs, List<String> docs) {}

// harvest = fraction of the data reflected in the answer. Returning it lets the caller
// (or the page) say "some results may be missing" instead of pretending the answer is whole.
record Answer(List<String> hits, double harvest) {}

public class Harvest {
    static Answer search(List<Shard> shards, String term, int deadlineMs, double minHarvest) {
        List<String> hits = new ArrayList<>();
        int answered = 0;
        for (Shard s : shards) {
            // A down shard and a shard slower than the deadline look the same to the caller:
            // both are missing from this answer. Waiting longer buys harvest with latency.
            if (!s.up() || s.latencyMs() > deadlineMs) continue;
            answered++;
            for (String d : s.docs()) if (d.contains(term)) hits.add(d);
        }
        double harvest = (double) answered / shards.size();
        // Below the floor, a partial answer would mislead more than it helps: refuse, which
        // costs yield (this request fails) to protect harvest.
        if (harvest < minHarvest) {
            throw new IllegalStateException("503: only " + answered + " of " + shards.size() + " shards answered");
        }
        return new Answer(hits, harvest);
    }

    public static void main(String[] args) {
        List<Shard> shards = List.of(
            new Shard("s1", true, 12, List.of("red shoes", "red hat")),
            new Shard("s2", false, 0, List.of("red scarf")), // down
            new Shard("s3", true, 250, List.of("red bag")), // GC pause: misses the deadline
            new Shard("s4", true, 18, List.of("blue shoes", "red socks")));

        Answer a = search(shards, "red", 100, 0.5);
        System.out.println(a.hits() + " harvest=" + a.harvest()); // [red shoes, red hat, red socks] harvest=0.5
    }
}

The minHarvest floor is the design decision. Below it, the answer would mislead, so the request fails. The deadline matters as much as the failed shard. A shard in a 250 ms garbage collection pause and a dead shard look identical, to a caller that stops waiting at 100 ms.

Options and tradeoffs

What an operation does when it cannot reach the other side

Option	What the user sees	Wins for	Avoid for
Refuse (CP)	An error or retry on the minority side	Money, inventory, uniqueness (usernames, seats), locks, leases	Anything users expect to always work, like browsing
Serve stale reads, refuse writes	Old data, clearly read-only	Catalog pages, profiles, settings pages during an outage	Reads that feed a decision (balance before a transfer)
Accept writes and merge later (AP)	Everything works; conflicts resolved on heal	Carts, likes, counters, presence, chat drafts, device sync	Data with invariants across writes, unless a CRDT fits
Degrade harvest	A partial answer, ideally labeled	Search, feeds, recommendations, dashboards	Anything where partial means wrong

The table is a menu per operation. A typical commerce design uses all four. Browsing serves stale reads. Carts accept and merge. Checkout refuses on the minority side. And search returns partial results.

TradeoffChoosing per operation instead of per system

We gain

The few operations that must be correct stay correct, and everything else stays up and fast during partitions and pays no cross-region latency when healthy.

We give up

Every operation needs a stated behavior, and code must route each one correctly. An operation on the wrong path is a bug that only shows during a partition, which is the hardest time to debug it.

How AP writes are reconciled

The mechanics are on the replication page. In CAP terms, the only question is what happens to acknowledged writes.

Go deeperReconciliation strategies compared2 min

Strategy	Acknowledged writes lost?	Who merges	Fits
Last writer wins	Yes, every concurrent write but one	Nobody	Caches, “last seen” timestamps, idempotent overwrites
Siblings with version vectors	No	The application, on read	Documents where a human or rule can merge
CRDTs	No	The data type itself	Counters, sets, flags, collaborative text
One home region per key	No conflicts, but the key is CP across regions	Nobody	Per-user or per-tenant data with a clear owner

The last row avoids the question. If every key has exactly one region that accepts its writes, two sides cannot diverge. The key is then CP across regions. Users far from its home pay a round trip, and a cut from the home region makes it read-only. Meanwhile, different keys are homed in different regions. Many large designs end here.

TradeoffLast writer wins for AP writes

We gain

No merge code, no siblings, and storage stays one value per key.

We give up

Concurrent writes are discarded silently after being acknowledged, and clock skew decides which one survives. Nothing alerts, because nothing failed.

Labels for real systems, with caveats

Abadi’s 2012 paper labeled default configurations: Dynamo, Cassandra and Riak PA/EL, Yahoo’s PNUTS PC/EL, and fully ACID systems (VoltDB/H-Store, Megastore) and BigTable/HBase PC/EC. A label describes a configuration and an operation, never a product. With that caveat:

System	Typical label	What changes it
Dynamo, Cassandra, Riak	PA/EL	Consistency level per request: QUORUM is neither C nor A; Cassandra lightweight transactions (Paxos) are PC/EC
DynamoDB, one region	PC/EL or PC/EC	A leader per partition takes writes; eventually consistent reads (the default, half the cost) are EL, ConsistentRead is EC
DynamoDB global tables (multi-active)	PA/EL	Each region accepts writes, replication is asynchronous, last writer wins; the newer multi-Region strong consistency mode is PC/EC
PNUTS	PC/EL	Each record has a master region: writes cut off from it fail, reads stay local. PC here means “keeps its usual timeline consistency”, not linearizable
VoltDB, BigTable/HBase	PC/EC	One owner per partition or region server: unreachable means unavailable for those keys, never divergent
Spanner, CockroachDB	PC/EC	Bounded-staleness and follower reads (AS OF SYSTEM TIME) are EL for those reads
etcd, ZooKeeper	PC/EC writes	etcd serializable reads and ZooKeeper’s default local reads are EL and can be stale
Postgres or MySQL with async replicas	PC/EL	Synchronous standby makes writes EC, and a lost standby blocks commits; async failover can lose acknowledged writes
MongoDB replica set	Depends	Abadi said PA/EC (2012 defaults). w: majority with readConcern: "linearizable" is PC/EC; w: 1 with secondary reads is closer to PA/EL
Kafka	PC/EC per partition	acks=all with min.insync.replicas=2 refuses writes without enough in-sync replicas; unclean leader election trades lost messages for availability
Redis with replicas or Redis Cluster	Neither, strictly	Replication is asynchronous, so a failover can lose acknowledged writes; a minority primary stops accepting writes after the node timeout

Go deeperSpanner: CP, and still five nines1 min

Spanner is CP. During a partition, the minority side of a Paxos group cannot commit. Brewer, who works at Google, wrote in 2017 that Spanner is still “effectively CA” for users. Google’s private network has redundant paths. So partitions that split a Paxos group are rare enough that their share of downtime is small, next to other causes such as bugs and bad configuration pushes.

The lesson is not that CAP was beaten. It is that “CP” and “highly available” are compatible, when partitions are rare and the majority side serves everyone who can reach it. The theorem limits behavior during a partition. Engineering limits how often partitions happen.

Where the replicas live decides what you pay

Placement	E cost of consistency	P risk	Typical choice
3 replicas, one region, 3 zones	1 to 2 ms	A zone split: the 2-zone side keeps a majority	EC for nearly everything: it is cheap
3 replicas across 3 regions	65 to 230 ms per write	A region cut off is the minority	EC only for small, critical data
Home region per key, async copies	Local for the home region	A cut makes remote copies stale and read-only	PC/EC at home, EL elsewhere
Every region accepts every write	Local everywhere	Divergence on every cut	PA/EL with CRDTs or careful merge rules

Within one region, choose consistency: it costs a millisecond or two. Across regions, consistency is a budget, spent only where a wrong answer costs more than 70 to 230 ms on every request.

TradeoffA home region per key

We gain

No conflicts to merge, linearizable per key, and local latency for the users near each key’s home, which is usually most of them.

We give up

Users far from the home region pay a round trip per write. And a region outage makes its keys read-only (or loses the last seconds of writes, on a forced failover), until it is back or moved.

How it fails

The first failure is assuming partitions are rare enough to ignore. Peter Bailis and Kyle Kingsbury collected years of public incident reports in 2014. They found partitions in every kind of environment: inside data centers, between them, and in the cloud. Gill, Jain and Nagappan studied Microsoft’s data center networks in 2011. Network redundancy reduced the median impact of a failure, measured in traffic lost, by only about 40%. Partitions are uncommon per link, and common across a fleet.

They also rarely look like a clean cut:

What happens	What the nodes see	Typical cause	What it breaks
Region or zone split	Two groups, each healthy, each sure the other is down	Fiber cut, BGP or routing change, inter-zone link fault	Any operation spanning the cut; failover tooling promotes a second leader
One-way link	A hears B, B does not hear A	Asymmetric firewall rule, bad NIC or switch port	A node keeps winning elections it cannot serve, or deposes a healthy leader
Process pause	A node stops answering for seconds, then resumes as if nothing happened	Stop-the-world GC, VM migration, swapping, a stuck disk	A paused leader wakes up and writes after a successor was elected
Gray failure, partial loss	5% of packets dropped, some peers reachable, some not	Failing optics, an overloaded switch buffer	Timeouts and retries everywhere, flapping membership, slow everything
Overload	Heartbeats miss because the process is too busy to send them	A traffic spike, a slow query, a retry storm	False failure detection, failovers that move the load and repeat

The common thread: a node cannot tell a crashed peer from a slow or unreachable one. It only sees a timeout. Every failure-handling decision rests on that guess.

Failure of the CAP choice	What you see	Detect	Fix
Split brain in a “CP” system	Two leaders accept writes, then conflicting data	Two nodes reporting leader, epoch gaps	Majority-based election, leases, and fencing tokens checked at the storage
LWW drops writes after heal	Users report edits or items that vanished	Count overwrites of a newer-timestamped sibling	Version vectors or CRDTs; a home region per key
Timeouts too short	Failovers during GC pauses or load spikes, repeated elections	Leader changes per hour, term numbers climbing	Timeouts well above p99 pause times; PreVote and CheckQuorum in Raft
Timeouts too long	Minutes of errors after a real failure	Time from fault to new leader	Tune detection, measure it in game days
Minority side serves stale reads	Reads go backwards after a failover	Staleness probe per replica	Linearizable reads through the leader or ReadIndex where it matters
A “CP” path that is really async	Acknowledged writes gone after failover	Compare acknowledged and recovered sequence numbers	Synchronous or majority acknowledgment on that path

Go deeperA 43 second partition, a day of repair1 min

GitHub’s outage of October 2018 is the standard example of a short partition with a long tail. Its US East Coast network hub lost connectivity to the main East Coast data center for 43 seconds. That led its failover tooling to promote database primaries on the West Coast. Writes had landed on both sides. The cross-country latency hurt the applications. And reconciling the two histories took about 24 hours of degraded service. The partition was the trigger. The damage came from a failover that did not know which writes the other side had.

Common mistake

"A timeout means the other node is down"

A timeout means no reply arrived in time. The node may be dead, paused, or overloaded. Or it may be fine, and unreachable from here, but reachable from its clients. A design that acts on a timeout (promote a replica, take over a lock) must also stop the old owner from acting. That needs a lease that expires first, and a fencing token checked where the data is written.

At scale

Quantity	Typical value
Round trip within a zone / across zones	Under 0.5 ms / 1 to 2 ms
Round trip across regions	65 to 230 ms, as in the table above
Raft or Paxos commit in one region, 3 zones	A few ms including fsync
Leader failover (detection plus election)	About 1 to 30 s, depending on timeouts
Partitions that split a well-run region	Rare: minutes a year, but some every year across a fleet
Process pauses that look like partitions	Hundreds of ms to tens of seconds (GC, VM migration, I/O stalls)

At 10 times the nodes, partitions become background: some link is cut, flaky or one-way, and some process is paused, at any moment. Designs that fail over by hand start paging every day. Make partition behavior routine: majority leadership per shard, automatic failover with fencing, AP merges that need no human, and CP refusals the client already retries.

Go deeperAt 100 times: data across regions1 min

At 100 times, the data spans regions, and the “else” cost dominates. Linearizable operations across regions cost 65 to 230 ms each, so designs shrink their scope:

• Home each key near its users. Spanner places leaders by configuration. CockroachDB homes rows by region (REGIONAL BY ROW). So most writes commit within one region.
• Split the invariant. A global seat or stock count becomes per-region allotments, each enforced locally, rebalanced occasionally. The global “never oversell” holds without a global round trip per sale.
• Read stale on purpose. Bounded-staleness reads (data at most 10 seconds old) serve from the nearest replica without coordination.

What to monitor:

• Round trip and packet loss between every pair of zones and regions, measured from both ends (one-way failures show up only one way). And which peers each node considers unreachable.
• Requests refused for lack of a quorum, per region: the cost of CP, measured.
• Leader changes and term increments per shard: false failovers show up here first.
• Conflicts on heal (siblings, LWW overwrites of acknowledged writes) and the anti-entropy backlog.
• A staleness probe: write a timestamp to a canary key every second in each region, read it in the others, graph the age. It measures the EL window users actually see.

In the interview

Interviewer asksExplain the CAP theorem.

Take a replicated piece of data. When the network partitions, an operation that spans the cut must either refuse (keeping linearizability), or answer from local state (keeping availability). C here means linearizability. A means every non-failing node answers. Partitions are not optional. So the real choice is C or A during a partition, made per operation. Without a partition, you can have both. Then the tradeoff becomes latency against consistency, which is PACELC.

Interviewer asksYour design runs in two regions. The link between them goes down for five minutes. What happens?

Per operation. Browsing and search keep serving from each region’s replicas, possibly stale, and possibly with partial results. Carts accept writes in both regions, and merge by union when the link heals. Checkout and payments live in each user’s home region. So users whose home is on their side of the cut are unaffected. Users routed to the other region get a retryable error, for checkout only. There’s no automatic cross-region failover of primaries during a cut. Promoting a second primary while the first is still serving is how writes get lost. After the heal, carts reconcile, and the monitoring shows how many conflicts merged.

Interviewer asksWhy not make everything strongly consistent?

Because consistency is paid on every request, not just during partitions. Within one region, it costs 1 to 2 ms, so I do pay it there for most data. Across regions, a linearizable write waits 65 to 230 ms for a remote majority. And it cannot complete at all when that majority is unreachable. So I keep cross-region consistency for the few operations where a wrong answer is expensive: money, inventory, uniqueness. I home each of those keys in one region. Everything else is local, and eventually consistent with a stated bound.

Interviewer asksIs Cassandra (or MongoDB) CP or AP?

It depends on how it is configured and which operation. Cassandra at consistency level ONE is PA/EL. At QUORUM it refuses on the minority side but is still not linearizable, so it is neither. Lightweight transactions use Paxos and are CP. MongoDB with w: majority and linearizable reads from the primary is CP. With w: 1 and secondary reads, it acknowledges writes a failover can roll back and serves stale reads. I would state the settings and the anomaly a user could see, not a letter.

Common mistake

"We chose AP, so users never see errors"

CAP availability covers requests that reach a live node. A client that cannot reach any replica, because its own network or region is down, gets nothing from an AP system either. AP also does not mean fast: a node can answer after 30 seconds and still count as available. And AP moves the cost to later: divergent data has to be merged, and a careless merge loses acknowledged writes.

Mid-level covers

States CAP as a choice during a partition, not as “pick two”. Knows C means linearizability. Explains that the majority side of a CP system keeps working. Names PACELC’s latency tradeoff, with a cross-region number.

Senior adds

Chooses per operation, and says what each one does on the minority side. Prices consistency in milliseconds, by replica placement. Picks a merge strategy for AP data, and says what it loses. Uses home regions or split invariants to keep strong operations local. Treats timeouts, pauses and one-way links as partitions, with leases and fencing.

Seen in

• Key-value store: PA/EL by default, QUORUM as neither C nor A, and Paxos compare-and-set as the one PC/EC path.
• Payment system: fails closed when the database is unreachable, and commits on a synchronous standby so a failover loses nothing.
• Ticket booking: each event has a home region, so two regions never sell the same seat and no hold needs cross-region consensus.
• Hotel reservation: a home region per shard of hotels. Two regions never take writes for one hotel.
• Chat: a home region per conversation orders messages, at the price of one cross-region hop for distant members.
• Distributed rate limiter: a local fallback (AP) for most rules when the counter store is unreachable, failing closed (CP) for login and OTP rules.
• Unique ID generator: regions mint IDs with no coordination at all, and issuance fails closed if worker leases cannot be renewed.
• Message queue: acks=all with a minimum in-sync set refuses writes, and unclean leader election is the explicit switch to availability.
• Dropbox: asynchronous cross-region replication, choosing latency over 70 to 100 ms on every save.
• Distributed cache: a primary cut off from the cluster manager stops serving when its lease runs out, so it never serves beside its successor.

Recap

CAP, precisely. During a partition, an operation that spans the cut gives up linearizability, or availability (every non-failing node answers). Without a partition, it can have both. C means linearizability only. Weaker models can stay available.

Not “pick two”. Partitions are not optional, so there is no distributed CA system. Choose per operation, not per product. Many real configurations are neither C nor A.

CP in practice. The majority side keeps serving. The minority side refuses, and a stranded leader steps down. Local reads on the minority side break CP for those reads.

AP in practice. Both sides accept and diverge. On heal: last writer wins (loses acknowledged writes), siblings (the application merges), or CRDTs (the type merges).

PACELC. Else, latency or consistency. A consistent write waits for remote replicas, on every request: 1 to 2 ms across zones, and 65 to 230 ms across regions.

Harvest and yield. Answer from part of the data (lower harvest) or refuse (lower yield). Set a floor below which partial means wrong.

Partitions in practice. Zone and region splits, one-way links, GC pauses, gray failures and overload. A timeout cannot tell dead from slow, so act on it with leases and fencing.

Further reading

• On this site: consistency models, for linearizability and the weaker guarantees. Replication, for lag, quorums and conflict resolution. And consensus, for majorities, leases and linearizable reads.
• Eric Brewer, CAP Twelve Years Later: How the “Rules” Have Changed, IEEE Computer 2012. It’s the author’s own correction of “two of three”, with partition recovery.
• Martin Kleppmann, A Critique of the CAP Theorem, 2015: why the CAP definitions fit few real systems, and a latency-based alternative.
• Peter Bailis and Kyle Kingsbury, The Network is Reliable, ACM Queue 2014: a catalog of real partitions and what they did to real systems.