Skip to content

System DesignWorked problems

Design a collaborative editor

Design a collaborative editor like Google Docs for 5 million open sockets. Start with one server and one database, grow it with gateways and one owner per document, ordered edits, an op log with snapshots and signed tickets, then stress it with OT against CRDTs, a crashed owner, offline editing, undo and one very popular document, with code.

Senior26 min core · 45 min full depth

“Design Google Docs. Several people edit the same document at the same time and see each other’s changes live.”

A collaborative editor looks like a text box with a save button that runs very often. The hard part is that two people type into the same sentence at the same moment. Each types on their own copy, over networks that delay and drop messages. Every copy must end up identical, and nobody may lose a keystroke. Locking a paragraph, or keeping the last save, isn’t an answer. Both throw away work the user saw on screen.

The single-user side (commands, undo stacks, the text buffer) is the text editor with undo design. This page is about what changes when the document is shared. It starts with one server and one database, and lets the numbers grow it. Then it tests the result against two edits at once, a crashed owner, a long flight and undo.

If you only have a minute, this is the design. The rest of the page explains each part.

  1. Type locally, at once. Each client shows its own edit immediately, and sends it with the revision (the version number) it was typed on.
  2. One owner per document decides the order. Every editor of a document reaches the same server. It rewrites each edit past the ones accepted since, using operational transformation (OT), and gives it the next revision.
  3. Ack only after a durable write. Each revision is written to an op log “only if this revision is empty”. That one rule stops two owners from both writing it.
  4. Catch up by revision. A client that was offline asks for everything after its last revision. It rewrites its own pending edits past them, and sends them.
  5. Cursors are lossy. They go through the owner and are never stored, so they never cost what edits cost.

Three questions pick the merge algorithm: what is edited, how many people edit one document, and whether offline editing is in scope.

Interviewerminute 0

Design Google Docs. Several people edit one document at once.

What they are testing

Whether you pin down what is edited, how many people edit one document, and whether offline editing is in scope. Those three pick the merge algorithm. The red flag is jumping to “use CRDTs” before asking any of them.

These four questions matter most:

  • What is edited? Rich text: characters plus formatting, lists and tables.
  • How many editors per document? Usually 1 to 5, at most 100 live editors. Viewers unlimited.
  • Offline? Yes, for hours, then merge on reconnect.
  • Version history? Every change for 30 days, named versions forever.

From these answers we make five promises:

  • Instant typing. My own edits show at once. A colleague sees them within 200 ms at p99 in one region.
  • Converge. Once edits stop, every copy is identical and contains every edit anyone saw.
  • Never lose an acked edit. An acknowledged edit is stored in 2 of 3 zones.
  • Always editable. 99.95% available. A failover can pause syncing for about 15 seconds, while typing continues locally.
  • Revokes work. A user whose access is removed is cut off from an open document within seconds.

We leave out the rich-text renderer, full-text search, and spreadsheets (the same skeleton with different edits).

Go deeperMore questions, and what each changes1 min
Question We assume If the answer were different
Plain text only? No, rich text Only insert and delete to transform
Thousands of live editors? No, 100 at most A different fan-out and merge design
Online only? No, offline for hours The server could reject stale edits instead of merging them
Real-time? Yes, about 200 ms “Save and merge every minute” is file sync, a much simpler system
End-to-end encryption? No The server couldn’t transform edits, which pushes the design to CRDTs
Largest document? About a million characters Book length needs loading in chunks

Now we work out the numbers. Count open sockets and edits, not stored bytes.

Assumptions
Daily active users
50 million
Minutes editing per user a day
60
Peak over average
2.5×
Op batches per session a minute
12
Bytes per stored op, with metadata
100 bytes
Days of raw ops kept
30
Document opens per user a day
10
Documents stored
5 billion
Average snapshot size
50 KB
Sessions2.1Mopen sessions; 5.2M at peak

50 million daily users, each with a document open about an hour a day.

So we hold millions of long-lived WebSockets, on gateways that do nothing else.

Edits420,000/s1M/s at peak

A client sends a batch of whatever was typed since its last batch was acknowledged, one batch in flight at a time. Most sessions are idle or reading, so it averages about 12 batches a minute.

So we can't write edits to one database: they go to a log split by document.

Owners87servers; 400 GB live state

About 20,000 edits a second per server, run at 60% busy. 4 million documents open at once, at about 100 KB each.

So we keep every open document in memory on its owner.

Storage3.6 TBof edits a day

At 100 bytes per stored edit. Thirty days is 110 TB before copies.

So we keep raw edits for 30 days, fold older ones into snapshots, and keep snapshots in object storage.

Send every keystroke on its own, and the peak passes 5 million edits a second. That’s why clients batch: batching is a client decision that sets the whole write load. Try it in the calculator.

Change an assumption
Open sessions, average
2.1 million
Open sessions (sockets), peak
5.2 million
At 100,000 sockets a gateway, a fleet of gateways that only hold connections.
Ops, average
420,000/s
Ops, peak
1,000,000/s
Past what one database primary writes: ops go to a log store partitioned by document.
Session servers at peak (20,000 ops/s each, 60% busy)
87
Each owns a share of the documents, by slot lease.
Open documents in memory at peak (1.3 sessions each)
400 GB
About 100 KB each with recent revisions. Spread over the session servers, a few GB per server.
Opens, peak
14,000/s
With reconnects, past what one Postgres answers: cache permissions in the doc service.
Op log written per day
3.6 TB
Raw op history kept
110 TB
Before replication. Older ops are folded into snapshots and named versions.
Latest snapshots of every document
250 TB
Object storage. Cheap per byte, and only the newest snapshot of each document is hot.

We start with the simplest design that is correct. Then we let the numbers break it, one at a time. Each step fixes only what broke, adds only the boxes that fix needs, and says what the fix costs. Throughout, follow one scenario: Alice and Bob have the same document open, and Alice types a word.

The simplest version is one doc server that holds both WebSockets and the document in memory, with Postgres behind it. Alice’s client shows the word at once and sends it. The server merges it, writes it to Postgres, sends it to Bob, and acks Alice. The merge works (the first stress test explains it).

Collaborative editor · architecture

Stage 1: One server, one database

EditorsDoc serversockets + OT10000% busyPostgresdocs + every op5000% busy
Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional

Stage 1 of 6

One server, one database

Why this stage

Nothing yet.

This is the smallest design that lets two people edit one document.

What we add

One doc server that holds every WebSocket, keeps each open document in memory, orders and transforms edits, and writes every op to Postgres.

What it costs

One box holds about 50,000 sockets and the peak needs 5.2 million.

Every op is a Postgres write, and one crash stops every document.

At 1M ops per second, Gateways is 10000% busy. Stage 2 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
The numbers behind the busy bars

How busy it gets

  • There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
  • Opens and reconnects add about 23,000 a second at peak.

What one machine can do

  • One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
  • One session server handles about 20,000 edits a second, including the durable write and the broadcast.
  • One op log node takes about 25,000 conditional writes a second.
  • One Postgres handles about 20,000 simple writes or reads a second.

What saves work

  • The doc service answers about 90% of permission checks from its cache.
  • There is one snapshot every 500 edits.
  • Relays deliver, on average, about half a viewer copy per edit.

But one box holds about 50,000 sockets, the peak needs 5.2 million, and one Postgres primary takes about 20,000 writes a second, not a million.

Step 2. Gateways, and one owner per document

Section titled “Step 2. Gateways, and one owner per document”

Split the server in two. Gateways hold WebSockets, and nothing else. They check the ticket, answer pings and forward frames. Session servers do the work. Each document has exactly one owner among them, because the order of a document’s edits must be decided in one place.

Document IDs hash to 16,384 slots. Each session server leases a few hundred slots in etcd, and renews them every few seconds. Gateways cache the slot map, so sending a frame to the owner is a hash and a lookup. A dead server’s leases expire after 10 seconds, and others take its slots.

Collaborative editor · architecture

Stage 2: Gateways and one owner per doc

EditorsLoad balancerL4, TLSnewGatewaysWebSockets only84 servers, 60% busySession serversone owner per docnew84 servers, 60% busyPostgresdocs + every op5000% busyPlacementslot leases, etcdnew
Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional
Stage

Stage 2 of 6

Gateways and one owner per doc

Why this stage

5.2 million sockets and 1 million ops/s at peak.

The ops for one document must still meet in one place, or nobody can order them.

What we add

An L4 load balancer, gateways that only hold sockets, session servers that each own a share of documents, and a placement service (etcd) that leases 16,384 slots to session servers.

What it costs

Editing a document pauses for seconds while its owner fails over, and every op is still a write to one Postgres.

At 1M ops per second, Metadata DB is 5000% busy. Stage 3 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
The numbers behind the busy bars

How busy it gets

  • There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
  • Opens and reconnects add about 23,000 a second at peak.

What one machine can do

  • One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
  • One session server handles about 20,000 edits a second, including the durable write and the broadcast.
  • One op log node takes about 25,000 conditional writes a second.
  • One Postgres handles about 20,000 simple writes or reads a second.

What saves work

  • The doc service answers about 90% of permission checks from its cache.
  • There is one snapshot every 500 edits.
  • Relays deliver, on average, about half a viewer copy per edit.
Go deeperWhy slots and leases, and what one owner costs1 min

Why 16,384 slots, not a hash of the server list. If documents hashed straight onto servers, adding a server would move documents in a pattern nobody chose. With slots, the map from slot to server is a small table in etcd. Moving a slot is an explicit step. The old owner gives up its lease, the new one takes it, and gateways see the new map. Rebalancing becomes a list of slot moves we can pace and watch.

Why a lease, not just a heartbeat. A lease is a claim with an end time. A server that can’t renew must stop accepting edits before its lease ends, even if it feels healthy. That rule is what keeps a document from having two owners for long. The stress tests come back to the case where a pause makes a server miss its own deadline.

What one owner costs. All of a document’s edits pass through one thread on one server. That thread can order thousands of edits a second, far more than people can type into one document. The real limits are failover, about 10 seconds while a lease runs out, and sending to many watchers, which a stress test fixes.

Two people can now type into one document at once, through one owner. Every edit carries the revision it was made on, plus a client ID and sequence number. The owner transforms it past every revision accepted since: an insert typed before someone else’s insert shifts right, and so on. It gives the edit the next revision, and sends it to everyone. For the sender, that message is the ack.

The pair (client ID, sequence) makes a retry safe. A client resends an unacked batch with the same sequence. The owner remembers the last sequence per client. So it answers with the original revision, instead of applying it twice.

Edits are now ordered in one place. But every edit still goes to one Postgres primary, which takes about 20,000 writes a second, not a million.

Edits go to an op log split by document: one row per revision, keyed (doc, rev). The owner writes row (doc, 18345) only if it doesn’t already exist, and acks only after that write commits in 2 of 3 zones. A crash can never take back an edit anyone was told about.

A snapshotter follows the log. Every 500 edits, it writes the whole document to object storage. A server that has never seen a document loads the newest snapshot, plus the revisions after it, in one range read. That’s never more than about 500 edits to replay.

Collaborative editor · architecture

Stage 3: Op log and snapshots

EditorsLoad balancerL4, TLSGatewaysWebSockets only84 servers, 60% busySession serversone owner per doc84 servers, 60% busyOp logdoc, revnew67 servers, 60% busyMetadata DBtitles, ACLs115% busyPlacementslot leases, etcdSnapshotterevery 500 opsnew7 servers, 57% busySnapshotsobject storagenew
Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional
Stage

Stage 3 of 6

Op log and snapshots

Why this stage

1 million op writes a second against a Postgres primary that takes about 20,000, and 3.6 TB of ops a day.

What we add

An op log partitioned by document, one row per revision written only if absent, a snapshotter that saves each document every 500 ops to object storage, and Postgres kept for titles and permissions.

What it costs

Opening a cold document reads a snapshot plus the ops after it, and every open and reconnect still checks permissions in Postgres: about 23,000 reads/s at peak.

At 1M ops per second, Metadata DB is 115% busy. Stage 4 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
The numbers behind the busy bars

How busy it gets

  • There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
  • Opens and reconnects add about 23,000 a second at peak.

What one machine can do

  • One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
  • One session server handles about 20,000 edits a second, including the durable write and the broadcast.
  • One op log node takes about 25,000 conditional writes a second.
  • One Postgres handles about 20,000 simple writes or reads a second.

What saves work

  • The doc service answers about 90% of permission checks from its cache.
  • There is one snapshot every 500 edits.
  • Relays deliver, on average, about half a viewer copy per edit.

Follow one word from Alice’s keyboard to Bob’s screen:

Collaborative editor · architecture

Trace a request: Type a word

EditorsLoad balancerL4, TLSGatewaysWebSockets onlySession serversone owner per docOp logdoc, revMetadata DBtitles, ACLsPlacementslot leases, etcdSnapshotterevery 500 opsSnapshotsobject storage
Color keyServiceDatabaseBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Type a word: step 1 of 7

Editors to Load balancer

The text is already on screen: the client applied it locally. It sends the batch typed since the last send (at most one in flight, every 100 ms or so) on the open WebSocket: base rev 18342, client c7, seq 12.

20 ms of a 200 ms budget
  1. Editors → Load balancer20 ms
  2. Load balancer → Gateways1 ms
  3. Gateways → Session servers1 ms
  4. Session servers → Op log6 ms
  5. Session servers → Gateways1 ms
  6. Gateways → Editors20 ms
  7. Response
Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

About 49 ms end to end in one region, against a 200 ms budget, and Alice saw her word at 0 ms.

Edits are now fast and safe. But anyone who can reach a gateway can open a socket, and a reconnect storm would check permissions in the database millions of times.

Opening also catches the client up: it gets the newest snapshot and the edits after it, the same range read a new owner uses.

Opening a document is a normal HTTPS call to a doc service. It checks the user’s role through a cache, and returns a signed ticket, valid for 5 minutes, for the editing socket. Gateways check the signature themselves, so a reconnect storm costs CPU, not database reads.

A sharing change publishes an event. The doc service drops its cache entries, and the owner closes the sessions of anyone who lost access.

Collaborative editor · architecture

Stage 4: Opening, permissions and catch-up

EditorsLoad balancerL4, TLSGatewaysWebSockets only84 servers, 60% busySession serversone owner per doc84 servers, 60% busyOp logdoc, rev67 servers, 60% busyDoc serviceopen, ACL checknew8 servers, 57% busyMetadata DB+ read replicas5% busyPlacementslot leases, etcdSnapshotterevery 500 ops7 servers, 57% busySnapshotsobject storage
Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional
Stage

Stage 4 of 6

Opening, permissions and catch-up

Why this stage

About 23,000 opens and reconnects a second at peak, each needing a permission check, against a Postgres that serves about 20,000.

A gateway crash adds 60,000 reconnects at once.

What we add

A doc service that checks permissions from a cache fed by revoke events and issues short-lived signed tickets the gateways verify locally, plus metadata read replicas.

Reconnects catch up by revision.

What it costs

A ticket lives 5 minutes, so a revoke must also reach the owner, which closes the revoked sessions itself.

At 1M ops per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
The numbers behind the busy bars

How busy it gets

  • There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
  • Opens and reconnects add about 23,000 a second at peak.

What one machine can do

  • One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
  • One session server handles about 20,000 edits a second, including the durable write and the broadcast.
  • One op log node takes about 25,000 conditional writes a second.
  • One Postgres handles about 20,000 simple writes or reads a second.

What saves work

  • The doc service answers about 90% of permission checks from its cache.
  • There is one snapshot every 500 edits.
  • Relays deliver, on average, about half a viewer copy per edit.
Collaborative editor · architecture

Trace a request: Open a document

EditorsLoad balancerL4, TLSGatewaysWebSockets onlySession serversone owner per docOp logdoc, revDoc serviceopen, ACL checkMetadata DB+ read replicasPlacementslot leases, etcdSnapshotterevery 500 opsSnapshotsobject storage
Color keyServiceDatabaseBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Open a document: step 1 of 11

Editors to Load balancer

GET /api/v1/docs/d_9f3 with the session cookie.

20 ms of a 300 ms budget
  1. Editors → Load balancer20 ms
  2. Load balancer → Doc service1 ms
  3. Doc service → Metadata DB4 ms
  4. Editors → Load balancer20 ms
  5. Load balancer → Gateways1 ms
  6. Gateways → Session servers1 ms
  7. Session servers → Snapshots30 ms
  8. Session servers → Op log6 ms
  9. Session servers → Gateways1 ms
  10. Gateways → Editors20 ms
  11. Response
Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

This is enough for a mid-level answer. Gateways. One owner per document, by slot lease. OT against a single order. An op log with conditional writes and snapshots. And signed tickets. The stress tests are what make it a senior answer.

Go deeperThe API2 min
GET /api/v1/docs/d_9f3
200 OK
{ "title": "Q3 plan", "role": "editor", "rev": 18342,
"socket": "wss://collab.example.com/docs/d_9f3",
"ticket": "eyJ... (signed: user, doc, role, expires in 5 minutes)" }

Everything after that runs over one WebSocket per open document:

// client → server: an edit made on revision 18342, the client's 12th batch
{ "type": "op", "baseRev": 18342, "clientId": "c7", "seq": 12,
"ops": [{ "insert": "launch ", "pos": 120 }, { "delete": 4, "pos": 131 }] }
// server → every session, including the sender (for whom it is the ack)
{ "type": "rev", "rev": 18345, "clientId": "c7", "seq": 12, "ops": [ ... transformed ... ] }
// either way, lossy: the newest one wins, nothing is stored
{ "type": "cursor", "clientId": "c7", "rev": 18345, "anchor": 127, "head": 127 }
// client → server after a reconnect
{ "type": "resume", "lastRev": 18342 }
{ "type": "error", "code": "rev_too_old" } // older than the op log keeps: reload
{ "type": "error", "code": "access_revoked" } // the socket closes after this

Restoring an old version is a new edit that turns the current text into the old one, so history stays a straight line.

Go deeperThe stores, and how routing could work2 min
-- Op log: a store with strongly consistent conditional writes (Bigtable, HBase, DynamoDB).
-- Key (doc_id, rev). Written "only if (doc_id, rev) is absent".
op_log(doc_id, rev, client_id, client_seq, user_id, ts, ops BLOB)
-- Snapshots: object storage, docs/{doc_id}/{rev}.snap. The document at that rev, plus each
-- recent client's last seq, so a new owner can still spot retries.
-- Metadata: Postgres.
CREATE TABLE documents (
doc_id TEXT PRIMARY KEY,
title TEXT NOT NULL,
snapshot_rev BIGINT NOT NULL, -- newest snapshot; the log after it is the tail
acl_version BIGINT NOT NULL -- bumped on every sharing change
);
CREATE TABLE acl (
doc_id TEXT,
principal TEXT, -- 'user:42', 'group:eng', 'domain:example.com', 'link'
role TEXT NOT NULL, -- owner, editor, commenter, viewer
PRIMARY KEY (doc_id, principal)
);
Routing Problem
Any server, shared database Every edit is a database race; transforms need a history nobody holds
Sticky load balancing by doc ID Adding a server moves documents with no handover; a crash leaves no owner
A directory row per document A write on every open, and a lease per document to renew
Slot leases in etcd One more critical part; a slot moves all its documents together

Saving the whole document on every change would turn a 200-byte edit into a 50 KB write. That’s 50 GB/s at peak. And it would throw away the history that OT, catch-up and undo need. Store edits. The snapshot is derived.

The design is complete. Now we test it where shared editing is hard. Two edits to one sentence. An owner that crashes or pauses. Three hours offline. Cursors and undo. And one very popular document. Only the last one adds boxes.

Interviewerminute 20

Alice and Bob both edit the same sentence at the same moment. How do their copies end up the same?

What they are testing

Whether you can explain one algorithm with an example, and compare it honestly to the other.

The document is “abc”. Alice inserts “X” at position 1, while Bob deletes the “b” at position 1, each on their own copy. Applied naively in either order, one of them deletes the wrong character.

OT with a central server fixes this by rewriting. The server orders Alice’s insert first, rev 1, “aXbc”. Bob’s delete was made on rev 0, so it is transformed past the insert: “delete at 2”, giving “aXc”. On Bob’s screen, the incoming insert is transformed past his own unacked delete. It stays at 1, and he also gets “aXc”. When two inserts land on the same spot, the one the server ordered first goes first, on every copy. That keeps them converging.

Ot.javadelete.pyOt.cse.goop.ts
// An edit is a list of these primitives, applied left to right. Each position counts characters
// in the text as it is when that primitive applies, so a list can be transformed piece by piece.
sealed interface Op permits Insert, Delete {}
record Insert(int pos, String text) implements Op {}
record Delete(int pos, int len) implements Op {}
// Two lists made on the same text, each rewritten to follow the other.
record Pair(List<Op> a, List<Op> b) {}
final class Ot {
// a rewritten to apply after b, when both were made on the same text. aFirst breaks the one
// real tie, two inserts at the same spot: the op the server ordered first goes first, so
// every copy of the document breaks the tie the same way.
static List<Op> transformOne(Op a, Op b, boolean aFirst) {
if (a instanceof Insert x && b instanceof Insert y) {
boolean before = x.pos() < y.pos() || (x.pos() == y.pos() && aFirst);
return List.of(before ? x : new Insert(x.pos() + y.text().length(), x.text()));
}
if (a instanceof Insert x && b instanceof Delete y) {
if (x.pos() <= y.pos()) return List.of(x);
// Inside the deleted range, the insert lands where the range was: the typing survives.
return List.of(new Insert(Math.max(y.pos(), x.pos() - y.len()), x.text()));
}
Delete x = (Delete) a;
int end = x.pos() + x.len();
if (b instanceof Insert y) {
if (y.pos() <= x.pos()) return List.of(new Delete(x.pos() + y.text().length(), x.len()));
if (y.pos() >= end) return List.of(x);
// Someone typed inside the range we deleted: delete around their text, not over it.
return List.of(new Delete(x.pos(), y.pos() - x.pos()),
new Delete(x.pos() + y.text().length(), end - y.pos()));
}
Delete y = (Delete) b;
int yEnd = y.pos() + y.len();
if (end <= y.pos()) return List.of(x);
if (x.pos() >= yEnd) return List.of(new Delete(x.pos() - y.len(), x.len()));
// They overlap, and b already removed the shared part. Delete what is left, or nothing.
int left = x.len() - (Math.min(end, yEnd) - Math.max(x.pos(), y.pos()));
return left == 0 ? List.of() : List.of(new Delete(Math.min(x.pos(), y.pos()), left));
}
// Lists, one primitive at a time. Both halves of the result matter: each piece of b must
// also move past the pieces of a it has crossed.
static Pair transform(List<Op> a, List<Op> b, boolean aFirst) {
if (a.isEmpty() || b.isEmpty()) return new Pair(a, b);
if (a.size() == 1 && b.size() == 1) {
return new Pair(transformOne(a.get(0), b.get(0), aFirst),
transformOne(b.get(0), a.get(0), !aFirst));
}
if (a.size() > 1) {
Pair head = transform(a.subList(0, 1), b, aFirst);
Pair tail = transform(a.subList(1, a.size()), head.b(), aFirst);
return new Pair(concat(head.a(), tail.a()), tail.b());
}
Pair head = transform(a, b.subList(0, 1), aFirst);
Pair tail = transform(head.a(), b.subList(1, b.size()), aFirst);
return new Pair(tail.a(), concat(head.b(), tail.b()));
}
static String apply(String text, List<Op> ops) {
StringBuilder sb = new StringBuilder(text);
for (Op op : ops) {
// An op that does not fit is a bug or a hostile client: refuse it, never guess.
switch (op) {
case Insert i -> sb.insert(i.pos(), i.text()); // throws when out of range
case Delete d -> {
if (d.pos() < 0 || d.pos() + d.len() > sb.length()) {
throw new IllegalArgumentException(d + " does not fit"); // delete would clip
}
sb.delete(d.pos(), d.pos() + d.len());
}
}
}
return sb.toString();
}
private static List<Op> concat(List<Op> a, List<Op> b) {
List<Op> all = new ArrayList<>(a);
all.addAll(b);
return all;
}
}
// The owner of one document. Every edit to it passes through here, so this one place picks the
// order, the revision number, and OT only ever has to transform two ops at a time.
final class DocSession {
record Accepted(int rev, List<Op> ops) {}
// log.get(i) made revision i + 1. Production keeps the last 1,000 here and reads older ones
// from the op log.
private final List<List<Op>> log = new ArrayList<>();
private String text;
DocSession(String text) { this.text = text; }
synchronized int rev() { return log.size(); }
synchronized String text() { return text; }
// baseRev is the revision the client's edit was made on. Everything accepted since then is
// concurrent with it, so the edit is rewritten past each of those, oldest first.
synchronized Accepted receive(int baseRev, List<Op> ops) {
if (baseRev < 0 || baseRev > log.size()) {
throw new IllegalArgumentException("unknown base revision " + baseRev);
}
for (int r = baseRev; r < log.size(); r++) {
ops = Ot.transform(ops, log.get(r), false).a(); // the logged op was ordered first
}
text = Ot.apply(text, ops); // throws before anything changes if it does not fit
log.add(ops);
// Stage 3: here the owner writes (doc, rev) to the op log, and acks and broadcasts only
// once that write has committed.
return new Accepted(log.size(), ops);
}
}

A CRDT works differently. Every character gets a unique ID, and an insert says “after the character with ID 7”, not “at position 1”. Merges then work in any order, with no server deciding. Yjs and Automerge are mature examples.

We pick OT with a central server. This system has a server in the middle anyway, for permissions, storage and history. And with one owner per document, the hard part of OT disappears. Without a server, OT needs a property called TP2, which several published algorithms got wrong. With one total order, each client only transforms against one sequence, and the simpler TP1 is enough.

Go deeperFour approaches, compared1 min
Approach How it works Loses
Locks or last writer wins Lock a paragraph while someone types, or keep the newest save Blocks people or loses their work
Differential sync Diff the copies now and then and patch fuzzily Fuzzy patches can misplace edits; no edit history for undo or cursors
OT with a central server The server orders edits; each is rewritten past earlier ones One authority per document; a transform for every pair of edit types
CRDT (Yjs, Automerge) Every character has an ID; inserts reference neighbors by ID IDs and tombstones per character; interleaving in some algorithms

Interleaving. In some CRDTs, “Alice” and “Bob” typed at the same spot can merge as “ABloibce”. OT can interleave too, when it transforms one character at a time, since the tie rule decides, not intent. Batching makes it rare. A delete that must split around text typed inside it is the case naive OT versions get wrong.

Formatting. Bold and italic on overlapping ranges commute, so both apply. On the same attribute (one bolds, one unbolds), the later revision wins on the overlap, the same on every copy.

The owner of a document crashes right after writing an edit and before acking it. A minute later, a garbage-collection pause makes another server think it still owns the document.

Three rules make the owner safe:

  1. Ack only after the write commits, so a crash loses only edits nobody was told about.
  2. Write each revision only if it’s empty. Row (doc, rev) is the fence: two owners can never both commit rev 18345.
  3. Rebuild from storage. A new owner loads snapshot plus tail: the text, the latest revision and each recent client’s last sequence.

In the crash case, Alice resends her batch to the new owner with the same sequence. The new owner finds it already committed as rev 18345, and simply tells her: that was your ack.

An owner crashes between the commit and the ack
Owner 2Op logOwner 1Alice's clientOwner 2Op logOwner 1Alice's clientslot lease expires after 10 s, owner 2 takes the slotop, base 18342, seq 12write rev 18345 if absentcommittedack lost, owner 1 crashesresume, last rev 18344snapshot plus tail up to 18345rev 18345 is c7 seq 12, that is your ackOwner 2Op logOwner 1Alice's clientOwner 2Op logOwner 1Alice's clientslot lease expires after 10 s, owner 2 takes the slotop, base 18342, seq 12write rev 18345 if absentcommittedack lost, owner 1 crashesresume, last rev 18344snapshot plus tail up to 18345rev 18345 is c7 seq 12, that is your ack
Scroll or pinch to zoom, drag to pan
  • Color key
  • Service
  • Database

The paused owner is the subtle case. Its lease expired during the pause. A new owner committed rev 18346. Then the old one wakes up, and tries to write its own rev 18346. Checking the lease just before writing doesn’t help. The pause can fall between the check and the write. The conditional write refuses it, and the old owner drops the document and closes its sessions.

A paused owner wakes up after a takeover
Owner 2Op logOwner 1 (paused)Owner 2Op logOwner 1 (paused)GC pause outlives the leasewakes, still thinks it owns the docno ack, drops the doc, closes its sessionswrite rev 18346 if absentcommittedwrite rev 18346 if absentrefused, row existsOwner 2Op logOwner 1 (paused)Owner 2Op logOwner 1 (paused)GC pause outlives the leasewakes, still thinks it owns the docno ack, drops the doc, closes its sessionswrite rev 18346 if absentcommittedwrite rev 18346 if absentrefused, row exists
Scroll or pinch to zoom, drag to pan
  • Color key
  • Service
  • Database

One detail makes this airtight. An owner has one write in flight per document. It writes rev N+1 only after N commits, so a successful write proves the log matches its memory. Say the zombie wins a race, and slips in rev 18346 first. The new owner’s write is refused. It checks it still holds the lease, reads the tail, transforms its pending edit past 18346, and retries as 18347.

Planned moves aren’t failures. On a deploy, the old owner stops accepting edits for a slot. It finishes its writes, releases the lease, and tells the gateways. That pauses edits for well under a second. The 10-second lease timeout is only paid on real crashes.

Go deeperWays to guard against a second owner1 min
Guard How it works Verdict
Lease timing margins Stop writing before the lease ends, take over after Unsafe: any pause longer than the margin puts two writers on one revision
Fencing token checked by the store Each lease has a rising number; writes carry it Correct, but needs a store that checks it on every write
Conditional write per revision Row (doc, rev) written only if absent The revision slot is the fence, at the cost of a normal write

Bob edits on a plane for three hours, then lands. What happens to his edits, and to the ones made while he was gone?

The client keeps three things. The last revision it applied. At most one batch in flight, with its sequence. And a buffer of everything typed since.

On reconnect, it asks for every revision after the last one it applied. If one of them carries its own client ID and in-flight sequence, only the ack was lost. It transforms its pending edits past the others, and resends whatever is still in flight under the same sequence. That is the same code as live editing, just with a longer gap.

DocClient.javadoc_client.pyDocClient.csdoc_client.godoc-client.ts
// One accepted edit as the owner broadcasts it. clientId and seq say who sent it, so a client
// can recognize its own edit coming back: that is its ack.
record Revision(int rev, String clientId, long seq, List<Op> ops) {}
interface DocServer {
// Every revision after rev: recent ones from the owner's memory, older from the op log.
List<Revision> opsSince(String docId, int rev);
// Safe to repeat: the owner keeps the last seq per client and answers a repeat by sending
// the original revision again, never by applying the edit twice.
void submit(String docId, String clientId, long seq, int baseRev, List<Op> ops);
}
// The client half of the protocol. Edits apply locally at once, at most one batch is in flight,
// and everything typed meanwhile, or while offline, waits in the buffer.
final class DocClient {
private final String docId;
private final String clientId;
private final DocServer server;
private String text;
private int serverRev; // the last revision this client has applied
private long inFlightSeq = 0; // 0: nothing in flight
private List<Op> inFlight = List.of();
private List<Op> buffer = new ArrayList<>();
private long nextSeq = 1;
private boolean connected = true;
DocClient(String docId, String clientId, DocServer server, String text, int rev) {
this.docId = docId;
this.clientId = clientId;
this.server = server;
this.text = text;
this.serverRev = rev;
}
String text() { return text; }
void edit(List<Op> ops) {
text = Ot.apply(text, ops); // local echo: typing never waits for the network
buffer.addAll(ops);
flush();
}
// Every revision from the owner, live or during catch-up, in revision order.
void onRevision(Revision r) {
if (r.rev() <= serverRev) return; // a repeat caused by a resent batch, already applied
serverRev = r.rev();
if (inFlightSeq != 0 && r.clientId().equals(clientId) && r.seq() == inFlightSeq) {
inFlightSeq = 0; // our own batch came back: the ack
inFlight = List.of();
flush();
return;
}
// Someone else's edit, ordered before everything of ours not yet acked. Move it past our
// pending edits, and our pending edits past it, then apply it.
Pair p = Ot.transform(inFlight, r.ops(), false);
inFlight = p.a();
Pair q = Ot.transform(buffer, p.b(), false);
buffer = new ArrayList<>(q.a());
text = Ot.apply(text, q.b());
}
void onDisconnect() { connected = false; } // keep typing: edits pile up in the buffer
// Catch up by revision, not by time: the owner says exactly what this client missed.
void onReconnect() {
connected = true;
for (Revision r : server.opsSince(docId, serverRev)) onRevision(r);
if (inFlightSeq != 0) {
// Not in the log yet. Resend under the same seq, rebased on what we just applied: if
// the old copy lands after all, the owner recognizes the seq and keeps only one.
server.submit(docId, clientId, inFlightSeq, serverRev, inFlight);
} else {
flush();
}
}
private void flush() {
if (!connected || inFlightSeq != 0 || buffer.isEmpty()) return;
inFlightSeq = nextSeq++;
inFlight = buffer;
buffer = new ArrayList<>();
server.submit(docId, clientId, inFlightSeq, serverRev, inFlight);
}
}
Collaborative editor · architecture

Trace a request: Reconnect after offline

EditorsLoad balancerL4, TLSGatewaysWebSockets onlySession serversone owner per docOp logdoc, revDoc serviceopen, ACL checkMetadata DB+ read replicasPlacementslot leases, etcdSnapshotterevery 500 opsSnapshotsobject storage
Color keyServiceDatabaseBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Reconnect after offline: step 1 of 9

Editors to Load balancer

The socket dropped 3 minutes ago in a train tunnel. The user kept typing: 40 edits sit in the local buffer, and batch seq 12 (base rev 18342) was in flight with no ack.

20 ms of a 300 ms budget
  1. Editors → Load balancer20 ms
  2. Load balancer → Doc service1 ms
  3. Editors → Load balancer20 ms
  4. Load balancer → Gateways1 ms
  5. Gateways → Session servers1 ms
  6. Session servers → Op log8 ms
  7. Session servers → Gateways1 ms
  8. Gateways → Editors20 ms
  9. Response
Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

What we can’t promise. A client offline longer than the 30 days of raw edits gets rev_too_old. The honest fallback keeps its text as a separate copy, “your offline changes”, instead of guessing. And a long offline merge is correct, but can read oddly. If Bob rewrote a paragraph Alice deleted, his insertions survive as fragments. OT keeps every character. It can’t keep meaning.

Go deeperOther ways to reconnect1 min
On reconnect What happens Verdict
Reload the document Fetch the latest snapshot and drop local state Loses everything typed offline
Upload the whole text, three-way merge The server diffs base, theirs and mine Fuzzy; loses intent on overlapping edits
Catch up by revision, rebase the buffer Fetch revisions after the last applied, transform pending edits Exact, and the same code as live editing

The owner keeps the last 1,000 revisions in memory, so most catch-ups never touch storage. After a gateway dies, its 60,000 clients back off with jitter, so they don’t all return in the same second.

Everyone wants to see each other’s cursor live. And when I press undo, what does it undo?

Cursors move on every keystroke and click, so they can cost more than the edits. They ride the same socket, on a lossy path. The owner passes them on, at most 10 a second per user. The newest wins, and they’re never stored. Under load, they are dropped first.

A cursor message carries the revision it refers to. A receiver that is further ahead transforms the position past the later revisions, like an empty insert, so it sits on the right character. Presence is the owner’s list of attached sessions, expired 30 seconds after the last heartbeat.

Undo means “undo my last edit”, not the document’s last edit. The client keeps its own edits, with the revision each became. To undo one, it inverts it: an insert becomes a delete of that text. It transforms the inverse past every later revision, and sends it as a new edit. If someone already deleted that text, the inverse transforms to nothing, and undo does nothing. That’s right.

Go deeperWhere cursors could live, and the snapshot cadence1 min
Approach Cost Verdict
Cursor moves as edits in the op log Doubles or triples the log; history full of noise No: nobody needs yesterday’s cursor
A separate presence service A second system that must know who is in each doc Fine, but the owner already knows
Passed on by the owner, never stored Coalesced to 10 a second per user Dropped first under load; rebuilt on attach

Undo stacks, and grouping keystrokes into undo steps, are in the text editor design.

Snapshot every Snapshot writes at peak Edits replayed on a cold load Fits
50 edits 20,000/s up to 50 Huge documents where a replay is costly
500 edits 2,000/s up to 500, a few ms Typical documents
10,000 edits 100/s up to 10,000 Logs that are rarely reopened
Section titled “One very popular document, and far-away editors”

Traffic grows ten times, one document is an all-hands agenda with ten thousand people watching, and some editors are in Tokyo. What breaks?

At ten times the load, nothing breaks by itself: every tier is split by document, so we add gateways, owners and log nodes. One document is different. An all-hands agenda has 30 people typing, about 60 edits a second, and 10,000 viewers. Sending every edit to every viewer is 600,000 deliveries a second from one owner, which can send about 20,000.

Ordering is cheap. The fan-out is the problem. So split them. Live editors, capped at 100, stay attached to the owner. Viewers attach through viewer relays. The owner sends each revision once to a few relays, which pass it to the gateways holding viewers. Viewers are read only, so nothing they do needs ordering.

Collaborative editor · architecture

Stage 5: Relays for hot documents

EditorsLoad balancerL4, TLSGatewaysWebSockets only84 servers, 60% busySession serversone owner per doc84 servers, 60% busyOp logdoc, rev67 servers, 60% busyDoc serviceopen, ACL check8 servers, 57% busyMetadata DB+ read replicas5% busyPlacementslot leases, etcdViewer relayshot docsnew9 servers, 56% busySnapshotterevery 500 ops7 servers, 57% busySnapshotsobject storage
Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional
Stage

Stage 5 of 6

Relays for hot documents

Why this stage

Not total load: one document.

An all-hands agenda with 10,000 viewers and 30 typing editors is 600,000 deliveries a second from one owner, which pushes about 20,000.

What we add

Viewer relays.

The owner sends each revision once to a few relays, which fan it out to the gateways holding viewers.

Live editors stay capped at 100 per document.

What it costs

Viewers see edits one hop later and are read only.

A viewer who starts editing moves to the owner, and an editor over the cap waits.

At 1M ops per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
The numbers behind the busy bars

How busy it gets

  • There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
  • Opens and reconnects add about 23,000 a second at peak.

What one machine can do

  • One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
  • One session server handles about 20,000 edits a second, including the durable write and the broadcast.
  • One op log node takes about 25,000 conditional writes a second.
  • One Postgres handles about 20,000 simple writes or reads a second.

What saves work

  • The doc service answers about 90% of permission checks from its cache.
  • There is one snapshot every 500 edits.
  • Relays deliver, on average, about half a viewer copy per edit.

Far-away editors. An editor in Tokyo, on a document owned in Virginia, waits about 150 ms for every ack. So gateways and doc services run in every region. Each document has a home region for its owner and log, near most of its editors. A Tokyo edit takes one cross-region hop, which local echo hides.

Collaborative editor · architecture

Stage 6: Regions

Every region runs gatewaysA doc’s owner and op loglive in its home region.Edits from elsewhere takeone cross-region hop.EditorsLoad balancerL4, TLSGatewaysWebSockets only84 servers, 60% busySession serversone owner per doc84 servers, 60% busyOp logdoc, rev67 servers, 60% busyDoc serviceopen, ACL check8 servers, 57% busyMetadata DB+ read replicas5% busyPlacementslot leases, etcdViewer relayshot docs9 servers, 56% busySnapshotterevery 500 ops7 servers, 57% busySnapshotsobject storage
Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional
Stage

Stage 6 of 6

Regions

Why this stage

An editor in Tokyo pays about 150 ms to a US owner for every ack and every remote edit, and one region is one failure domain.

What we add

Gateways and doc services in every region.

Each document has a home region near its editors, where its owner and op log live; the op log is copied asynchronously to a second region.

What it costs

Editors outside the home region wait one cross-region round trip for acks (local echo hides it), and losing the home region can lose the last few seconds of acked edits.

At 1M ops per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
The numbers behind the busy bars

How busy it gets

  • There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
  • Opens and reconnects add about 23,000 a second at peak.

What one machine can do

  • One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
  • One session server handles about 20,000 edits a second, including the durable write and the broadcast.
  • One op log node takes about 25,000 conditional writes a second.
  • One Postgres handles about 20,000 simple writes or reads a second.

What saves work

  • The doc service answers about 90% of permission checks from its cache.
  • There is one snapshot every 500 edits.
  • Relays deliver, on average, about half a viewer copy per edit.

The log copies synchronously across 3 zones in the home region, and asynchronously to a second region. Synchronous copies across regions would add 70 to 150 ms to every ack, for an event that happens a few times a decade. So losing the home region can drop a few seconds of acked edits: the stated recovery point.

Go deeperFan-out options, and very large documents1 min
Fan-out for 10,000 viewers Owner sends per revision Problem
Owner to every viewer session 10,000 600,000 deliveries a second from one server
Owner to relays, relays to gateways 5 to 20 One more tier; viewers one hop, about 5 ms, behind
Viewers poll a snapshot every few seconds 0 Not live; right for a document published to the world

Why not let each region order its own editors’ changes? That is two owners for one document, which is exactly where a CRDT would fit. For documents whose editors are mostly in one place, one cross-region hop is cheaper than a second merge algorithm.

Very large documents, a million characters, stress the snapshot and the client, not the transform. Snapshots are stored in 64 KB chunks, so the snapshotter rewrites only changed chunks. The client loads chunks near the screen first. And the owner keeps the text in a rope, so an insert in the middle doesn’t copy megabytes.

Interviewerminute 40

A session server dies. What do the people in its documents see?

What they are testing

Whether typing survives, nothing acked is lost, and users see only a short “reconnecting” badge.

Typing never stops, and acked edits never disappear, even when syncing pauses.

When a session server dies, its slot leases stop renewing. After 10 seconds, other servers take its slots and load each document from snapshot plus tail. Meanwhile, clients show “reconnecting” and keep typing into their buffers. When they reattach, they catch up by revision and resend by sequence, exactly as after a flight.

Under stress, shed cursors and presence first, then viewer relays (viewers fall back to polling a snapshot), then new opens. Edits from people already in a document go last.

Click any box to break it and see what users would notice:

Collaborative editor · architecture

Break it: what fails, who notices, what users see

Every region runs gatewaysA doc’s owner and op loglive in its home region.Edits from elsewhere takeone cross-region hop.EditorsLoad balancerL4, TLSGatewaysWebSockets onlySession serversone owner per docOp logdoc, revDoc serviceopen, ACL checkMetadata DB+ read replicasPlacementslot leases, etcdViewer relayshot docsSnapshotterevery 500 opsSnapshotsobject storage
Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional
Break

Failure: click any box to break it

A session server goes down

What breaks

Every document it owns, about 1 in 85 of the open ones, has no owner.

Edits to them cannot be ordered.

How we notice

Its slot leases in etcd stop being renewed; they expire after 10 seconds.

What users see

A "reconnecting" badge for about 10 to 15 seconds.

Typing keeps working locally and nothing acked is lost.

What we do

Other servers take the slots, load each document from its snapshot and op log tail when the first client reattaches, and clients resend unacked batches with the same seq.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
Go deeperEvery failure, and what users see1 min
Part Failure What users see What we do
Session server Crash Sync pauses 10 to 15 s for its documents; typing continues Slots move; load from snapshot plus tail; clients resend by seq
Gateway Crash One or two seconds of reconnecting Jittered reconnect; tickets checked locally
Op log partition Loses quorum Edits show locally but don’t sync (“saving”) Nothing acked or sent uncommitted; retries drain in order
Metadata DB Primary down New opens on cache misses fail; open documents fine Failover in about 30 s; serve cached permissions
Whole region Outage Documents homed there pause; a few seconds of acks at risk Move homes to the region with the copy; clients resend by seq
Operation Guarantee What users can see
Order of edits in a doc One order, decided by the owner, fenced by the log Everyone ends with the same text
My own edits Immediate (local echo) My typing is never delayed or reordered on my screen
Others’ edits Converge; in order per sender A colleague’s edit appears 50 to 200 ms later, maybe shifted by mine
Acked edit In 2 of 3 zones; async to a second region Never lost short of a region loss
Cursors and presence Best effort A cursor can lag, jump, or linger 30 s after someone leaves

Convergence holds only because each document has one strict order. A last-write-wins op log would let two owners both write rev 18346 and silently drop one. The clients converge because the server is strict.

Go deeperRunning it: alerts, deploys and cost1 min

Wake someone up when:

  • Ack latency p99 passes 500 ms for 5 minutes.
  • The oldest unacked batch on clients passes 60 seconds, for more than 0.1% of sessions. (It catches every way sync can break.)
  • Conditional writes fail on a log partition.
  • Any slot has no owner for 30 seconds.

Catch divergence. Every few hundred revisions, clients send a hash of their text with its revision, and the owner compares it with its own. A mismatch is a transform bug: that client reloads from the owner, and the edit sequence is saved for the bug report. Without this check, divergence is silent.

Deploys. Slots move by handover, a few percent at a time. Transform code is the riskiest change, because a bug corrupts documents silently. New edit types ship behind a protocol version, tested with randomized multi-client fuzzing.

Cost is mostly the op log (writes and 110 TB of history) and the owners’ memory. Batching edits per row and folding old history into snapshots are the two biggest levers.

Here is everything we built, in the order we built it. Use the buttons to step through it.

Collaborative editor · architecture

The full design, stage by stage

Every region runs gatewaysA doc’s owner and op loglive in its home region.Edits from elsewhere takeone cross-region hop.EditorsLoad balancerL4, TLSGatewaysWebSockets only84 servers, 60% busySession serversone owner per doc84 servers, 60% busyOp logdoc, rev67 servers, 60% busyDoc serviceopen, ACL check8 servers, 57% busyMetadata DB+ read replicas5% busyPlacementslot leases, etcdViewer relayshot docs9 servers, 56% busySnapshotterevery 500 ops7 servers, 57% busySnapshotsobject storage
Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional
Stage

Stage 6 of 6

Regions

Why this stage

An editor in Tokyo pays about 150 ms to a US owner for every ack and every remote edit, and one region is one failure domain.

What we add

Gateways and doc services in every region.

Each document has a home region near its editors, where its owner and op log live; the op log is copied asynchronously to a second region.

What it costs

Editors outside the home region wait one cross-region round trip for acks (local echo hides it), and losing the home region can lose the last few seconds of acked edits.

At 1M ops per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.The whole design. Step through it: one server and one database, gateways and one owner per document, an op log and snapshots, opening and permissions, relays for hot documents, then regions.
The numbers behind the busy bars

How busy it gets

  • There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
  • Opens and reconnects add about 23,000 a second at peak.

What one machine can do

  • One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
  • One session server handles about 20,000 edits a second, including the durable write and the broadcast.
  • One op log node takes about 25,000 conditional writes a second.
  • One Postgres handles about 20,000 simple writes or reads a second.

What saves work

  • The doc service answers about 90% of permission checks from its cache.
  • There is one snapshot every 500 edits.
  • Relays deliver, on average, about half a viewer copy per edit.
Decision We chose We gave up We’d choose differently if
Merge OT with a central owner per document Serverless merging, long offline drift Peer to peer or end-to-end encrypted: a CRDT
Routing Slot leases in etcd, gateways cache the map The simplicity of sticky balancing Documents never moved and servers never failed
Durability Ack after a 2-of-3 zone commit, async copy Zero loss on region failure Regulation required it: synchronous across regions
Permissions Cached roles, signed tickets, revoke events Instant revocation in every case Documents were public
Hot documents Viewer relays, 100 live editors Unlimited live editors Thousands edited at once: a different product
Regions A home region per document Equal latency for far editors Editors of one document were always spread worldwide
  • 0 to 4. Rich text, editors per document, offline, history.
  • 4 to 8. 5.2 million sockets, a million edits a second, and why clients batch.
  • 8 to 20. Build it: gateways, one owner per document, the transform, the op log with conditional writes. A mid-level answer can stop here.
  • 20 to 30. The stress tests: OT or CRDT with the “abc” example, then owner failover.
  • 30 to 40. Offline catch-up, then cursors and undo, or the hot document.
  • 40 to 45. A session server dying, then your own questions.

Never skip the merge algorithm, the single owner, or acking after the write.

Interviewer asksWould you build this with a CRDT today?

For a product that must work offline for days, peer to peer, or end-to-end encrypted, yes: Yjs and Automerge are mature. For a server-centric product like this one, OT with an owner is simpler to store and reason about.

Interviewer asksProduct wants comments anchored to text. Where do they go?

In their own table, with an anchor: a range plus the revision it was made on. Like a cursor, the owner transforms the anchor past later revisions and saves current anchors with each snapshot. If the text is deleted, the comment shows as orphaned.

Go deeperMore follow-up questions1 min

Product wants suggestion mode (tracked changes). What changes? A suggestion is an edit with a mark. Inserted text is marked “suggested by Bob”. Deletions are marked, not removed. Accepting is a new edit that clears the mark. Rejecting is its inverse. Only the edit types change, not the server.

A revoked user still has the document open. How fast are they cut off? Within seconds: the sharing change publishes an event, and the owner closes their session. If the event is lost, a stale cache entry can let them reopen it for up to a minute.

Why a WebSocket and not REST calls per edit? The server must push remote edits, acks and cursors the moment they exist, in revision order, on one stream. Server-sent events plus POST would work. But they split one ordered stream into two.