Design a collaborative editor
Design a collaborative editor like Google Docs for 5 million open sockets. Start with one server and one database, grow it with gateways and one owner per document, ordered edits, an op log with snapshots and signed tickets, then stress it with OT against CRDTs, a crashed owner, offline editing, undo and one very popular document, with code.
“Design Google Docs. Several people edit the same document at the same time and see each other’s changes live.”
A collaborative editor looks like a text box with a save button that runs very often. The hard part is that two people type into the same sentence at the same moment. Each types on their own copy, over networks that delay and drop messages. Every copy must end up identical, and nobody may lose a keystroke. Locking a paragraph, or keeping the last save, isn’t an answer. Both throw away work the user saw on screen.
The single-user side (commands, undo stacks, the text buffer) is the text editor with undo design. This page is about what changes when the document is shared. It starts with one server and one database, and lets the numbers grow it. Then it tests the result against two edits at once, a crashed owner, a long flight and undo.
The answer in one minute
Section titled “The answer in one minute”If you only have a minute, this is the design. The rest of the page explains each part.
- Type locally, at once. Each client shows its own edit immediately, and sends it with the revision (the version number) it was typed on.
- One owner per document decides the order. Every editor of a document reaches the same server. It rewrites each edit past the ones accepted since, using operational transformation (OT), and gives it the next revision.
- Ack only after a durable write. Each revision is written to an op log “only if this revision is empty”. That one rule stops two owners from both writing it.
- Catch up by revision. A client that was offline asks for everything after its last revision. It rewrites its own pending edits past them, and sends them.
- Cursors are lossy. They go through the owner and are never stored, so they never cost what edits cost.
Scope it
Section titled “Scope it”Three questions pick the merge algorithm: what is edited, how many people edit one document, and whether offline editing is in scope.
Design Google Docs. Several people edit one document at once.
Whether you pin down what is edited, how many people edit one document, and whether offline editing is in scope. Those three pick the merge algorithm. The red flag is jumping to “use CRDTs” before asking any of them.
These four questions matter most:
- What is edited? Rich text: characters plus formatting, lists and tables.
- How many editors per document? Usually 1 to 5, at most 100 live editors. Viewers unlimited.
- Offline? Yes, for hours, then merge on reconnect.
- Version history? Every change for 30 days, named versions forever.
From these answers we make five promises:
- Instant typing. My own edits show at once. A colleague sees them within 200 ms at p99 in one region.
- Converge. Once edits stop, every copy is identical and contains every edit anyone saw.
- Never lose an acked edit. An acknowledged edit is stored in 2 of 3 zones.
- Always editable. 99.95% available. A failover can pause syncing for about 15 seconds, while typing continues locally.
- Revokes work. A user whose access is removed is cut off from an open document within seconds.
We leave out the rich-text renderer, full-text search, and spreadsheets (the same skeleton with different edits).
Go deeperMore questions, and what each changes1 min
| Question | We assume | If the answer were different |
|---|---|---|
| Plain text only? | No, rich text | Only insert and delete to transform |
| Thousands of live editors? | No, 100 at most | A different fan-out and merge design |
| Online only? | No, offline for hours | The server could reject stale edits instead of merging them |
| Real-time? | Yes, about 200 ms | “Save and merge every minute” is file sync, a much simpler system |
| End-to-end encryption? | No | The server couldn’t transform edits, which pushes the design to CRDTs |
| Largest document? | About a million characters | Book length needs loading in chunks |
The numbers that decide it
Section titled “The numbers that decide it”Now we work out the numbers. Count open sockets and edits, not stored bytes.
- Daily active users
- 50 million
- Minutes editing per user a day
- 60
- Peak over average
- 2.5×
- Op batches per session a minute
- 12
- Bytes per stored op, with metadata
- 100 bytes
- Days of raw ops kept
- 30
- Document opens per user a day
- 10
- Documents stored
- 5 billion
- Average snapshot size
- 50 KB
50 million daily users, each with a document open about an hour a day.
So we hold millions of long-lived WebSockets, on gateways that do nothing else.
A client sends a batch of whatever was typed since its last batch was acknowledged, one batch in flight at a time. Most sessions are idle or reading, so it averages about 12 batches a minute.
So we can't write edits to one database: they go to a log split by document.
About 20,000 edits a second per server, run at 60% busy. 4 million documents open at once, at about 100 KB each.
So we keep every open document in memory on its owner.
At 100 bytes per stored edit. Thirty days is 110 TB before copies.
So we keep raw edits for 30 days, fold older ones into snapshots, and keep snapshots in object storage.
Send every keystroke on its own, and the peak passes 5 million edits a second. That’s why clients batch: batching is a client decision that sets the whole write load. Try it in the calculator.
- Open sessions, average
- 2.1 million
- Open sessions (sockets), peak
- 5.2 million
- At 100,000 sockets a gateway, a fleet of gateways that only hold connections.
- Ops, average
- 420,000/s
- Ops, peak
- 1,000,000/s
- Past what one database primary writes: ops go to a log store partitioned by document.
- Session servers at peak (20,000 ops/s each, 60% busy)
- 87
- Each owns a share of the documents, by slot lease.
- Open documents in memory at peak (1.3 sessions each)
- 400 GB
- About 100 KB each with recent revisions. Spread over the session servers, a few GB per server.
- Opens, peak
- 14,000/s
- With reconnects, past what one Postgres answers: cache permissions in the doc service.
- Op log written per day
- 3.6 TB
- Raw op history kept
- 110 TB
- Before replication. Older ops are folded into snapshots and named versions.
- Latest snapshots of every document
- 250 TB
- Object storage. Cheap per byte, and only the newest snapshot of each document is hot.
Build the design
Section titled “Build the design”We start with the simplest design that is correct. Then we let the numbers break it, one at a time. Each step fixes only what broke, adds only the boxes that fix needs, and says what the fix costs. Throughout, follow one scenario: Alice and Bob have the same document open, and Alice types a word.
Step 1. One server, one database
Section titled “Step 1. One server, one database”The simplest version is one doc server that holds both WebSockets and the document in memory, with Postgres behind it. Alice’s client shows the word at once and sends it. The server merges it, writes it to Postgres, sends it to Bob, and acks Alice. The merge works (the first stress test explains it).
Stage 1: One server, one database
Stage 1 of 6
One server, one database
Why this stage
Nothing yet.
This is the smallest design that lets two people edit one document.
What we add
One doc server that holds every WebSocket, keeps each open document in memory, orders and transforms edits, and writes every op to Postgres.
What it costs
One box holds about 50,000 sockets and the peak needs 5.2 million.
Every op is a Postgres write, and one crash stops every document.
At 1M ops per second, Gateways is 10000% busy. Stage 2 is the fix.
The numbers behind the busy bars
How busy it gets
- There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
- Opens and reconnects add about 23,000 a second at peak.
What one machine can do
- One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
- One session server handles about 20,000 edits a second, including the durable write and the broadcast.
- One op log node takes about 25,000 conditional writes a second.
- One Postgres handles about 20,000 simple writes or reads a second.
What saves work
- The doc service answers about 90% of permission checks from its cache.
- There is one snapshot every 500 edits.
- Relays deliver, on average, about half a viewer copy per edit.
But one box holds about 50,000 sockets, the peak needs 5.2 million, and one Postgres primary takes about 20,000 writes a second, not a million.
Step 2. Gateways, and one owner per document
Section titled “Step 2. Gateways, and one owner per document”Split the server in two. Gateways hold WebSockets, and nothing else. They check the ticket, answer pings and forward frames. Session servers do the work. Each document has exactly one owner among them, because the order of a document’s edits must be decided in one place.
Document IDs hash to 16,384 slots. Each session server leases a few hundred slots in etcd, and renews them every few seconds. Gateways cache the slot map, so sending a frame to the owner is a hash and a lookup. A dead server’s leases expire after 10 seconds, and others take its slots.
Stage 2: Gateways and one owner per doc
Stage 2 of 6
Gateways and one owner per doc
Why this stage
5.2 million sockets and 1 million ops/s at peak.
The ops for one document must still meet in one place, or nobody can order them.
What we add
An L4 load balancer, gateways that only hold sockets, session servers that each own a share of documents, and a placement service (etcd) that leases 16,384 slots to session servers.
What it costs
Editing a document pauses for seconds while its owner fails over, and every op is still a write to one Postgres.
At 1M ops per second, Metadata DB is 5000% busy. Stage 3 is the fix.
The numbers behind the busy bars
How busy it gets
- There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
- Opens and reconnects add about 23,000 a second at peak.
What one machine can do
- One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
- One session server handles about 20,000 edits a second, including the durable write and the broadcast.
- One op log node takes about 25,000 conditional writes a second.
- One Postgres handles about 20,000 simple writes or reads a second.
What saves work
- The doc service answers about 90% of permission checks from its cache.
- There is one snapshot every 500 edits.
- Relays deliver, on average, about half a viewer copy per edit.
Go deeperWhy slots and leases, and what one owner costs1 min
Why 16,384 slots, not a hash of the server list. If documents hashed straight onto servers, adding a server would move documents in a pattern nobody chose. With slots, the map from slot to server is a small table in etcd. Moving a slot is an explicit step. The old owner gives up its lease, the new one takes it, and gateways see the new map. Rebalancing becomes a list of slot moves we can pace and watch.
Why a lease, not just a heartbeat. A lease is a claim with an end time. A server that can’t renew must stop accepting edits before its lease ends, even if it feels healthy. That rule is what keeps a document from having two owners for long. The stress tests come back to the case where a pause makes a server miss its own deadline.
What one owner costs. All of a document’s edits pass through one thread on one server. That thread can order thousands of edits a second, far more than people can type into one document. The real limits are failover, about 10 seconds while a lease runs out, and sending to many watchers, which a stress test fixes.
Step 3. Order the edits
Section titled “Step 3. Order the edits”Two people can now type into one document at once, through one owner. Every edit carries the revision it was made on, plus a client ID and sequence number. The owner transforms it past every revision accepted since: an insert typed before someone else’s insert shifts right, and so on. It gives the edit the next revision, and sends it to everyone. For the sender, that message is the ack.
The pair (client ID, sequence) makes a retry safe. A client resends an unacked batch with the same sequence. The owner remembers the last sequence per client. So it answers with the original revision, instead of applying it twice.
Edits are now ordered in one place. But every edit still goes to one Postgres primary, which takes about 20,000 writes a second, not a million.
Step 4. An op log, and snapshots
Section titled “Step 4. An op log, and snapshots”Edits go to an op log split by document: one row per revision, keyed (doc, rev). The owner writes row (doc, 18345) only if it doesn’t already exist, and acks only after that write commits in 2 of 3 zones. A crash can never take back an edit anyone was told about.
A snapshotter follows the log. Every 500 edits, it writes the whole document to object storage. A server that has never seen a document loads the newest snapshot, plus the revisions after it, in one range read. That’s never more than about 500 edits to replay.
Stage 3: Op log and snapshots
Stage 3 of 6
Op log and snapshots
Why this stage
1 million op writes a second against a Postgres primary that takes about 20,000, and 3.6 TB of ops a day.
What we add
An op log partitioned by document, one row per revision written only if absent, a snapshotter that saves each document every 500 ops to object storage, and Postgres kept for titles and permissions.
What it costs
Opening a cold document reads a snapshot plus the ops after it, and every open and reconnect still checks permissions in Postgres: about 23,000 reads/s at peak.
At 1M ops per second, Metadata DB is 115% busy. Stage 4 is the fix.
The numbers behind the busy bars
How busy it gets
- There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
- Opens and reconnects add about 23,000 a second at peak.
What one machine can do
- One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
- One session server handles about 20,000 edits a second, including the durable write and the broadcast.
- One op log node takes about 25,000 conditional writes a second.
- One Postgres handles about 20,000 simple writes or reads a second.
What saves work
- The doc service answers about 90% of permission checks from its cache.
- There is one snapshot every 500 edits.
- Relays deliver, on average, about half a viewer copy per edit.
Follow one word from Alice’s keyboard to Bob’s screen:
Trace a request: Type a word
Type a word: step 1 of 7
Editors to Load balancer
The text is already on screen: the client applied it locally. It sends the batch typed since the last send (at most one in flight, every 100 ms or so) on the open WebSocket: base rev 18342, client c7, seq 12.
- Editors → Load balancer20 ms
- Load balancer → Gateways1 ms
- Gateways → Session servers1 ms
- Session servers → Op log6 ms
- Session servers → Gateways1 ms
- Gateways → Editors20 ms
- Response
About 49 ms end to end in one region, against a 200 ms budget, and Alice saw her word at 0 ms.
Step 5. Opening, permissions and catch-up
Section titled “Step 5. Opening, permissions and catch-up”Edits are now fast and safe. But anyone who can reach a gateway can open a socket, and a reconnect storm would check permissions in the database millions of times.
Opening also catches the client up: it gets the newest snapshot and the edits after it, the same range read a new owner uses.
Opening a document is a normal HTTPS call to a doc service. It checks the user’s role through a cache, and returns a signed ticket, valid for 5 minutes, for the editing socket. Gateways check the signature themselves, so a reconnect storm costs CPU, not database reads.
A sharing change publishes an event. The doc service drops its cache entries, and the owner closes the sessions of anyone who lost access.
Stage 4: Opening, permissions and catch-up
Stage 4 of 6
Opening, permissions and catch-up
Why this stage
About 23,000 opens and reconnects a second at peak, each needing a permission check, against a Postgres that serves about 20,000.
A gateway crash adds 60,000 reconnects at once.
What we add
A doc service that checks permissions from a cache fed by revoke events and issues short-lived signed tickets the gateways verify locally, plus metadata read replicas.
Reconnects catch up by revision.
What it costs
A ticket lives 5 minutes, so a revoke must also reach the owner, which closes the revoked sessions itself.
At 1M ops per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
How busy it gets
- There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
- Opens and reconnects add about 23,000 a second at peak.
What one machine can do
- One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
- One session server handles about 20,000 edits a second, including the durable write and the broadcast.
- One op log node takes about 25,000 conditional writes a second.
- One Postgres handles about 20,000 simple writes or reads a second.
What saves work
- The doc service answers about 90% of permission checks from its cache.
- There is one snapshot every 500 edits.
- Relays deliver, on average, about half a viewer copy per edit.
Trace a request: Open a document
Open a document: step 1 of 11
Editors to Load balancer
GET /api/v1/docs/d_9f3 with the session cookie.
- Editors → Load balancer20 ms
- Load balancer → Doc service1 ms
- Doc service → Metadata DB4 ms
- Editors → Load balancer20 ms
- Load balancer → Gateways1 ms
- Gateways → Session servers1 ms
- Session servers → Snapshots30 ms
- Session servers → Op log6 ms
- Session servers → Gateways1 ms
- Gateways → Editors20 ms
- Response
This is enough for a mid-level answer. Gateways. One owner per document, by slot lease. OT against a single order. An op log with conditional writes and snapshots. And signed tickets. The stress tests are what make it a senior answer.
Go deeperThe API2 min
GET /api/v1/docs/d_9f3
200 OK{ "title": "Q3 plan", "role": "editor", "rev": 18342, "socket": "wss://collab.example.com/docs/d_9f3", "ticket": "eyJ... (signed: user, doc, role, expires in 5 minutes)" }Everything after that runs over one WebSocket per open document:
// client → server: an edit made on revision 18342, the client's 12th batch{ "type": "op", "baseRev": 18342, "clientId": "c7", "seq": 12, "ops": [{ "insert": "launch ", "pos": 120 }, { "delete": 4, "pos": 131 }] }
// server → every session, including the sender (for whom it is the ack){ "type": "rev", "rev": 18345, "clientId": "c7", "seq": 12, "ops": [ ... transformed ... ] }
// either way, lossy: the newest one wins, nothing is stored{ "type": "cursor", "clientId": "c7", "rev": 18345, "anchor": 127, "head": 127 }
// client → server after a reconnect{ "type": "resume", "lastRev": 18342 }
{ "type": "error", "code": "rev_too_old" } // older than the op log keeps: reload{ "type": "error", "code": "access_revoked" } // the socket closes after thisRestoring an old version is a new edit that turns the current text into the old one, so history stays a straight line.
Go deeperThe stores, and how routing could work2 min
-- Op log: a store with strongly consistent conditional writes (Bigtable, HBase, DynamoDB).-- Key (doc_id, rev). Written "only if (doc_id, rev) is absent".op_log(doc_id, rev, client_id, client_seq, user_id, ts, ops BLOB)
-- Snapshots: object storage, docs/{doc_id}/{rev}.snap. The document at that rev, plus each-- recent client's last seq, so a new owner can still spot retries.
-- Metadata: Postgres.CREATE TABLE documents ( doc_id TEXT PRIMARY KEY, title TEXT NOT NULL, snapshot_rev BIGINT NOT NULL, -- newest snapshot; the log after it is the tail acl_version BIGINT NOT NULL -- bumped on every sharing change);CREATE TABLE acl ( doc_id TEXT, principal TEXT, -- 'user:42', 'group:eng', 'domain:example.com', 'link' role TEXT NOT NULL, -- owner, editor, commenter, viewer PRIMARY KEY (doc_id, principal));| Routing | Problem |
|---|---|
| Any server, shared database | Every edit is a database race; transforms need a history nobody holds |
| Sticky load balancing by doc ID | Adding a server moves documents with no handover; a crash leaves no owner |
| A directory row per document | A write on every open, and a lease per document to renew |
| Slot leases in etcd | One more critical part; a slot moves all its documents together |
Saving the whole document on every change would turn a 200-byte edit into a 50 KB write. That’s 50 GB/s at peak. And it would throw away the history that OT, catch-up and undo need. Store edits. The snapshot is derived.
Stress it
Section titled “Stress it”The design is complete. Now we test it where shared editing is hard. Two edits to one sentence. An owner that crashes or pauses. Three hours offline. Cursors and undo. And one very popular document. Only the last one adds boxes.
OT or CRDT
Section titled “OT or CRDT”Alice and Bob both edit the same sentence at the same moment. How do their copies end up the same?
Whether you can explain one algorithm with an example, and compare it honestly to the other.
The document is “abc”. Alice inserts “X” at position 1, while Bob deletes the “b” at position 1, each on their own copy. Applied naively in either order, one of them deletes the wrong character.
OT with a central server fixes this by rewriting. The server orders Alice’s insert first, rev 1, “aXbc”. Bob’s delete was made on rev 0, so it is transformed past the insert: “delete at 2”, giving “aXc”. On Bob’s screen, the incoming insert is transformed past his own unacked delete. It stays at 1, and he also gets “aXc”. When two inserts land on the same spot, the one the server ordered first goes first, on every copy. That keeps them converging.
// An edit is a list of these primitives, applied left to right. Each position counts characters// in the text as it is when that primitive applies, so a list can be transformed piece by piece.sealed interface Op permits Insert, Delete {}record Insert(int pos, String text) implements Op {}record Delete(int pos, int len) implements Op {}
// Two lists made on the same text, each rewritten to follow the other.record Pair(List<Op> a, List<Op> b) {}
final class Ot { // a rewritten to apply after b, when both were made on the same text. aFirst breaks the one // real tie, two inserts at the same spot: the op the server ordered first goes first, so // every copy of the document breaks the tie the same way. static List<Op> transformOne(Op a, Op b, boolean aFirst) { if (a instanceof Insert x && b instanceof Insert y) { boolean before = x.pos() < y.pos() || (x.pos() == y.pos() && aFirst); return List.of(before ? x : new Insert(x.pos() + y.text().length(), x.text())); } if (a instanceof Insert x && b instanceof Delete y) { if (x.pos() <= y.pos()) return List.of(x); // Inside the deleted range, the insert lands where the range was: the typing survives. return List.of(new Insert(Math.max(y.pos(), x.pos() - y.len()), x.text())); } Delete x = (Delete) a; int end = x.pos() + x.len(); if (b instanceof Insert y) { if (y.pos() <= x.pos()) return List.of(new Delete(x.pos() + y.text().length(), x.len())); if (y.pos() >= end) return List.of(x); // Someone typed inside the range we deleted: delete around their text, not over it. return List.of(new Delete(x.pos(), y.pos() - x.pos()), new Delete(x.pos() + y.text().length(), end - y.pos())); } Delete y = (Delete) b; int yEnd = y.pos() + y.len(); if (end <= y.pos()) return List.of(x); if (x.pos() >= yEnd) return List.of(new Delete(x.pos() - y.len(), x.len())); // They overlap, and b already removed the shared part. Delete what is left, or nothing. int left = x.len() - (Math.min(end, yEnd) - Math.max(x.pos(), y.pos())); return left == 0 ? List.of() : List.of(new Delete(Math.min(x.pos(), y.pos()), left)); }
// Lists, one primitive at a time. Both halves of the result matter: each piece of b must // also move past the pieces of a it has crossed. static Pair transform(List<Op> a, List<Op> b, boolean aFirst) { if (a.isEmpty() || b.isEmpty()) return new Pair(a, b); if (a.size() == 1 && b.size() == 1) { return new Pair(transformOne(a.get(0), b.get(0), aFirst), transformOne(b.get(0), a.get(0), !aFirst)); } if (a.size() > 1) { Pair head = transform(a.subList(0, 1), b, aFirst); Pair tail = transform(a.subList(1, a.size()), head.b(), aFirst); return new Pair(concat(head.a(), tail.a()), tail.b()); } Pair head = transform(a, b.subList(0, 1), aFirst); Pair tail = transform(head.a(), b.subList(1, b.size()), aFirst); return new Pair(tail.a(), concat(head.b(), tail.b())); }
static String apply(String text, List<Op> ops) { StringBuilder sb = new StringBuilder(text); for (Op op : ops) { // An op that does not fit is a bug or a hostile client: refuse it, never guess. switch (op) { case Insert i -> sb.insert(i.pos(), i.text()); // throws when out of range case Delete d -> { if (d.pos() < 0 || d.pos() + d.len() > sb.length()) { throw new IllegalArgumentException(d + " does not fit"); // delete would clip } sb.delete(d.pos(), d.pos() + d.len()); } } } return sb.toString(); }
private static List<Op> concat(List<Op> a, List<Op> b) { List<Op> all = new ArrayList<>(a); all.addAll(b); return all; }}
// The owner of one document. Every edit to it passes through here, so this one place picks the// order, the revision number, and OT only ever has to transform two ops at a time.final class DocSession { record Accepted(int rev, List<Op> ops) {}
// log.get(i) made revision i + 1. Production keeps the last 1,000 here and reads older ones // from the op log. private final List<List<Op>> log = new ArrayList<>(); private String text;
DocSession(String text) { this.text = text; }
synchronized int rev() { return log.size(); } synchronized String text() { return text; }
// baseRev is the revision the client's edit was made on. Everything accepted since then is // concurrent with it, so the edit is rewritten past each of those, oldest first. synchronized Accepted receive(int baseRev, List<Op> ops) { if (baseRev < 0 || baseRev > log.size()) { throw new IllegalArgumentException("unknown base revision " + baseRev); } for (int r = baseRev; r < log.size(); r++) { ops = Ot.transform(ops, log.get(r), false).a(); // the logged op was ordered first } text = Ot.apply(text, ops); // throws before anything changes if it does not fit log.add(ops); // Stage 3: here the owner writes (doc, rev) to the op log, and acks and broadcasts only // once that write has committed. return new Accepted(log.size(), ops); }}@dataclass(frozen=True)class Insert: pos: int text: str
@dataclass(frozen=True)class Delete: pos: int length: int
# An edit is a list of these primitives, applied left to right. Each position counts characters# in the text as it is when that primitive applies, so a list can be transformed piece by piece.Op = Insert | Delete
# a rewritten to apply after b, when both were made on the same text. a_first breaks the one# real tie, two inserts at the same spot: the op the server ordered first goes first, so every# copy of the document breaks the tie the same way.def transform_one(a: Op, b: Op, a_first: bool) -> list[Op]: if isinstance(a, Insert) and isinstance(b, Insert): before = a.pos < b.pos or (a.pos == b.pos and a_first) return [a if before else Insert(a.pos + len(b.text), a.text)] if isinstance(a, Insert): # b deletes if a.pos <= b.pos: return [a] # Inside the deleted range, the insert lands where the range was: the typing survives. return [Insert(max(b.pos, a.pos - b.length), a.text)] end = a.pos + a.length if isinstance(b, Insert): # a deletes if b.pos <= a.pos: return [Delete(a.pos + len(b.text), a.length)] if b.pos >= end: return [a] # Someone typed inside the range we deleted: delete around their text, not over it. return [Delete(a.pos, b.pos - a.pos), Delete(a.pos + len(b.text), end - b.pos)] b_end = b.pos + b.length # both delete if end <= b.pos: return [a] if a.pos >= b_end: return [Delete(a.pos - b.length, a.length)] # They overlap, and b already removed the shared part. Delete what is left, or nothing. left = a.length - (min(end, b_end) - max(a.pos, b.pos)) return [Delete(min(a.pos, b.pos), left)] if left else []
# Two lists made on the same text. Returns a rewritten to follow b, and b rewritten to follow# a. Both halves matter: each piece of b must also move past the pieces of a it has crossed.def transform(a: list[Op], b: list[Op], a_first: bool) -> tuple[list[Op], list[Op]]: if not a or not b: return a, b if len(a) == 1 and len(b) == 1: return transform_one(a[0], b[0], a_first), transform_one(b[0], a[0], not a_first) if len(a) > 1: a1, b1 = transform(a[:1], b, a_first) a2, b2 = transform(a[1:], b1, a_first) return a1 + a2, b2 a1, b1 = transform(a, b[:1], a_first) a2, b2 = transform(a1, b[1:], a_first) return a2, b1 + b2
def apply(text: str, ops: list[Op]) -> str: for op in ops: # An op that does not fit is a bug or a hostile client: refuse it, never guess. if op.pos < 0 or op.pos > len(text) or ( isinstance(op, Delete) and op.pos + op.length > len(text) ): raise ValueError(f"{op} does not fit a text of {len(text)} characters") if isinstance(op, Insert): text = text[: op.pos] + op.text + text[op.pos :] else: text = text[: op.pos] + text[op.pos + op.length :] return text
# The owner of one document. Every edit to it passes through here, so this one place picks# the order, the revision number, and OT only ever has to transform two ops at a time.class DocSession: def __init__(self, text: str = ""): self.text = text # log[i] made revision i + 1. Production keeps the last 1,000 here and reads older ones # from the op log. self._log: list[list[Op]] = [] self._lock = threading.Lock()
@property def rev(self) -> int: return len(self._log)
# base_rev is the revision the client's edit was made on. Everything accepted since then # is concurrent with it, so the edit is rewritten past each of those, oldest first. def receive(self, base_rev: int, ops: list[Op]) -> tuple[int, list[Op]]: with self._lock: if not 0 <= base_rev <= self.rev: raise ValueError(f"unknown base revision {base_rev}") for logged in self._log[base_rev:]: ops, _ = transform(ops, logged, a_first=False) # the logged op was ordered first self.text = apply(self.text, ops) # raises before anything changes if it does not fit self._log.append(ops) # Stage 3: here the owner writes (doc, rev) to the op log, and acks and broadcasts # only once that write has committed. return self.rev, ops// An edit is a list of these primitives, applied left to right. Each position counts characters// in the text as it is when that primitive applies, so a list can be transformed piece by piece.abstract record Op;sealed record Insert(int Pos, string Text) : Op;sealed record Delete(int Pos, int Len) : Op;
static class Ot{ // a rewritten to apply after b, when both were made on the same text. aFirst breaks the one // real tie, two inserts at the same spot: the op the server ordered first goes first, so // every copy of the document breaks the tie the same way. public static List<Op> TransformOne(Op a, Op b, bool aFirst) { switch (a, b) { case (Insert x, Insert y): return [x.Pos < y.Pos || (x.Pos == y.Pos && aFirst) ? x : x with { Pos = x.Pos + y.Text.Length }]; case (Insert x, Delete y): if (x.Pos <= y.Pos) return [x]; // Inside the deleted range, the insert lands where the range was: the typing survives. return [x with { Pos = Math.Max(y.Pos, x.Pos - y.Len) }]; case (Delete x, Insert y): { int end = x.Pos + x.Len; if (y.Pos <= x.Pos) return [x with { Pos = x.Pos + y.Text.Length }]; if (y.Pos >= end) return [x]; // Someone typed inside the range we deleted: delete around their text, not over it. return [new Delete(x.Pos, y.Pos - x.Pos), new Delete(x.Pos + y.Text.Length, end - y.Pos)]; } case (Delete x, Delete y): { int end = x.Pos + x.Len, yEnd = y.Pos + y.Len; if (end <= y.Pos) return [x]; if (x.Pos >= yEnd) return [x with { Pos = x.Pos - y.Len }]; // They overlap, and b already removed the shared part. Delete what is left, or nothing. int left = x.Len - (Math.Min(end, yEnd) - Math.Max(x.Pos, y.Pos)); if (left == 0) return []; return [new Delete(Math.Min(x.Pos, y.Pos), left)]; } default: throw new ArgumentException($"unknown ops {a}, {b}"); } }
// Two lists made on the same text. Returns a rewritten to follow b, and b rewritten to follow // a. Both halves matter: each piece of b must also move past the pieces of a it has crossed. public static (List<Op> A, List<Op> B) Transform(List<Op> a, List<Op> b, bool aFirst) { if (a.Count == 0 || b.Count == 0) return (a, b); if (a.Count == 1 && b.Count == 1) return (TransformOne(a[0], b[0], aFirst), TransformOne(b[0], a[0], !aFirst)); if (a.Count > 1) { var (a1, b1) = Transform(a.GetRange(0, 1), b, aFirst); var (a2, b2) = Transform(a.GetRange(1, a.Count - 1), b1, aFirst); return (a1.Concat(a2).ToList(), b2); } var (x1, y1) = Transform(a, b.GetRange(0, 1), aFirst); var (x2, y2) = Transform(x1, b.GetRange(1, b.Count - 1), aFirst); return (x2, y1.Concat(y2).ToList()); }
public static string Apply(string text, IEnumerable<Op> ops) { var sb = new StringBuilder(text); foreach (var op in ops) { // An op that does not fit is a bug or a hostile client: refuse it, never guess. Both // calls throw ArgumentOutOfRangeException when the position or range is outside the text. switch (op) { case Insert i: sb.Insert(i.Pos, i.Text); break; case Delete d: sb.Remove(d.Pos, d.Len); break; } } return sb.ToString(); }}
// The owner of one document. Every edit to it passes through here, so this one place picks the// order, the revision number, and OT only ever has to transform two ops at a time.sealed class DocSession(string text){ private readonly Lock _gate = new(); // _log[i] made revision i + 1. Production keeps the last 1,000 here and reads older ones // from the op log. private readonly List<List<Op>> _log = new();
public string Text { get; private set; } = text;
public int Rev { get { lock (_gate) return _log.Count; } }
// baseRev is the revision the client's edit was made on. Everything accepted since then is // concurrent with it, so the edit is rewritten past each of those, oldest first. public (int Rev, List<Op> Ops) Receive(int baseRev, List<Op> ops) { lock (_gate) { if (baseRev < 0 || baseRev > _log.Count) throw new ArgumentOutOfRangeException(nameof(baseRev), $"unknown base revision {baseRev}"); for (int r = baseRev; r < _log.Count; r++) ops = Ot.Transform(ops, _log[r], aFirst: false).A; // the logged op was ordered first Text = Ot.Apply(Text, ops); // throws before anything changes if it does not fit _log.Add(ops); // Stage 3: here the owner writes (doc, rev) to the op log, and acks and broadcasts // only once that write has committed. return (_log.Count, ops); } }}// Op is one primitive of an edit. An edit is a list of them, applied left to right. Each// position counts characters in the text as it is when that primitive applies, so a list can be// transformed piece by piece.type Op interface{ isOp() }
type Insert struct { Pos int Text string}
type Delete struct { Pos, Len int}
func (Insert) isOp() {}func (Delete) isOp() {}
// TransformOne rewrites a to apply after b, when both were made on the same text. aFirst breaks// the one real tie, two inserts at the same spot: the op the server ordered first goes first, so// every copy of the document breaks the tie the same way. Positions count runes.func TransformOne(a, b Op, aFirst bool) []Op { switch x := a.(type) { case Insert: switch y := b.(type) { case Insert: if x.Pos < y.Pos || (x.Pos == y.Pos && aFirst) { return []Op{x} } return []Op{Insert{x.Pos + utf8.RuneCountInString(y.Text), x.Text}} case Delete: if x.Pos <= y.Pos { return []Op{x} } // Inside the deleted range, the insert lands where the range was: the typing survives. return []Op{Insert{max(y.Pos, x.Pos-y.Len), x.Text}} } case Delete: end := x.Pos + x.Len switch y := b.(type) { case Insert: n := utf8.RuneCountInString(y.Text) if y.Pos <= x.Pos { return []Op{Delete{x.Pos + n, x.Len}} } if y.Pos >= end { return []Op{x} } // Someone typed inside the range we deleted: delete around their text, not over it. return []Op{Delete{x.Pos, y.Pos - x.Pos}, Delete{x.Pos + n, end - y.Pos}} case Delete: yEnd := y.Pos + y.Len if end <= y.Pos { return []Op{x} } if x.Pos >= yEnd { return []Op{Delete{x.Pos - y.Len, x.Len}} } // They overlap, and b already removed the shared part. Delete what is left, or nothing. left := x.Len - (min(end, yEnd) - max(x.Pos, y.Pos)) if left == 0 { return nil } return []Op{Delete{min(x.Pos, y.Pos), left}} } } panic("unknown op")}
// Transform takes two lists made on the same text and returns a rewritten to follow b, and b// rewritten to follow a. Both halves matter: each piece of b must also move past the pieces of a// it has crossed.func Transform(a, b []Op, aFirst bool) ([]Op, []Op) { if len(a) == 0 || len(b) == 0 { return a, b } if len(a) == 1 && len(b) == 1 { return TransformOne(a[0], b[0], aFirst), TransformOne(b[0], a[0], !aFirst) } if len(a) > 1 { a1, b1 := Transform(a[:1], b, aFirst) a2, b2 := Transform(a[1:], b1, aFirst) return append(append([]Op{}, a1...), a2...), b2 } a1, b1 := Transform(a, b[:1], aFirst) a2, b2 := Transform(a1, b[1:], aFirst) return a2, append(append([]Op{}, b1...), b2...)}
// Apply returns the text with ops applied, or an error if one does not fit: that is a bug or a// hostile client, so the edit is refused rather than guessed at.func Apply(text string, ops []Op) (string, error) { rs := []rune(text) for _, op := range ops { switch o := op.(type) { case Insert: if o.Pos < 0 || o.Pos > len(rs) { return "", fmt.Errorf("%+v does not fit %d characters", o, len(rs)) } rs = append(rs[:o.Pos], append([]rune(o.Text), rs[o.Pos:]...)...) case Delete: if o.Pos < 0 || o.Pos+o.Len > len(rs) { return "", fmt.Errorf("%+v does not fit %d characters", o, len(rs)) } rs = append(rs[:o.Pos], rs[o.Pos+o.Len:]...) } } return string(rs), nil}
// DocSession is the owner of one document. Every edit to it passes through here, so this one// place picks the order, the revision number, and OT only ever has to transform two ops at a time.type DocSession struct { mu sync.Mutex text string // log[i] made revision i + 1. Production keeps the last 1,000 here and reads older ones // from the op log. log [][]Op}
// Receive takes an edit made on baseRev. Everything accepted since then is concurrent with it,// so the edit is rewritten past each of those, oldest first.func (s *DocSession) Receive(baseRev int, ops []Op) (int, []Op, error) { s.mu.Lock() defer s.mu.Unlock() if baseRev < 0 || baseRev > len(s.log) { return 0, nil, fmt.Errorf("unknown base revision %d", baseRev) } for _, logged := range s.log[baseRev:] { ops, _ = Transform(ops, logged, false) // the logged op was ordered first } text, err := Apply(s.text, ops) if err != nil { return 0, nil, err // nothing has changed } s.text = text s.log = append(s.log, ops) // Stage 3: here the owner writes (doc, rev) to the op log, and acks and broadcasts only // once that write has committed. return len(s.log), ops, nil}// An edit is a list of these primitives, applied left to right. Each position counts characters// in the text as it is when that primitive applies, so a list can be transformed piece by piece.type Op = { kind: 'insert'; pos: number; text: string } | { kind: 'delete'; pos: number; len: number };
const ins = (pos: number, text: string): Op => ({ kind: 'insert', pos, text });const del = (pos: number, len: number): Op => ({ kind: 'delete', pos, len });
// a rewritten to apply after b, when both were made on the same text. aFirst breaks the one real// tie, two inserts at the same spot: the op the server ordered first goes first, so every copy// of the document breaks the tie the same way.function transformOne(a: Op, b: Op, aFirst: boolean): Op[] { if (a.kind === 'insert') { if (b.kind === 'insert') { const before = a.pos < b.pos || (a.pos === b.pos && aFirst); return [before ? a : ins(a.pos + b.text.length, a.text)]; } if (a.pos <= b.pos) return [a]; // Inside the deleted range, the insert lands where the range was: the typing survives. return [ins(Math.max(b.pos, a.pos - b.len), a.text)]; } const end = a.pos + a.len; if (b.kind === 'insert') { if (b.pos <= a.pos) return [del(a.pos + b.text.length, a.len)]; if (b.pos >= end) return [a]; // Someone typed inside the range we deleted: delete around their text, not over it. return [del(a.pos, b.pos - a.pos), del(a.pos + b.text.length, end - b.pos)]; } const bEnd = b.pos + b.len; if (end <= b.pos) return [a]; if (a.pos >= bEnd) return [del(a.pos - b.len, a.len)]; // They overlap, and b already removed the shared part. Delete what is left, or nothing. const left = a.len - (Math.min(end, bEnd) - Math.max(a.pos, b.pos)); return left === 0 ? [] : [del(Math.min(a.pos, b.pos), left)];}
// Two lists made on the same text. Returns a rewritten to follow b, and b rewritten to follow a.// Both halves matter: each piece of b must also move past the pieces of a it has crossed.function transform(a: Op[], b: Op[], aFirst: boolean): [Op[], Op[]] { if (a.length === 0 || b.length === 0) return [a, b]; if (a.length === 1 && b.length === 1) { return [transformOne(a[0], b[0], aFirst), transformOne(b[0], a[0], !aFirst)]; } if (a.length > 1) { const [a1, b1] = transform(a.slice(0, 1), b, aFirst); const [a2, b2] = transform(a.slice(1), b1, aFirst); return [[...a1, ...a2], b2]; } const [a1, b1] = transform(a, b.slice(0, 1), aFirst); const [a2, b2] = transform(a1, b.slice(1), aFirst); return [a2, [...b1, ...b2]];}
function apply(text: string, ops: Op[]): string { for (const op of ops) { // An op that does not fit is a bug or a hostile client: refuse it, never guess. const end = op.kind === 'delete' ? op.pos + op.len : op.pos; if (op.pos < 0 || end > text.length) throw new RangeError(`${JSON.stringify(op)} does not fit`); text = op.kind === 'insert' ? text.slice(0, op.pos) + op.text + text.slice(op.pos) : text.slice(0, op.pos) + text.slice(end); } return text;}
// The owner of one document. Every edit to it passes through here, so this one place picks the// order, the revision number, and OT only ever has to transform two ops at a time. Node runs it// on one thread, so receive never interleaves with another receive.class DocSession { // log[i] made revision i + 1. Production keeps the last 1,000 here and reads older ones from // the op log. private readonly log: Op[][] = [];
constructor(public text = '') {}
get rev(): number { return this.log.length; }
// baseRev is the revision the client's edit was made on. Everything accepted since then is // concurrent with it, so the edit is rewritten past each of those, oldest first. receive(baseRev: number, ops: Op[]): { rev: number; ops: Op[] } { if (baseRev < 0 || baseRev > this.rev) throw new RangeError(`unknown base revision ${baseRev}`); for (const logged of this.log.slice(baseRev)) { [ops] = transform(ops, logged, false); // the logged op was ordered first } this.text = apply(this.text, ops); // throws before anything changes if it does not fit this.log.push(ops); // Stage 3: here the owner writes (doc, rev) to the op log, and acks and broadcasts only once // that write has committed. return { rev: this.rev, ops }; }}A CRDT works differently. Every character gets a unique ID, and an insert says “after the character with ID 7”, not “at position 1”. Merges then work in any order, with no server deciding. Yjs and Automerge are mature examples.
We pick OT with a central server. This system has a server in the middle anyway, for permissions, storage and history. And with one owner per document, the hard part of OT disappears. Without a server, OT needs a property called TP2, which several published algorithms got wrong. With one total order, each client only transforms against one sequence, and the simpler TP1 is enough.
Go deeperFour approaches, compared1 min
| Approach | How it works | Loses |
|---|---|---|
| Locks or last writer wins | Lock a paragraph while someone types, or keep the newest save | Blocks people or loses their work |
| Differential sync | Diff the copies now and then and patch fuzzily | Fuzzy patches can misplace edits; no edit history for undo or cursors |
| OT with a central server | The server orders edits; each is rewritten past earlier ones | One authority per document; a transform for every pair of edit types |
| CRDT (Yjs, Automerge) | Every character has an ID; inserts reference neighbors by ID | IDs and tombstones per character; interleaving in some algorithms |
Interleaving. In some CRDTs, “Alice” and “Bob” typed at the same spot can merge as “ABloibce”. OT can interleave too, when it transforms one character at a time, since the tie rule decides, not intent. Batching makes it rare. A delete that must split around text typed inside it is the case naive OT versions get wrong.
Formatting. Bold and italic on overlapping ranges commute, so both apply. On the same attribute (one bolds, one unbolds), the later revision wins on the overlap, the same on every copy.
The owner that crashes, or pauses
Section titled “The owner that crashes, or pauses”The owner of a document crashes right after writing an edit and before acking it. A minute later, a garbage-collection pause makes another server think it still owns the document.
Three rules make the owner safe:
- Ack only after the write commits, so a crash loses only edits nobody was told about.
- Write each revision only if it’s empty. Row (doc, rev) is the fence: two owners can never both commit rev 18345.
- Rebuild from storage. A new owner loads snapshot plus tail: the text, the latest revision and each recent client’s last sequence.
In the crash case, Alice resends her batch to the new owner with the same sequence. The new owner finds it already committed as rev 18345, and simply tells her: that was your ack.
- Color key
- Service
- Database
The paused owner is the subtle case. Its lease expired during the pause. A new owner committed rev 18346. Then the old one wakes up, and tries to write its own rev 18346. Checking the lease just before writing doesn’t help. The pause can fall between the check and the write. The conditional write refuses it, and the old owner drops the document and closes its sessions.
- Color key
- Service
- Database
One detail makes this airtight. An owner has one write in flight per document. It writes rev N+1 only after N commits, so a successful write proves the log matches its memory. Say the zombie wins a race, and slips in rev 18346 first. The new owner’s write is refused. It checks it still holds the lease, reads the tail, transforms its pending edit past 18346, and retries as 18347.
Planned moves aren’t failures. On a deploy, the old owner stops accepting edits for a slot. It finishes its writes, releases the lease, and tells the gateways. That pauses edits for well under a second. The 10-second lease timeout is only paid on real crashes.
Go deeperWays to guard against a second owner1 min
| Guard | How it works | Verdict |
|---|---|---|
| Lease timing margins | Stop writing before the lease ends, take over after | Unsafe: any pause longer than the margin puts two writers on one revision |
| Fencing token checked by the store | Each lease has a rising number; writes carry it | Correct, but needs a store that checks it on every write |
| Conditional write per revision | Row (doc, rev) written only if absent | The revision slot is the fence, at the cost of a normal write |
Three hours on a plane
Section titled “Three hours on a plane”Bob edits on a plane for three hours, then lands. What happens to his edits, and to the ones made while he was gone?
The client keeps three things. The last revision it applied. At most one batch in flight, with its sequence. And a buffer of everything typed since.
On reconnect, it asks for every revision after the last one it applied. If one of them carries its own client ID and in-flight sequence, only the ack was lost. It transforms its pending edits past the others, and resends whatever is still in flight under the same sequence. That is the same code as live editing, just with a longer gap.
// One accepted edit as the owner broadcasts it. clientId and seq say who sent it, so a client// can recognize its own edit coming back: that is its ack.record Revision(int rev, String clientId, long seq, List<Op> ops) {}
interface DocServer { // Every revision after rev: recent ones from the owner's memory, older from the op log. List<Revision> opsSince(String docId, int rev);
// Safe to repeat: the owner keeps the last seq per client and answers a repeat by sending // the original revision again, never by applying the edit twice. void submit(String docId, String clientId, long seq, int baseRev, List<Op> ops);}
// The client half of the protocol. Edits apply locally at once, at most one batch is in flight,// and everything typed meanwhile, or while offline, waits in the buffer.final class DocClient { private final String docId; private final String clientId; private final DocServer server; private String text; private int serverRev; // the last revision this client has applied private long inFlightSeq = 0; // 0: nothing in flight private List<Op> inFlight = List.of(); private List<Op> buffer = new ArrayList<>(); private long nextSeq = 1; private boolean connected = true;
DocClient(String docId, String clientId, DocServer server, String text, int rev) { this.docId = docId; this.clientId = clientId; this.server = server; this.text = text; this.serverRev = rev; }
String text() { return text; }
void edit(List<Op> ops) { text = Ot.apply(text, ops); // local echo: typing never waits for the network buffer.addAll(ops); flush(); }
// Every revision from the owner, live or during catch-up, in revision order. void onRevision(Revision r) { if (r.rev() <= serverRev) return; // a repeat caused by a resent batch, already applied serverRev = r.rev(); if (inFlightSeq != 0 && r.clientId().equals(clientId) && r.seq() == inFlightSeq) { inFlightSeq = 0; // our own batch came back: the ack inFlight = List.of(); flush(); return; } // Someone else's edit, ordered before everything of ours not yet acked. Move it past our // pending edits, and our pending edits past it, then apply it. Pair p = Ot.transform(inFlight, r.ops(), false); inFlight = p.a(); Pair q = Ot.transform(buffer, p.b(), false); buffer = new ArrayList<>(q.a()); text = Ot.apply(text, q.b()); }
void onDisconnect() { connected = false; } // keep typing: edits pile up in the buffer
// Catch up by revision, not by time: the owner says exactly what this client missed. void onReconnect() { connected = true; for (Revision r : server.opsSince(docId, serverRev)) onRevision(r); if (inFlightSeq != 0) { // Not in the log yet. Resend under the same seq, rebased on what we just applied: if // the old copy lands after all, the owner recognizes the seq and keeps only one. server.submit(docId, clientId, inFlightSeq, serverRev, inFlight); } else { flush(); } }
private void flush() { if (!connected || inFlightSeq != 0 || buffer.isEmpty()) return; inFlightSeq = nextSeq++; inFlight = buffer; buffer = new ArrayList<>(); server.submit(docId, clientId, inFlightSeq, serverRev, inFlight); }}# One accepted edit as the owner broadcasts it. client_id and seq say who sent it, so a# client can recognize its own edit coming back: that is its ack.@dataclass(frozen=True)class Revision: rev: int client_id: str seq: int ops: list[Op]
class DocServer(Protocol): # Every revision after rev: recent ones from the owner's memory, older from the op log. def ops_since(self, doc_id: str, rev: int) -> list[Revision]: ...
# Safe to repeat: the owner keeps the last seq per client and answers a repeat by sending # the original revision again, never by applying the edit twice. def submit(self, doc_id: str, client_id: str, seq: int, base_rev: int, ops: list[Op]) -> None: ...
# The client half of the protocol. Edits apply locally at once, at most one batch is in# flight, and everything typed meanwhile, or while offline, waits in the buffer.class DocClient: def __init__(self, doc_id: str, client_id: str, server: DocServer, text: str, rev: int): self.doc_id, self.client_id, self.server = doc_id, client_id, server self.text = text self.server_rev = rev # the last revision this client has applied self.in_flight_seq = 0 # 0: nothing in flight self.in_flight: list[Op] = [] self.buffer: list[Op] = [] self.next_seq = 1 self.connected = True
def edit(self, ops: list[Op]) -> None: self.text = apply(self.text, ops) # local echo: typing never waits for the network self.buffer += ops self._flush()
# Every revision from the owner, live or during catch-up, in revision order. def on_revision(self, r: Revision) -> None: if r.rev <= self.server_rev: return # a repeat caused by a resent batch, already applied self.server_rev = r.rev if self.in_flight_seq and r.client_id == self.client_id and r.seq == self.in_flight_seq: self.in_flight_seq, self.in_flight = 0, [] # our own batch came back: the ack self._flush() return # Someone else's edit, ordered before everything of ours not yet acked. Move it past our # pending edits, and our pending edits past it, then apply it. self.in_flight, theirs = transform(self.in_flight, r.ops, a_first=False) self.buffer, theirs = transform(self.buffer, theirs, a_first=False) self.text = apply(self.text, theirs)
def on_disconnect(self) -> None: self.connected = False # keep typing: edits pile up in the buffer
# Catch up by revision, not by time: the owner says exactly what this client missed. def on_reconnect(self) -> None: self.connected = True for r in self.server.ops_since(self.doc_id, self.server_rev): self.on_revision(r) if self.in_flight_seq: # Not in the log yet. Resend under the same seq, rebased on what we just applied: if # the old copy lands after all, the owner recognizes the seq and keeps only one. self.server.submit( self.doc_id, self.client_id, self.in_flight_seq, self.server_rev, self.in_flight ) else: self._flush()
def _flush(self) -> None: if not self.connected or self.in_flight_seq or not self.buffer: return self.in_flight_seq, self.next_seq = self.next_seq, self.next_seq + 1 self.in_flight, self.buffer = self.buffer, [] self.server.submit( self.doc_id, self.client_id, self.in_flight_seq, self.server_rev, self.in_flight )// One accepted edit as the owner broadcasts it. ClientId and Seq say who sent it, so a client// can recognize its own edit coming back: that is its ack.sealed record Revision(int Rev, string ClientId, long Seq, List<Op> Ops);
interface IDocServer{ // Every revision after rev: recent ones from the owner's memory, older from the op log. IReadOnlyList<Revision> OpsSince(string docId, int rev);
// Safe to repeat: the owner keeps the last seq per client and answers a repeat by sending // the original revision again, never by applying the edit twice. void Submit(string docId, string clientId, long seq, int baseRev, List<Op> ops);}
// The client half of the protocol. Edits apply locally at once, at most one batch is in flight,// and everything typed meanwhile, or while offline, waits in the buffer.sealed class DocClient(string docId, string clientId, IDocServer server, string text, int rev){ public string Text { get; private set; } = text; private int _serverRev = rev; // the last revision this client has applied private long _inFlightSeq; // 0: nothing in flight private List<Op> _inFlight = []; private List<Op> _buffer = []; private long _nextSeq = 1; private bool _connected = true;
public void Edit(List<Op> ops) { Text = Ot.Apply(Text, ops); // local echo: typing never waits for the network _buffer.AddRange(ops); Flush(); }
// Every revision from the owner, live or during catch-up, in revision order. public void OnRevision(Revision r) { if (r.Rev <= _serverRev) return; // a repeat caused by a resent batch, already applied _serverRev = r.Rev; if (_inFlightSeq != 0 && r.ClientId == clientId && r.Seq == _inFlightSeq) { _inFlightSeq = 0; // our own batch came back: the ack _inFlight = []; Flush(); return; } // Someone else's edit, ordered before everything of ours not yet acked. Move it past our // pending edits, and our pending edits past it, then apply it. (_inFlight, var theirs) = Ot.Transform(_inFlight, r.Ops, aFirst: false); (_buffer, theirs) = Ot.Transform(_buffer, theirs, aFirst: false); Text = Ot.Apply(Text, theirs); }
public void OnDisconnect() => _connected = false; // keep typing: edits pile up in the buffer
// Catch up by revision, not by time: the owner says exactly what this client missed. public void OnReconnect() { _connected = true; foreach (var r in server.OpsSince(docId, _serverRev)) OnRevision(r); if (_inFlightSeq != 0) { // Not in the log yet. Resend under the same seq, rebased on what we just applied: if // the old copy lands after all, the owner recognizes the seq and keeps only one. server.Submit(docId, clientId, _inFlightSeq, _serverRev, _inFlight); } else { Flush(); } }
private void Flush() { if (!_connected || _inFlightSeq != 0 || _buffer.Count == 0) return; _inFlightSeq = _nextSeq++; _inFlight = _buffer; _buffer = []; server.Submit(docId, clientId, _inFlightSeq, _serverRev, _inFlight); }}// Revision is one accepted edit as the owner broadcasts it. ClientID and Seq say who sent it, so// a client can recognize its own edit coming back: that is its ack.type Revision struct { Rev int ClientID string Seq int64 Ops []Op}
type DocServer interface { // OpsSince returns every revision after rev: recent ones from the owner's memory, older ones // from the op log. OpsSince(docID string, rev int) ([]Revision, error) // Submit is safe to repeat: the owner keeps the last seq per client and answers a repeat by // sending the original revision again, never by applying the edit twice. Submit(docID, clientID string, seq int64, baseRev int, ops []Op) error}
// DocClient is the client half of the protocol. Edits apply locally at once, at most one batch// is in flight, and everything typed meanwhile, or while offline, waits in the buffer.type DocClient struct { docID, clientID string server DocServer Text string serverRev int // the last revision this client has applied inFlightSeq int64 // 0: nothing in flight inFlight []Op buffer []Op nextSeq int64 connected bool}
func NewDocClient(docID, clientID string, server DocServer, text string, rev int) *DocClient { return &DocClient{docID: docID, clientID: clientID, server: server, Text: text, serverRev: rev, nextSeq: 1, connected: true}}
func (c *DocClient) Edit(ops []Op) error { text, err := Apply(c.Text, ops) // local echo: typing never waits for the network if err != nil { return err } c.Text = text c.buffer = append(c.buffer, ops...) return c.flush()}
// OnRevision takes every revision from the owner, live or during catch-up, in revision order.func (c *DocClient) OnRevision(r Revision) error { if r.Rev <= c.serverRev { return nil // a repeat caused by a resent batch, already applied } c.serverRev = r.Rev if c.inFlightSeq != 0 && r.ClientID == c.clientID && r.Seq == c.inFlightSeq { c.inFlightSeq, c.inFlight = 0, nil // our own batch came back: the ack return c.flush() } // Someone else's edit, ordered before everything of ours not yet acked. Move it past our // pending edits, and our pending edits past it, then apply it. var theirs []Op c.inFlight, theirs = Transform(c.inFlight, r.Ops, false) c.buffer, theirs = Transform(c.buffer, theirs, false) text, err := Apply(c.Text, theirs) if err != nil { return err } c.Text = text return nil}
func (c *DocClient) OnDisconnect() { c.connected = false } // keep typing: edits pile up
// OnReconnect catches up by revision, not by time: the owner says exactly what was missed.func (c *DocClient) OnReconnect() error { c.connected = true missed, err := c.server.OpsSince(c.docID, c.serverRev) if err != nil { return err } for _, r := range missed { if err := c.OnRevision(r); err != nil { return err } } if c.inFlightSeq != 0 { // Not in the log yet. Resend under the same seq, rebased on what we just applied: if the // old copy lands after all, the owner recognizes the seq and keeps only one. return c.server.Submit(c.docID, c.clientID, c.inFlightSeq, c.serverRev, c.inFlight) } return c.flush()}
func (c *DocClient) flush() error { if !c.connected || c.inFlightSeq != 0 || len(c.buffer) == 0 { return nil } c.inFlightSeq, c.nextSeq = c.nextSeq, c.nextSeq+1 c.inFlight, c.buffer = c.buffer, nil return c.server.Submit(c.docID, c.clientID, c.inFlightSeq, c.serverRev, c.inFlight)}// One accepted edit as the owner broadcasts it. clientId and seq say who sent it, so a client can// recognize its own edit coming back: that is its ack.interface Revision { rev: number; clientId: string; seq: number; ops: Op[];}
interface DocServer { // Every revision after rev: recent ones from the owner's memory, older from the op log. opsSince(docId: string, rev: number): Revision[]; // Safe to repeat: the owner keeps the last seq per client and answers a repeat by sending the // original revision again, never by applying the edit twice. submit(docId: string, clientId: string, seq: number, baseRev: number, ops: Op[]): void;}
// The client half of the protocol. Edits apply locally at once, at most one batch is in flight,// and everything typed meanwhile, or while offline, waits in the buffer.class DocClient { private inFlightSeq = 0; // 0: nothing in flight private inFlight: Op[] = []; private buffer: Op[] = []; private nextSeq = 1; private connected = true;
constructor( private readonly docId: string, private readonly clientId: string, private readonly server: DocServer, public text: string, private serverRev: number, // the last revision this client has applied ) {}
edit(ops: Op[]): void { this.text = apply(this.text, ops); // local echo: typing never waits for the network this.buffer.push(...ops); this.flush(); }
// Every revision from the owner, live or during catch-up, in revision order. onRevision(r: Revision): void { if (r.rev <= this.serverRev) return; // a repeat caused by a resent batch, already applied this.serverRev = r.rev; if (this.inFlightSeq && r.clientId === this.clientId && r.seq === this.inFlightSeq) { this.inFlightSeq = 0; // our own batch came back: the ack this.inFlight = []; this.flush(); return; } // Someone else's edit, ordered before everything of ours not yet acked. Move it past our // pending edits, and our pending edits past it, then apply it. let theirs: Op[]; [this.inFlight, theirs] = transform(this.inFlight, r.ops, false); [this.buffer, theirs] = transform(this.buffer, theirs, false); this.text = apply(this.text, theirs); }
onDisconnect(): void { this.connected = false; // keep typing: edits pile up in the buffer }
// Catch up by revision, not by time: the owner says exactly what this client missed. onReconnect(): void { this.connected = true; for (const r of this.server.opsSince(this.docId, this.serverRev)) this.onRevision(r); if (this.inFlightSeq) { // Not in the log yet. Resend under the same seq, rebased on what we just applied: if the // old copy lands after all, the owner recognizes the seq and keeps only one. this.server.submit(this.docId, this.clientId, this.inFlightSeq, this.serverRev, this.inFlight); } else { this.flush(); } }
private flush(): void { if (!this.connected || this.inFlightSeq || this.buffer.length === 0) return; this.inFlightSeq = this.nextSeq++; [this.inFlight, this.buffer] = [this.buffer, []]; this.server.submit(this.docId, this.clientId, this.inFlightSeq, this.serverRev, this.inFlight); }}Trace a request: Reconnect after offline
Reconnect after offline: step 1 of 9
Editors to Load balancer
The socket dropped 3 minutes ago in a train tunnel. The user kept typing: 40 edits sit in the local buffer, and batch seq 12 (base rev 18342) was in flight with no ack.
- Editors → Load balancer20 ms
- Load balancer → Doc service1 ms
- Editors → Load balancer20 ms
- Load balancer → Gateways1 ms
- Gateways → Session servers1 ms
- Session servers → Op log8 ms
- Session servers → Gateways1 ms
- Gateways → Editors20 ms
- Response
What we can’t promise. A client offline longer than the 30 days of raw edits gets
rev_too_old. The honest fallback keeps its text as a separate copy, “your offline changes”,
instead of guessing. And a long offline merge is correct, but can read oddly. If Bob rewrote a
paragraph Alice deleted, his insertions survive as fragments. OT keeps every character. It can’t
keep meaning.
Go deeperOther ways to reconnect1 min
| On reconnect | What happens | Verdict |
|---|---|---|
| Reload the document | Fetch the latest snapshot and drop local state | Loses everything typed offline |
| Upload the whole text, three-way merge | The server diffs base, theirs and mine | Fuzzy; loses intent on overlapping edits |
| Catch up by revision, rebase the buffer | Fetch revisions after the last applied, transform pending edits | Exact, and the same code as live editing |
The owner keeps the last 1,000 revisions in memory, so most catch-ups never touch storage. After a gateway dies, its 60,000 clients back off with jitter, so they don’t all return in the same second.
Cursors, presence and undo
Section titled “Cursors, presence and undo”Everyone wants to see each other’s cursor live. And when I press undo, what does it undo?
Cursors move on every keystroke and click, so they can cost more than the edits. They ride the same socket, on a lossy path. The owner passes them on, at most 10 a second per user. The newest wins, and they’re never stored. Under load, they are dropped first.
A cursor message carries the revision it refers to. A receiver that is further ahead transforms the position past the later revisions, like an empty insert, so it sits on the right character. Presence is the owner’s list of attached sessions, expired 30 seconds after the last heartbeat.
Undo means “undo my last edit”, not the document’s last edit. The client keeps its own edits, with the revision each became. To undo one, it inverts it: an insert becomes a delete of that text. It transforms the inverse past every later revision, and sends it as a new edit. If someone already deleted that text, the inverse transforms to nothing, and undo does nothing. That’s right.
Go deeperWhere cursors could live, and the snapshot cadence1 min
| Approach | Cost | Verdict |
|---|---|---|
| Cursor moves as edits in the op log | Doubles or triples the log; history full of noise | No: nobody needs yesterday’s cursor |
| A separate presence service | A second system that must know who is in each doc | Fine, but the owner already knows |
| Passed on by the owner, never stored | Coalesced to 10 a second per user | Dropped first under load; rebuilt on attach |
Undo stacks, and grouping keystrokes into undo steps, are in the text editor design.
| Snapshot every | Snapshot writes at peak | Edits replayed on a cold load | Fits |
|---|---|---|---|
| 50 edits | 20,000/s | up to 50 | Huge documents where a replay is costly |
| 500 edits | 2,000/s | up to 500, a few ms | Typical documents |
| 10,000 edits | 100/s | up to 10,000 | Logs that are rarely reopened |
One very popular document, and far-away editors
Section titled “One very popular document, and far-away editors”Traffic grows ten times, one document is an all-hands agenda with ten thousand people watching, and some editors are in Tokyo. What breaks?
At ten times the load, nothing breaks by itself: every tier is split by document, so we add gateways, owners and log nodes. One document is different. An all-hands agenda has 30 people typing, about 60 edits a second, and 10,000 viewers. Sending every edit to every viewer is 600,000 deliveries a second from one owner, which can send about 20,000.
Ordering is cheap. The fan-out is the problem. So split them. Live editors, capped at 100, stay attached to the owner. Viewers attach through viewer relays. The owner sends each revision once to a few relays, which pass it to the gateways holding viewers. Viewers are read only, so nothing they do needs ordering.
Stage 5: Relays for hot documents
Stage 5 of 6
Relays for hot documents
Why this stage
Not total load: one document.
An all-hands agenda with 10,000 viewers and 30 typing editors is 600,000 deliveries a second from one owner, which pushes about 20,000.
What we add
Viewer relays.
The owner sends each revision once to a few relays, which fan it out to the gateways holding viewers.
Live editors stay capped at 100 per document.
What it costs
Viewers see edits one hop later and are read only.
A viewer who starts editing moves to the owner, and an editor over the cap waits.
At 1M ops per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
How busy it gets
- There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
- Opens and reconnects add about 23,000 a second at peak.
What one machine can do
- One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
- One session server handles about 20,000 edits a second, including the durable write and the broadcast.
- One op log node takes about 25,000 conditional writes a second.
- One Postgres handles about 20,000 simple writes or reads a second.
What saves work
- The doc service answers about 90% of permission checks from its cache.
- There is one snapshot every 500 edits.
- Relays deliver, on average, about half a viewer copy per edit.
Far-away editors. An editor in Tokyo, on a document owned in Virginia, waits about 150 ms for every ack. So gateways and doc services run in every region. Each document has a home region for its owner and log, near most of its editors. A Tokyo edit takes one cross-region hop, which local echo hides.
Stage 6: Regions
Stage 6 of 6
Regions
Why this stage
An editor in Tokyo pays about 150 ms to a US owner for every ack and every remote edit, and one region is one failure domain.
What we add
Gateways and doc services in every region.
Each document has a home region near its editors, where its owner and op log live; the op log is copied asynchronously to a second region.
What it costs
Editors outside the home region wait one cross-region round trip for acks (local echo hides it), and losing the home region can lose the last few seconds of acked edits.
At 1M ops per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
How busy it gets
- There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
- Opens and reconnects add about 23,000 a second at peak.
What one machine can do
- One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
- One session server handles about 20,000 edits a second, including the durable write and the broadcast.
- One op log node takes about 25,000 conditional writes a second.
- One Postgres handles about 20,000 simple writes or reads a second.
What saves work
- The doc service answers about 90% of permission checks from its cache.
- There is one snapshot every 500 edits.
- Relays deliver, on average, about half a viewer copy per edit.
The log copies synchronously across 3 zones in the home region, and asynchronously to a second region. Synchronous copies across regions would add 70 to 150 ms to every ack, for an event that happens a few times a decade. So losing the home region can drop a few seconds of acked edits: the stated recovery point.
Go deeperFan-out options, and very large documents1 min
| Fan-out for 10,000 viewers | Owner sends per revision | Problem |
|---|---|---|
| Owner to every viewer session | 10,000 | 600,000 deliveries a second from one server |
| Owner to relays, relays to gateways | 5 to 20 | One more tier; viewers one hop, about 5 ms, behind |
| Viewers poll a snapshot every few seconds | 0 | Not live; right for a document published to the world |
Why not let each region order its own editors’ changes? That is two owners for one document, which is exactly where a CRDT would fit. For documents whose editors are mostly in one place, one cross-region hop is cheaper than a second merge algorithm.
Very large documents, a million characters, stress the snapshot and the client, not the transform. Snapshots are stored in 64 KB chunks, so the snapshotter rewrites only changed chunks. The client loads chunks near the screen first. And the owner keeps the text in a rope, so an insert in the middle doesn’t copy megabytes.
When things break
Section titled “When things break”A session server dies. What do the people in its documents see?
Whether typing survives, nothing acked is lost, and users see only a short “reconnecting” badge.
Typing never stops, and acked edits never disappear, even when syncing pauses.
When a session server dies, its slot leases stop renewing. After 10 seconds, other servers take its slots and load each document from snapshot plus tail. Meanwhile, clients show “reconnecting” and keep typing into their buffers. When they reattach, they catch up by revision and resend by sequence, exactly as after a flight.
Under stress, shed cursors and presence first, then viewer relays (viewers fall back to polling a snapshot), then new opens. Edits from people already in a document go last.
Click any box to break it and see what users would notice:
Break it: what fails, who notices, what users see
Failure: click any box to break it
A session server goes down
What breaks
Every document it owns, about 1 in 85 of the open ones, has no owner.
Edits to them cannot be ordered.
How we notice
Its slot leases in etcd stop being renewed; they expire after 10 seconds.
What users see
A "reconnecting" badge for about 10 to 15 seconds.
Typing keeps working locally and nothing acked is lost.
What we do
Other servers take the slots, load each document from its snapshot and op log tail when the first client reattaches, and clients resend unacked batches with the same seq.
Go deeperEvery failure, and what users see1 min
| Part | Failure | What users see | What we do |
|---|---|---|---|
| Session server | Crash | Sync pauses 10 to 15 s for its documents; typing continues | Slots move; load from snapshot plus tail; clients resend by seq |
| Gateway | Crash | One or two seconds of reconnecting | Jittered reconnect; tickets checked locally |
| Op log partition | Loses quorum | Edits show locally but don’t sync (“saving”) | Nothing acked or sent uncommitted; retries drain in order |
| Metadata DB | Primary down | New opens on cache misses fail; open documents fine | Failover in about 30 s; serve cached permissions |
| Whole region | Outage | Documents homed there pause; a few seconds of acks at risk | Move homes to the region with the copy; clients resend by seq |
| Operation | Guarantee | What users can see |
|---|---|---|
| Order of edits in a doc | One order, decided by the owner, fenced by the log | Everyone ends with the same text |
| My own edits | Immediate (local echo) | My typing is never delayed or reordered on my screen |
| Others’ edits | Converge; in order per sender | A colleague’s edit appears 50 to 200 ms later, maybe shifted by mine |
| Acked edit | In 2 of 3 zones; async to a second region | Never lost short of a region loss |
| Cursors and presence | Best effort | A cursor can lag, jump, or linger 30 s after someone leaves |
Convergence holds only because each document has one strict order. A last-write-wins op log would let two owners both write rev 18346 and silently drop one. The clients converge because the server is strict.
Go deeperRunning it: alerts, deploys and cost1 min
Wake someone up when:
- Ack latency p99 passes 500 ms for 5 minutes.
- The oldest unacked batch on clients passes 60 seconds, for more than 0.1% of sessions. (It catches every way sync can break.)
- Conditional writes fail on a log partition.
- Any slot has no owner for 30 seconds.
Catch divergence. Every few hundred revisions, clients send a hash of their text with its revision, and the owner compares it with its own. A mismatch is a transform bug: that client reloads from the owner, and the edit sequence is saved for the bug report. Without this check, divergence is silent.
Deploys. Slots move by handover, a few percent at a time. Transform code is the riskiest change, because a bug corrupts documents silently. New edit types ship behind a protocol version, tested with randomized multi-client fuzzing.
Cost is mostly the op log (writes and 110 TB of history) and the owners’ memory. Batching edits per row and folding old history into snapshots are the two biggest levers.
Wrap up
Section titled “Wrap up”The whole design
Section titled “The whole design”Here is everything we built, in the order we built it. Use the buttons to step through it.
The full design, stage by stage
Stage 6 of 6
Regions
Why this stage
An editor in Tokyo pays about 150 ms to a US owner for every ack and every remote edit, and one region is one failure domain.
What we add
Gateways and doc services in every region.
Each document has a home region near its editors, where its owner and op log live; the op log is copied asynchronously to a second region.
What it costs
Editors outside the home region wait one cross-region round trip for acks (local echo hides it), and losing the home region can lose the last few seconds of acked edits.
At 1M ops per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
How busy it gets
- There are about 5 open sessions for every edit a second. At the 1 million edits a second peak, that is 5.2 million sockets.
- Opens and reconnects add about 23,000 a second at peak.
What one machine can do
- One gateway holds about 100,000 sockets. One server doing everything manages about 50,000.
- One session server handles about 20,000 edits a second, including the durable write and the broadcast.
- One op log node takes about 25,000 conditional writes a second.
- One Postgres handles about 20,000 simple writes or reads a second.
What saves work
- The doc service answers about 90% of permission checks from its cache.
- There is one snapshot every 500 edits.
- Relays deliver, on average, about half a viewer copy per edit.
The decisions, and what each one costs
Section titled “The decisions, and what each one costs”| Decision | We chose | We gave up | We’d choose differently if |
|---|---|---|---|
| Merge | OT with a central owner per document | Serverless merging, long offline drift | Peer to peer or end-to-end encrypted: a CRDT |
| Routing | Slot leases in etcd, gateways cache the map | The simplicity of sticky balancing | Documents never moved and servers never failed |
| Durability | Ack after a 2-of-3 zone commit, async copy | Zero loss on region failure | Regulation required it: synchronous across regions |
| Permissions | Cached roles, signed tickets, revoke events | Instant revocation in every case | Documents were public |
| Hot documents | Viewer relays, 100 live editors | Unlimited live editors | Thousands edited at once: a different product |
| Regions | A home region per document | Equal latency for far editors | Editors of one document were always spread worldwide |
How to spend 45 minutes
Section titled “How to spend 45 minutes”- 0 to 4. Rich text, editors per document, offline, history.
- 4 to 8. 5.2 million sockets, a million edits a second, and why clients batch.
- 8 to 20. Build it: gateways, one owner per document, the transform, the op log with conditional writes. A mid-level answer can stop here.
- 20 to 30. The stress tests: OT or CRDT with the “abc” example, then owner failover.
- 30 to 40. Offline catch-up, then cursors and undo, or the hot document.
- 40 to 45. A session server dying, then your own questions.
Never skip the merge algorithm, the single owner, or acking after the write.
Follow-up questions
Section titled “Follow-up questions”Interviewer asksWould you build this with a CRDT today?
For a product that must work offline for days, peer to peer, or end-to-end encrypted, yes: Yjs and Automerge are mature. For a server-centric product like this one, OT with an owner is simpler to store and reason about.
Interviewer asksProduct wants comments anchored to text. Where do they go?
In their own table, with an anchor: a range plus the revision it was made on. Like a cursor, the owner transforms the anchor past later revisions and saves current anchors with each snapshot. If the text is deleted, the comment shows as orphaned.
Go deeperMore follow-up questions1 min
Product wants suggestion mode (tracked changes). What changes? A suggestion is an edit with a mark. Inserted text is marked “suggested by Bob”. Deletions are marked, not removed. Accepting is a new edit that clears the mark. Rejecting is its inverse. Only the edit types change, not the server.
A revoked user still has the document open. How fast are they cut off? Within seconds: the sharing change publishes an event, and the owner closes their session. If the event is lost, a stale cache entry can let them reopen it for up to a minute.
Why a WebSocket and not REST calls per edit? The server must push remote edits, acks and cursors the moment they exist, in revision order, on one stream. Server-sent events plus POST would work. But they split one ordered stream into two.
Further reading
Section titled “Further reading”- The chat system for WebSocket gateways, reconnect storms and conversation owners; the text editor with undo for the single-user editor; and consistency models and partitioning on this site.
- How Figma’s multiplayer technology works, Figma engineering: why a server-authoritative design beat both pure OT and pure CRDTs for them.
- Interleaving anomalies in collaborative text editors, Kleppmann and others, 2019: the CRDT interleaving problem, with examples.
- How to do distributed locking, Martin Kleppmann: why a lease needs the store to refuse the old holder, the reason every revision is a conditional write.
The designbeyondcode digest, every Saturday
New deep dives, interview tips and site news in one short email. Free. Leave with one click.
We never sell or share your email. We only use it to send this newsletter.