Design a stock exchange

“Design a stock exchange like Nasdaq. Members send orders, the exchange matches them, and everyone sees the market data.”

An order book behind a load balancer, with a database underneath, is wrong for an exchange. Matching takes about a microsecond, and a database commit takes about a millisecond. Every copy of the engine must match orders in the same order, or a failover produces different trades. And a thousand firms must see each book change at the same moment, because seeing it first is worth money.

The order book classes are the LLD problem. This page is the system around them. It starts with one engine in memory and lets the numbers grow it. Then it tests the result against threads, lost packets and fairness.

The framework

1. 01Answer
2. 02Scope
3. 03Numbers
4. 04Build
5. 05Stress
6. 06Failures
7. 07Wrap up

The answer in one minute

If you only have a minute, this is the design. The rest of the page explains each part.

1. One sequencer gives every order a number. That number is the order of arrival. It is written to a journal before anything acts on it.
2. The engine is deterministic. It matches in memory on one thread, and never reads a clock. The same journal always produces exactly the same trades.
3. Credit is checked per account before the book. One risk shard owns each account, across all symbols.
4. Market data goes out once, over multicast. Every firm gets the same packet at the same moment, and repairs its own gaps.
5. A hot standby replays the same stream. It holds the same books, so a failover takes under a second. Every fill has a stable ID, so none is lost or counted twice.

Scope it

Three questions decide whether this is a microsecond system or a web app: the message rate, the latency target, and who reads market data.

Interviewerminute 0

Design a stock exchange like Nasdaq.

What they are testing

Whether you pin down message rates, the latency target and who consumes market data before drawing anything. Opening with “users place orders through a REST API into a database” is the red flag: it designs a broker, not an exchange.

These four questions matter most:

• How busy? 10,000 symbols and 1 billion order messages a day.
• How fast? p99 under 50 microseconds (µs) from an order reaching us to its acknowledgement leaving, inside the data centre.
• Who reads market data? About 1,000 member firms, colocated in our data centre, wanting every change.
• Do we check risk? Yes: price limits, size limits, credit per member, and a kill switch.

From these answers we make five promises:

• Fast and steady. p99 under 50 µs, p99.99 under 200 µs. Jitter matters as much as the average.
• Never lost. No acknowledged order or fill is ever lost.
• Deterministic. The same inputs produce byte-identical outputs on any machine.
• Always up. 99.99% of trading hours, about 10 minutes a year. A failover takes under a second, with no lost or repeated fill.
• Fair. Every member gets the same path in, and every subscriber gets market data at the same moment.

We leave out auctions in detail (a mode of the same engine), routing orders to other venues, and settlement, which is the clearing house’s job.

Go deeperMore questions, and what each changes1 min

Question	We assume	If the answer were different
A small venue?	No, 1 billion messages a day	1 million a day fits one engine forever
Retail users directly?	No, member firms over binary sessions	Accounts, a web API and huge fan-out: a broker’s system
A 10 ms target?	No, 50 µs	Kafka and a database could sit on the order path (a crypto venue)
A handful of subscribers?	No, about 1,000	Each could get a TCP stream
Trading 24/7?	No, 9:30 to 16:00 with auctions	No nightly window for snapshots, deploys and rebalancing
Do we clear trades?	No, a clearing house does	Margin, netting and settlement: a payment system bolted on

The numbers that decide it

Now we work out the numbers. Storage is the one number that doesn’t matter here.

Assumptions

Order messages a day

1 billion

Peak second over average

10×

Messages that trade

2%

Bytes per journal entry

100 bytes

Direct feed subscribers

1,000

Messages one engine thread matches a second

1 million

Messages43,000/s430,000/s at peak

1 billion messages over 6.5 trading hours. Most are cancels and replaces from market makers updating quotes. The open, the close and news push that ten times.

So we size for latency under bursts, not just for throughput.

Matching1M/sper engine thread

One thread matches an order in about 1 µs when the books are in the CPU cache. LMAX reported 6 million a second on one thread; 1 million leaves room for journaling and risk.

So we can match the whole market's peak on one thread, at 43% busy. The margin is thin, so partitions come early.

Replay1,000 sworst case

Rebuilding the books from nothing means applying the whole day: 1 billion entries at 1 million a second, about 17 minutes. The yearly downtime budget is about 10 minutes.

So we need a hot standby that is already caught up, not just a journal.

Market data21 MB/s21 GB/s fanned out

About 1.2 feed messages of 40 bytes per order message. One TCP stream for each of 1,000 subscribers would be 21 GB/s, far past a 10 Gb/s link, and the exchange would wait on the slowest reader.

So we send every packet once, over multicast.

About 2% of messages trade: 20 million executions a day, which an ordinary database handles off the hot path. The journal is 100 GB a day. Try the calculator: at 1.2 billion messages a day, one thread at half load no longer carries the peak.

Change an assumptionReset to the page's numbers

Order messages a day

1 billion

Peak second over average

10×

Messages that trade

2%

Bytes per journal entry

100 bytes

Direct feed subscribers

1,000

Messages one engine thread matches a second

1 million

Order messages, average (6.5 trading hours)

43,000/s

Order messages, peak second

430,000/s

Engine threads needed at peak, each under 50% busy

1

One single-threaded engine carries the whole market at peak.

Executions a day

20 million

Each one goes to clearing and to two drop copies. A database handles this easily.

Journal a day

100 GB

Fits on one NVMe drive many times over. Sequential appends only.

Journal a year (252 trading days)

25 TB

Kept for years for regulators, in cheap object storage. Not a design driver.

Seconds to replay a full day

1,000

Too slow to recover by replay alone: keep a hot standby that is already caught up.

Market data, peak, sent once (multicast)

21 MB/s

About 1.2 feed messages of 40 bytes per order message. One copy on the wire.

Market data, peak, one TCP stream per subscriber

21 GB/s

Past a 10 Gb/s link: send each packet once over multicast, not a stream per subscriber.

Build the design

We start with the simplest design that is correct. Then we let the numbers break it, one at a time. Each step fixes only what broke, adds only the boxes that fix needs, and says what the fix costs. The numbers already told us that nothing on the order path can touch a database, and that one thread can do the matching.

Step 1. One engine, in memory

The simplest version is gateways that forward each order to one matching engine, which keeps every book in memory on one thread. One thread can carry the whole peak.

Stock exchange · architecture

Stage 1: One engine, in memory

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional

Order messages per second:

430,000

43k average430k peak4.3M (10×)20M

Stage 1 of 6

One engine, in memory

Why this stage

Nothing yet.

This is the smallest design that accepts orders and matches them.

What we add

Gateways that terminate member sessions, and one matching engine that keeps every order book in memory and matches on a single thread.

What it costs

A crash loses every resting order, and nothing defines one order of arrival across gateways.

At 430k order messages per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One gateway handles about 50,000 messages a second across its sessions.
• One risk shard runs about 200,000 checks a second.
• One engine thread handles about 1,000,000 messages a second, about 1 µs each, with the books in CPU cache.
• One sequencer handles about 2,000,000 a second, and one journal about 3,000,000 appends with batching.
• One feed channel sends about 5,000,000 packets a second.

How the traffic is shaped

• There are about 1.2 market data messages for every order message.
• About 2% of messages trade.
• Partition capacity assumes symbols are placed so that no partition carries more than its share.

Each symbol has a book: buy orders (bids) and sell orders (asks), sorted by price. Matching follows price-time priority: the best price trades first, and at one price, the order that arrived first trades first. An incoming buy trades against the lowest asks it crosses, oldest first at each price, and what is left rests on the bid side. Prices are whole numbers of ticks, never floats.

JBook.javaPybook.pyC#Book.csGobook.goTSbook.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

// The heart of one partition's engine: price-time priority on one symbol's book. The full
// order book design (cancel, replace, order types) is on the LLD page; this is the match loop.
enum Side { BUY, SELL }

// Prices are integer ticks, never doubles: replay on another machine must produce exactly
// the same comparisons, and 0.1 + 0.2 != 0.3 in floating point.
final class Order {
    final long id, seq, price;
    final Side side;
    long remaining;

    Order(long id, long seq, Side side, long price, long qty) {
        this.id = id; this.seq = seq; this.side = side; this.price = price; this.remaining = qty;
    }
}

// An execution's ID is (input sequence number, n-th fill of that input). The primary, the
// standby and a replay all mint the same IDs, which is how downstream drops duplicates.
record Fill(long seq, int n, long makerId, long takerId, long price, long qty) {}

final class Book {
    // Best bid is the highest price, best ask the lowest. Each level is a FIFO queue:
    // that queue is what time priority means.
    private final TreeMap<Long, ArrayDeque<Order>> bids = new TreeMap<>(Comparator.reverseOrder());
    private final TreeMap<Long, ArrayDeque<Order>> asks = new TreeMap<>();

    // Called by exactly one thread, in sequence order. No locks, no clock, no randomness:
    // the same inputs always give the same fills.
    List<Fill> submit(Order in) {
        var fills = new ArrayList<Fill>();
        var opposite = in.side == Side.BUY ? asks : bids;
        while (in.remaining > 0 && !opposite.isEmpty()) {
            var best = opposite.firstEntry();
            long level = best.getKey();
            boolean crosses = in.side == Side.BUY ? in.price >= level : in.price <= level;
            if (!crosses) break; // price priority: better prices always trade first

            ArrayDeque<Order> queue = best.getValue();
            Order maker = queue.peekFirst(); // time priority: the oldest order at this price
            long qty = Math.min(in.remaining, maker.remaining);
            // The trade prints at the resting order's price: it was there first, the
            // incoming order crossed it. Price improvement goes to the taker.
            fills.add(new Fill(in.seq, fills.size(), maker.id, in.id, level, qty));
            in.remaining -= qty;
            maker.remaining -= qty;
            if (maker.remaining == 0) queue.pollFirst();
            if (queue.isEmpty()) opposite.pollFirstEntry();
        }
        if (in.remaining > 0) {
            // What is left rests at the back of its level, behind everyone already there.
            var own = in.side == Side.BUY ? bids : asks;
            own.computeIfAbsent(in.price, p -> new ArrayDeque<>()).addLast(in);
        }
        return fills;
    }
}

This version is fast, but it has two flaws. A crash loses every resting order. And with 15 gateways forwarding at once, the order the engine sees depends on network timing. So nobody can reproduce it, not even a second copy of the engine.

Step 2. Sequence, then journal

If the engine is deterministic and its inputs have one agreed order, the inputs are the only thing that must survive. So every message passes through a sequencer first. It gives the message the next sequence number and a timestamp, appends it to the journal, and only then releases it to the engine. The sequence number is the order of arrival. Whatever reached the sequencer first is first, by definition.

Recovery is then mechanical. Start from the latest snapshot, which records the sequence number it was taken at. Then apply the journal from the next entry. The same wrapper applies entries on the primary, on a standby and during a replay. It refuses to skip a gap, or to apply anything twice:

JSequencer.javaPysequencer.pyC#Sequencer.csGoreplica.goTSsequencer.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

// One inbound message after sequencing. The sequence number is the only order that counts,
// and the timestamp is the sequencer's, so nothing downstream ever reads its own clock.
record Entry(long seq, long tsNanos, String command) {}

// An engine output, numbered by the input that caused it. (seq, n) is the same on every
// replica, so a consumer that sees an ID twice drops the second copy.
record Event(long seq, int n, String body) {}

interface Journal {
    void append(Entry e);              // returns once the entry is on two machines: committed
    Iterable<Entry> readFrom(long seq);
    long lastSeq();                    // highest committed sequence number
}

// The matching engine of one partition: every book in memory.
interface StateMachine {
    List<String> apply(Entry e);       // deterministic: no clock, no randomness, no I/O
}

// One thread per partition calls submit, so numbering needs no lock and no compare-and-set.
final class Sequencer {
    private final Journal journal;
    private final Consumer<Entry> release; // to the primary engine and the standby, in order
    private final LongSupplier clock;
    private long next;
    private boolean steppedDown;

    Sequencer(Journal journal, Consumer<Entry> release, LongSupplier clock) {
        this.journal = journal;
        this.release = release;
        this.clock = clock;
        // A restarted or promoted sequencer continues after the last committed entry, so a
        // number is never handed out twice.
        this.next = journal.lastSeq() + 1;
    }

    long submit(String command) {
        if (steppedDown) throw new IllegalStateException("stepped down");
        var e = new Entry(next, clock.getAsLong(), command);
        // Commit first: no engine acts on an entry a crash could lose.
        try {
            journal.append(e);
        } catch (RuntimeException ex) {
            // A failed append may still have reached one copy. Reusing this number for another
            // message would give the copies two different entries N, so step down and let the
            // successor settle the tail from the longest surviving log.
            steppedDown = true;
            throw ex;
        }
        next++;
        release.accept(e);
        return e.seq();
    }
}

// Wraps the engine on the primary, on the standby and during recovery. One code path, so all
// three end in the same state.
final class Replica {
    private final StateMachine engine;
    private long applied; // highest sequence number applied; a snapshot records it too

    Replica(StateMachine engine, long snapshotSeq) {
        this.engine = engine;
        this.applied = snapshotSeq;
    }

    List<Event> apply(Entry e) {
        if (e.seq() <= applied) return List.of(); // already applied: a resend after failover
        if (e.seq() != applied + 1)                // a gap: never skip ahead, fetch it first
            throw new IllegalStateException("gap: want " + (applied + 1) + ", got " + e.seq());
        applied = e.seq();
        var bodies = engine.apply(e);
        var out = new ArrayList<Event>(bodies.size());
        for (int n = 0; n < bodies.size(); n++) out.add(new Event(e.seq(), n, bodies.get(n)));
        return out;
    }

    // Recovery: start from a snapshot taken at sequence N (or empty, N = 0) and apply the
    // journal from N + 1. Same entries in the same order give the same books and fills.
    void replay(Journal journal) {
        for (Entry e : journal.readFrom(applied + 1)) apply(e);
    }

    long applied() {
        return applied;
    }
}

Determinism takes discipline. The engine may not read a clock (the sequencer’s timestamp is in the entry). It may not use random numbers, or loop over a hash map whose order varies. It may not use floating-point prices, or do I/O. Order IDs and execution IDs come from sequence numbers, so every copy makes the same ones. Timers and reference data (“halt XYZ”) are journal entries too. So they happen at the same point everywhere.

The journal is made durable by group commit to a local fast disk. Each flush carries whatever arrived during the last one. So throughput stays high, and each entry waits about 20 µs.

Stock exchange · architecture

Stage 2: Sequence, then journal

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

12

Order messages per second:

430,000

43k average430k peak4.3M (10×)20M

Stage 2 of 6

Sequence, then journal

Why this stage

Durability: a crash may not lose an acknowledged order.

And 15 gateways need one agreed order of arrival, or replay gives different trades.

What we add

A sequencer that stamps every inbound message with the next sequence number and a timestamp, and an append-only journal written before the engine sees the message.

What it costs

About 20 µs of group commit on every order, and one sequencer that every message passes through.

At 430k order messages per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One gateway handles about 50,000 messages a second across its sessions.
• One risk shard runs about 200,000 checks a second.
• One engine thread handles about 1,000,000 messages a second, about 1 µs each, with the books in CPU cache.
• One sequencer handles about 2,000,000 a second, and one journal about 3,000,000 appends with batching.
• One feed channel sends about 5,000,000 packets a second.

How the traffic is shaped

• There are about 1.2 market data messages for every order message.
• About 2% of messages trade.
• Partition capacity assumes symbols are placed so that no partition carries more than its share.

Now the books survive a crash, and any copy of the engine makes the same trades. But the engine still trusts every order, and nobody outside can see what traded.

Go deeperWhy not Kafka or a database as the journal1 min

Kafka is a durable, ordered log, so it looks like a natural journal. But a Kafka write that waits for copies on other brokers takes about a millisecond, often more. A database commit is similar. The whole order path has a budget of 50 µs, so either one would use the budget twenty times over.

So the journal is a plain file on a fast local disk, written by the sequencer itself. Group commit keeps each entry’s wait near 20 µs. It is still a log, in the same sense as Kafka: an append-only list of numbered entries that anyone can replay. It is just a log built for microseconds.

Snapshots keep replays short. Replaying a full day of journal takes about 17 minutes. So the engine’s state is saved every few minutes as a snapshot that records its sequence number. A restart loads the snapshot and replays only the entries after it. The failover stress test shows why even that is too slow to be the main recovery plan.

Step 3. Risk checks before the book

Simple checks need only the order and the day’s reference data: the price within 10% of a reference price, the size below the member’s maximum. Gateways do these. Credit checks need the account’s open exposure, across every symbol and every gateway it uses. So each account belongs to one risk shard, a single thread, and two orders from one account can never race past the limit together.

The shard reserves an order’s value when it passes, and releases it when a cancel or fill comes back. A late release only makes the check stricter for a moment, never looser. The kill switch blocks the account and sends “cancel everything for this member” into the stream.

Stock exchange · architecture

Stage 3: Risk checks before the book

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

123

Order messages per second:

430,000

43k average430k peak4.3M (10×)20M

Stage 3 of 6

Risk checks before the book

Why this stage

A member’s runaway algorithm can send 100,000 bad orders a second.

The exchange must stop them before they trade, not after.

What we add

Risk shards, one owner per member account, that check price collars, order size and credit before an order is sequenced.

What it costs

A few microseconds on every order, and risk state that must be rebuilt after a crash.

At 430k order messages per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One gateway handles about 50,000 messages a second across its sessions.
• One risk shard runs about 200,000 checks a second.
• One engine thread handles about 1,000,000 messages a second, about 1 µs each, with the books in CPU cache.
• One sequencer handles about 2,000,000 a second, and one journal about 3,000,000 appends with batching.
• One feed channel sends about 5,000,000 packets a second.

How the traffic is shaped

• There are about 1.2 market data messages for every order message.
• About 2% of messages trade.
• Partition capacity assumes symbols are placed so that no partition carries more than its share.

Step 4. Market data and post-trade

Trades now happen safely. But nobody outside can see them yet. The engine’s outputs form a stream in sequence order. The market data publisher sends each packet once over UDP multicast, on two separate networks (the A and B feeds). The switches copy it to every subscriber. Post-trade sends copies to members’ risk systems, and stores executions. It also builds the end-of-day file for the clearing house. It has no microsecond budget, so Kafka is fine there.

Stock exchange · architecture

Stage 4: Market data and post-trade

Grow itTrace a request

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

1234

Order messages per second:

430,000

43k average430k peak4.3M (10×)20M

Stage 4 of 6

Market data and post-trade

Why this stage

About 1,000 subscribers want every book change.

At 21 MB/s each, a TCP stream per subscriber is 21 GB/s.

What we add

A market data publisher that sends each change once over multicast (A and B feeds), with snapshots for late joiners, and a post-trade service that feeds clearing and drop copies.

What it costs

Multicast is UDP: subscribers must detect gaps and recover themselves.

At 430k order messages per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One gateway handles about 50,000 messages a second across its sessions.
• One risk shard runs about 200,000 checks a second.
• One engine thread handles about 1,000,000 messages a second, about 1 µs each, with the books in CPU cache.
• One sequencer handles about 2,000,000 a second, and one journal about 3,000,000 appends with batching.
• One feed channel sends about 5,000,000 packets a second.

How the traffic is shaped

• There are about 1.2 market data messages for every order message.
• About 2% of messages trade.
• Partition capacity assumes symbols are placed so that no partition carries more than its share.

Follow one order, then one trade onto the feed. The bar is in milliseconds, so 0.001 is one microsecond:

Stock exchange · architecture

Trace a request: New order, to its fill

New order, to its fillA trade on the feed

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Play

PreviousNext

New order, to its fill: step 1 of 7

Members to Gateways

A binary order over a colocation cross-connect. The gateway parses it with kernel-bypass networking, checks the session sequence number and the per-session throttle, and routes it by symbol: 4 µs.

0.004 ms of a 0.05 ms budget

1. Members → Gateways0.004 ms
2. Gateways → Risk checks0.003 ms
3. Risk checks → Sequencer0.001 ms
4. Sequencer → Journal0.02 ms
5. Sequencer → Engines0.003 ms
6. Engines → Market dataasync
7. Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Stock exchange · architecture

Trace a request: A trade on the feed

New order, to its fillA trade on the feed

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Play

PreviousNext

A trade on the feed: step 1 of 3

Engines to Market data

The engine’s output events carry the input sequence number. The publisher turns each into a compact binary message (an executed message is about 30 bytes) and packs several into one UDP packet with a feed sequence number.

0.002 ms of a 0.02 ms budget

1. Engines → Market data0.002 ms
2. Market data → Subscribers0.004 ms
3. Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The order spends more than half of its 36 µs waiting for the journal flush. Matching itself is about 1 µs.

This is enough for a mid-level answer. A sequencer and journal in front of a deterministic, in-memory engine, risk per account, and multicast market data. Step 5 and the stress tests are what make it a senior answer.

Go deeperThe API2 min

Order entry is a binary protocol over a long-lived TCP session, in the style of Nasdaq’s OUCH. Fixed fields mean a gateway parses a message in nanoseconds.

EnterOrder            member → exchange, about 48 bytes
  clOrdId    u64      the member's ID for this order, unique per session per day
  symbol     u32      index into today's symbol directory
  side       u8       B or S
  qty        u32
  price      i64      integer ticks: dollars × 10,000, never a float
  tif        u8       DAY or IOC

Accepted     clOrdId, orderId u64, seq u64, timestamp u64
Executed     clOrdId, orderId, execId (seq, n), qty, price, liquidity (maker or taker)
Rejected     clOrdId, reason: CREDIT | PRICE_COLLAR | SIZE | DUPLICATE_ID | THROTTLED | HALTED
CancelOrder  clOrdId, orderId          ReplaceOrder  clOrdId, orderId, newQty, newPrice

clOrdId makes a resend safe: it gets DUPLICATE_ID, not a second order. Every outbound message carries a session sequence number. On reconnect, the member says which number it expects next, and the gateway replays the rest, so no execution report is lost. A throttle per session rejects excess messages with THROTTLED instead of queueing them, because a queue would only add latency.

Market data is one-way UDP multicast in the style of Nasdaq’s ITCH. Each packet carries the sequence number of its first message and a count, so a subscriber spots a gap at once.

Go deeperThe journal entry, and the durability options2 min

Journal entry    (one per sequenced input, about 100 bytes)
  seq        u64    the position in the partition's total order; no gaps
  epoch      u32    which sequencer leader wrote it; fences a deposed one
  ts         u64    the sequencer's timestamp; the engine never reads a clock
  type       u8     NEW | CANCEL | REPLACE | KILL | TIMER | REFDATA
  payload           the order fields, as received

-- Post-trade only. Written from the engines' output stream, never on the order path.
CREATE TABLE executions (
  partition   SMALLINT NOT NULL,
  seq         BIGINT   NOT NULL,  -- the input that caused the fill
  n           SMALLINT NOT NULL,  -- the n-th fill of that input
  symbol      INT      NOT NULL,
  price       BIGINT   NOT NULL,  -- ticks
  qty         INT      NOT NULL,
  PRIMARY KEY (partition, seq, n) -- a repeated event is a no-op insert
);

Option	Commit latency	Survives
fsync every entry	20 to 100 µs each	Power loss on this machine; caps throughput
Group commit to local NVMe	About 20 µs per batch	Power loss, not losing the machine
Copy to a second machine’s memory	About 7 µs round trip	Losing either machine (deep dive 3)
A consensus round per entry	10 to 30 µs	Losing a minority of three or five

Never order orders by gateway timestamps. Even with clocks within 100 ns, two gateways stamp orders in the same microsecond. Then clock skew decides who was first. That’s unfair, and impossible to replay.

Where credit is checked	Problem
In each gateway	An account on four gateways gets four times its limit
One risk shard per account	One more hop; the shard must rebuild exposure after a crash
Inside the matching engine	Once symbols are split, an account spans 32 engines

Step 5. Partition by symbol

The last thing to break is size. Message rates grow ten times.

At 4.3 million messages a second, the engine is at 430%, the sequencer at 215%, and the journal and feed past 100%. They all sit on one ordered stream, so they all saturate together. Gateways, risk shards and post-trade already scale out on their own.

Price-time priority is defined within one symbol’s book. Nothing in matching compares orders on two different symbols. So the order only has to be total per symbol. The market splits cleanly into partitions, each with its own sequencer, journal, engine thread and feed channel. Gateways route each order by a symbol directory loaded at the start of the day.

Stock exchange · architecture

Stage 5: Partition by symbol

Grow itTrace a request

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

12345

Order messages per second:

430,000

43k average430k peak4.3M (10×)20M

Stage 5 of 6

Partition by symbol

Why this stage

At 10× (4.3 million messages/s) one engine thread is 430% busy and the single sequencer and journal are past their limits.

What we add

32 partitions, each with its own sequencer, journal, single-threaded engine and feed channel.

A symbol lives on exactly one partition.

What it costs

No total order across partitions, so nothing may depend on the order of two different symbols.

At 430k order messages per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One gateway handles about 50,000 messages a second across its sessions.
• One risk shard runs about 200,000 checks a second.
• One engine thread handles about 1,000,000 messages a second, about 1 µs each, with the books in CPU cache.
• One sequencer handles about 2,000,000 a second, and one journal about 3,000,000 appends with batching.
• One feed channel sends about 5,000,000 packets a second.

How the traffic is shaped

• There are about 1.2 market data messages for every order message.
• About 2% of messages trade.
• Partition capacity assumes symbols are placed so that no partition carries more than its share.

We use 32 partitions. Nine would carry ten times today’s peak. But message rates per symbol are very uneven, and more partitions mean a smaller blast radius when one fails over. Placement uses yesterday’s message counts, so the busiest symbols are spread out. One hot symbol can’t be split. It doesn’t need to be. Even at ten times the peak, the busiest symbol is a few hundred thousand messages a second, inside one thread’s million.

The cost: two orders on different symbols now have no defined “first”. Anything that spans symbols, such as a multi-leg spread order, needs its own mechanism.

Go deeperPartition keys compared, and the kill switch across partitions1 min

Partition by	Every match inside one partition	Problem
Symbol, placed by measured rate	Yes	Moves only overnight
Symbol, alphabetical ranges	Yes	Hot symbols cluster on one partition
Member or account	No: a book would be split	Two members’ orders on one symbol must meet
Price range of a symbol	No	Price-time priority across ranges breaks

Is the kill switch atomic across 32 partitions? No, and it doesn’t need to be. The risk shard blocks the account first, so no new order gets through. Each partition then cancels at its own sequence number, within microseconds of the others. A fill just before a partition’s cancel is a real trade, reported as usual.

Stress it

The design is complete. Now we test it where exchanges are hard. The choice of one thread. A failover in the middle of an order. A subscriber that drops a packet. And two firms that arrive at the same moment. Only the failover adds boxes.

Why one thread per book

Interviewerminute 20

One thread for the whole engine? Wouldn’t more threads be faster?

What they are testing

Whether you can show with numbers that coordination costs more than the matching work itself.

Matching an order is a few hundred nanoseconds of work, on data that fits in the CPU cache. It compares with the best price, adjusts two quantities, and adds to a queue. Anything that coordinates threads costs as much or more. A cache line moving between two cores takes 50 to 100 ns. A contended lock, where a thread sleeps and wakes, takes 5 to 50 µs. That’s fifty times the work it protects.

And orders on one book must be applied one at a time anyway: price-time priority is a sequential definition. Several threads on one book would also make replay impossible, because the interleaving would differ on every run.

So each engine is one thread, pinned to a core. It spins on a lock-free ring buffer (the LMAX Disruptor pattern). It never allocates memory, so garbage collection never pauses it. It owns the books outright. That is also why replay works: there is exactly one interleaving, the sequence order.

Go deeperThe numbers behind it1 min

Operation	Typical cost
Read from L1 cache	About 1 ns
Uncontended lock acquire and release	About 20 ns
Cache line moving between two cores	About 50 to 100 ns
Read from main memory	About 100 ns
Contended lock where a thread sleeps, wakes	5 to 50 µs
Matching one order, books in cache	About 0.5 to 1 µs

Option	Deterministic replay	Verdict
Thread pool, lock per book	Only if lock order is journaled	The LLD answer; fine for one machine
Lock-free concurrent book	No: interleaving varies per run	Complex, and slower for this work
One thread, pinned core	Yes, by construction	The pick

Martin Fowler’s article on LMAX describes the same design at 6 million orders a second.

Failover with no lost or repeated fills

A partition’s primary engine has sent the first fill of an order, and dies before sending the second. The budget is about ten minutes of downtime a year. What does every downstream system see?

Each partition gets a hot standby: a second engine on another machine, applying the same sequenced stream. Being deterministic, it holds the same books and computes the same fills, but it emits nothing unless promoted.

It also changes how entries become durable. An entry is committed once it is in memory on two machines. The sequencer sends it to the standby and a journal host. It releases it to the primary when either one confirms. Both write to disk in the background. Losing any single machine loses no committed entry. And the order path drops from 36 µs to 23 µs, because the disk flush is gone.

Stock exchange · architecture

Stage 6: Hot standby per partition

Grow itTrace a requestBreak it

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

123456

Order messages per second:

430,000

43k average430k peak4.3M (10×)20M

Stage 6 of 6

Hot standby per partition

Why this stage

Availability: replaying a full day takes about 17 minutes, and 99.99% of trading hours allows about 10 minutes a year.

What we add

A standby engine per partition that applies the same sequenced stream.

An entry is committed once it is in memory on two machines, so the journal disk leaves the hot path.

What it costs

Double the engine hardware, and a failover protocol that must never let two primaries emit.

At 430k order messages per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One gateway handles about 50,000 messages a second across its sessions.
• One risk shard runs about 200,000 checks a second.
• One engine thread handles about 1,000,000 messages a second, about 1 µs each, with the books in CPU cache.
• One sequencer handles about 2,000,000 a second, and one journal about 3,000,000 appends with batching.
• One feed channel sends about 5,000,000 packets a second.

How the traffic is shaped

• There are about 1.2 market data messages for every order message.
• About 2% of messages trade.
• Partition capacity assumes symbols are placed so that no partition carries more than its share.

Stock exchange · architecture

Trace a request: New order, replicated

New order, to its fillA trade on the feedNew order, replicated

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Play

PreviousNext

New order, replicated: step 1 of 7

Members to Gateways

Parse, session checks, throttle, and route by symbol to the partition that owns it.

0.004 ms of a 0.05 ms budget

1. Members → Gateways0.004 ms
2. Gateways → Risk checks0.003 ms
3. Risk checks → Sequencer0.001 ms
4. Sequencer → Hot standby0.007 ms
5. Sequencer → Journalasync
6. Sequencer → Engines0.003 ms
7. Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Three rules make the handover exact:

1. Nothing uncommitted is acted on. The primary sees an entry only after it is on two machines. So the standby has every entry the primary ever applied.
2. Outputs have stable IDs. Every event is numbered by the input that caused it. Execution (seq 1002, n 1) is the same on the primary, the standby and a replay. Consumers keep the highest ID they’ve seen per partition, and drop anything at or below it.
3. Only the current epoch emits. Promotion raises the epoch. Gateways, the feed and post-trade reject events from an older epoch. So a primary that was only slow, not dead, can’t keep emitting after it’s been replaced.

The primary dies between two fills of one order

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

The standby computed both fills itself while the primary was alive. After promotion, it asks each consumer for the last event ID it holds, and resumes after it. A fill can be sent twice by accident, but it can never be processed twice or skipped. Detection is three missed 10 ms heartbeats, about 30 ms.

Common mistake

"The system delivers each fill exactly once"

No network delivers exactly once. The exchange guarantees that each fill is produced exactly once (determinism plus committed inputs) and applied exactly once (stable IDs plus consumers that drop repeats). Delivery itself is at least once.

Go deeperWays to keep a spare, and sequencer failover2 min

Option	Recovery time	Lost or repeated fills
Cold restart from snapshot and journal	Up to minutes	None, but far over the target
Primary ships its state or outputs	Seconds	Possible: it can die between applying and shipping
Hot standby applies the same inputs	Under a second	None, given the three rules
Consensus cluster (Raft) over the inputs	Under a second	None; the same idea with three or five nodes

Every five minutes the standby snapshots its books from a copy-on-write fork, so a replacement replays minutes of journal, not a day.

Sequencer failover has one subtle step. A committed entry is on at least one of the standby and the journal host, but not necessarily both. So the new sequencer takes the longer of the two logs, and copies the missing tail to the other. Only then does it number new messages, in a new epoch. Taking the shorter log would let the copies disagree about entry N.

Is memory on two machines less durable than fsync? An fsync survives power loss on one machine, but not a dead motherboard. A second machine’s memory survives losing either machine, the more common failure. A power loss that takes both is a data-centre disaster, handled by an asynchronous copy at a second site.

Market data to a thousand subscribers

A subscriber’s network card drops a packet. Another joins at 11:00. How does each get a correct book?

The feed carries only changes, so a missed message silently corrupts a subscriber’s book unless it is noticed. Every packet carries a sequence number: seeing 511 after 509 says exactly what is missing.

The feed goes out twice, A and B, through separate switches. A subscriber listens to both and keeps the first copy of each sequence number, so one lost packet almost never becomes a gap. For a real gap, there are two repair paths. A retransmission service holds the last few minutes of packets and answers “send 510 to 510”. A snapshot service, itself a feed subscriber, keeps every book and hands it out as of sequence N.

A subscriber joins late and builds a correct book

Scroll or pinch to zoom, drag to pan

• Color key
• Service

The order of the steps matters. Joining the feed first and buffering means nothing is lost while the snapshot is fetched. And the snapshot’s sequence number says exactly where to start. Retail apps never see this feed. Vendors pass it on over WebSockets.

Go deeperFan-out options compared1 min

Option	Fairness	Slow subscriber
TCP stream per subscriber	The last stream sent is last to know	Back-pressure or a big buffer
Message broker (Kafka, pub/sub)	Depends on consumer position	Falls behind on its own
UDP multicast, A and B feeds	Same packet, same moment for all	Drops packets, repairs itself

Multicast works only inside a network we control (colocation), not over the internet, and every subscriber must implement gap detection and recovery correctly.

Who is first?

Two firms send an order at the same nanosecond. Who is first, and how do we make that fair?

The sequencer decides: whichever reached it first. So fairness means an equal path to it for everyone. Colocation cabinets get the same length of fibre whatever their distance. Every gateway is identical, with no priority lanes. And per-session throttles stop one member flooding a shared gateway.

Equal paths aren’t equal waits: an order behind a burst on a shared gateway queues for microseconds, so fast firms open sessions on many gateways. Capping sessions per member limits that. The promise is the same path for the same connection, not identical outcomes. On the way out, multicast gives every subscriber each packet at the same time.

Two other designs exist. A speed bump delays every order by a fixed time (IEX uses 350 µs). That protects resting orders from the fastest traders. Frequent batch auctions collect orders for, say, 1 ms, and match them at one price. That removes the speed race. Both are different market structures. Most large exchanges use equal paths.

Go deeperFairness designs compared1 min

Approach	How it works	Cost
Equal paths, first come first served	Same cable length and gateway path for everyone	A race measured in nanoseconds, won by the fastest firm
Speed bump	Delay every inbound order by a fixed time (IEX: 350 µs)	Protects resting orders from the fastest traders; slower for all
Frequent batch auctions	Collect orders for, say, 1 ms, then match them at one price	Removes the speed race; a different market structure

When things break

Interviewerminute 40

One partition’s primary engine dies at 10:30. What do members see?

What they are testing

Whether the hot standby takes over at the next sequence number, with no fill lost or repeated.

The goal: any single machine can fail in the busiest second. Every acknowledged order and fill survives, with at most a sub-second pause for one partition.

The primary engine dies. Its hot standby has applied every committed entry, so it holds the same books. After three missed heartbeats, about 30 ms, it is promoted with a higher epoch. It asks each consumer for the last event ID it holds, and carries on from the next sequence number. Members trading that partition’s symbols see a pause of well under a second. No fill is lost or sent twice, for the reasons in the failover stress test.

A sequencer dies. Its partition holds orders for under a second. Entries that weren’t committed were never released, so nothing acted on them. Members simply resend, and their clOrdId stops any double order. The standby sequencer takes the longer log, raises the epoch, and continues.

Bugs are the failure replication doesn’t fix: both replicas make the same bad trade, faithfully. The defences are risk checks, price limits, circuit breakers per symbol, and the power to cancel clearly erroneous trades afterwards.

Click any box to break it and see what members would notice:

Stock exchange · architecture

Break it: what fails, who notices, what users see

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional

Break

A primary engineA standby engineA sequencerA journal hostA gatewayA risk shardA market data publisherA slow subscriberPost-trade

Failure: click any box to break it

A primary engine goes down

What breaks

Matching stops for the symbols on that partition.

Other partitions do not notice.

How we notice

Three missed heartbeats at 10 ms intervals.

What users see

Orders on those symbols pause for well under a second, then continue.

No fill is lost or repeated.

What we do

Promote the standby with a new epoch.

It has, or fetches from the journal host, every committed entry, so it resumes emitting after the last event each consumer holds; consumers drop duplicates by event ID.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Go deeperEvery failure, and what users see1 min

Part	What members see	What we do
Primary engine	One partition pauses under a second	Promote the standby in a new epoch; resume after last ID
Standby engine	Nothing	New standby from snapshot plus journal
Sequencer	One partition holds orders under a second	Standby sequencer, new epoch, after the longer log
Journal host	Nothing	Commit via the standby; copy the journal to a new host
Gateway	Its sessions reconnect	A backup gateway replays each session’s numbering
Risk shard	Its accounts rejected for seconds	Fail closed; standby rebuilds exposure from output streams
Feed publisher	Nothing while the other feed runs	Restart from the output stream; subscribers repair gaps
Post-trade	Copies arrive late	Resume after the last stored ID
Data centre	Trading halts, then resumes at the other site	A failover declared by people

Path	Guarantee	What users can see
Orders on one symbol	One total order: the sequence number	Priority is exactly arrival at the sequencer
Orders on different symbols	No order between partitions	Two orders sent together may be sequenced either way
Fills	Produced once, applied once by ID; delivered at least once	A rare repeated report, with the same execution ID
Risk exposure	Reserve on accept, release on cancel or fill	A brief reject for credit the member has just freed
Post-trade	Seconds behind at worst	Back-office screens lag trading

Go deeperRunning it: alerts, deploys and end of day1 min

Wake someone up when:

• p99 latency per partition passes 100 µs for a minute (latency spikes come before outages)
• a standby falls more than 1,000 entries behind
• feed gaps or retransmission requests suddenly rise
• end-of-day reconciliation finds any mismatch

Deploys never happen during trading hours. Determinism gives a strong test. After the close, replay the day’s journals through the new release, and compare outputs byte for byte with production. Releases then go to standbys first. Both engines also publish a running checksum of their outputs. A mismatch blocks promotion, because it means non-determinism slipped in.

End of day. At 16:00, sequenced commands run the closing auction and cancel every remaining DAY order. Executions are counted three ways (engine journals, post-trade database, drop copies) and must match exactly. Then the trade file goes to the clearing house, and the journals go to object storage.

Wrap up

The whole design

Here is everything we built, in the order we built it. Use the buttons to step through it.

Stock exchange · architecture

The full design, stage by stage

Color keyServiceDatabaseBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

123456

Order messages per second:

430,000

43k average430k peak4.3M (10×)20M

Stage 6 of 6

Hot standby per partition

Why this stage

Availability: replaying a full day takes about 17 minutes, and 99.99% of trading hours allows about 10 minutes a year.

What we add

A standby engine per partition that applies the same sequenced stream.

An entry is committed once it is in memory on two machines, so the journal disk leaves the hot path.

What it costs

Double the engine hardware, and a failover protocol that must never let two primaries emit.

At 430k order messages per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.The whole design. Step through it: one engine in memory, a sequencer and journal, risk checks, market data and post-trade, partitions by symbol, then a hot standby per partition.

The numbers behind the busy bars

What one machine can do

• One gateway handles about 50,000 messages a second across its sessions.
• One risk shard runs about 200,000 checks a second.
• One engine thread handles about 1,000,000 messages a second, about 1 µs each, with the books in CPU cache.
• One sequencer handles about 2,000,000 a second, and one journal about 3,000,000 appends with batching.
• One feed channel sends about 5,000,000 packets a second.

How the traffic is shaped

• There are about 1.2 market data messages for every order message.
• About 2% of messages trade.
• Partition capacity assumes symbols are placed so that no partition carries more than its share.

The decisions, and what each one costs

Decision	We chose	We gave up	We’d choose differently if
Order of arrival	One sequencer per partition	Ordering across partitions	We needed atomic orders across symbols
Matching	One pinned thread per partition	Parallelism inside a book	Never, for price-time priority
State	A journal, with books in memory	Ad hoc queries on live state	Latency targets were in milliseconds
Durability	Commit in two machines’ memory	Survival if both die at once	We had one machine per partition (group commit)
Availability	A hot standby replaying the same stream	Half the engine hardware sits idle	The budget allowed a 17-minute replay
Market data	Multicast, A and B feeds, snapshots, repair	Delivery guarantees inside the network	Subscribers were few, or on the internet (TCP)
Partitioning	By symbol, placed by measured rate	Live rebalancing	The market traded 24/7

How to spend 45 minutes

• 0 to 4. Message rate, latency target, who reads market data, risk.
• 4 to 8. 430,000 messages a second against a million per thread, 17 minutes of replay, 21 GB/s of unicast.
• 8 to 20. Build it: matching, the sequencer and journal, risk shards, multicast. A mid-level answer can stop here.
• 20 to 30. The stress tests: one thread per book, then failover with exact fills.
• 30 to 40. Market data repair or fairness.
• 40 to 45. An engine failing, then your own questions.

Never skip the sequencer, determinism, or the failover rules.

Follow-up questions

Interviewer asksA member says they were filled at a price they never should have been. How do you investigate?

Replay the partition’s journal from the snapshot before the trade, in a test environment. Determinism reproduces the exact trade and book, and the journal shows exactly which orders were ahead of theirs.

Interviewer asksHow do Good-'til-time orders expire if the engine never reads a clock?

The sequencer adds timer entries, for example one per millisecond while trading, carrying its timestamp. The engine expires orders when it applies a timer entry past their expiry. On replay, the same timer entries arrive at the same sequence numbers.

Go deeperMore follow-up questions1 min

What changes at 100 times today’s traffic? Partitions grow to a few hundred, which is mechanical. Feed bandwidth reaches 2 GB/s, so the feed splits into more multicast channels and subscribers listen only to their symbols. The hottest symbol may near one thread’s limit, so the code path per message gets attention first.

Product wants opening and closing auctions. Where do they go? In the engine, as a mode per symbol. Orders build up without matching, while the feed publishes the likely price. A sequenced command ends the auction, and everything fills at the price that trades the most. It’s new engine logic, but no new part.

Why not run a FIX gateway instead of a binary protocol? Most exchanges also run one. It translates for slower members, at tens of microseconds more per message. The binary protocol is for members who care about every microsecond.

Further reading

• Design a stock exchange order book, the LLD problem: the order book classes, cancel and replace, and order types.
• Replication, consensus and partitioning on this site, and the payment system for reconciliation and ledgers.
• The LMAX Architecture, Martin Fowler: a single-threaded business logic processor, input journaling and replicated replay, at 6 million orders a second.
• Event Sourcing, Martin Fowler: state as the replay of an ordered event log.
• Nasdaq TotalView-ITCH 5.0 specification: a real incremental market data feed, message by message.