Design a ticket booking system
Design a ticket booking system like Ticketmaster. Start with one service and one database, grow it with browsing on its own path, a cached seat map, a 10-minute hold, payment outside the transaction and a waiting room, then stress it with contention, late payments, bots and many on-sales at once, with interview prompts and code.
“Design a ticket booking system like Ticketmaster. Fans pick their own seats, and a big tour goes on sale at 10:00 sharp.”
On an ordinary day this is a modest system, a few thousand requests a second. Then a stadium tour goes on sale. Two million people arrive in the same minute for 50,000 seats, and every weakness shows at once. Two people click the same seat. A payment succeeds a second after its hold ran out. And bots take the front of the line.
So this page starts with one service and one database that is already correct. Then it lets the numbers grow it, one step at a time, until it survives 10:00. Then it tests the result against the on-sale’s worst moments.
The answer in one minute
Section titled “The answer in one minute”If you only have a minute, this is the design. The rest of the page explains each part.
- Hold a seat with one conditional update. “Mark this seat held, only if it’s free or its hold expired.” Two clicks on one seat give one hold and one “taken”. No timer is needed for expiry.
- Keep the card network out of the seat transaction. Authorize the card, then sell the seats, then capture the money. A sale that fails becomes a void, not a refund.
- Serve the seat map from a cache, up to 2 seconds stale. The map is only a guide. The hold decides, so a stale map costs a fan one “taken” message, never a double sale.
- Put a waiting room in front of a big on-sale. Admit people at a pace set by the seats left. Booking sees 8,000 shoppers instead of 2 million.
- Split by event. Every hold touches one event’s rows, so each big on-sale gets its own shard and its own line.
Scope it
Section titled “Scope it”Two questions decide almost everything: are seats assigned, and how big is the spike?
Design a ticket booking system like Ticketmaster.
Whether you ask about assigned seats, the biggest on-sale and hold time, and call “never sold twice” the hard promise.
These four questions matter most:
- Do fans pick specific seats? Yes, assigned seating for most events.
- How big is the biggest on-sale? A stadium: 50,000 seats, 2 million people at 10:00.
- How long can a fan hold seats? 10 minutes to pay.
- Is overselling ever acceptable? Never. A sold seat is sold to exactly one person.
From these answers we make five promises:
- Never sold twice. A seat is sold to at most one order, under any failure or retry. This is a hard promise, not a percentage.
- Always up. Browse and search work 99.99% of the time, holds and checkout 99.95%. The waiting room fails closed, and keeps everyone’s place.
- Fast. Search under 300 ms, the seat map under 200 ms, and a hold under 300 ms. Checkout under 3 seconds, including the card network.
- Durable. A confirmed order survives the loss of a data center zone. A hold may be lost in a failover: the fan just picks again.
- Fair. People who arrive before the on-sale opens get a random place. Later arrivals are first come, first served. One place per verified account.
We leave out resale and transfers, venue and seat-map authoring, dynamic pricing, and the payment provider itself.
Go deeperMore questions, and what each changes1 min
| Question | We assume | If the answer were different |
|---|---|---|
| General admission instead? | Assigned seats | A counter per section; contention moves from rows to one number |
| A small theater? | A stadium | 2,000 seats never needs a waiting room; one row per seat copes |
| Do we take payments ourselves? | Through a provider with separate authorize and capture | Without them, an expired hold after payment means a refund |
| Limits per buyer? | Yes, 4 to 8 tickets, set per event | No limit makes bots and resellers far more damaging |
| How fresh must the map be? | About 2 seconds stale is fine | Instant freshness means pushing every change to every browser |
The numbers that decide it
Section titled “The numbers that decide it”Now we work out the numbers. Size from one on-sale, not the average day.
- People in line at opening
- 2 million
- Seats on sale
- 50,000
- Tickets per order
- 2.5
- Seat map loads per shopper a minute
- 15
- Minutes a shopper spends inside
- 4
- Shoppers admitted a minute
- 2,000
500 million tickets a year, at 2.5 tickets an order, is about 6 orders a second. Browsing is the real load: 20 million daily users at 15 requests each.
So we the ordinary day runs on a handful of servers and one database. None of this is why the problem is hard.
2 million people for 20,000 orders, 100 people per order. If everyone is let in at once and reloads the seat map every 4 seconds, that’s 500,000 reads a second. If 5% of requests are hold attempts, that’s 25,000 holds a second, all on one event’s rows.
So we the map needs a cache, and something must stop 25,000 holds a second from reaching the database.
Two bits a seat (free, held, sold) for 50,000 seats.
So we send one section at a time, and don't let two million people load it at once.
Admit 2,000 people a minute. Each spends about 4 minutes inside, so 8,000 shoppers at once, fewer than the 20,000 orders on sale. They make about 100 holds a second, and the show sells out in about 12 minutes.
So we one database primary per event, with room to spare.
2 million entries at about 100 bytes is one Redis node. Checking your place every 15 seconds is 130,000 requests a second.
So we publish "admitted up to position N" as one small file the CDN caches for 2 seconds.
More people than tickets means the answer isn’t more servers. It’s letting fewer people in at once. Try the calculator: drop the crowd to 30,000, and one database serves the map without a cache. Admit 6,000 a minute instead of 2,000, and there are more shoppers than orders, so clicks start colliding.
- Orders until sold out
- 20,000
- People in line per order
- 100
- More people than tickets: most will leave empty-handed, so the line must be fair and say so early.
- Seat map reads with no waiting room
- 500,000/s
- Past one Redis node and far past any database: gate entry with a waiting room.
- Seat map bandwidth with no waiting room
- 6.3 GB/s
- Gigabytes a second of maps: send one section at a time, and admit fewer people.
- Hold attempts with no waiting room
- 25,000/s
- Past what one primary does on one event’s hot rows: admission control, not a bigger database.
- Shoppers inside at once
- 8,000
- Shoppers inside per order on sale
- 0.4
- Fewer shoppers than orders: most clicks find a free seat.
- Seat map reads with the waiting room
- 2,000/s
- Booking servers answer from their 1 second copy.
- Hold attempts with the waiting room
- 100/s
- Three attempts per shopper. One primary per event takes this easily.
- Minutes to sell out (80% of shoppers buy)
- 13
- Fast enough that the line moves visibly.
- Queue memory (100 bytes a person)
- 200 MB
- Fits one Redis node per on-sale.
Build the design
Section titled “Build the design”We start with the simplest design that is correct. Then we let the numbers break it, one at a time. Each step fixes only what broke, adds only the boxes that fix needs, and says what the fix costs. Throughout, follow one scenario: a fan picks two seats and buys them.
Step 1. One service, one database
Section titled “Step 1. One service, one database”The smallest version is one booking service and one Postgres database, with one row per seat per event. A fan picks seats, and the service marks them sold, only if they are still free, and inserts the tickets, all in one transaction. Even this smallest version must already be correct, because no later step can fix a seat sold twice.
Stage 1: One service, one database
Stage 1 of 6
One service, one database
Why this stage
Nothing yet.
This is the smallest design that sells a seat exactly once.
What we add
A booking service and a Postgres table with one row per seat per event.
A hold is a conditional update that sets the status, a hold id and an expiry.
What it costs
Browsing, seat maps and holds all share one server and one database, and nobody pays yet.
At 500k requests per second at the front door, Booking servers is 10000% busy. Stage 2 is the fix.
The numbers behind the busy bars
What one machine can do
- One booking server handles about 5,000 requests a second.
- One Postgres primary does about 10,000 point reads a second, but only about 3,000 hold transactions when they all hit one event's rows.
- One Redis node does about 100,000 to 150,000 simple operations a second.
- One waiting room server handles about 20,000 requests a second.
Outside limits
- We negotiated a payment provider limit of about 2,000 calls a second.
What the waiting room saves
- Writes are 5% of requests.
- Behind the waiting room, only about 1% of requests and 2% of hold attempts reach booking.
- At the edge, the CDN answers 90% of the line's status checks.
A second, independent guard sits underneath: the tickets table has one row per seat as its primary
key. Even a bug above the database can’t create two tickets for one seat.
This version is correct. But browsing and booking now fight over the same rows, and during an on-sale two million people reload the seat map every 4 seconds.
Step 2. Browse apart from booking
Section titled “Step 2. Browse apart from booking”On an ordinary day about 90% of requests are browsing. They need full-text and location search, and a minute of staleness is fine. Holds need the opposite. So the paths split. Event pages go through the CDN with a 60-second cache. An event service answers search from Elasticsearch, fed from Postgres by change data capture, which reads the database’s own log of commits. The index can always be rebuilt, and nothing that decides money or seats reads it.
Stage 2: Browse apart from booking
Stage 2 of 6
Browse apart from booking
Why this stage
Browsing and search are about 90% of an ordinary day, need full-text and geo search, and can be seconds stale.
Holds cannot.
What we add
A CDN for event pages, an API gateway, an event service on a search index fed by change data capture, and many booking servers.
What it costs
The search index lags the database by a few seconds, and there are now two read paths to keep apart.
At 500k requests per second at the front door, Booking shards is 2500% busy. Stage 3 is the fix.
The numbers behind the busy bars
What one machine can do
- One booking server handles about 5,000 requests a second.
- One Postgres primary does about 10,000 point reads a second, but only about 3,000 hold transactions when they all hit one event's rows.
- One Redis node does about 100,000 to 150,000 simple operations a second.
- One waiting room server handles about 20,000 requests a second.
Outside limits
- We negotiated a payment provider limit of about 2,000 calls a second.
What the waiting room saves
- Writes are 5% of requests.
- Behind the waiting room, only about 1% of requests and 2% of hold attempts reach booking.
- At the edge, the CDN answers 90% of the line's status checks.
The seat map, from a cache
Section titled “The seat map, from a cache”At 500,000 reads a second, the seat map is 50 times what one Postgres serves. It’s also the perfect thing to cache, because it can be wrong without harm. A fan who clicks a seat that was just taken gets a “taken” message and picks another. The map is never the authority.
Each section’s statuses are packed into one Redis value, 2 bits a seat. After every hold, release or sale commits, the booking server sets those bits with one atomic command, so servers never overwrite each other. A crash between the commit and that write can lose an update, so a worker rebuilds hot sections from the database every 5 seconds. Each booking server also keeps hot sections in memory for 1 second. So 500,000 reads a second become one Redis read per section, per server, per second.
Stage 3: Seat map from a cache
Stage 3 of 6
Seat map from a cache
Why this stage
In an on-sale, 2 million people reload the seat map every 4 seconds: 500,000 reads/s, 50 times what one Postgres serves.
What we add
A packed seat status array per event section in Redis, updated with an atomic BITFIELD after every commit, and a 1 second copy in each booking server.
What it costs
The map can be up to about 2 seconds stale, so some clicks land on seats that are already gone.
At 500k requests per second at the front door, Booking shards is 833% busy. Stage 4 is the fix.
The numbers behind the busy bars
What one machine can do
- One booking server handles about 5,000 requests a second.
- One Postgres primary does about 10,000 point reads a second, but only about 3,000 hold transactions when they all hit one event's rows.
- One Redis node does about 100,000 to 150,000 simple operations a second.
- One waiting room server handles about 20,000 requests a second.
Outside limits
- We negotiated a payment provider limit of about 2,000 calls a second.
What the waiting room saves
- Writes are 5% of requests.
- Behind the waiting room, only about 1% of requests and 2% of hold attempts reach booking.
- At the edge, the CDN answers 90% of the line's status checks.
Go deeperWhy change data capture, and not two writes1 min
The booking service could write to Postgres and then update the search index itself. But that is two writes to two systems with no transaction across them. A crash between them leaves the index wrong, and nothing notices. Retrying from the service also means every write path must remember to do it.
Change data capture avoids both. A connector, such as Debezium, reads Postgres’s own log of committed changes and turns each one into an event. A small indexer reads the events and updates Elasticsearch. If the indexer stops, it resumes from its place in the log, so no change is skipped. If the index is lost, it is rebuilt by reading the tables once, then following the log again.
The cost is a delay. A new event or a changed price usually reaches search within a second or two, and up to a minute under load. That is fine for browsing, and it is why nothing that sells a seat ever reads the index.
Step 3. Hold a seat
Section titled “Step 3. Hold a seat”Browsing no longer touches the seat rows. Now booking itself must be safe: a fan needs time to pay, and nobody else may take the seats meanwhile.
The hold lives on the seat row itself: a status (available, held, sold), the hold’s ID, and when it ends. A hold is one conditional update per seat, inside one transaction. It says: “set this seat to held, for my hold, until 10 minutes from now, only if it’s available or its hold has ended.” If any seat changes 0 rows, someone else got it, and the whole transaction rolls back.
That makes expiry exact without a timer. A seat whose hold time has passed is simply free to the next update that looks at it. “Now” comes from the database’s clock, so every server agrees.
The race this must survive is two fans clicking the same seat in the same millisecond:
- Color key
- Service
- Database
Ben’s update waits for Ana’s lock, then checks its condition again on the new row, finds the seat held, and changes nothing. This works under Postgres’s default isolation level. The first stress test compares it with locking and Redis.
Follow a seat map load and a hold:
Trace a request: Load the seat map
Load the seat map: step 1 of 5
Fans to CDN edge
The browser asks for one section of the map. The CDN passes it through: the map changes every second, so edge caching would only add staleness.
- Fans → CDN edge10 ms
- CDN edge → API gateway10 ms
- API gateway → Booking servers1 ms
- Booking servers → Seat map cache1 ms
- Response
Step 4. Pay outside the transaction
Section titled “Step 4. Pay outside the transaction”A card authorization takes 300 ms to a couple of seconds. It can never sit inside a database transaction holding locks on the hottest rows. So checkout runs in three short steps, with the slow call between them:
- Create the order as pending, keyed by the client’s idempotency key. Extend the hold by 5 minutes, so it can’t expire while the card network thinks.
- Authorize the card with the payment provider. Authorizing reserves the money without taking it. No transaction is open.
- Sell, in one transaction. Mark the seats sold where the hold is still ours. Insert the tickets, and confirm the order. Also write an outbox row: a message stored in the same transaction, so it can’t be lost.
A relay ships outbox rows to Kafka. Workers then capture the payment (actually take the money), render tickets and send email. They also sweep expired holds back to available, which only tidies the seat map. Correctness never depends on it.
Stage 4: Checkout with payments
Stage 4 of 6
Checkout with payments
Why this stage
A hold must turn into a paid order before it expires, and a card authorization takes about a second, far too long to hold database locks.
What we add
A payment provider called outside any transaction, an outbox feeding Kafka, and workers that capture payments, issue tickets, send email and sweep expired holds.
What it costs
A payment can succeed after its hold expired.
Deep dive 3 handles it with authorize first, capture later.
At 500k requests per second at the front door, Booking shards is 833% busy. Stage 5 is the fix.
The numbers behind the busy bars
What one machine can do
- One booking server handles about 5,000 requests a second.
- One Postgres primary does about 10,000 point reads a second, but only about 3,000 hold transactions when they all hit one event's rows.
- One Redis node does about 100,000 to 150,000 simple operations a second.
- One waiting room server handles about 20,000 requests a second.
Outside limits
- We negotiated a payment provider limit of about 2,000 calls a second.
What the waiting room saves
- Writes are 5% of requests.
- Behind the waiting room, only about 1% of requests and 2% of hold attempts reach booking.
- At the edge, the CDN answers 90% of the line's status checks.
Follow one checkout through it:
Trace a request: Pay for a hold
Pay for a hold: step 1 of 9
Fans to CDN edge
POST /orders with the hold id, a payment method token and an Idempotency-Key.
- Fans → CDN edge10 ms
- CDN edge → API gateway10 ms
- API gateway → Booking servers1 ms
- Booking servers → Booking shards6 ms
- Booking servers → Payments800 ms
- Booking servers → Booking shards8 ms
- Booking servers → Kafkaasync
- Kafka → Workersasync
- Response
This is enough for a mid-level answer. A conditional update for holds, payment outside the transaction, a cached seat map, search on its own index. Step 5 and the stress tests are what make it a senior answer.
Go deeperThe API2 min
GET /v1/events/81231/seatmap?section=A12X-Access-Token: <signed token from the waiting room>200 OK { "section": "A12", "version": 88412, "seats": "base64 of 2 bits per seat" }
POST /v1/events/81231/holdsIdempotency-Key: 2b7f1c0e-...{ "seatIds": ["A12-7-14", "A12-7-15"] }201 Created { "holdId": "f3a9...", "expiresAt": "2027-03-01T10:14:12Z" }409 Conflict { "error": "SEATS_TAKEN", "unavailable": ["A12-7-15"] }
POST /v1/ordersIdempotency-Key: 9d44e2aa-...{ "holdId": "f3a9...", "paymentMethod": "pm_tok_..." }201 Created { "orderId": "o_7Q...", "status": "CONFIRMED", "tickets": [ ... ] }402 Payment Required { "status": "DECLINED" }409 Conflict { "status": "SEATS_GONE" }202 Accepted { "status": "PAYING" }A 409 on a hold is an expected outcome. The client refreshes the section, and the fan picks again.
A retried checkout with the same key returns the first result. 202 means the card network was
slow, and the client polls the order. SEATS_GONE means the hold expired, and a seat went to
someone else. The authorization is voided. The seat layout itself (rows, numbers, positions) is
static, from the CDN.
Hold and pay are separate calls on purpose. One “book seat” call that holds and charges would put the card network inside the seat transaction.
Go deeperThe tables1 min
CREATE TABLE seats ( event_id BIGINT NOT NULL, seat_id TEXT NOT NULL, -- 'A12-7-14': section, row, number status TEXT NOT NULL DEFAULT 'AVAILABLE', -- AVAILABLE | HELD | SOLD hold_id UUID, -- which hold owns it; the fence for checkout held_until TIMESTAMPTZ, -- compared with the database clock order_id TEXT, PRIMARY KEY (event_id, seat_id));
CREATE TABLE tickets ( event_id BIGINT NOT NULL, seat_id TEXT NOT NULL, order_id TEXT NOT NULL, PRIMARY KEY (event_id, seat_id) -- one ticket per seat, whatever the code above does);Holds and orders each carry a unique (user_id, idem_key), so a retried call finds the first one.
| Database | Fits because | Pick when |
|---|---|---|
| Postgres | Conditional updates, transactions, unique keys | The default: correctness is cheap to express |
| DynamoDB | Conditional writes; transactions up to 100 items | You’re on AWS and want no database to run |
| Redis | Atomic holds at 100,000+ a second | Holds alone, in front of a durable sale |
Step 5. A virtual waiting room
Section titled “Step 5. A virtual waiting room”Now it is 10:00, and two million people hit the on-sale at once.
More booking servers only send more conflicting updates to the same rows. And even with an infinite database, 2 million people picking from 50,000 seats at once means nearly every click collides. So we put a line in front of booking.
Everyone who arrives joins it and gets a signed ticket with a position. A pacing loop admits people by moving one number forward: “admitted up to position N”. When your position is reached, your ticket is swapped for an access token, valid for 15 minutes. The gateway refuses booking calls without one. Behind the line, booking sees about 8,000 shoppers instead of 2 million.
Stage 5: Virtual waiting room
Stage 5 of 6
Virtual waiting room
Why this stage
With reads cached, holds still arrive at 25,000/s on one event’s rows, 8 times what one primary takes on hot rows, and 2.5 times the payment provider’s limit.
What we add
A waiting room in front of booking.
People join a line, and a signed access token admits about 2,000 a minute, matched to the seats left.
What it costs
Most people wait, and most never get a ticket.
The line itself is now a system that must not fail.
At 500k requests per second at the front door, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One booking server handles about 5,000 requests a second.
- One Postgres primary does about 10,000 point reads a second, but only about 3,000 hold transactions when they all hit one event's rows.
- One Redis node does about 100,000 to 150,000 simple operations a second.
- One waiting room server handles about 20,000 requests a second.
Outside limits
- We negotiated a payment provider limit of about 2,000 calls a second.
What the waiting room saves
- Writes are 5% of requests.
- Behind the waiting room, only about 1% of requests and 2% of hold attempts reach booking.
- At the edge, the CDN answers 90% of the line's status checks.
The admission rate is set by the seats, not the servers. It’s the configured rate, but never more than about two shoppers per order still for sale. When every seat is held, it stops. It resumes as holds expire. When everything is sold, the line closes and tells everyone at once, instead of letting them wait an hour to find out.
// The waiting room for one on-sale. Its whole state is a counter, one place per account, and// one number: everyone at or below admittedUpTo may shop. That number is all the CDN serves.interface QueueStore { // INCR and SET NX in one Lua script, so a second tab or a retry gets the same place. long joinOnce(String accountId); long admittedUpTo(); // read from memory, refreshed every second void advanceTo(long position);}
class WaitingRoom { private static final long ACCESS_MILLIS = 15 * 60_000; // time to shop; checkout extends the hold, not this private final long eventId; private final QueueStore store; private final byte[] key; // shared with the gateway, which checks every booking call private final LongSupplier clock;
WaitingRoom(long eventId, QueueStore store, byte[] key, LongSupplier clock) { this.eventId = eventId; this.store = store; this.key = key; this.clock = clock; }
// The ticket carries the position, signed, so checking it later needs no store lookup and a // lost counter can restart above the highest ticket anyone shows. String join(String accountId) { long position = store.joinOnce(accountId); return sign("ticket|" + eventId + "|" + accountId + "|" + position); }
// Runs once a second on one leader per event. Admission follows the seats, not the servers: // letting in more shoppers than there are orders left only makes their clicks collide. void tick(long perSecond, long ordersLeft, long shoppersInside) { long room = Math.max(0, 2 * ordersLeft - shoppersInside); // about two shoppers per order left long step = Math.min(perSecond, room); // zero when everything is held if (step > 0) store.advanceTo(store.admittedUpTo() + step); }
// Swap a ticket whose turn has come for an access token. Empty means "not yet": keep polling. Optional<String> admit(String ticket) { String[] f = verify(ticket).split("\\|"); // a forged or edited ticket throws here if (!f[0].equals("ticket") || Long.parseLong(f[1]) != eventId) throw new SecurityException("wrong ticket"); if (Long.parseLong(f[3]) > store.admittedUpTo()) return Optional.empty(); long expires = clock.getAsLong() + ACCESS_MILLIS; return Optional.of(sign("access|" + eventId + "|" + f[2] + "|" + expires)); }
private String sign(String payload) { return payload + "." + Base64.getUrlEncoder().withoutPadding().encodeToString(mac(payload)); }
private String verify(String token) { int dot = token.lastIndexOf('.'); String payload = token.substring(0, Math.max(dot, 0)); byte[] given = Base64.getUrlDecoder().decode(token.substring(dot + 1)); // Constant time, so response timing does not leak how much of a guessed signature matched. if (!MessageDigest.isEqual(mac(payload), given)) throw new SecurityException("bad signature"); return payload; }
private byte[] mac(String payload) { try { Mac m = Mac.getInstance("HmacSHA256"); m.init(new SecretKeySpec(key, "HmacSHA256")); return m.doFinal(payload.getBytes(StandardCharsets.UTF_8)); } catch (GeneralSecurityException e) { throw new IllegalStateException(e); } }}# The waiting room for one on-sale. Its whole state is a counter, one place per account, and# one number: everyone at or below admitted_up_to may shop. That number is all the CDN serves.class QueueStore(Protocol): # INCR and SET NX in one Lua script, so a second tab or a retry gets the same place. def join_once(self, account_id: str) -> int: ... def admitted_up_to(self) -> int: ... # read from memory, refreshed every second def advance_to(self, position: int) -> None: ...
class BadToken(Exception): pass
class WaitingRoom: ACCESS_SECONDS = 15 * 60 # time to shop; checkout extends the hold, not this
def __init__(self, event_id: int, store: QueueStore, key: bytes, clock=time.time): self._event_id = event_id self._store = store self._key = key # shared with the gateway, which checks every booking call self._clock = clock
def join(self, account_id: str) -> str: # The ticket carries the position, signed, so checking it later needs no store lookup # and a lost counter can restart above the highest ticket anyone shows. position = self._store.join_once(account_id) return self._sign(f"ticket|{self._event_id}|{account_id}|{position}")
def tick(self, per_second: int, orders_left: int, shoppers_inside: int) -> None: # Runs once a second on one leader per event. Admission follows the seats, not the # servers: letting in more shoppers than there are orders left only makes clicks collide. room = max(0, 2 * orders_left - shoppers_inside) # about two shoppers per order left step = min(per_second, room) # zero when everything is held if step > 0: self._store.advance_to(self._store.admitted_up_to() + step)
# Swap a ticket whose turn has come for an access token. None means: keep polling. def admit(self, ticket: str) -> str | None: kind, event, account, position = self._verify(ticket).split("|") # forged tickets raise if kind != "ticket" or int(event) != self._event_id: raise BadToken("wrong ticket") if int(position) > self._store.admitted_up_to(): return None expires = int(self._clock()) + self.ACCESS_SECONDS return self._sign(f"access|{self._event_id}|{account}|{expires}")
def _sign(self, payload: str) -> str: return payload + "." + self._mac(payload).hex()
def _verify(self, token: str) -> str: payload, _, sig = token.rpartition(".") # Constant time, so response timing does not leak how much of a guessed signature matched. if not hmac.compare_digest(self._mac(payload).hex(), sig): raise BadToken("bad signature") return payload
def _mac(self, payload: str) -> bytes: return hmac.new(self._key, payload.encode(), hashlib.sha256).digest()using System.Security.Cryptography;
// The waiting room for one on-sale. Its whole state is a counter, one place per account, and// one number: everyone at or below AdmittedUpTo may shop. That number is all the CDN serves.interface IQueueStore{ // INCR and SET NX in one Lua script, so a second tab or a retry gets the same place. long JoinOnce(string accountId); long AdmittedUpTo(); // read from memory, refreshed every second void AdvanceTo(long position);}
class WaitingRoom(long eventId, IQueueStore store, byte[] key, TimeProvider clock){ // Time to shop; checkout extends the hold, not this. static readonly TimeSpan AccessFor = TimeSpan.FromMinutes(15);
// The ticket carries the position, signed, so checking it later needs no store lookup and a // lost counter can restart above the highest ticket anyone shows. public string Join(string accountId) { long position = store.JoinOnce(accountId); return Sign($"ticket|{eventId}|{accountId}|{position}"); }
// Runs once a second on one leader per event. Admission follows the seats, not the servers: // letting in more shoppers than there are orders left only makes their clicks collide. public void Tick(long perSecond, long ordersLeft, long shoppersInside) { long room = Math.Max(0, 2 * ordersLeft - shoppersInside); // about two shoppers per order left long step = Math.Min(perSecond, room); // zero when everything is held if (step > 0) store.AdvanceTo(store.AdmittedUpTo() + step); }
// Swap a ticket whose turn has come for an access token. Null means "not yet": keep polling. public string? Admit(string ticket) { var f = Verify(ticket).Split('|'); // a forged or edited ticket throws here if (f[0] != "ticket" || long.Parse(f[1]) != eventId) throw new CryptographicException("wrong ticket"); if (long.Parse(f[3]) > store.AdmittedUpTo()) return null; var expires = (clock.GetUtcNow() + AccessFor).ToUnixTimeSeconds(); return Sign($"access|{eventId}|{f[2]}|{expires}"); }
string Sign(string payload) => payload + "." + Convert.ToHexString(Mac(payload));
string Verify(string token) { int dot = token.LastIndexOf('.'); if (dot < 0) throw new CryptographicException("malformed token"); var payload = token[..dot]; var given = Convert.FromHexString(token[(dot + 1)..]); // Constant time, so response timing does not leak how much of a guessed signature matched. if (!CryptographicOperations.FixedTimeEquals(Mac(payload), given)) throw new CryptographicException("bad signature"); return payload; }
byte[] Mac(string payload) => HMACSHA256.HashData(key, Encoding.UTF8.GetBytes(payload));}// QueueStore holds one on-sale's line. JoinOnce runs INCR and SET NX in one Lua script, so a// second tab or a retry gets the same place.type QueueStore interface { JoinOnce(accountID string) (int64, error) AdmittedUpTo() int64 // read from memory, refreshed every second AdvanceTo(position int64) error}
// WaitingRoom's whole state is a counter, one place per account, and one number: everyone at// or below AdmittedUpTo may shop. That number is all the CDN serves.type WaitingRoom struct { eventID int64 store QueueStore key []byte // shared with the gateway, which checks every booking call now func() time.Time}
const accessFor = 15 * time.Minute // time to shop; checkout extends the hold, not this
var ErrBadToken = errors.New("bad token")
// Join returns a signed ticket. It carries the position, so checking it later needs no store// lookup and a lost counter can restart above the highest ticket anyone shows.func (w *WaitingRoom) Join(accountID string) (string, error) { pos, err := w.store.JoinOnce(accountID) if err != nil { return "", err } return w.sign(fmt.Sprintf("ticket|%d|%s|%d", w.eventID, accountID, pos)), nil}
// Tick runs once a second on one leader per event. Admission follows the seats, not the// servers: letting in more shoppers than there are orders left only makes clicks collide.func (w *WaitingRoom) Tick(perSecond, ordersLeft, shoppersInside int64) error { room := max(0, 2*ordersLeft-shoppersInside) // about two shoppers per order left step := min(perSecond, room) // zero when everything is held if step == 0 { return nil } return w.store.AdvanceTo(w.store.AdmittedUpTo() + step)}
// Admit swaps a ticket whose turn has come for an access token. "" means: keep polling.func (w *WaitingRoom) Admit(ticket string) (string, error) { payload, err := w.verify(ticket) // a forged or edited ticket fails here if err != nil { return "", err } f := strings.Split(payload, "|") if len(f) != 4 || f[0] != "ticket" || f[1] != strconv.FormatInt(w.eventID, 10) { return "", ErrBadToken } pos, err := strconv.ParseInt(f[3], 10, 64) if err != nil { return "", ErrBadToken } if pos > w.store.AdmittedUpTo() { return "", nil } expires := w.now().Add(accessFor).Unix() return w.sign(fmt.Sprintf("access|%d|%s|%d", w.eventID, f[2], expires)), nil}
func (w *WaitingRoom) mac(payload string) []byte { m := hmac.New(sha256.New, w.key) m.Write([]byte(payload)) return m.Sum(nil)}
func (w *WaitingRoom) sign(payload string) string { return payload + "." + hex.EncodeToString(w.mac(payload))}
func (w *WaitingRoom) verify(token string) (string, error) { dot := strings.LastIndexByte(token, '.') if dot < 0 { return "", ErrBadToken } given, err := hex.DecodeString(token[dot+1:]) // hmac.Equal is constant time, so response timing does not leak how much of a guessed // signature matched. if err != nil || !hmac.Equal(w.mac(token[:dot]), given) { return "", ErrBadToken } return token[:dot], nil}import { createHmac, timingSafeEqual } from 'node:crypto';
// The waiting room for one on-sale. Its whole state is a counter, one place per account, and// one number: everyone at or below admittedUpTo may shop. That number is all the CDN serves.interface QueueStore { // INCR and SET NX in one Lua script, so a second tab or a retry gets the same place. joinOnce(accountId: string): Promise<number>; admittedUpTo(): number; // read from memory, refreshed every second advanceTo(position: number): Promise<void>;}
const ACCESS_MS = 15 * 60_000; // time to shop; checkout extends the hold, not this
class WaitingRoom { constructor( private readonly eventId: number, private readonly store: QueueStore, private readonly key: Buffer, // shared with the gateway, which checks every booking call private readonly now: () => number = Date.now, ) {}
// The ticket carries the position, signed, so checking it later needs no store lookup and a // lost counter can restart above the highest ticket anyone shows. async join(accountId: string): Promise<string> { const position = await this.store.joinOnce(accountId); return this.sign(`ticket|${this.eventId}|${accountId}|${position}`); }
// Runs once a second on one leader per event. Admission follows the seats, not the servers: // letting in more shoppers than there are orders left only makes their clicks collide. async tick(perSecond: number, ordersLeft: number, shoppersInside: number): Promise<void> { const room = Math.max(0, 2 * ordersLeft - shoppersInside); // about two shoppers per order left const step = Math.min(perSecond, room); // zero when everything is held if (step > 0) await this.store.advanceTo(this.store.admittedUpTo() + step); }
// Swap a ticket whose turn has come for an access token. null means "not yet": keep polling. admit(ticket: string): string | null { const [kind, event, account, position] = this.verify(ticket).split('|'); // forged tickets throw if (kind !== 'ticket' || Number(event) !== this.eventId) throw new Error('wrong ticket'); if (Number(position) > this.store.admittedUpTo()) return null; const expires = this.now() + ACCESS_MS; return this.sign(`access|${this.eventId}|${account}|${expires}`); }
private mac(payload: string): Buffer { return createHmac('sha256', this.key).update(payload).digest(); }
private sign(payload: string): string { return `${payload}.${this.mac(payload).toString('hex')}`; }
private verify(token: string): string { const dot = token.lastIndexOf('.'); const payload = token.slice(0, Math.max(dot, 0)); const given = Buffer.from(token.slice(dot + 1), 'hex'); const want = this.mac(payload); // Constant time, so response timing does not leak how much of a guessed signature matched. // timingSafeEqual throws on unequal lengths, so check that first. if (dot < 0 || given.length !== want.length || !timingSafeEqual(given, want)) { throw new Error('bad signature'); } return payload; }}Four details make it work:
- Waiting costs almost nothing. Every browser fetches the same status file every 15 seconds, cached for 2 seconds at the CDN. Two million people polling becomes about one request per CDN location every 2 seconds.
- The line’s state is tiny. A counter, a map from account to position (so a second tab gets the same place), and the number N. A join is one atomic Redis script.
- Early arrivals are shuffled. People who arrive before 10:00 get random places at 10:00. Without that, the line rewards whichever script fires first at 10:00:00.000.
- Tokens are the enforcement. The gateway checks each token’s signature and expiry, with no lookup. A token is tied to an account and an event. The ticket limit is counted in the hold transaction, so sharing a token gains nothing.
Go deeperWhy not just rate limit, and other options1 min
| Option | Wins | Loses |
|---|---|---|
| Let everyone in, autoscale | Nothing to build | The database and the seats don’t scale; clicks collide |
Rate limit with 429 |
Simple; protects the backend | Random winners; rewards bots and fast retries |
| Virtual waiting room | Fair order, bounded load, cheap to wait | A new critical component; people see their odds |
| Lottery for registered fans | Bots filtered before sale day | Days of lead time; only for the biggest shows |
A rate limiter sheds load at random: whoever’s retry lands in a free slot wins, which rewards the fastest scripts. It does no less work, because rejected clients retry at once. A line gives each person one place, and costs one cached check every 15 seconds. Rate limits stay at the gateway as a backstop. For the very largest on-sales, a registration lottery sits in front of the line.
Stress it
Section titled “Stress it”The design is complete. Now we test it against the on-sale’s worst moments. Thousands of clicks on the same seats. A payment that lands too late. Bots. And ten big on-sales on one morning. Only the last one adds new pieces.
Never selling a seat twice, under contention
Section titled “Never selling a seat twice, under contention”How exactly do you make sure two people can’t hold the same seat? Optimistic or pessimistic locking?
Whether you compare the locking options with numbers for a hot event.
| Approach | How it works | At 25,000 attempts a second on one event |
|---|---|---|
| Read, then write | Read the status, then update if free | Both readers see free; both write. Wrong |
Pessimistic: SELECT ... FOR UPDATE |
Lock the rows, check, update, commit | Waiters pile up; the connection pool runs out |
| Optimistic: version column | Update only if the version matches; retry if not | No waiting, but losers retry, multiplying load |
| Conditional update | Update where free or expired; 0 rows means taken | Locks held a few ms; losers fail fast |
The contention math. Connections in use equal the arrival rate times how long each transaction holds one. This is Little’s law. A pessimistic hold keeps its locks across several round trips, about 10 ms. It also waits for one or two others ahead of it on a hot seat: say 30 ms in all. At 25,000 attempts a second, that’s 25,000 × 0.03 = 750 connections busy, against a pool of a few hundred. The database falls over at the moment it matters most.
The conditional update has no application code between check and write. So it holds locks for 2 to 4 ms, and losers fail in a millisecond. That’s 75 connections. Survivable, but 25,000 commits a second is still 8 times what one primary does on hot rows. Behind the waiting room, 100 holds a second × 3 ms is less than one connection busy on average.
Optimistic retries have their own trap. If a fraction p of attempts lose and every loser retries,
the database sees 1 / (1 - p) times the load. At p = 0.9, late in a sale, that’s 10 times.
So a “taken” answer is never retried blindly. The fan chooses again, or asks for “best available”.
// Holding seats: one transaction, one conditional update per seat, all or nothing.interface Tx { int execute(String sql, Object... args); // returns the number of rows changed UUID queryUuid(String sql, Object... args); // null when there is no row}
interface Db { <T> T inTransaction(Function<Tx, T> work); // commits on return, rolls back on throw}
record Hold(UUID holdId, List<String> seats) {}
class SeatsTakenException extends RuntimeException { final String seat; SeatsTakenException(String seat) { super("seat " + seat + " is taken"); this.seat = seat; }}
class HoldService { // The idempotency row goes in first. A concurrent retry with the same key blocks on the // unique index until we commit, then finds our hold instead of competing for the seats. static final String INSERT_HOLD = """ INSERT INTO holds (hold_id, event_id, user_id, idem_key, expires_at) VALUES (?, ?, ?, ?, now() + interval '10 minutes') ON CONFLICT (user_id, idem_key) DO NOTHING"""; static final String FIND_HOLD = "SELECT hold_id FROM holds WHERE user_id = ? AND idem_key = ?";
// Free, or held by someone whose time ran out: an expired hold is free the moment it // expires, with no sweeper involved. now() is the database clock, so app servers whose // clocks drift can never disagree about when a hold ends. static final String TAKE_SEAT = """ UPDATE seats SET status = 'HELD', hold_id = ?, held_until = now() + interval '10 minutes' WHERE event_id = ? AND seat_id = ? AND (status = 'AVAILABLE' OR (status = 'HELD' AND held_until < now()))""";
private final Db db;
HoldService(Db db) { this.db = db; }
Hold hold(long eventId, long userId, String idemKey, List<String> seatIds) { // Everyone locks seats in the same order, so two overlapping requests queue behind // each other instead of deadlocking. List<String> seats = seatIds.stream().sorted().toList(); UUID holdId = UUID.randomUUID(); return db.inTransaction(tx -> { if (tx.execute(INSERT_HOLD, holdId, eventId, userId, idemKey) == 0) return new Hold(tx.queryUuid(FIND_HOLD, userId, idemKey), seats); // a retry for (String seat : seats) { // Under READ COMMITTED a second writer on this row waits for the first to // commit, then re-checks the WHERE clause against the new row. The loser // changes 0 rows. No SELECT first: reading, then writing, is the race. if (tx.execute(TAKE_SEAT, holdId, eventId, seat) == 0) throw new SeatsTakenException(seat); // rollback frees the seats taken so far } return new Hold(holdId, seats); }); }}# Holding seats: one transaction, one conditional update per seat, all or nothing.class Tx(Protocol): def execute(self, sql: str, *args) -> int: ... # returns the number of rows changed def query_uuid(self, sql: str, *args) -> UUID | None: ...
class Db(Protocol): def transaction(self) -> ContextManager[Tx]: ... # commits on exit, rolls back on raise
@dataclass(frozen=True)class Hold: hold_id: UUID seats: list[str]
class SeatsTaken(Exception): def __init__(self, seat: str): super().__init__(f"seat {seat} is taken") self.seat = seat
# The idempotency row goes in first. A concurrent retry with the same key blocks on the# unique index until we commit, then finds our hold instead of competing for the seats.INSERT_HOLD = """ INSERT INTO holds (hold_id, event_id, user_id, idem_key, expires_at) VALUES (%s, %s, %s, %s, now() + interval '10 minutes') ON CONFLICT (user_id, idem_key) DO NOTHING"""FIND_HOLD = "SELECT hold_id FROM holds WHERE user_id = %s AND idem_key = %s"
# Free, or held by someone whose time ran out: an expired hold is free the moment it# expires, with no sweeper involved. now() is the database clock, so app servers whose# clocks drift can never disagree about when a hold ends.TAKE_SEAT = """ UPDATE seats SET status = 'HELD', hold_id = %s, held_until = now() + interval '10 minutes' WHERE event_id = %s AND seat_id = %s AND (status = 'AVAILABLE' OR (status = 'HELD' AND held_until < now()))"""
class HoldService: def __init__(self, db: Db): self._db = db
def hold(self, event_id: int, user_id: int, idem_key: str, seat_ids: list[str]) -> Hold: # Everyone locks seats in the same order, so two overlapping requests queue behind # each other instead of deadlocking. seats = sorted(seat_ids) hold_id = uuid4() with self._db.transaction() as tx: if tx.execute(INSERT_HOLD, hold_id, event_id, user_id, idem_key) == 0: return Hold(tx.query_uuid(FIND_HOLD, user_id, idem_key), seats) # a retry for seat in seats: # Under READ COMMITTED a second writer on this row waits for the first to # commit, then re-checks the WHERE clause against the new row. The loser # changes 0 rows. No SELECT first: reading, then writing, is the race. if tx.execute(TAKE_SEAT, hold_id, event_id, seat) == 0: raise SeatsTaken(seat) # rollback frees the seats taken so far return Hold(hold_id, seats)// Holding seats: one transaction, one conditional update per seat, all or nothing.interface ITx{ int Execute(string sql, params object[] args); // returns the number of rows changed Guid? QueryGuid(string sql, params object[] args);}
interface IDb{ T InTransaction<T>(Func<ITx, T> work); // commits on return, rolls back on throw}
record Hold(Guid HoldId, IReadOnlyList<string> Seats);
class SeatsTakenException(string seat) : Exception($"seat {seat} is taken"){ public string Seat { get; } = seat;}
class HoldService(IDb db){ // The idempotency row goes in first. A concurrent retry with the same key blocks on the // unique index until we commit, then finds our hold instead of competing for the seats. const string InsertHold = """ INSERT INTO holds (hold_id, event_id, user_id, idem_key, expires_at) VALUES (@p0, @p1, @p2, @p3, now() + interval '10 minutes') ON CONFLICT (user_id, idem_key) DO NOTHING """; const string FindHold = "SELECT hold_id FROM holds WHERE user_id = @p0 AND idem_key = @p1";
// Free, or held by someone whose time ran out: an expired hold is free the moment it // expires, with no sweeper involved. now() is the database clock, so app servers whose // clocks drift can never disagree about when a hold ends. const string TakeSeat = """ UPDATE seats SET status = 'HELD', hold_id = @p0, held_until = now() + interval '10 minutes' WHERE event_id = @p1 AND seat_id = @p2 AND (status = 'AVAILABLE' OR (status = 'HELD' AND held_until < now())) """;
public Hold HoldSeats(long eventId, long userId, string idemKey, IEnumerable<string> seatIds) { // Everyone locks seats in the same order, so two overlapping requests queue behind // each other instead of deadlocking. var seats = seatIds.Order(StringComparer.Ordinal).ToList(); var holdId = Guid.NewGuid(); return db.InTransaction(tx => { if (tx.Execute(InsertHold, holdId, eventId, userId, idemKey) == 0) return new Hold(tx.QueryGuid(FindHold, userId, idemKey)!.Value, seats); // a retry foreach (var seat in seats) { // Under READ COMMITTED a second writer on this row waits for the first to // commit, then re-checks the WHERE clause against the new row. The loser // changes 0 rows. No SELECT first: reading, then writing, is the race. if (tx.Execute(TakeSeat, holdId, eventId, seat) == 0) throw new SeatsTakenException(seat); // rollback frees the seats taken so far } return new Hold(holdId, seats); }); }}// Holding seats: one transaction, one conditional update per seat, all or nothing.
// Tx is one open database transaction.type Tx interface { Exec(sql string, args ...any) (int64, error) // returns the number of rows changed QueryUUID(sql string, args ...any) (string, error)}
// DB commits when fn returns nil and rolls back when it returns an error.type DB interface { InTx(fn func(Tx) error) error}
type Hold struct { HoldID string Seats []string}
// SeatsTakenError names the first seat someone else got to.type SeatsTakenError struct{ Seat string }
func (e *SeatsTakenError) Error() string { return "seat " + e.Seat + " is taken" }
// The idempotency row goes in first. A concurrent retry with the same key blocks on the// unique index until we commit, then finds our hold instead of competing for the seats.const insertHold = ` INSERT INTO holds (hold_id, event_id, user_id, idem_key, expires_at) VALUES ($1, $2, $3, $4, now() + interval '10 minutes') ON CONFLICT (user_id, idem_key) DO NOTHING`const findHold = `SELECT hold_id FROM holds WHERE user_id = $1 AND idem_key = $2`
// Free, or held by someone whose time ran out: an expired hold is free the moment it// expires, with no sweeper involved. now() is the database clock, so app servers whose// clocks drift can never disagree about when a hold ends.const takeSeat = ` UPDATE seats SET status = 'HELD', hold_id = $1, held_until = now() + interval '10 minutes' WHERE event_id = $2 AND seat_id = $3 AND (status = 'AVAILABLE' OR (status = 'HELD' AND held_until < now()))`
type HoldService struct { db DB newID func() string // a UUID generator, injected}
func (s *HoldService) Hold(eventID, userID int64, idemKey string, seatIDs []string) (Hold, error) { // Everyone locks seats in the same order, so two overlapping requests queue behind // each other instead of deadlocking. seats := slices.Sorted(slices.Values(seatIDs)) hold := Hold{HoldID: s.newID(), Seats: seats} err := s.db.InTx(func(tx Tx) error { n, err := tx.Exec(insertHold, hold.HoldID, eventID, userID, idemKey) if err != nil { return err } if n == 0 { // a retry: return the first hold hold.HoldID, err = tx.QueryUUID(findHold, userID, idemKey) return err } for _, seat := range seats { // Under READ COMMITTED a second writer on this row waits for the first to // commit, then re-checks the WHERE clause against the new row. The loser // changes 0 rows. No SELECT first: reading, then writing, is the race. n, err := tx.Exec(takeSeat, hold.HoldID, eventID, seat) if err != nil { return err } if n == 0 { return &SeatsTakenError{Seat: seat} // rollback frees the seats taken so far } } return nil }) return hold, err}// Holding seats: one transaction, one conditional update per seat, all or nothing.interface Tx { execute(sql: string, ...args: unknown[]): Promise<number>; // rows changed queryUuid(sql: string, ...args: unknown[]): Promise<string | null>;}
interface Db { inTransaction<T>(work: (tx: Tx) => Promise<T>): Promise<T>; // commits, or rolls back on throw}
interface Hold { holdId: string; seats: string[];}
class SeatsTakenError extends Error { constructor(readonly seat: string) { super(`seat ${seat} is taken`); }}
// The idempotency row goes in first. A concurrent retry with the same key blocks on the// unique index until we commit, then finds our hold instead of competing for the seats.const INSERT_HOLD = ` INSERT INTO holds (hold_id, event_id, user_id, idem_key, expires_at) VALUES ($1, $2, $3, $4, now() + interval '10 minutes') ON CONFLICT (user_id, idem_key) DO NOTHING`;const FIND_HOLD = `SELECT hold_id FROM holds WHERE user_id = $1 AND idem_key = $2`;
// Free, or held by someone whose time ran out: an expired hold is free the moment it// expires, with no sweeper involved. now() is the database clock, so app servers whose// clocks drift can never disagree about when a hold ends.const TAKE_SEAT = ` UPDATE seats SET status = 'HELD', hold_id = $1, held_until = now() + interval '10 minutes' WHERE event_id = $2 AND seat_id = $3 AND (status = 'AVAILABLE' OR (status = 'HELD' AND held_until < now()))`;
class HoldService { constructor(private readonly db: Db) {}
hold(eventId: number, userId: number, idemKey: string, seatIds: string[]): Promise<Hold> { // Everyone locks seats in the same order, so two overlapping requests queue behind // each other instead of deadlocking. const seats = [...seatIds].sort(); const holdId = crypto.randomUUID(); return this.db.inTransaction(async (tx) => { if ((await tx.execute(INSERT_HOLD, holdId, eventId, userId, idemKey)) === 0) { return { holdId: (await tx.queryUuid(FIND_HOLD, userId, idemKey))!, seats }; // a retry } for (const seat of seats) { // Under READ COMMITTED a second writer on this row waits for the first to commit, // then re-checks the WHERE clause against the new row. The loser changes 0 rows. // No SELECT first: reading, then writing, is the race. if ((await tx.execute(TAKE_SEAT, holdId, eventId, seat)) === 0) { throw new SeatsTakenError(seat); // rollback frees the seats taken so far } } return { holdId, seats }; }); }}The code sorts seat IDs before updating. Two holds on overlapping seats, in different orders, would each wait for the row the other locked. Postgres detects that deadlock after a second and aborts one. Sorting makes everyone lock in the same order, so they queue instead.
Go deeperHolds in Redis, and isolation levels1 min
| Hold stored in | Expiry | What it risks |
|---|---|---|
| The seat row | Lazy: expired is free to the next update | A durable write per hold; about 3,000 a second on hot rows |
| A Redis key per seat, with an expiry | Redis deletes the key | A failover can lose a hold, so two fans hold one seat |
Redis isn’t wrong. It moves the risk. Redis copies writes to replicas after answering, so a failover
can lose a hold. That’s acceptable only because the sale is still decided in the database. So one
of the two fans gets SEATS_GONE at checkout. That’s a bad experience, but never a double sale.
Pick it when one event needs more holds than one primary can take.
“Only SERIALIZABLE isolation prevents double booking” is a common claim, and wrong. The bug is reading and then writing in two statements, not the isolation level. One conditional update is safe under Postgres’s default. SERIALIZABLE also stops the two-statement bug, but under contention it aborts transactions, which brings back retries.
The card is approved, but the hold expired
Section titled “The card is approved, but the hold expired”A fan’s card is approved, but their 10-minute hold expired a second earlier, and someone else took one of the seats. What happens to the money?
Three mechanisms together make this safe.
Shrink the window. When checkout starts, the hold is extended by 5 minutes, only if it’s still ours and live. Most fans never get near the edge.
Fence the sale with the hold ID, not the clock. The final update sells seats where the hold ID is ours and the status is held, with no time check. If the hold ran out but nobody took the seats, we still sell them. If someone took a seat, their hold replaced ours. Then the update matches fewer rows than the order has seats, and the transaction rolls back.
Authorize, then capture. A failed sale is undone by voiding the authorization: no money moved. Capture happens only after the sale commits, from the outbox.
- Color key
- Service
- Database
// Turning a hold into an order. The card is authorized first and captured only after the// seats are ours in the database, so "paid, but no seats" is undone by a void, not a refund.interface Tx { int execute(String sql, Object... args); // returns the number of rows changed}
interface Db { boolean commitIf(Predicate<Tx> work); // commits on true, rolls back on false}
interface Payments { // The order id doubles as the provider's idempotency key: a retried call after a // timeout returns the first authorization instead of making a second one. Auth authorize(String orderId, long amountCents, String paymentToken); void voidAuth(String orderId, String authId);}
record Auth(boolean approved, String authId) {}
record Order(String orderId, long eventId, UUID holdId, int seatCount, long amountCents) {}
enum Outcome { CONFIRMED, DECLINED, SEATS_GONE, IN_PROGRESS }
class Checkout { // Only one request per order gets past this, so a double submit cannot authorize twice // and then void the authorization the other request just used. static final String CLAIM = "UPDATE orders SET status = 'PAYING' WHERE order_id = ? AND status = 'PENDING'";
// Payment grace: five more minutes so the hold cannot run out while the card network // thinks. Only a hold that is still ours and still live; an expired one is not revived. static final String EXTEND = """ UPDATE seats SET held_until = held_until + interval '5 minutes' WHERE event_id = ? AND hold_id = ? AND status = 'HELD' AND held_until > now()""";
// No held_until check on purpose. If the hold ran out but nobody took the seats, they are // still ours. If someone did, their hold replaced our hold_id and this matches fewer rows. static final String SELL = """ UPDATE seats SET status = 'SOLD', order_id = ? WHERE event_id = ? AND hold_id = ? AND status = 'HELD'""";
// The unique index on tickets (event_id, seat_id) is the last line of defense: even a bug // above cannot issue two tickets for one seat. static final String ISSUE = """ INSERT INTO tickets (event_id, seat_id, order_id) SELECT event_id, seat_id, order_id FROM seats WHERE event_id = ? AND order_id = ?""";
private final Db db; private final Payments payments;
Checkout(Db db, Payments payments) { this.db = db; this.payments = payments; }
Outcome pay(Order order, String paymentToken) { boolean claimed = db.commitIf(tx -> { if (tx.execute(CLAIM, order.orderId()) != 1) return false; tx.execute(EXTEND, order.eventId(), order.holdId()); return true; }); if (!claimed) return Outcome.IN_PROGRESS; // the client polls GET /orders/{id}
// No transaction is open during the slow call: a second of card network time must // never hold row locks on seats that thousands of people are clicking. Auth auth = payments.authorize(order.orderId(), order.amountCents(), paymentToken); if (!auth.approved()) { db.commitIf(tx -> tx.execute( "UPDATE orders SET status = 'DECLINED' WHERE order_id = ?", order.orderId()) == 1); return Outcome.DECLINED; // the hold is untouched: a new order can try another card } // If this process dies here, a sweeper finds PAYING orders older than a few minutes, // asks the provider about the order id, and finishes or voids them.
boolean sold = db.commitIf(tx -> { if (tx.execute(SELL, order.orderId(), order.eventId(), order.holdId()) != order.seatCount()) return false; // some seats went to someone else: sell none of them tx.execute(ISSUE, order.eventId(), order.orderId()); tx.execute("UPDATE orders SET status = 'CONFIRMED', auth_id = ? WHERE order_id = ?", auth.authId(), order.orderId()); // Capture, ticket rendering and email happen from this outbox row, retried until // done. It commits with the sale, so the sale and its follow-up cannot diverge. tx.execute("INSERT INTO outbox (topic, msg_key) VALUES ('order-confirmed', ?)", order.orderId()); return true; }); if (sold) return Outcome.CONFIRMED;
payments.voidAuth(order.orderId(), auth.authId()); // the hold expired and was lost: no money moves db.commitIf(tx -> tx.execute( "UPDATE orders SET status = 'SEATS_GONE' WHERE order_id = ?", order.orderId()) == 1); return Outcome.SEATS_GONE; }}# Turning a hold into an order. The card is authorized first and captured only after the# seats are ours in the database, so "paid, but no seats" is undone by a void, not a refund.class Tx(Protocol): def execute(self, sql: str, *args) -> int: ... # returns the number of rows changed
class Db(Protocol): def commit_if(self, work: Callable[[Tx], bool]) -> bool: ... # commits on True
class Payments(Protocol): # The order id doubles as the provider's idempotency key: a retried call after a # timeout returns the first authorization instead of making a second one. def authorize(self, order_id: str, amount_cents: int, payment_token: str) -> "Auth": ... def void_auth(self, order_id: str, auth_id: str) -> None: ...
@dataclass(frozen=True)class Auth: approved: bool auth_id: str
@dataclass(frozen=True)class Order: order_id: str event_id: int hold_id: UUID seat_count: int amount_cents: int
class Outcome(Enum): CONFIRMED = "confirmed" DECLINED = "declined" SEATS_GONE = "seats_gone" IN_PROGRESS = "in_progress"
# Only one request per order gets past this, so a double submit cannot authorize twice# and then void the authorization the other request just used.CLAIM = "UPDATE orders SET status = 'PAYING' WHERE order_id = %s AND status = 'PENDING'"
# Payment grace: five more minutes so the hold cannot run out while the card network# thinks. Only a hold that is still ours and still live; an expired one is not revived.EXTEND = """ UPDATE seats SET held_until = held_until + interval '5 minutes' WHERE event_id = %s AND hold_id = %s AND status = 'HELD' AND held_until > now()"""
# No held_until check on purpose. If the hold ran out but nobody took the seats, they are# still ours. If someone did, their hold replaced our hold_id and this matches fewer rows.SELL = """ UPDATE seats SET status = 'SOLD', order_id = %s WHERE event_id = %s AND hold_id = %s AND status = 'HELD'"""
# The unique index on tickets (event_id, seat_id) is the last line of defense: even a bug# above cannot issue two tickets for one seat.ISSUE = """ INSERT INTO tickets (event_id, seat_id, order_id) SELECT event_id, seat_id, order_id FROM seats WHERE event_id = %s AND order_id = %s"""
class Checkout: def __init__(self, db: Db, payments: Payments): self._db = db self._payments = payments
def _set_status(self, order_id: str, status: str) -> None: self._db.commit_if( lambda tx: tx.execute("UPDATE orders SET status = %s WHERE order_id = %s", status, order_id) == 1 )
def pay(self, order: Order, payment_token: str) -> Outcome: def claim(tx: Tx) -> bool: if tx.execute(CLAIM, order.order_id) != 1: return False tx.execute(EXTEND, order.event_id, order.hold_id) return True
if not self._db.commit_if(claim): return Outcome.IN_PROGRESS # the client polls GET /orders/{id}
# No transaction is open during the slow call: a second of card network time must # never hold row locks on seats that thousands of people are clicking. auth = self._payments.authorize(order.order_id, order.amount_cents, payment_token) if not auth.approved: self._set_status(order.order_id, "DECLINED") return Outcome.DECLINED # the hold is untouched: a new order can try another card # If this process dies here, a sweeper finds PAYING orders older than a few minutes, # asks the provider about the order id, and finishes or voids them.
def sell(tx: Tx) -> bool: if tx.execute(SELL, order.order_id, order.event_id, order.hold_id) != order.seat_count: return False # some seats went to someone else: sell none of them tx.execute(ISSUE, order.event_id, order.order_id) tx.execute( "UPDATE orders SET status = 'CONFIRMED', auth_id = %s WHERE order_id = %s", auth.auth_id, order.order_id, ) # Capture, ticket rendering and email happen from this outbox row, retried until # done. It commits with the sale, so the sale and its follow-up cannot diverge. tx.execute("INSERT INTO outbox (topic, msg_key) VALUES ('order-confirmed', %s)", order.order_id) return True
if self._db.commit_if(sell): return Outcome.CONFIRMED
self._payments.void_auth(order.order_id, auth.auth_id) # hold expired and lost: no money moves self._set_status(order.order_id, "SEATS_GONE") return Outcome.SEATS_GONE// Turning a hold into an order. The card is authorized first and captured only after the// seats are ours in the database, so "paid, but no seats" is undone by a void, not a refund.interface ITx{ int Execute(string sql, params object[] args); // returns the number of rows changed}
interface IDb{ bool CommitIf(Func<ITx, bool> work); // commits on true, rolls back on false or throw}
interface IPayments{ // The order id doubles as the provider's idempotency key: a retried call after a // timeout returns the first authorization instead of making a second one. Auth Authorize(string orderId, long amountCents, string paymentToken); void VoidAuth(string orderId, string authId);}
record Auth(bool Approved, string AuthId);
record Order(string OrderId, long EventId, Guid HoldId, int SeatCount, long AmountCents);
enum Outcome { Confirmed, Declined, SeatsGone, InProgress }
class Checkout(IDb db, IPayments payments){ // Only one request per order gets past this, so a double submit cannot authorize twice // and then void the authorization the other request just used. const string Claim = "UPDATE orders SET status = 'PAYING' WHERE order_id = @p0 AND status = 'PENDING'";
// Payment grace: five more minutes so the hold cannot run out while the card network // thinks. Only a hold that is still ours and still live; an expired one is not revived. const string Extend = """ UPDATE seats SET held_until = held_until + interval '5 minutes' WHERE event_id = @p0 AND hold_id = @p1 AND status = 'HELD' AND held_until > now() """;
// No held_until check on purpose. If the hold ran out but nobody took the seats, they are // still ours. If someone did, their hold replaced our hold_id and this matches fewer rows. const string Sell = """ UPDATE seats SET status = 'SOLD', order_id = @p0 WHERE event_id = @p1 AND hold_id = @p2 AND status = 'HELD' """;
// The unique index on tickets (event_id, seat_id) is the last line of defense: even a bug // above cannot issue two tickets for one seat. const string Issue = """ INSERT INTO tickets (event_id, seat_id, order_id) SELECT event_id, seat_id, order_id FROM seats WHERE event_id = @p0 AND order_id = @p1 """;
void SetStatus(string orderId, string status) => db.CommitIf(tx => tx.Execute( "UPDATE orders SET status = @p0 WHERE order_id = @p1", status, orderId) == 1);
public Outcome Pay(Order order, string paymentToken) { var claimed = db.CommitIf(tx => { if (tx.Execute(Claim, order.OrderId) != 1) return false; tx.Execute(Extend, order.EventId, order.HoldId); return true; }); if (!claimed) return Outcome.InProgress; // the client polls GET /orders/{id}
// No transaction is open during the slow call: a second of card network time must // never hold row locks on seats that thousands of people are clicking. var auth = payments.Authorize(order.OrderId, order.AmountCents, paymentToken); if (!auth.Approved) { SetStatus(order.OrderId, "DECLINED"); return Outcome.Declined; // the hold is untouched: a new order can try another card } // If this process dies here, a sweeper finds PAYING orders older than a few minutes, // asks the provider about the order id, and finishes or voids them.
var sold = db.CommitIf(tx => { if (tx.Execute(Sell, order.OrderId, order.EventId, order.HoldId) != order.SeatCount) return false; // some seats went to someone else: sell none of them tx.Execute(Issue, order.EventId, order.OrderId); tx.Execute("UPDATE orders SET status = 'CONFIRMED', auth_id = @p0 WHERE order_id = @p1", auth.AuthId, order.OrderId); // Capture, ticket rendering and email happen from this outbox row, retried until // done. It commits with the sale, so the sale and its follow-up cannot diverge. tx.Execute("INSERT INTO outbox (topic, msg_key) VALUES ('order-confirmed', @p0)", order.OrderId); return true; }); if (sold) return Outcome.Confirmed;
payments.VoidAuth(order.OrderId, auth.AuthId); // the hold expired and was lost: no money moves SetStatus(order.OrderId, "SEATS_GONE"); return Outcome.SeatsGone; }}// Turning a hold into an order. The card is authorized first and captured only after the// seats are ours in the database, so "paid, but no seats" is undone by a void, not a refund.
// Tx is one open database transaction.type Tx interface { Exec(sql string, args ...any) (int64, error) // returns the number of rows changed}
// DB commits when fn returns true and a nil error, and rolls back otherwise.type DB interface { CommitIf(fn func(Tx) (bool, error)) (bool, error)}
// Payments is the card provider. The order id doubles as its idempotency key: a retried// call after a timeout returns the first authorization instead of making a second one.type Payments interface { Authorize(orderID string, amountCents int64, paymentToken string) (Auth, error) Void(orderID, authID string) error}
type Auth struct { Approved bool AuthID string}
type Order struct { OrderID string EventID int64 HoldID string SeatCount int64 AmountCents int64}
type Outcome int
const ( Confirmed Outcome = iota Declined SeatsGone InProgress)
// Only one request per order gets past this, so a double submit cannot authorize twice// and then void the authorization the other request just used.const claim = `UPDATE orders SET status = 'PAYING' WHERE order_id = $1 AND status = 'PENDING'`
// Payment grace: five more minutes so the hold cannot run out while the card network// thinks. Only a hold that is still ours and still live; an expired one is not revived.const extend = ` UPDATE seats SET held_until = held_until + interval '5 minutes' WHERE event_id = $1 AND hold_id = $2 AND status = 'HELD' AND held_until > now()`
// No held_until check on purpose. If the hold ran out but nobody took the seats, they are// still ours. If someone did, their hold replaced our hold_id and this matches fewer rows.const sell = ` UPDATE seats SET status = 'SOLD', order_id = $1 WHERE event_id = $2 AND hold_id = $3 AND status = 'HELD'`
// The unique index on tickets (event_id, seat_id) is the last line of defense: even a bug// above cannot issue two tickets for one seat.const issue = ` INSERT INTO tickets (event_id, seat_id, order_id) SELECT event_id, seat_id, order_id FROM seats WHERE event_id = $1 AND order_id = $2`
type Checkout struct { db DB payments Payments}
func (c *Checkout) setStatus(orderID, status string) error { _, err := c.db.CommitIf(func(tx Tx) (bool, error) { n, err := tx.Exec(`UPDATE orders SET status = $1 WHERE order_id = $2`, status, orderID) return n == 1, err }) return err}
func (c *Checkout) Pay(o Order, paymentToken string) (Outcome, error) { claimed, err := c.db.CommitIf(func(tx Tx) (bool, error) { n, err := tx.Exec(claim, o.OrderID) if err != nil || n != 1 { return false, err } _, err = tx.Exec(extend, o.EventID, o.HoldID) return err == nil, err }) if err != nil || !claimed { return InProgress, err // the client polls GET /orders/{id} }
// No transaction is open during the slow call: a second of card network time must // never hold row locks on seats that thousands of people are clicking. auth, err := c.payments.Authorize(o.OrderID, o.AmountCents, paymentToken) if err != nil { return InProgress, err // the sweeper resolves PAYING orders with the provider } if !auth.Approved { return Declined, c.setStatus(o.OrderID, "DECLINED") // the hold is untouched: a new order can try another card } // If this process dies here, a sweeper finds PAYING orders older than a few minutes, // asks the provider about the order id, and finishes or voids them.
sold, err := c.db.CommitIf(func(tx Tx) (bool, error) { n, err := tx.Exec(sell, o.OrderID, o.EventID, o.HoldID) if err != nil || n != o.SeatCount { return false, err // some seats went to someone else: sell none of them } if _, err := tx.Exec(issue, o.EventID, o.OrderID); err != nil { return false, err } if _, err := tx.Exec(`UPDATE orders SET status = 'CONFIRMED', auth_id = $1 WHERE order_id = $2`, auth.AuthID, o.OrderID); err != nil { return false, err } // Capture, ticket rendering and email happen from this outbox row, retried until // done. It commits with the sale, so the sale and its follow-up cannot diverge. _, err = tx.Exec(`INSERT INTO outbox (topic, msg_key) VALUES ('order-confirmed', $1)`, o.OrderID) return err == nil, err }) if err != nil { return InProgress, err } if sold { return Confirmed, nil }
if err := c.payments.Void(o.OrderID, auth.AuthID); err != nil { // hold expired and lost: no money moves return InProgress, err } return SeatsGone, c.setStatus(o.OrderID, "SEATS_GONE")}// Turning a hold into an order. The card is authorized first and captured only after the// seats are ours in the database, so "paid, but no seats" is undone by a void, not a refund.interface Tx { execute(sql: string, ...args: unknown[]): Promise<number>; // rows changed}
interface Db { commitIf(work: (tx: Tx) => Promise<boolean>): Promise<boolean>; // commits on true}
interface Payments { // The order id doubles as the provider's idempotency key: a retried call after a // timeout returns the first authorization instead of making a second one. authorize(orderId: string, amountCents: number, paymentToken: string): Promise<Auth>; voidAuth(orderId: string, authId: string): Promise<void>;}
interface Auth { approved: boolean; authId: string;}
interface Order { orderId: string; eventId: number; holdId: string; seatCount: number; amountCents: number;}
type Outcome = 'CONFIRMED' | 'DECLINED' | 'SEATS_GONE' | 'IN_PROGRESS';
// Only one request per order gets past this, so a double submit cannot authorize twice// and then void the authorization the other request just used.const CLAIM = `UPDATE orders SET status = 'PAYING' WHERE order_id = $1 AND status = 'PENDING'`;
// Payment grace: five more minutes so the hold cannot run out while the card network// thinks. Only a hold that is still ours and still live; an expired one is not revived.const EXTEND = ` UPDATE seats SET held_until = held_until + interval '5 minutes' WHERE event_id = $1 AND hold_id = $2 AND status = 'HELD' AND held_until > now()`;
// No held_until check on purpose. If the hold ran out but nobody took the seats, they are// still ours. If someone did, their hold replaced our hold_id and this matches fewer rows.const SELL = ` UPDATE seats SET status = 'SOLD', order_id = $1 WHERE event_id = $2 AND hold_id = $3 AND status = 'HELD'`;
// The unique index on tickets (event_id, seat_id) is the last line of defense: even a bug// above cannot issue two tickets for one seat.const ISSUE = ` INSERT INTO tickets (event_id, seat_id, order_id) SELECT event_id, seat_id, order_id FROM seats WHERE event_id = $1 AND order_id = $2`;
class Checkout { constructor( private readonly db: Db, private readonly payments: Payments, ) {}
private setStatus(orderId: string, status: string) { return this.db.commitIf( async (tx) => (await tx.execute(`UPDATE orders SET status = $1 WHERE order_id = $2`, status, orderId)) === 1, ); }
async pay(order: Order, paymentToken: string): Promise<Outcome> { const claimed = await this.db.commitIf(async (tx) => { if ((await tx.execute(CLAIM, order.orderId)) !== 1) return false; await tx.execute(EXTEND, order.eventId, order.holdId); return true; }); if (!claimed) return 'IN_PROGRESS'; // the client polls GET /orders/{id}
// No transaction is open during the slow call: a second of card network time must // never hold row locks on seats that thousands of people are clicking. const auth = await this.payments.authorize(order.orderId, order.amountCents, paymentToken); if (!auth.approved) { await this.setStatus(order.orderId, 'DECLINED'); return 'DECLINED'; // the hold is untouched: a new order can try another card } // If this process dies here, a sweeper finds PAYING orders older than a few minutes, // asks the provider about the order id, and finishes or voids them.
const sold = await this.db.commitIf(async (tx) => { if ( (await tx.execute(SELL, order.orderId, order.eventId, order.holdId)) !== order.seatCount ) { return false; // some seats went to someone else: sell none of them } await tx.execute(ISSUE, order.eventId, order.orderId); await tx.execute( `UPDATE orders SET status = 'CONFIRMED', auth_id = $1 WHERE order_id = $2`, auth.authId, order.orderId, ); // Capture, ticket rendering and email happen from this outbox row, retried until // done. It commits with the sale, so the sale and its follow-up cannot diverge. await tx.execute( `INSERT INTO outbox (topic, msg_key) VALUES ('order-confirmed', $1)`, order.orderId, ); return true; }); if (sold) return 'CONFIRMED';
await this.payments.voidAuth(order.orderId, auth.authId); // the hold expired and was lost: no money moves await this.setStatus(order.orderId, 'SEATS_GONE'); return 'SEATS_GONE'; }}Idempotency at three levels. The client’s key makes a retried checkout return the stored order. Moving the order from pending to paying, conditionally, lets only one of two simultaneous submits run. And the order ID is the provider’s idempotency key, so our retry of a timed-out authorization returns the original result.
Go deeperEvery checkout outcome1 min
| Situation | Outcome |
|---|---|
| Card approved, hold valid | Sold; money captured from the outbox |
| Card approved, hold expired, seats untouched | Sold anyway, unless the sweeper already released them |
| Card approved, hold expired, a seat taken | Roll back, void the authorization, SEATS_GONE |
| Card declined | Order declined; the hold stays, and the fan tries another card |
| Server crash after authorization | Order stuck paying; reconciliation asks the provider, then sells or voids |
| Capture fails after the sale (rare) | Retry; authorizations last days. If it finally fails, release and cancel |
“Exactly-once payment” is a phrase to avoid. Networks only deliver at most once or at least once. What the system provides is an exactly-once effect: every call may repeat, and idempotency keys at each step make the repeat harmless.
Half the traffic on a big on-sale is bots. How do we keep the seats for people?
Bots win by being faster, more numerous and more persistent than people. Each layer removes one of those advantages:
| Layer | What it does | Removes |
|---|---|---|
| Registration lottery | Fans register days ahead; accounts are scored | Numbers, before sale day |
| Verified accounts | One place per account, tied to a phone and a card | A thousand tabs get one place |
| Shuffled early arrivals | Random places for everyone before opening | Speed at 10:00:00.000 |
| Bot checks at the CDN | IP reputation, device signals, challenges | Volume, before it reaches us |
| Bound tokens and limits | Tokens tied to account and event; limit in the hold | Skipping the line, bulk buying |
The scale is real: Ticketmaster said the 2022 Eras Tour presale drew 3.5 billion requests, four times its previous peak, mostly from bots.
Many big on-sales at once
Section titled “Many big on-sales at once”Next week, ten stadium shows go on sale the same morning. That means about 5,000 hold writes a second on one booking database, and 250,000 joins a second on one queue store. So each on-sale gets its own resources. Booking data is split by event, so no transaction ever spans shards. Events are assigned to shards by hand, so a big on-sale gets a shard of its own the day before. Each on-sale gets its own queue store, and the CDN serves the status file and the waiting page.
Stage 6: Shard by event, line at the edge
Stage 6 of 6
Shard by event, line at the edge
Why this stage
Ten stadium shows go on sale the same morning: 5,000 hold writes/s on one primary and 250,000 queue joins/s on one queue store.
What we add
Booking data sharded by event, one queue store per on-sale, and the line’s status and bot checks served from the CDN.
What it costs
Cross-event queries (a fan’s orders) need their own table, and one event can never use more than one shard.
At the 20M mark, forty on-sales at once, the search cluster needs more replicas and payments need a second provider.
At 500k requests per second at the front door, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One booking server handles about 5,000 requests a second.
- One Postgres primary does about 10,000 point reads a second, but only about 3,000 hold transactions when they all hit one event's rows.
- One Redis node does about 100,000 to 150,000 simple operations a second.
- One waiting room server handles about 20,000 requests a second.
Outside limits
- We negotiated a payment provider limit of about 2,000 calls a second.
What the waiting room saves
- Writes are 5% of requests.
- Behind the waiting room, only about 1% of requests and 2% of hold attempts reach booking.
- At the edge, the CDN answers 90% of the line's status checks.
Each event also has a home region, where its shard, line and payments run. Browsing runs in every region. Two regions never sell the same event, because that would need agreement across regions on every hold. A distant fan pays 100 ms more per booking call, unnoticed in a 4-minute visit.
Follow a fan through the line:
Trace a request: Join the line
Join the line: step 1 of 7
Fans to CDN edge
The waiting room page is static and comes from the edge. Bot checks run here: IP reputation, a challenge for suspicious clients.
- Fans → CDN edge15 ms
- CDN edge → API gateway10 ms
- API gateway → Waiting room1 ms
- Waiting room → Queue store1 ms
- Fans → CDN edgeasync
- API gateway → Waiting roomasync
- Response
Go deeperOther split keys1 min
| Split by | One transaction per hold | Isolates on-sales | Problem |
|---|---|---|---|
| Event | Yes | Yes | One event is capped at one primary; the line keeps it under |
| Event and section | Yes, if holds stay in one section | Yes | The map and per-account limits span sections |
| Seat hash | No: a 4-seat hold spans shards | No | Every hold is a distributed transaction |
| User | No | No | The same seat can be written from any shard |
“All my orders” now spans shards, so it moves to a table keyed by user, fed from the outbox. One event that outgrew a primary would split by section, with holds kept within one section.
When things break
Section titled “When things break”The booking primary for a hot event dies in the middle of its on-sale. What happens?
Whether paid orders survive, nobody loses their place, and holds are not silently lost.
The goal is that a failure pauses an on-sale rather than corrupting it. The primary has a standby in another zone that confirms every commit before the primary answers, so no confirmed order is lost. The standby takes over in about 30 seconds. Meanwhile, the pacer pauses admission, and every live hold is extended by the length of the outage. The line is the shock absorber: the people inside keep their seats, and nobody loses their place.
Click any box to break it and see what fans would notice:
Break it: what fails, who notices, what users see
Failure: click any box to break it
Booking shard primary goes down
What breaks
Holds, payments and confirmations for every event on that shard fail.
How we notice
Write error rate and failed health checks on the primary.
What users see
For about 30 seconds, seat clicks fail with a retry message.
Paid orders are not lost: the standby has every commit.
What we do
Promote the synchronous standby.
The waiting room pauses admission for the shard’s events so people do not burn their 15 minutes.
Holds keep their expiry, extended by the outage.
The waiting room must fail closed. If it fails open, two million people reach booking at once, which is exactly the collapse it exists to prevent. And if its Redis loses the last second of joins, fans still hold signed tickets. Admission works from the ticket alone, so they keep their place.
Go deeperEvery failure, and what users see1 min
| Part | Failure | What users see | What we do |
|---|---|---|---|
| Booking primary | Down | Holds and checkout fail for about 30 s on that shard | Promote the standby; pause admission; extend holds |
| Seat map cache | Down | Maps freeze, then show section counts only | Replica takes over; rebuild hot sections from the database |
| Waiting room | Down | New arrivals see a holding page; shoppers inside fine | Stateless restart; the gateway fails closed |
| Queue store | Down | Joining pauses; places kept in signed tickets | Restart the counter above the highest ticket seen |
| Payment provider | Slow or down | Checkout slow or failing; holds were extended | Same-key retries; a second provider; pause admission |
| Kafka or workers | Down | Orders confirm; emails and captures arrive late | The outbox waits in Postgres; workers catch up |
| Search index | Down | Search fails; event pages from the CDN still work | Rebuild from change data capture |
| Home region | Outage | Its on-sales pause; the line keeps its order | Promote the standby region; reconcile the last seconds |
Under stress, keep checkout working first, then holds, then the live seat map (fall back to section counts and server-picked seats), then search freshness.
Go deeperRunning it: alerts and on-sale readiness1 min
Wake someone up when:
- hold success drops below 99.9%, counting “taken” as success (that’s the system working)
- checkout passes 3 seconds at p99, or authorization errors pass 2%
- admission runs under half its target while seats remain
- any ticket insert is rejected by its primary key (always a bug above the database)
- orders sit in “paying” for over 10 minutes
Before a big on-sale, treat it as a launch. Move the event to its own shard the day before. Warm up servers and the CDN. Load test the line at the expected crowd. Freeze deploys for the hour. And agree on the admission rate, and when to pause.
Proving no double sale. The ticket primary key makes it impossible in the database. A nightly job also checks that every captured payment has a confirmed order, and every order has tickets for exactly its seats. Anything odd goes to a person.
Wrap up
Section titled “Wrap up”The whole design
Section titled “The whole design”Here is everything we built, in the order we built it. Use the buttons to step through it.
The full design, stage by stage
Stage 6 of 6
Shard by event, line at the edge
Why this stage
Ten stadium shows go on sale the same morning: 5,000 hold writes/s on one primary and 250,000 queue joins/s on one queue store.
What we add
Booking data sharded by event, one queue store per on-sale, and the line’s status and bot checks served from the CDN.
What it costs
Cross-event queries (a fan’s orders) need their own table, and one event can never use more than one shard.
At the 20M mark, forty on-sales at once, the search cluster needs more replicas and payments need a second provider.
At 500k requests per second at the front door, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One booking server handles about 5,000 requests a second.
- One Postgres primary does about 10,000 point reads a second, but only about 3,000 hold transactions when they all hit one event's rows.
- One Redis node does about 100,000 to 150,000 simple operations a second.
- One waiting room server handles about 20,000 requests a second.
Outside limits
- We negotiated a payment provider limit of about 2,000 calls a second.
What the waiting room saves
- Writes are 5% of requests.
- Behind the waiting room, only about 1% of requests and 2% of hold attempts reach booking.
- At the edge, the CDN answers 90% of the line's status checks.
The decisions, and what each one costs
Section titled “The decisions, and what each one costs”| Decision | We chose | We gave up | We’d choose differently if |
|---|---|---|---|
| Concurrency | Conditional update per seat, in sorted order | Row locking’s simpler reasoning | Contention were low everywhere |
| Hold storage | The seat row, with lazy expiry | Redis’s speed for holds | One event needed more than one primary’s holds |
| Seat map | Redis per section, up to 2 seconds stale | A perfectly fresh map | Venues were small enough to read the database |
| The spike | A waiting room paced by seats left | Instant access for everyone | Demand never exceeded supply |
| Payments | Authorize, sell, then capture, all safe to repeat | One-step charging | The provider lacked authorize and capture |
| Split | By event, one home region per event | Queries across events; writes everywhere | One event outgrew a primary (event and section) |
How to spend 45 minutes
Section titled “How to spend 45 minutes”- 0 to 4. Assigned seats, the biggest on-sale, hold time, payments. “Never sold twice” is the hard promise.
- 4 to 9. 2 million people, 50,000 seats, 25,000 holds a second unprotected, 100 behind a line.
- 9 to 20. Build it: browse apart, the cached map, the conditional update, checkout outside the transaction. A mid-level answer can stop here.
- 20 to 26. The waiting room.
- 26 to 40. The stress tests: locking with the contention math, the late payment, then bots and the split by event.
- 40 to 45. A primary failing mid-sale, then your own questions.
Never skip the conditional update, the payment after expiry, or the waiting room.
Follow-up questions
Section titled “Follow-up questions”Interviewer asksHow would general admission (standing) tickets change the design?
Inventory becomes a counter per section, and a hold becomes a conditional decrement: “subtract 2 where at least 2 remain”. One hot row instead of thousands makes contention worse. So split the counter into, say, 16 rows of 1,000 tickets. Pick one at random, and fall back to the others when it’s empty.
Interviewer asksCould the whole on-sale run in memory, without a database?
Yes: one process per event owns its seats in memory, and writes every change to a durable log before answering, like a trading engine. It handles hundreds of thousands of holds a second. The cost is failover: a standby replays the log while the event pauses. Behind a waiting room, one Postgres primary is enough, so the simpler design wins.
Go deeperMore follow-up questions1 min
Why 10 minutes for a hold? How would you choose? From data: the 95th percentile checkout time, plus a margin. Too short fails people typing card numbers. Too long keeps seats out of sale during the most valuable minutes. The payment extension only applies to people who reach checkout.
The payment provider has a regional outage during an on-sale. What do you do? Send new orders to a second provider, if one is set up. Pause admission, so people already inside can finish. And extend holds for orders stuck in paying. Reconciliation settles every order once the provider is back, keyed by order ID.
What changes at 100 times this load? Little in booking, because each on-sale is isolated: more shards and queue stores, the same pace per event. What grows is the edge: hundreds of millions checking a status file, and bots with them. That’s CDN capacity and bot management, which is why the status is a cacheable file.
Further reading
Section titled “Further reading”- Movie ticket booking on this site: the same race at the object level.
- Rate limiting, message queues, partitioning and the payment system on this site.
- How to do distributed locking, Martin Kleppmann: why a lock with a TTL needs a fencing token, which is the role the conditional update plays here.
- Taylor Swift | The Eras Tour onsale explained, Ticketmaster: a real on-sale spike and bot surge.
- Transaction isolation, PostgreSQL
docs: how READ COMMITTED re-checks an
UPDATE’s condition after waiting for a concurrent writer.
The designbeyondcode digest, every Saturday
New deep dives, interview tips and site news in one short email. Free. Leave with one click.
We never sell or share your email. We only use it to send this newsletter.