Design a hotel reservation system
Design a hotel reservation system. Start with one service and one database, grow it with counts per night, one transaction for the whole stay, search on a copy and a hold around payment, then stress it with date ranges, late payments, agencies and changed stays, with interview prompts and code.
“Design a hotel reservation system, like the one behind Marriott.com or Booking.com. Guests search a city for some dates, pick a room and book it.”
A platform for 20,000 hotels takes only a few bookings a second, so the difficulty isn’t volume. It hides in three places. A stay is several nights that must all be taken, or none. Search runs a thousand times for every booking. And the same rooms sell on other websites at the same time, from those websites’ own copy of our counts.
So this page starts with one service and one database, and lets the numbers grow it. Then it tests the result against date ranges, late payments, agencies and changed stays.
The answer in one minute
Section titled “The answer in one minute”If you only have a minute, this is the design. The rest of the page explains each part.
- Count rooms per room type per night. A guest books “a double”, not room 101. The hotel picks the room at check-in. So inventory is a number: doubles left on the 5th.
- Book a stay in one transaction. Lock the nights in date order. Add one to each night only where a room is left. Roll back unless every night changed.
- Let search read a stale copy. Search is 1,000 times the bookings, so it never touches the inventory tables. A room shown that just sold is caught by the booking.
- Hold the room while the card is checked. The card network never runs inside a transaction. A worker gives back holds that are never paid.
- Sync the agencies, and expect them to be late. Push counts within a second. Record their bookings even when a night is full, because their guest is already confirmed. A small overbooking allowance absorbs the rare collision.
Scope it
Section titled “Scope it”A few questions decide what a booking even is.
Design a hotel reservation system.
Whether you ask whether guests book a room or a room type, how many searches per booking, and about agencies.
These four questions matter most:
- How many hotels? 20,000, about 100 rooms each: 2 million rooms.
- Does a guest book a room or a type? A room type for some nights. The room is picked at check-in.
- How many searches per booking? About 1,000, counting partner and price-comparison traffic.
- Do other channels sell the same rooms? Yes. About 40% sell through online travel agencies, from one shared pool.
From these answers we make five promises:
- Never oversold through us. Take any hotel, room type and night. Rooms sold through our own channels never pass the rooms in service plus the hotel’s overbooking allowance. This is a hard promise, under any retry or failure.
- Booked once. A retried booking never creates a second reservation or a second charge.
- Always up. Search works 99.99% of the time, booking 99.95%.
- Fast. Search under 500 ms. Booking under 2 seconds with a card, under 300 ms when paying at the hotel.
- Fresh enough. Search may be a few seconds old, a minute at worst. Agencies get changes within about 5 seconds.
We leave out four things. Assigning physical rooms (the hotel’s own system does that at check-in). Dynamic pricing (we store the prices hotels send us). Reviews and photos. And the payment provider itself.
Go deeperMore questions, and what each changes1 min
| Question | We assume | If the answer were different |
|---|---|---|
| One hotel only? | 20,000 hotels | One hotel fits one server with a lock in memory: the LLD answer |
| Pay now or at the hotel? | Both: prepaid rates, and pay-at-hotel with a card guarantee | Pay-at-hotel only removes holds and payment calls |
| Can a hotel sell more rooms than it has? | Yes, a small allowance per type and night | Without it, every agency race is an oversell handled by hand |
| Fixed allotments per agency? | No, one shared pool | Allotments remove the race but leave rooms unsold in one channel |
The numbers that decide it
Section titled “The numbers that decide it”Now we work out the numbers. Bookings and searches differ by a factor of a thousand, so we size them apart.
- Hotels on the platform
- 20,000
- Rooms per hotel
- 100
- Room types per hotel
- 5
- Occupancy
- 70%
- Nights per stay
- 3
- Bookings later cancelled
- 35%
- Searches per booking
- 1,000 to 1
- Peak over average
- 3×
- Hotels checked per search
- 100
- Rate updates per room type a day
- 20
2 million rooms at 70% full, 3 nights a stay, and 35% of bookings later cancelled. About 25 a second at peak.
So we bookings need no distributed database.
1,000 searches per booking. A city search checks about 100 hotels × 3 nights × 5 room types.
So we search never reads the inventory tables. At 1,500 rows a search, one database answers about 2,000 searches a second.
20,000 hotels × 5 room types × 365 nights. A count and a lowest rate per row is 6 bytes.
So we we split for writes, never for size. All availability fits in one server's memory.
The heavy writes aren’t bookings. Hotels’ pricing systems change rates about 20 times a day per room type, each covering 30 nights. 90% of row writes are rate and availability updates.
So we one primary carries it, if updates are batched per hotel.
Try the calculator: double the searches per booking, and the cache passes what a 20-node Redis cluster serves. Multiply the hotels by ten, and row writes reach 23,000 a second, the trigger for splitting by hotel.
- Bookings a day, cancellations included
- 720,000
- Bookings per second (average)
- 8.3/s
- A handful a second: write volume from bookings alone is never the problem.
- Searches per second at peak
- 25,000/s
- Past the 2,000 searches/s one database can answer: search reads a cache, never the inventory tables.
- Cache lookups per second at peak (one per hotel checked)
- 2,500,000/s
- A Redis cluster of about a dozen primaries serves this.
- Inventory rows (one per room type per night, 365 nights)
- 37 million
- Small: one database holds all of it, so sharding is for writes, not size.
- Availability in memory (6 bytes a row)
- 220 MB
- Fits in every search server’s memory.
- Inventory row writes per second at peak
- 2,300/s
- One primary with a synchronous standby takes this.
- Reservations stored over 5 years (1 KB each)
- 1.3 TB
- Fits one primary; archive stays more than a year old.
Build the design
Section titled “Build the design”We start with the simplest design that works. Then we let the numbers break it, one at a time. Each step fixes only what broke, adds only the boxes that fix needs, and says what the fix costs. Throughout, follow one scenario: a guest searches Paris, picks a double for three nights, and pays.
Step 1. One service, one database
Section titled “Step 1. One service, one database”The smallest version is one booking service and one Postgres database. Search queries the hotel and room tables directly, and a booking marks a room as taken for each night of the stay.
Stage 1: One service, one database
Stage 1 of 6
One service, one database
Why this stage
Nothing yet.
This is the smallest design that never sells a night twice.
What we add
A booking service and a Postgres table with one row per hotel, room type and night.
A booking is one transaction that adds to every night of the stay, or to none.
What it costs
Search scans inventory rows in the same database the bookings write to, and one server does everything.
At 25k searches per second, Inventory shards is 1250% busy. Stage 2 is the fix.
The numbers behind the busy bars
What one machine can do
- One app server handles about 2,000 to 5,000 requests a second.
- One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
- The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.
How big the shared systems are
- A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
- An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
- The payment provider allows about 1,000 calls a second.
What the writes are
- Inventory writes are 9% of the search rate.
- About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
- From stage 6, search servers cache location queries, so only 30% reach the index.
It works for a few hotels. But two things go wrong as it grows. Search runs about a thousand times for every booking, against the same tables booking writes, so search load slows booking. And we haven’t yet said what “a room is free” means for a stay of several nights.
Step 2. Count rooms per night
Section titled “Step 2. Count rooms per night”The inventory table has one row per hotel, room type and night. Each row holds the rooms in service, the overbooking allowance, and how many are reserved (held or confirmed, from every channel). Prices live in a separate table, because they change far more often, and several rate plans (non-refundable, flexible) sell the same doubles.
Why counts, not a calendar per room? Per-room calendars refuse guests while some double is free every night, just not the same double. The hotel assigns rooms at check-in, so it doesn’t need us to.
Step 3. Book a stay in one transaction
Section titled “Step 3. Book a stay in one transaction”A booking is one transaction. It inserts the reservation, then adds one to reserved on each night
of the stay, only where a room is left. If any night is full, fewer rows change than there are
nights, and the transaction rolls back everything. The check and the write are one step, so two
guests can never both take the last room.
The race this must survive is two overlapping stays competing for the last double on a shared night. Both lock their nights in date order, so they queue instead of each holding a night the other needs:
- Color key
- Service
- Database
Step 4. Search reads a copy
Section titled “Step 4. Search reads a copy”Search needs location and filter queries, and availability for many hotels at once. Neither needs to be exact, because the booking transaction checks again. So search never touches the inventory tables.
An Elasticsearch index holds the facts that rarely change: location, stars, amenities. A Redis cache holds one hash per hotel. For each room type and night, it stores the rooms left and the lowest rate. A search asks the index for the best 100 matching hotels. It reads their nights from Redis in one round trip. It drops hotels with no type free on every night, sums the rates, and returns 25. Booking servers update the hotel’s hash after every change.
Stage 2: Search apart from booking
Stage 2 of 6
Search apart from booking
Why this stage
At the 25,000 searches/s peak, each search checks about 100 hotels over 3 nights: far past the 2,000 searches/s one database can answer.
What we add
An API gateway, a fleet of search servers, a hotel index on Elasticsearch for location and filters, and an availability cache in Redis, written after every commit.
Booking servers become a stateless fleet.
What it costs
Search shows availability a few seconds old.
The booking transaction, not the cache, decides whether a room is left.
At 25k searches per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One app server handles about 2,000 to 5,000 requests a second.
- One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
- The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.
How big the shared systems are
- A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
- An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
- The payment provider allows about 1,000 calls a second.
What the writes are
- Inventory writes are 9% of the search rate.
- About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
- From stage 6, search servers cache location queries, so only 30% reach the index.
Follow one search:
Trace a request: Search Paris for 3 nights
Search Paris for 3 nights: step 1 of 5
Guests to API gateway
GET /search with a city, dates and guests. The gateway checks the session or API key and the caller’s rate limit; metasearch partners get their own quota.
- Guests → API gateway15 ms
- API gateway → Search servers1 ms
- Search servers → Hotel index25 ms
- Search servers → Availability6 ms
- Response
Go deeperKeeping the search copy in step1 min
The booking server updates the hotel’s Redis hash right after each commit. If it crashes between the commit and that update, the copy is wrong until something fixes it. So a background job rebuilds each hotel’s hash from Postgres every few minutes, and at once for any hotel that just changed.
A wrong copy can only be wrong in two ways, and they cost different things. Say it shows a room that is gone. The guest clicks, the booking transaction checks the real count, and the guest sees “sold out”. That is annoying, but nothing is sold twice. Now say it shows “sold out” for a room that is free. The hotel loses a sale, and nobody notices. That second case is why the rebuild job exists. It’s also why we watch how often searches show a hotel as full when Postgres says it isn’t.
Step 5. Pay with a hold
Section titled “Step 5. Pay with a hold”A card check takes 300 ms to a few seconds, longer with a bank’s identity challenge, so it never runs inside the inventory transaction. A prepaid booking runs in three short steps:
- Hold. The booking transaction inserts the reservation as held, with a 15-minute expiry. Held rooms count as reserved, so the room is safe while the guest pays.
- Authorize the card, with the reservation ID as the provider’s idempotency key. No transaction is open.
- Confirm, only if the reservation is still held, and write an outbox row in the same transaction. A relay sends it to Kafka, and workers take the money, send the confirmation and update the cache.
Pay-at-hotel bookings skip the card call and confirm in the first transaction.
Stage 3: Holds and payments
Stage 3 of 6
Holds and payments
Why this stage
Prepaid rates must take the money before the booking is final, and a card authorization takes about a second, too long to keep inventory rows locked.
What we add
A payment provider called outside any transaction, a 15 minute hold that already counts against inventory, an outbox feeding Kafka, and workers that capture payments, release expired holds and keep the cache fresh.
What it costs
A held room is unsellable until it is paid or released, and holds on counts cannot expire by themselves: a sweeper must give them back.
At 25k searches per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One app server handles about 2,000 to 5,000 requests a second.
- One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
- The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.
How big the shared systems are
- A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
- An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
- The payment provider allows about 1,000 calls a second.
What the writes are
- Inventory writes are 9% of the search rate.
- About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
- From stage 6, search servers cache location queries, so only 30% reach the index.
A count of 37 doesn’t say which of its 37 are holds. So unlike a seat row, an expired hold can’t simply be ignored. A sweeper on each database runs every 30 seconds. It marks expired holds and gives their nights back, in one statement. So a hold is released exactly once.
This is enough for a mid-level answer. Counts per night, one transaction for the stay, search on a copy, and a hold around the card. The stress tests are what make it a senior answer.
Go deeperThe sweeper, in SQL1 min
WITH expired AS ( UPDATE reservations SET status = 'EXPIRED' WHERE status = 'HELD' AND hold_expires_at < now() RETURNING hotel_id, room_type_id, check_in, check_out, rooms), give_back AS ( -- one row per night: two holds on the 5th must give back 2 SELECT hotel_id, room_type_id, n::date AS night, SUM(rooms) AS rooms FROM expired, generate_series(check_in, check_out - 1, interval '1 day') AS n GROUP BY 1, 2, 3)UPDATE room_type_inventory iSET reserved = i.reserved - g.roomsFROM give_back gWHERE i.hotel_id = g.hotel_id AND i.room_type_id = g.room_type_id AND i.night = g.night;The status change and the give-back are one statement, so a hold is released exactly once. The
give_back step matters. An update that joins one inventory row to two expired holds would apply
only one of them. The real version takes a few hundred holds per run. It locks the nights in date
order first, like a booking. Abandoned holds keep rooms out of sale for up to 15 minutes and 30
seconds. That’s the price of keeping the room safe through a slow bank.
Go deeperWhy the hold lasts 15 minutes1 min
Most card payments finish in a few seconds. But some banks ask the guest to confirm in their banking app, and that can take a few minutes while they find their phone. A hold that ends too soon would lose the room while the guest is still paying.
Fifteen minutes covers nearly every real payment. The price is that an abandoned checkout keeps a room out of sale for up to 15 minutes and 30 seconds. On a quiet night that costs nothing. On the last few rooms of a busy weekend it can turn a guest away. Some platforms shorten the hold when only a few rooms are left, and show the guest a countdown.
Go deeperThe API2 min
GET /v1/search?area=paris&checkIn=2027-05-03&checkOut=2027-05-06&guests=2&rooms=1200 OK { "hotels": [ { "id": 81231, "fromPrice": { "amount": 41700, "currency": "EUR" }, "fewLeft": true } ], "asOf": "2027-03-01T10:04:12Z" }
POST /v1/quotes{ "hotelId": 81231, "roomTypeId": 12, "ratePlanId": 3, "checkIn": "2027-05-03", "checkOut": "2027-05-06" }201 Created { "quoteId": "q_9f2...", "total": { "amount": 41700 }, "expiresAt": "..." }
POST /v1/reservationsIdempotency-Key: 4c1e7a90-...{ "quoteId": "q_9f2...", "guest": { ... }, "payment": { "token": "pm_tok_..." } }201 Created { "reservationId": "R81231-7Q2XK", "status": "CONFIRMED" }409 Conflict { "error": "SOLD_OUT", "nights": ["2027-05-05"] }checkOut is the morning the guest leaves, so this stay is the nights of the 3rd, 4th and 5th. A
quote is priced from the database when the booking page opens. It fixes the price for 30 minutes.
It reserves nothing, so a guest can still lose the last room while typing a card number. That’s
cheaper than holding rooms for every page view.
A retry with the same key and body returns the first response. 422 means a key reused with a
different body. The reservation ID starts with the hotel ID, so later calls find the right database
without a lookup. Agencies and hotel systems use the channel manager instead, often with the travel
industry’s standard XML messages.
Go deeperThe tables1 min
CREATE TABLE room_type_inventory ( hotel_id BIGINT NOT NULL, room_type_id BIGINT NOT NULL, night DATE NOT NULL, total_rooms SMALLINT NOT NULL, -- rooms of this type in service that night overbook_limit SMALLINT NOT NULL DEFAULT 0, -- extra rooms the hotel is willing to sell reserved SMALLINT NOT NULL DEFAULT 0, -- held plus confirmed, every channel PRIMARY KEY (hotel_id, room_type_id, night));
CREATE TABLE reservations ( hotel_id BIGINT NOT NULL, reservation_id UUID NOT NULL, room_type_id BIGINT NOT NULL, check_in DATE NOT NULL, check_out DATE NOT NULL, -- nights are check_in up to, not including, check_out rooms SMALLINT NOT NULL, status TEXT NOT NULL, -- HELD | CONFIRMED | EXPIRED | CANCELLED | NO_SHOW hold_expires_at TIMESTAMPTZ, idem_key TEXT NOT NULL, -- the client's key, or 'agency:their reference' PRIMARY KEY (hotel_id, reservation_id), UNIQUE (hotel_id, idem_key));Every table starts with hotel_id, so splitting by hotel later needs no schema change.
| Where search could read availability | At 25,000 searches a second | Verdict |
|---|---|---|
| The primary | 12 times what it can answer, and it slows bookings | Wrong at this ratio |
| Read replicas | About 13 replicas, each still scanning | Works, at a high cost per search |
| A Redis hash per hotel | 2.5 million lookups a second, about a dozen nodes | The pick for now |
| Inside the search index | Every booking and rate change reindexes a hotel | The index is built for reads |
Stress it
Section titled “Stress it”The design is complete. Now we test it where hotel booking is hard. Two overlapping stays at once. Retries, and a payment that outlasts its hold. The agencies that sell the same rooms. And a guest who changes their dates. The agencies add one box.
Booking a date range atomically
Section titled “Booking a date range atomically”Show me how a three-night booking avoids double-selling when two overlapping stays arrive at once.
Whether the check and the write are one step, all nights succeed or none do, and lock order is clear.
| Approach | Correct? | Cost here |
|---|---|---|
| Read the counts, then write | No: two readers both see a room | None, which is why it ships by mistake |
| SERIALIZABLE isolation | Yes | Aborts under contention; every caller retries |
| A version number per night | Yes | Retries on a popular weekend |
| Lock nights in order + conditional update | Yes | Two round trips, locks held a few ms |
| Lock the whole hotel | Yes | Blocks unrelated room types and stays |
The pick locks only the nights it takes, in a fixed order, with no application code between the check and the write. It’s correct under Postgres’s default isolation: a writer waiting on a row checks its condition again against the committed row.
// Reserving a stay: one transaction takes every night of the range, or none of them.interface Tx { int execute(String sql, Object... args); // returns the number of rows changed int countRows(String sql, Object... args); // runs a SELECT, returns its row count Reservation findByKey(long hotelId, String idemKey);}
interface Db { <T> T inTransaction(Function<Tx, T> work); // commits on return, rolls back on throw}
record StayRequest(long hotelId, long roomTypeId, LocalDate checkIn, LocalDate checkOut, int rooms, boolean prepaid, String idemKey, String requestHash) {}
record Reservation(UUID id, String status, String requestHash) {}
class SoldOutException extends RuntimeException { SoldOutException(String why) { super(why); }}
class KeyReusedException extends RuntimeException { KeyReusedException() { super("idempotency key reused for a different request"); }}
class ReservationService { // The reservation row goes in first, unique on (hotel_id, idem_key). The hotel is part of // the key because a unique index is only enforced inside one shard. A concurrent retry // blocks on this index until we commit, then finds our row instead of booking again. // Prepaid stays start HELD with a 15 minute expiry; pay-at-hotel stays are final at once. static final String INSERT_RESERVATION = """ INSERT INTO reservations (reservation_id, hotel_id, room_type_id, check_in, check_out, rooms, status, hold_expires_at, idem_key, request_hash) VALUES (?, ?, ?, ?, ?, ?, ?, CASE WHEN ? THEN now() + interval '15 minutes' END, ?, ?) ON CONFLICT (hotel_id, idem_key) DO NOTHING""";
// Lock the nights in date order. Two overlapping stays (the 3rd to the 6th and the 5th to // the 8th) then queue on the 5th instead of each locking part of the other's range and // deadlocking. A plain multi-row UPDATE gives no guarantee about the order it locks rows in. static final String LOCK_NIGHTS = """ SELECT night FROM room_type_inventory WHERE hotel_id = ? AND room_type_id = ? AND night >= ? AND night < ? ORDER BY night FOR UPDATE""";
// Take a room on every night, only where one is left. The limit includes the hotel's // overbooking allowance for that night, so the policy is data, not code. No SELECT and // compare in the application: the check and the write are one statement. static final String TAKE_NIGHTS = """ UPDATE room_type_inventory SET reserved = reserved + ?, version = version + 1 WHERE hotel_id = ? AND room_type_id = ? AND night >= ? AND night < ? AND reserved + ? <= total_rooms + overbook_limit""";
private final Db db;
ReservationService(Db db) { this.db = db; }
Reservation reserve(StayRequest r) { // Half-open range: check-out day is not a night, so back-to-back stays share nothing. int nights = (int) ChronoUnit.DAYS.between(r.checkIn(), r.checkOut()); UUID id = UUID.randomUUID(); String status = r.prepaid() ? "HELD" : "CONFIRMED"; return db.inTransaction(tx -> { if (tx.execute(INSERT_RESERVATION, id, r.hotelId(), r.roomTypeId(), r.checkIn(), r.checkOut(), r.rooms(), status, r.prepaid(), r.idemKey(), r.requestHash()) == 0) { // A retry. Same key, same request: return the first answer. Same key with a // different body is a client bug, and silently returning the old booking would // hide it. Reservation first = tx.findByKey(r.hotelId(), r.idemKey()); if (!first.requestHash().equals(r.requestHash())) throw new KeyReusedException(); return first; } // A missing row means the night is not on sale (beyond the horizon, or closed). if (tx.countRows(LOCK_NIGHTS, r.hotelId(), r.roomTypeId(), r.checkIn(), r.checkOut()) != nights) throw new SoldOutException("not on sale for every night"); // All or nothing: if one night is full, the update changes fewer rows than there // are nights, and the throw rolls back the nights it did take and the reservation // row. A sold-out answer is not stored, so a retry later may succeed. if (tx.execute(TAKE_NIGHTS, r.rooms(), r.hotelId(), r.roomTypeId(), r.checkIn(), r.checkOut(), r.rooms()) != nights) throw new SoldOutException("no room left on at least one night"); return new Reservation(id, status, r.requestHash()); }); }}# Reserving a stay: one transaction takes every night of the range, or none of them.class Tx(Protocol): def execute(self, sql: str, *args) -> int: ... # returns the number of rows changed def count_rows(self, sql: str, *args) -> int: ... # runs a SELECT, returns its row count def find_by_key(self, hotel_id: int, idem_key: str) -> "Reservation": ...
class Db(Protocol): def transaction(self) -> ContextManager[Tx]: ... # commits on exit, rolls back on raise
@dataclass(frozen=True)class StayRequest: hotel_id: int room_type_id: int check_in: date check_out: date rooms: int prepaid: bool idem_key: str request_hash: str
@dataclass(frozen=True)class Reservation: id: UUID status: str request_hash: str
class SoldOut(Exception): pass
class KeyReused(Exception): pass
# The reservation row goes in first, unique on (hotel_id, idem_key). The hotel is part of# the key because a unique index is only enforced inside one shard. A concurrent retry# blocks on this index until we commit, then finds our row instead of booking again.# Prepaid stays start HELD with a 15 minute expiry; pay-at-hotel stays are final at once.INSERT_RESERVATION = """ INSERT INTO reservations (reservation_id, hotel_id, room_type_id, check_in, check_out, rooms, status, hold_expires_at, idem_key, request_hash) VALUES (%s, %s, %s, %s, %s, %s, %s, CASE WHEN %s THEN now() + interval '15 minutes' END, %s, %s) ON CONFLICT (hotel_id, idem_key) DO NOTHING"""
# Lock the nights in date order. Two overlapping stays (the 3rd to the 6th and the 5th to# the 8th) then queue on the 5th instead of each locking part of the other's range and# deadlocking. A plain multi-row UPDATE gives no guarantee about the order it locks rows in.LOCK_NIGHTS = """ SELECT night FROM room_type_inventory WHERE hotel_id = %s AND room_type_id = %s AND night >= %s AND night < %s ORDER BY night FOR UPDATE"""
# Take a room on every night, only where one is left. The limit includes the hotel's# overbooking allowance for that night, so the policy is data, not code. No SELECT and# compare in the application: the check and the write are one statement.TAKE_NIGHTS = """ UPDATE room_type_inventory SET reserved = reserved + %s, version = version + 1 WHERE hotel_id = %s AND room_type_id = %s AND night >= %s AND night < %s AND reserved + %s <= total_rooms + overbook_limit"""
class ReservationService: def __init__(self, db: Db): self._db = db
def reserve(self, r: StayRequest) -> Reservation: # Half-open range: check-out day is not a night, so back-to-back stays share nothing. nights = (r.check_out - r.check_in).days rid = uuid4() status = "HELD" if r.prepaid else "CONFIRMED" with self._db.transaction() as tx: inserted = tx.execute( INSERT_RESERVATION, rid, r.hotel_id, r.room_type_id, r.check_in, r.check_out, r.rooms, status, r.prepaid, r.idem_key, r.request_hash, ) if inserted == 0: # A retry. Same key, same request: return the first answer. Same key with a # different body is a client bug, and silently returning the old booking would # hide it. first = tx.find_by_key(r.hotel_id, r.idem_key) if first.request_hash != r.request_hash: raise KeyReused() return first # A missing row means the night is not on sale (beyond the horizon, or closed). args = (r.hotel_id, r.room_type_id, r.check_in, r.check_out) if tx.count_rows(LOCK_NIGHTS, *args) != nights: raise SoldOut("not on sale for every night") # All or nothing: if one night is full, the update changes fewer rows than there # are nights, and the raise rolls back the nights it did take and the reservation # row. A sold-out answer is not stored, so a retry later may succeed. if tx.execute(TAKE_NIGHTS, r.rooms, *args, r.rooms) != nights: raise SoldOut("no room left on at least one night") return Reservation(rid, status, r.request_hash)// Reserving a stay: one transaction takes every night of the range, or none of them.interface ITx{ int Execute(string sql, params object[] args); // returns the number of rows changed int CountRows(string sql, params object[] args); // runs a SELECT, returns its row count Reservation FindByKey(long hotelId, string idemKey);}
interface IDb{ T InTransaction<T>(Func<ITx, T> work); // commits on return, rolls back on throw}
record StayRequest(long HotelId, long RoomTypeId, DateOnly CheckIn, DateOnly CheckOut, int Rooms, bool Prepaid, string IdemKey, string RequestHash);
record Reservation(Guid Id, string Status, string RequestHash);
class SoldOutException(string why) : Exception(why);
class KeyReusedException() : Exception("idempotency key reused for a different request");
class ReservationService(IDb db){ // The reservation row goes in first, unique on (hotel_id, idem_key). The hotel is part of // the key because a unique index is only enforced inside one shard. A concurrent retry // blocks on this index until we commit, then finds our row instead of booking again. // Prepaid stays start HELD with a 15 minute expiry; pay-at-hotel stays are final at once. const string InsertReservation = """ INSERT INTO reservations (reservation_id, hotel_id, room_type_id, check_in, check_out, rooms, status, hold_expires_at, idem_key, request_hash) VALUES (@p0, @p1, @p2, @p3, @p4, @p5, @p6, CASE WHEN @p7 THEN now() + interval '15 minutes' END, @p8, @p9) ON CONFLICT (hotel_id, idem_key) DO NOTHING """;
// Lock the nights in date order. Two overlapping stays (the 3rd to the 6th and the 5th to // the 8th) then queue on the 5th instead of each locking part of the other's range and // deadlocking. A plain multi-row UPDATE gives no guarantee about the order it locks rows in. const string LockNights = """ SELECT night FROM room_type_inventory WHERE hotel_id = @p0 AND room_type_id = @p1 AND night >= @p2 AND night < @p3 ORDER BY night FOR UPDATE """;
// Take a room on every night, only where one is left. The limit includes the hotel's // overbooking allowance for that night, so the policy is data, not code. No SELECT and // compare in the application: the check and the write are one statement. const string TakeNights = """ UPDATE room_type_inventory SET reserved = reserved + @p0, version = version + 1 WHERE hotel_id = @p1 AND room_type_id = @p2 AND night >= @p3 AND night < @p4 AND reserved + @p5 <= total_rooms + overbook_limit """;
public Reservation Reserve(StayRequest r) { // Half-open range: check-out day is not a night, so back-to-back stays share nothing. int nights = r.CheckOut.DayNumber - r.CheckIn.DayNumber; var id = Guid.NewGuid(); var status = r.Prepaid ? "HELD" : "CONFIRMED"; return db.InTransaction(tx => { if (tx.Execute(InsertReservation, id, r.HotelId, r.RoomTypeId, r.CheckIn, r.CheckOut, r.Rooms, status, r.Prepaid, r.IdemKey, r.RequestHash) == 0) { // A retry. Same key, same request: return the first answer. Same key with a // different body is a client bug, and silently returning the old booking would // hide it. var first = tx.FindByKey(r.HotelId, r.IdemKey); if (first.RequestHash != r.RequestHash) throw new KeyReusedException(); return first; } // A missing row means the night is not on sale (beyond the horizon, or closed). if (tx.CountRows(LockNights, r.HotelId, r.RoomTypeId, r.CheckIn, r.CheckOut) != nights) throw new SoldOutException("not on sale for every night"); // All or nothing: if one night is full, the update changes fewer rows than there // are nights, and the throw rolls back the nights it did take and the reservation // row. A sold-out answer is not stored, so a retry later may succeed. if (tx.Execute(TakeNights, r.Rooms, r.HotelId, r.RoomTypeId, r.CheckIn, r.CheckOut, r.Rooms) != nights) throw new SoldOutException("no room left on at least one night"); return new Reservation(id, status, r.RequestHash); }); }}// Reserving a stay: one transaction takes every night of the range, or none of them.
// Tx is one open database transaction.type Tx interface { Exec(sql string, args ...any) (int64, error) // returns the number of rows changed CountRows(sql string, args ...any) (int64, error) // runs a SELECT, returns its row count FindByKey(hotelID int64, idemKey string) (Reservation, error)}
// DB commits when fn returns nil and rolls back when it returns an error.type DB interface { InTx(fn func(Tx) error) error}
type StayRequest struct { HotelID, RoomTypeID int64 CheckIn, CheckOut time.Time // dates, at midnight UTC Rooms int Prepaid bool IdemKey string RequestHash string}
type Reservation struct { ID, Status, RequestHash string}
var ( ErrNotOnSale = errors.New("not on sale for every night") ErrSoldOut = errors.New("no room left on at least one night") ErrKeyReused = errors.New("idempotency key reused for a different request"))
// The reservation row goes in first, unique on (hotel_id, idem_key). The hotel is part of// the key because a unique index is only enforced inside one shard. A concurrent retry// blocks on this index until we commit, then finds our row instead of booking again.// Prepaid stays start HELD with a 15 minute expiry; pay-at-hotel stays are final at once.const insertReservation = ` INSERT INTO reservations (reservation_id, hotel_id, room_type_id, check_in, check_out, rooms, status, hold_expires_at, idem_key, request_hash) VALUES ($1, $2, $3, $4, $5, $6, $7, CASE WHEN $8 THEN now() + interval '15 minutes' END, $9, $10) ON CONFLICT (hotel_id, idem_key) DO NOTHING`
// Lock the nights in date order. Two overlapping stays (the 3rd to the 6th and the 5th to// the 8th) then queue on the 5th instead of each locking part of the other's range and// deadlocking. A plain multi-row UPDATE gives no guarantee about the order it locks rows in.const lockNights = ` SELECT night FROM room_type_inventory WHERE hotel_id = $1 AND room_type_id = $2 AND night >= $3 AND night < $4 ORDER BY night FOR UPDATE`
// Take a room on every night, only where one is left. The limit includes the hotel's// overbooking allowance for that night, so the policy is data, not code. No SELECT and// compare in the application: the check and the write are one statement.const takeNights = ` UPDATE room_type_inventory SET reserved = reserved + $1, version = version + 1 WHERE hotel_id = $2 AND room_type_id = $3 AND night >= $4 AND night < $5 AND reserved + $1 <= total_rooms + overbook_limit`
type ReservationService struct { db DB newID func() string // a UUID generator, injected}
func (s *ReservationService) Reserve(r StayRequest) (Reservation, error) { // Half-open range: check-out day is not a night, so back-to-back stays share nothing. nights := int64(r.CheckOut.Sub(r.CheckIn).Hours() / 24) res := Reservation{ID: s.newID(), Status: "CONFIRMED", RequestHash: r.RequestHash} if r.Prepaid { res.Status = "HELD" } err := s.db.InTx(func(tx Tx) error { n, err := tx.Exec(insertReservation, res.ID, r.HotelID, r.RoomTypeID, r.CheckIn, r.CheckOut, r.Rooms, res.Status, r.Prepaid, r.IdemKey, r.RequestHash) if err != nil { return err } if n == 0 { // A retry. Same key, same request: return the first answer. Same key with a // different body is a client bug, and silently returning the old booking would // hide it. first, err := tx.FindByKey(r.HotelID, r.IdemKey) if err != nil { return err } if first.RequestHash != r.RequestHash { return ErrKeyReused } res = first return nil } // A missing row means the night is not on sale (beyond the horizon, or closed). found, err := tx.CountRows(lockNights, r.HotelID, r.RoomTypeID, r.CheckIn, r.CheckOut) if err != nil { return err } if found != nights { return ErrNotOnSale } // All or nothing: if one night is full, the update changes fewer rows than there // are nights, and the error rolls back the nights it did take and the reservation // row. A sold-out answer is not stored, so a retry later may succeed. taken, err := tx.Exec(takeNights, r.Rooms, r.HotelID, r.RoomTypeID, r.CheckIn, r.CheckOut) if err != nil { return err } if taken != nights { return ErrSoldOut } return nil }) return res, err}// Reserving a stay: one transaction takes every night of the range, or none of them.interface Tx { execute(sql: string, ...args: unknown[]): Promise<number>; // rows changed countRows(sql: string, ...args: unknown[]): Promise<number>; // runs a SELECT, returns its row count findByKey(hotelId: number, idemKey: string): Promise<Reservation>;}
interface Db { inTransaction<T>(work: (tx: Tx) => Promise<T>): Promise<T>; // commits, or rolls back on throw}
interface StayRequest { hotelId: number; roomTypeId: number; checkIn: string; // 'YYYY-MM-DD' checkOut: string; rooms: number; prepaid: boolean; idemKey: string; requestHash: string;}
interface Reservation { id: string; status: 'HELD' | 'CONFIRMED'; requestHash: string;}
class SoldOutError extends Error {}class KeyReusedError extends Error {}
// The reservation row goes in first, unique on (hotel_id, idem_key). The hotel is part of// the key because a unique index is only enforced inside one shard. A concurrent retry// blocks on this index until we commit, then finds our row instead of booking again.// Prepaid stays start HELD with a 15 minute expiry; pay-at-hotel stays are final at once.const INSERT_RESERVATION = ` INSERT INTO reservations (reservation_id, hotel_id, room_type_id, check_in, check_out, rooms, status, hold_expires_at, idem_key, request_hash) VALUES ($1, $2, $3, $4, $5, $6, $7, CASE WHEN $8 THEN now() + interval '15 minutes' END, $9, $10) ON CONFLICT (hotel_id, idem_key) DO NOTHING`;
// Lock the nights in date order. Two overlapping stays (the 3rd to the 6th and the 5th to// the 8th) then queue on the 5th instead of each locking part of the other's range and// deadlocking. A plain multi-row UPDATE gives no guarantee about the order it locks rows in.const LOCK_NIGHTS = ` SELECT night FROM room_type_inventory WHERE hotel_id = $1 AND room_type_id = $2 AND night >= $3 AND night < $4 ORDER BY night FOR UPDATE`;
// Take a room on every night, only where one is left. The limit includes the hotel's// overbooking allowance for that night, so the policy is data, not code. No SELECT and// compare in the application: the check and the write are one statement.const TAKE_NIGHTS = ` UPDATE room_type_inventory SET reserved = reserved + $1, version = version + 1 WHERE hotel_id = $2 AND room_type_id = $3 AND night >= $4 AND night < $5 AND reserved + $1 <= total_rooms + overbook_limit`;
const DAY_MS = 86_400_000;
class ReservationService { constructor( private readonly db: Db, private readonly newId: () => string = () => crypto.randomUUID(), ) {}
reserve(r: StayRequest): Promise<Reservation> { // Half-open range: check-out day is not a night, so back-to-back stays share nothing. const nights = (Date.parse(r.checkOut) - Date.parse(r.checkIn)) / DAY_MS; const id = this.newId(); const status = r.prepaid ? 'HELD' : 'CONFIRMED'; return this.db.inTransaction(async (tx) => { const inserted = await tx.execute( INSERT_RESERVATION, id, r.hotelId, r.roomTypeId, r.checkIn, r.checkOut, r.rooms, status, r.prepaid, r.idemKey, r.requestHash, ); if (inserted === 0) { // A retry. Same key, same request: return the first answer. Same key with a // different body is a client bug, and silently returning the old booking would // hide it. const first = await tx.findByKey(r.hotelId, r.idemKey); if (first.requestHash !== r.requestHash) throw new KeyReusedError(); return first; } // A missing row means the night is not on sale (beyond the horizon, or closed). const range = [r.hotelId, r.roomTypeId, r.checkIn, r.checkOut]; if ((await tx.countRows(LOCK_NIGHTS, ...range)) !== nights) throw new SoldOutError('not on sale for every night'); // All or nothing: if one night is full, the update changes fewer rows than there // are nights, and the throw rolls back the nights it did take and the reservation // row. A sold-out answer is not stored, so a retry later may succeed. if ((await tx.execute(TAKE_NIGHTS, r.rooms, ...range)) !== nights) throw new SoldOutError('no room left on at least one night'); return { id, status, requestHash: r.requestHash }; }); }}Why lock order matters. If Ana and Ben locked nights in whatever order the database found them,
each could hold a night the other needs. Postgres detects that deadlock after a second and aborts
one. A single multi-row update makes no promise about the order it locks rows, so an explicit
SELECT ... ORDER BY night FOR UPDATE is what turns date order into a guarantee.
How much contention? Transactions in flight equal the booking rate times how long each holds its locks (Little’s law). That’s 25 a second × 5 ms, or 0.125 across the whole platform. Even a flash sale of 100 bookings a second on one room type is 0.5. Row locks are cheap here, and no waiting room like ticket booking’s is needed.
Retries, and a payment that outlasts its hold
Section titled “Retries, and a payment that outlasts its hold”The guest clicks Book, the request times out, and they click again. Then a slow bank approves a payment after the hold expired.
A retried click. The page creates the idempotency key when it loads. The key lives in the reservation row, unique per hotel, inserted in the same transaction as the nights. A retry after the commit finds the row. A retry during it waits on the unique index, then finds it. A crash before commit leaves no row, so the retry books normally.
A payment that outlasts its hold. Confirming says “confirmed, only if still held”. The sweeper says “expired, only if still held and past its time”. They lock the same row, so exactly one wins. If the sweeper won, the nights went back on sale. The booking service then tries once more: expired to confirmed, only if still expired, plus the same ordered, conditional update over the nights. At hotel contention levels, the rooms are almost always still there. If a night is now full, it voids the authorization: no money moved.
- Color key
- Service
- Database
Go deeperA key at every step1 min
| Step | Key | What a repeat does |
|---|---|---|
| Guest to booking service | Idempotency-Key, unique per hotel |
Returns the first response |
| Booking service to provider | Reservation ID | Returns the first authorization, never a second |
| Agency booking delivery | The agency and its booking reference | Records nothing new, acknowledges again |
| Outbox to workers | Event ID; each job keyed by reservation | Capture, email and fee jobs run once in effect |
| Channel manager to agency | None needed: it sends absolute counts | Sets the same count again |
“Exactly-once booking” is a phrase to avoid. Networks deliver at most once or at least once. What the system gives is an exactly-once effect: every call may repeat, and a key at each step makes the repeat harmless. Naming the keys is the answer.
| Situation | Outcome |
|---|---|
| Approved, hold still valid | Confirmed; money taken from the outbox |
| Approved, hold expired, rooms still free | Nights taken again, confirmed |
| Approved, hold expired, a night now full | Void the authorization, 409 SOLD_OUT |
| Server crash after authorization | The guest’s retry re-authorizes under the same key, and confirms |
Overbooking, and the agencies
Section titled “Overbooking, and the agencies”Agencies sell the same rooms we do. What happens when one sells the last double on the 5th, two seconds after our website sold it, and how do we keep that rare?
About 40% of rooms sell through online travel agencies. Most keep their own copy of each hotel’s availability, sell against it, confirm the guest, then send us the booking. A channel manager does three things:
- Pushes availability and rates to every agency selling a hotel.
- Records bookings the agencies send, keyed by the agency’s reference. These add to
reservedwithout the limit check, because the guest is already confirmed. - Takes rate and room count updates from hotels’ own systems, batched per hotel.
Stage 4: Channel manager
Stage 4 of 6
Channel manager
Why this stage
About 40% of rooms are sold by online travel agencies from their own copy of our availability, and hotels push rates and room counts from their own systems.
What we add
A channel manager that pushes availability and rates to every agency within seconds of a change, and records bookings they deliver, which they have already confirmed to the guest.
What it costs
An agency can sell the last room on a copy a few seconds old.
Overbooking allowances and alerts absorb it, because we cannot refuse a booking the agency already confirmed.
At 25k searches per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One app server handles about 2,000 to 5,000 requests a second.
- One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
- The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.
How big the shared systems are
- A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
- An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
- The payment provider allows about 1,000 calls a second.
What the writes are
- Inventory writes are 9% of the search rate.
- About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
- From stage 6, search servers cache location queries, so only 30% reach the index.
Follow one booking all the way to the agencies:
Trace a request: Book 3 nights, prepaid
Book 3 nights, prepaid: step 1 of 8
Guests to API gateway
POST /reservations with the hotel, room type, dates, rate plan, a quote id and an Idempotency-Key generated when the booking page loaded.
- Guests → API gateway15 ms
- API gateway → Booking servers1 ms
- Booking servers → Inventory shards8 ms
- Booking servers → Payments700 ms
- Booking servers → Inventory shards5 ms
- Booking servers → Kafkaasync
- Kafka → Channel managerasync
- Response
Overbooking is policy, stored as data. Some guests cancel late, and some never arrive. So a hotel that sells exactly its rooms has empty rooms most nights. Revenue managers set an allowance per type and night from history, say 3 extra doubles on a Saturday. A hotel that wants none sets it to 0. When it goes wrong, a guest is “walked” to a similar hotel, at the hotel’s cost.
The agency race. We record the agency’s booking with the same transaction as a direct booking. The one difference is that the add has no condition, and we flag any night past its limit. Refusing it wouldn’t undo the guest’s confirmation.
- Color key
- Service
Follow an agency booking through the design:
Trace a request: An agency delivers a booking
An agency delivers a booking: step 1 of 6
Partners to Channel manager
An agency has already sold the room to its guest from its copy of our availability, and sends the reservation with its own booking reference.
- Partners → Channel manager30 ms
- Channel manager → Booking servers1 ms
- Booking servers → Inventory shards8 ms
- Booking servers → Kafkaasync
- Kafka → Channel managerasync
- Response
Keeping the window short. Oversells are roughly the agency sales that land inside the push delay. So pushes go out within a second, in a shape that’s safe to lose, repeat or reorder: the whole current count, never “one fewer”. Changes are grouped per hotel, and one pusher owns each hotel, so pushes for a hotel don’t race.
// The channel manager's push to agencies: coalesce changes per hotel, send absolute counts.
record NightRange(LocalDate from, LocalDate to) { // half-open, like a stay NightRange union(NightRange o) { return new NightRange(from.isBefore(o.from) ? from : o.from, to.isAfter(o.to) ? to : o.to); }}
record RoomCount(long roomTypeId, LocalDate night, int available, int rateCents) {}
interface Inventory { // Reads the shard as it is now: the push carries the latest state, not the event's. List<RoomCount> current(long hotelId, NightRange nights);}
interface Agency { String name(); boolean sells(long hotelId); // "Set availability to n", never "subtract one". A repeated or late push is harmless, // because applying the same absolute value twice changes nothing. void setAvailability(long hotelId, List<RoomCount> counts) throws IOException;}
class AvailabilityPusher { // Hotels changed since the last flush, with the nights touched. A map, so fifty changes to // one hotel in a second (a group block, a rate reload) become one push per agency. private final Map<Long, NightRange> dirty = new HashMap<>(); private final Inventory inventory; private final List<Agency> agencies;
AvailabilityPusher(Inventory inventory, List<Agency> agencies) { this.inventory = inventory; this.agencies = agencies; }
// Called for every InventoryChanged event from Kafka. synchronized void onChange(long hotelId, NightRange nights) { dirty.merge(hotelId, nights, NightRange::union); }
// Runs every second. Agencies rate-limit pushes per hotel, so batching is required, and // one second keeps the window in which they sell on stale counts short. void flush() { Map<Long, NightRange> batch; synchronized (this) { batch = new HashMap<>(dirty); dirty.clear(); } batch.forEach((hotelId, nights) -> { List<RoomCount> counts = inventory.current(hotelId, nights); for (Agency agency : agencies) { if (!agency.sells(hotelId)) continue; try { agency.setAvailability(hotelId, counts); } catch (IOException e) { // Mark the hotel dirty again rather than retrying this payload: the next // flush reads fresher counts, so a slow agency never receives old ones. // (A real pusher keeps dirty state per agency, so one slow agency does not // make the others receive repeats.) onChange(hotelId, nights); } } }); }}# The channel manager's push to agencies: coalesce changes per hotel, send absolute counts.
@dataclass(frozen=True)class NightRange: # half-open, like a stay first: date last: date
def union(self, o: "NightRange") -> "NightRange": return NightRange(min(self.first, o.first), max(self.last, o.last))
@dataclass(frozen=True)class RoomCount: room_type_id: int night: date available: int rate_cents: int
class Inventory(Protocol): # Reads the shard as it is now: the push carries the latest state, not the event's. def current(self, hotel_id: int, nights: NightRange) -> list[RoomCount]: ...
class Agency(Protocol): name: str
def sells(self, hotel_id: int) -> bool: ...
# "Set availability to n", never "subtract one". A repeated or late push is harmless, # because applying the same absolute value twice changes nothing. def set_availability(self, hotel_id: int, counts: list[RoomCount]) -> None: ...
class AvailabilityPusher: def __init__(self, inventory: Inventory, agencies: list[Agency]): self._inventory = inventory self._agencies = agencies # Hotels changed since the last flush, with the nights touched. A dict, so fifty # changes to one hotel in a second (a group block, a rate reload) become one push # per agency. self._dirty: dict[int, NightRange] = {} self._lock = Lock()
# Called for every InventoryChanged event from Kafka. def on_change(self, hotel_id: int, nights: NightRange) -> None: with self._lock: seen = self._dirty.get(hotel_id) self._dirty[hotel_id] = seen.union(nights) if seen else nights
# Runs every second. Agencies rate-limit pushes per hotel, so batching is required, # and one second keeps the window in which they sell on stale counts short. def flush(self) -> None: with self._lock: batch, self._dirty = self._dirty, {} for hotel_id, nights in batch.items(): counts = self._inventory.current(hotel_id, nights) for agency in self._agencies: if not agency.sells(hotel_id): continue try: agency.set_availability(hotel_id, counts) except OSError: # Mark the hotel dirty again rather than retrying this payload: the next # flush reads fresher counts, so a slow agency never receives old ones. # (A real pusher keeps dirty state per agency, so one slow agency does not # make the others receive repeats.) self.on_change(hotel_id, nights)// The channel manager's push to agencies: coalesce changes per hotel, send absolute counts.
record NightRange(DateOnly From, DateOnly To) // half-open, like a stay{ public NightRange Union(NightRange o) => new(From < o.From ? From : o.From, To > o.To ? To : o.To);}
record RoomCount(long RoomTypeId, DateOnly Night, int Available, int RateCents);
interface IInventory{ // Reads the shard as it is now: the push carries the latest state, not the event's. IReadOnlyList<RoomCount> Current(long hotelId, NightRange nights);}
interface IAgency{ string Name { get; } bool Sells(long hotelId); // "Set availability to n", never "subtract one". A repeated or late push is harmless, // because applying the same absolute value twice changes nothing. void SetAvailability(long hotelId, IReadOnlyList<RoomCount> counts);}
class AvailabilityPusher(IInventory inventory, IReadOnlyList<IAgency> agencies){ // Hotels changed since the last flush, with the nights touched. A dictionary, so fifty // changes to one hotel in a second (a group block, a rate reload) become one push per agency. Dictionary<long, NightRange> dirty = new(); readonly object gate = new();
// Called for every InventoryChanged event from Kafka. public void OnChange(long hotelId, NightRange nights) { lock (gate) dirty[hotelId] = dirty.TryGetValue(hotelId, out var seen) ? seen.Union(nights) : nights; }
// Runs every second. Agencies rate-limit pushes per hotel, so batching is required, and // one second keeps the window in which they sell on stale counts short. public void Flush() { Dictionary<long, NightRange> batch; lock (gate) { batch = dirty; dirty = new(); } foreach (var (hotelId, nights) in batch) { var counts = inventory.Current(hotelId, nights); foreach (var agency in agencies) { if (!agency.Sells(hotelId)) continue; try { agency.SetAvailability(hotelId, counts); } catch (IOException) { // Mark the hotel dirty again rather than retrying this payload: the next // flush reads fresher counts, so a slow agency never receives old ones. // (A real pusher keeps dirty state per agency, so one slow agency does not // make the others receive repeats.) OnChange(hotelId, nights); } } } }}// The channel manager's push to agencies: coalesce changes per hotel, send absolute counts.
// NightRange is half-open, like a stay.type NightRange struct{ From, To time.Time }
func (r NightRange) Union(o NightRange) NightRange { if o.From.Before(r.From) { r.From = o.From } if o.To.After(r.To) { r.To = o.To } return r}
type RoomCount struct { RoomTypeID int64 Night time.Time Available int RateCents int}
// Inventory reads the shard as it is now: the push carries the latest state, not the event's.type Inventory interface { Current(hotelID int64, nights NightRange) ([]RoomCount, error)}
// Agency takes "set availability to n", never "subtract one". A repeated or late push is// harmless, because applying the same absolute value twice changes nothing.type Agency interface { Name() string Sells(hotelID int64) bool SetAvailability(hotelID int64, counts []RoomCount) error}
type AvailabilityPusher struct { inventory Inventory agencies []Agency
mu sync.Mutex // Hotels changed since the last flush, with the nights touched. A map, so fifty changes to // one hotel in a second (a group block, a rate reload) become one push per agency. dirty map[int64]NightRange}
// OnChange is called for every InventoryChanged event from Kafka.func (p *AvailabilityPusher) OnChange(hotelID int64, nights NightRange) { p.mu.Lock() defer p.mu.Unlock() if seen, ok := p.dirty[hotelID]; ok { nights = seen.Union(nights) } p.dirty[hotelID] = nights}
// Flush runs every second. Agencies rate-limit pushes per hotel, so batching is required,// and one second keeps the window in which they sell on stale counts short.func (p *AvailabilityPusher) Flush() { p.mu.Lock() batch := p.dirty p.dirty = map[int64]NightRange{} p.mu.Unlock()
for hotelID, nights := range batch { counts, err := p.inventory.Current(hotelID, nights) if err != nil { p.OnChange(hotelID, nights) // try again next second continue } for _, agency := range p.agencies { if !agency.Sells(hotelID) { continue } if err := agency.SetAvailability(hotelID, counts); err != nil { // Mark the hotel dirty again rather than retrying this payload: the next // flush reads fresher counts, so a slow agency never receives old ones. // (A real pusher keeps dirty state per agency, so one slow agency does not // make the others receive repeats.) p.OnChange(hotelID, nights) } } }}// The channel manager's push to agencies: coalesce changes per hotel, send absolute counts.
// Half-open, like a stay. Dates as 'YYYY-MM-DD', which compare correctly as strings.interface NightRange { from: string; to: string;}
const union = (a: NightRange, b: NightRange): NightRange => ({ from: a.from < b.from ? a.from : b.from, to: a.to > b.to ? a.to : b.to,});
interface RoomCount { roomTypeId: number; night: string; available: number; rateCents: number;}
interface Inventory { // Reads the shard as it is now: the push carries the latest state, not the event's. current(hotelId: number, nights: NightRange): Promise<RoomCount[]>;}
interface Agency { readonly name: string; sells(hotelId: number): boolean; // "Set availability to n", never "subtract one". A repeated or late push is harmless, // because applying the same absolute value twice changes nothing. setAvailability(hotelId: number, counts: RoomCount[]): Promise<void>;}
class AvailabilityPusher { // Hotels changed since the last flush, with the nights touched. A Map, so fifty changes to // one hotel in a second (a group block, a rate reload) become one push per agency. private dirty = new Map<number, NightRange>();
constructor( private readonly inventory: Inventory, private readonly agencies: Agency[], ) {}
// Called for every InventoryChanged event from Kafka. No lock: one event loop. onChange(hotelId: number, nights: NightRange): void { const seen = this.dirty.get(hotelId); this.dirty.set(hotelId, seen ? union(seen, nights) : nights); }
// Runs every second. Agencies rate-limit pushes per hotel, so batching is required, and // one second keeps the window in which they sell on stale counts short. async flush(): Promise<void> { const batch = this.dirty; this.dirty = new Map(); // changes arriving during the awaits below go to the next flush for (const [hotelId, nights] of batch) { const counts = await this.inventory.current(hotelId, nights); for (const agency of this.agencies) { if (!agency.sells(hotelId)) continue; try { await agency.setAvailability(hotelId, counts); } catch { // Mark the hotel dirty again rather than retrying this payload: the next flush // reads fresher counts, so a slow agency never receives old ones. (A real pusher // keeps dirty state per agency, so one slow agency does not make the others // receive repeats.) this.onChange(hotelId, nights); } } } }}Three more habits help. Close out early. At the last one or two rooms of a type, push 0 to the agencies, and keep the rest for direct sale, where the check is exact. Repair drift. Every night, push all 365 nights for every hotel. Watch it. A rising oversell rate for one agency means a slow push or a broken mapping.
Go deeperOther ways to sync agencies1 min
| Sync design | Wins | Loses |
|---|---|---|
| Push every change as “one fewer” | Simple to send | A lost or doubled message corrupts the agency’s count forever |
| Group per hotel, push absolute counts | Repeats and losses are harmless | A second of delay before each push |
| Agencies poll us | No push system | Minutes of staleness; load grows with every agency |
| Fixed allotments per agency | No race inside an allotment | Rooms stuck in one agency; release deadlines to manage |
After a rebalance, a late push from the old owner is overwritten by the next one, because every push reads the whole current state. Two agencies selling the same last room both deliver, both are recorded, and the night ends one past its limit: the same alert.
Changes and cancellations
Section titled “Changes and cancellations”A guest changes their stay from the 3rd–6th to the 4th–8th. What does the system do?
Cancel is one transaction: confirmed to cancelled, only if still confirmed, and subtract its rooms from every night. A second cancel changes no rows and gives nothing back twice. Any fee or refund is an outbox job keyed by reservation ID.
Change locks the old and new nights together, in date order. It adds to the new nights where a room is left, subtracts from the old, and updates the reservation. Shared nights (the 4th and 5th) are untouched, so only the new nights must be free. If one is full, it rolls back, and the guest keeps the original booking.
Go deeperNo-shows, closed rooms and group bookings1 min
No-shows are a morning job. Confirmed stays that started yesterday with no arrival become no-shows. Their remaining nights go back on sale, and the fee is charged.
A flooded room is out of service for a week. The hotel lowers total_rooms on those nights. If
a night is now oversold, the update still succeeds. Agencies get the lower counts, and an alert
shows which nights need a guest moved. Nothing is cancelled automatically. That’s the hotel’s call.
A tour operator booking 40 doubles uses the same transaction with rooms = 40: all nights or
none. Hotels usually want it as a block with a release date, so unsold rooms return to sale on their
own. That’s a hold with a long expiry, and its own sweep.
Go deeperTen times the hotels3 min
The platform grows ten times, to 200,000 hotels. Two things break: writes on one side, search on the other.
At ten times, row writes reach 23,000 a second, past one primary with a synchronous standby. Every booking, sweep and rate update touches one hotel, so we split inventory, rates and reservations by hotel, and every transaction stays on one database. Hotels map to 4,096 slots, and slots to databases, so a database splits by moving slots. A guest’s own bookings, the one query without a hotel, move to a table keyed by guest, fed from the outbox.
Stage 5: Shard by hotel
Stage 5 of 6
Shard by hotel
Why this stage
At 10 times the hotels, inventory takes about 23,000 row writes/s at peak, mostly rate and availability updates: twice what one primary sustains.
What we add
Inventory and reservations sharded by hotel id, so every booking, cancellation and update stays in one shard and one transaction.
What it costs
A guest’s list of bookings spans shards and moves to its own table, fed from the outbox.
One hotel can never use more than one shard, which no hotel comes near.
At 25k searches per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One app server handles about 2,000 to 5,000 requests a second.
- One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
- The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.
How big the shared systems are
- A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
- An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
- The payment provider allows about 1,000 calls a second.
What the writes are
- Inventory writes are 9% of the search rate.
- About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
- From stage 6, search servers cache location queries, so only 30% reach the index.
Search is next. It needs 25 million cache lookups a second, six times what a 20-node Redis cluster handles. But the data is small enough to copy everywhere instead. At 6 bytes a row, all availability is 2.2 GB, even at ten times today’s size. So every search server keeps all availability in its own memory. It loads a snapshot at start, then applies the stream of inventory changes from Kafka. A search reads every candidate hotel’s nights from local arrays, in microseconds. So it can check all 500 hotels in an area, and sort by price exactly. The Redis tier goes away.
Stage 6: Search from memory
Stage 6 of 6
Search from memory
Why this stage
At 10 times the platform, 250,000 searches/s each look up 100 hotels in the availability cache: 25 million lookups/s, about six times what a 20 primary Redis cluster serves.
What we add
Every search server keeps all availability in memory (about 220 MB, 2.2 GB at 10 times), fed by the inventory change stream, and caches location queries, which do not depend on dates.
The search tier runs in every region.
What it costs
Each server is a replica that must load a snapshot at start and apply changes in order.
At the 1M stress mark the hotel index needs more replicas and payments need a second provider.
At 25k searches per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One app server handles about 2,000 to 5,000 requests a second.
- One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
- The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.
How big the shared systems are
- A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
- An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
- The payment provider allows about 1,000 calls a second.
What the writes are
- Inventory writes are 9% of the search rate.
- About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
- From stage 6, search servers cache location queries, so only 30% reach the index.
// A search server's copy of all availability: applied from the change stream, read by every search.
// One inventory row after a change. `position` is where the commit sits in its shard's log.// A hotel lives on one shard and its changes travel on one Kafka partition, so positions for// one hotel only ever grow.record InventoryChange(long hotelId, int roomType, int night, int available, int rateCents, long position) {}
class AvailabilityReplica { static final int HORIZON = 366; // tonight plus 365 nights on sale
// Primitive arrays per hotel, one slot per room type and night, 6 bytes a row: 20,000 // hotels fit in about 220 MB. A map of small objects would take several times that. // Nights use a ring: night N and night N + 366 share a slot, and by the time the later one // goes on sale the earlier one is in the past. static final class HotelNights { final short[] available; final int[] rateCents; long appliedUpTo = -1; // the last log position applied for this hotel HotelNights(int roomTypes) { available = new short[roomTypes * HORIZON]; rateCents = new int[roomTypes * HORIZON]; } }
private final Map<Long, HotelNights> hotels = new ConcurrentHashMap<>(); private final Function<Long, Integer> roomTypesOf; // from the hotel catalog
AvailabilityReplica(Function<Long, Integer> roomTypesOf) { this.roomTypesOf = roomTypesOf; }
void apply(InventoryChange c) { HotelNights h = hotels.computeIfAbsent(c.hotelId(), id -> new HotelNights(roomTypesOf.apply(id))); int i = c.roomType() * HORIZON + Math.floorMod(c.night(), HORIZON); // A lock per hotel: changes for one hotel arrive a few times a second, and a read // holds it for microseconds, so nobody waits in practice. synchronized (h) { // The stream is at-least-once, and a new server replays it from a point before its // snapshot was taken, so changes it already has arrive again. One high-water mark // per hotel makes apply idempotent: anything at or below it is skipped. if (c.position() <= h.appliedUpTo) return; h.available[i] = (short) c.available(); h.rateCents[i] = c.rateCents(); h.appliedUpTo = c.position(); } }
// The cheapest total for `rooms` rooms of one type on every night of [from, to), or -1. // Nights are days since the epoch; the caller has checked they lie within the horizon. // At 500 candidate hotels and 3 nights this is a few thousand array reads per search. long cheapestStay(long hotelId, int from, int to, int rooms) { HotelNights h = hotels.get(hotelId); if (h == null) return -1; long best = -1; synchronized (h) { for (int t = 0; t < h.available.length / HORIZON; t++) { long total = 0; for (int n = from; n < to && total >= 0; n++) { int i = t * HORIZON + Math.floorMod(n, HORIZON); total = h.available[i] >= rooms ? total + (long) h.rateCents[i] * rooms : -1; } if (total >= 0 && (best < 0 || total < best)) best = total; } } // May be a second stale. That is fine: the booking transaction checks again, and // this answer only decides what the guest sees first. return best; }}# A search server's copy of all availability: applied from the change stream, read by every search.
# One inventory row after a change. `position` is where the commit sits in its shard's log.# A hotel lives on one shard and its changes travel on one Kafka partition, so positions for# one hotel only ever grow.@dataclass(frozen=True)class InventoryChange: hotel_id: int room_type: int night: int # days since the epoch available: int rate_cents: int position: int
HORIZON = 366 # tonight plus 365 nights on sale
class HotelNights: # Typed arrays per hotel, one slot per room type and night, 6 bytes a row: 20,000 hotels # fit in about 220 MB. A dict of small objects would take many times that. # Nights use a ring: night N and night N + 366 share a slot, and by the time the later one # goes on sale the earlier one is in the past. def __init__(self, room_types: int): self.available = array("h", bytes(2 * room_types * HORIZON)) self.rate_cents = array("i", bytes(4 * room_types * HORIZON)) self.room_types = room_types self.applied_up_to = -1 # the last log position applied for this hotel self.lock = Lock()
class AvailabilityReplica: def __init__(self, room_types_of: Callable[[int], int]): self._hotels: dict[int, HotelNights] = {} self._room_types_of = room_types_of # from the hotel catalog
def apply(self, c: InventoryChange) -> None: h = self._hotels.get(c.hotel_id) if h is None: h = self._hotels.setdefault(c.hotel_id, HotelNights(self._room_types_of(c.hotel_id))) i = c.room_type * HORIZON + c.night % HORIZON # A lock per hotel: changes for one hotel arrive a few times a second, and a read # holds it for microseconds, so nobody waits in practice. with h.lock: # The stream is at-least-once, and a new server replays it from a point before its # snapshot was taken, so changes it already has arrive again. One high-water mark # per hotel makes apply idempotent: anything at or below it is skipped. if c.position <= h.applied_up_to: return h.available[i] = c.available h.rate_cents[i] = c.rate_cents h.applied_up_to = c.position
# The cheapest total for `rooms` rooms of one type on every night of [first, last), # or -1. At 500 candidate hotels and 3 nights this is a few thousand reads per search. def cheapest_stay(self, hotel_id: int, first: int, last: int, rooms: int) -> int: h = self._hotels.get(hotel_id) if h is None: return -1 best = -1 with h.lock: for t in range(h.room_types): total = 0 for n in range(first, last): i = t * HORIZON + n % HORIZON if h.available[i] < rooms: total = -1 break total += h.rate_cents[i] * rooms if total >= 0 and (best < 0 or total < best): best = total # May be a second stale. That is fine: the booking transaction checks again, and # this answer only decides what the guest sees first. return best// A search server's copy of all availability: applied from the change stream, read by every search.
// One inventory row after a change. Position is where the commit sits in its shard's log.// A hotel lives on one shard and its changes travel on one Kafka partition, so positions for// one hotel only ever grow. Night is days since the epoch.record InventoryChange(long HotelId, int RoomType, int Night, int Available, int RateCents, long Position);
class AvailabilityReplica(Func<long, int> roomTypesOf) // room type count, from the hotel catalog{ const int Horizon = 366; // tonight plus 365 nights on sale
// Primitive arrays per hotel, one slot per room type and night, 6 bytes a row: 20,000 // hotels fit in about 220 MB. A dictionary of small objects would take several times that. // Nights use a ring: night N and night N + 366 share a slot, and by the time the later one // goes on sale the earlier one is in the past. sealed class HotelNights(int roomTypes) { public readonly short[] Available = new short[roomTypes * Horizon]; public readonly int[] RateCents = new int[roomTypes * Horizon]; public long AppliedUpTo = -1; // the last log position applied for this hotel }
readonly ConcurrentDictionary<long, HotelNights> hotels = new();
static int Slot(int roomType, int night) => roomType * Horizon + ((night % Horizon) + Horizon) % Horizon;
public void Apply(InventoryChange c) { var h = hotels.GetOrAdd(c.HotelId, id => new HotelNights(roomTypesOf(id))); int i = Slot(c.RoomType, c.Night); // A lock per hotel: changes for one hotel arrive a few times a second, and a read // holds it for microseconds, so nobody waits in practice. lock (h) { // The stream is at-least-once, and a new server replays it from a point before its // snapshot was taken, so changes it already has arrive again. One high-water mark // per hotel makes apply idempotent: anything at or below it is skipped. if (c.Position <= h.AppliedUpTo) return; h.Available[i] = (short)c.Available; h.RateCents[i] = c.RateCents; h.AppliedUpTo = c.Position; } }
// The cheapest total for `rooms` rooms of one type on every night of [from, to), or -1. // At 500 candidate hotels and 3 nights this is a few thousand array reads per search. public long CheapestStay(long hotelId, int from, int to, int rooms) { if (!hotels.TryGetValue(hotelId, out var h)) return -1; long best = -1; lock (h) { for (int t = 0; t < h.Available.Length / Horizon; t++) { long total = 0; for (int n = from; n < to && total >= 0; n++) { int i = Slot(t, n); total = h.Available[i] >= rooms ? total + (long)h.RateCents[i] * rooms : -1; } if (total >= 0 && (best < 0 || total < best)) best = total; } } // May be a second stale. That is fine: the booking transaction checks again, and // this answer only decides what the guest sees first. return best; }}// A search server's copy of all availability: applied from the change stream, read by every search.
// InventoryChange is one inventory row after a change. Position is where the commit sits in// its shard's log. A hotel lives on one shard and its changes travel on one Kafka partition,// so positions for one hotel only ever grow. Night is days since the epoch.type InventoryChange struct { HotelID int64 RoomType int Night int Available int RateCents int Position int64}
const horizon = 366 // tonight plus 365 nights on sale
// hotelNights holds plain slices per hotel, one slot per room type and night, 6 bytes a row:// 20,000 hotels fit in about 220 MB. A map of small structs would take several times that.// Nights use a ring: night N and night N + 366 share a slot, and by the time the later one// goes on sale the earlier one is in the past.type hotelNights struct { // A lock per hotel: changes for one hotel arrive a few times a second, and a read holds // it for microseconds, so nobody waits in practice. mu sync.Mutex available []int16 rateCents []int32 appliedUpTo int64 // the last log position applied for this hotel}
type AvailabilityReplica struct { mu sync.RWMutex // guards the map only, not the hotels in it hotels map[int64]*hotelNights roomTypesOf func(hotelID int64) int // from the hotel catalog}
func slot(roomType, night int) int { return roomType*horizon + ((night%horizon)+horizon)%horizon }
func (r *AvailabilityReplica) hotel(id int64, create bool) *hotelNights { r.mu.RLock() h := r.hotels[id] r.mu.RUnlock() if h != nil || !create { return h } r.mu.Lock() defer r.mu.Unlock() if h = r.hotels[id]; h == nil { n := r.roomTypesOf(id) * horizon h = &hotelNights{available: make([]int16, n), rateCents: make([]int32, n), appliedUpTo: -1} r.hotels[id] = h } return h}
func (r *AvailabilityReplica) Apply(c InventoryChange) { h := r.hotel(c.HotelID, true) h.mu.Lock() defer h.mu.Unlock() // The stream is at-least-once, and a new server replays it from a point before its // snapshot was taken, so changes it already has arrive again. One high-water mark per // hotel makes apply idempotent: anything at or below it is skipped. if c.Position <= h.appliedUpTo { return } i := slot(c.RoomType, c.Night) h.available[i] = int16(c.Available) h.rateCents[i] = int32(c.RateCents) h.appliedUpTo = c.Position}
// CheapestStay is the cheapest total for rooms rooms of one type on every night of// [from, to), or -1. At 500 candidate hotels and 3 nights it is a few thousand reads a search.func (r *AvailabilityReplica) CheapestStay(hotelID int64, from, to, rooms int) int64 { h := r.hotel(hotelID, false) if h == nil { return -1 } h.mu.Lock() defer h.mu.Unlock() best := int64(-1) for t := 0; t < len(h.available)/horizon; t++ { total := int64(0) for n := from; n < to && total >= 0; n++ { i := slot(t, n) if int(h.available[i]) < rooms { total = -1 } else { total += int64(h.rateCents[i]) * int64(rooms) } } if total >= 0 && (best < 0 || total < best) { best = total } } // May be a second stale. That is fine: the booking transaction checks again, and this // answer only decides what the guest sees first. return best}// A search server's copy of all availability: applied from the change stream, read by every search.
// One inventory row after a change. `position` is where the commit sits in its shard's log.// A hotel lives on one shard and its changes travel on one Kafka partition, so positions for// one hotel only ever grow. `night` is days since the epoch.interface InventoryChange { hotelId: number; roomType: number; night: number; available: number; rateCents: number; position: number;}
const HORIZON = 366; // tonight plus 365 nights on sale
// Typed arrays per hotel, one slot per room type and night, 6 bytes a row: 20,000 hotels fit// in about 220 MB. A Map of small objects would take many times that.// Nights use a ring: night N and night N + 366 share a slot, and by the time the later one// goes on sale the earlier one is in the past.class HotelNights { readonly available: Int16Array; readonly rateCents: Int32Array; appliedUpTo = -1; // the last log position applied for this hotel constructor(readonly roomTypes: number) { this.available = new Int16Array(roomTypes * HORIZON); this.rateCents = new Int32Array(roomTypes * HORIZON); }}
const slot = (roomType: number, night: number) => roomType * HORIZON + (((night % HORIZON) + HORIZON) % HORIZON);
// No locks: a Node process applies changes and answers searches on one thread, so a search// never sees half an update. A server runs one process per core, each with its own copy.class AvailabilityReplica { private readonly hotels = new Map<number, HotelNights>();
constructor(private readonly roomTypesOf: (hotelId: number) => number) {} // hotel catalog
apply(c: InventoryChange): void { let h = this.hotels.get(c.hotelId); if (!h) this.hotels.set(c.hotelId, (h = new HotelNights(this.roomTypesOf(c.hotelId)))); // The stream is at-least-once, and a new server replays it from a point before its // snapshot was taken, so changes it already has arrive again. One high-water mark per // hotel makes apply idempotent: anything at or below it is skipped. if (c.position <= h.appliedUpTo) return; const i = slot(c.roomType, c.night); h.available[i] = c.available; h.rateCents[i] = c.rateCents; h.appliedUpTo = c.position; }
// The cheapest total for `rooms` rooms of one type on every night of [from, to), or -1. // At 500 candidate hotels and 3 nights this is a few thousand array reads per search. cheapestStay(hotelId: number, from: number, to: number, rooms: number): number { const h = this.hotels.get(hotelId); if (!h) return -1; let best = -1; for (let t = 0; t < h.roomTypes; t++) { let total = 0; for (let n = from; n < to && total >= 0; n++) { const i = slot(t, n); total = h.available[i] >= rooms ? total + h.rateCents[i] * rooms : -1; } if (total >= 0 && (best < 0 || total < best)) best = total; } // May be a second stale. That is fine: the booking transaction checks again, and this // answer only decides what the guest sees first. return best; }}Trace a request: Search from memory
Search from memory: step 1 of 4
Guests to API gateway
The same request, served by the nearest region.
- Guests → API gateway15 ms
- API gateway → Search servers1 ms
- Search servers → Hotel index3 ms
- Response
| Split by | Booking on one database | Spreads load | Problem |
|---|---|---|---|
| Hotel | Yes | Yes; the largest hotel is tiny | A guest’s bookings span databases: a second table |
| Hotel + room type | Yes, for one-type bookings | Slightly better | A change from a double to a suite spans databases |
| Night | No: a stay spans nights | No: next weekend is one hot spot | Every booking is a distributed transaction |
Each database has a home region near its hotels. A distant guest pays one cross-region round trip, unnoticed next to the card network. Two regions never take writes for one hotel.
At 100 times, bookings reach about 2,500 a second at peak: more databases, nothing new. In-memory availability reaches 22 GB, near the 30 GB point where it’s split by geography. Most search traffic at that scale is machines, so quotas per partner and edge caching matter more than any server change.
When things break
Section titled “When things break”The channel manager to the agencies is down for an hour. What do guests and agencies see?
Whether our own sales carry on, and agency counts are put right afterwards.
The channel manager goes down. Our website and app keep selling, because booking needs only the inventory database. Agencies keep selling too, from their last copy of our counts, which gets staler every minute. Their bookings wait on their side and arrive when we are back. Each one is recorded by the agency’s reference, so a resend is harmless. Then the channel manager pushes fresh, absolute counts to every agency, which corrects their copies in one step. The cost of the hour is a few more oversells on nearly full nights, which the oversell check flags for the hotel.
An inventory database fails. A failure should delay bookings, never corrupt inventory, and search should survive almost anything. Each primary has a standby in another zone that confirms every commit before the primary answers, so no confirmed booking is lost. The standby takes over in about 30 seconds. Meanwhile search keeps working from its copy, and guests and agencies retry with the same keys, so nothing is booked twice.
Click any box to break it and see what guests would notice:
Break it: what fails, who notices, what users see
Failure: click any box to break it
Inventory shard primary goes down
What breaks
Bookings, cancellations and rate updates for the hotels on that shard fail.
How we notice
Write error rate and failed health checks on the primary.
What users see
For about 30 seconds, booking those hotels fails with a retry message.
Search keeps working from memory.
Confirmed bookings are not lost: the standby has every commit.
What we do
Promote the synchronous standby.
Clients retry with the same idempotency key.
Agencies retry deliveries until acknowledged, so none is lost.
The quiet failure is worse: a search copy that silently stops updating. Search keeps answering, just wrongly, and guests see more “sold out” after clicking. So the lag of the change stream is an alert, and so is the share of bookings that fail with “sold out”.
Go deeperEvery failure, and what users see1 min
| Part | Failure | What users see | What we do |
|---|---|---|---|
| Inventory primary | Down | Booking its hotels fails ~30 s; search continues | Promote the standby; clients and agencies retry with keys |
| Search server | Crash | Nothing | Load a snapshot, replay the stream, then serve |
| Hotel index | Down | Some areas unsearchable; booking fine | Replicas; rebuild from change data capture |
| Payment provider | Slow or down | Prepaid fails; pay-at-hotel still books | Same-key retries, a second provider |
| Kafka | Down | Bookings confirm; search and agencies go stale | The outbox waits; close out nearly full hotels on agencies |
| Workers | Stopped | Expired holds keep rooms out of sale | Jobs are safe to repeat and catch up on restart |
| Channel manager | Down | Agencies sell on frozen counts | Restart; full push for changed hotels |
| One agency’s API | Down | That agency shows stale availability | Retry; only the latest state is sent |
| A region | Outage | Search fails over; its hotels’ booking pauses | Promote replicas in another region |
Under stress, keep confirmed bookings correct first, then booking, then agency sync, then search freshness.
| Operation | Guarantee | What users can see |
|---|---|---|
| Direct booking | Exact per night: ordered locks, conditional update | Never past the limit |
| Agency booking | Recorded once in effect; may pass the limit | A rare oversold night, alerted |
| Hold expiry | Within 30 s, exactly once | A room shown sold for up to 30 s after a hold lapsed |
| Search | Usually under 2 s behind | A hotel shown available that isn’t: the booking says 409 |
Go deeperRunning it: alerts, abuse and proof1 min
Wake someone up when:
- booking success drops below 99.9% (counting “sold out” and “declined” as success)
- booking takes more than 2 seconds at p99
- the change stream or agency pushes lag more than 30 seconds
- oversold nights per hour jump for any agency
- holds sit past their expiry
Abuse. Most searches are machines: quotas per partner key and short edge caching. Holds never paid: limit open holds per account and card. Card testing through pay-at-hotel bookings: limit per account, device and card.
Proving no oversell. A nightly job recomputes reserved for every row from the held and
confirmed stays covering that night. A mismatch is a bug, repaired from the reservations, which are
the truth. Oversold nights are then split by cause: agency bookings (expected) or direct ones (a
bug).
Wrap up
Section titled “Wrap up”The whole design
Section titled “The whole design”Here is everything we built, in the order we built it. Use the buttons to step through it.
The full design, stage by stage
Stage 6 of 6
Search from memory
Why this stage
At 10 times the platform, 250,000 searches/s each look up 100 hotels in the availability cache: 25 million lookups/s, about six times what a 20 primary Redis cluster serves.
What we add
Every search server keeps all availability in memory (about 220 MB, 2.2 GB at 10 times), fed by the inventory change stream, and caches location queries, which do not depend on dates.
The search tier runs in every region.
What it costs
Each server is a replica that must load a snapshot at start and apply changes in order.
At the 1M stress mark the hotel index needs more replicas and payments need a second provider.
At 25k searches per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One app server handles about 2,000 to 5,000 requests a second.
- One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
- The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.
How big the shared systems are
- A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
- An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
- The payment provider allows about 1,000 calls a second.
What the writes are
- Inventory writes are 9% of the search rate.
- About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
- From stage 6, search servers cache location queries, so only 30% reach the index.
The decisions, and what each one costs
Section titled “The decisions, and what each one costs”| Decision | We chose | We gave up | We’d choose differently if |
|---|---|---|---|
| Inventory | Counts per room type per night | Booking a specific room | Each unit were unique (villas): a calendar each |
| Concurrency | Nights locked in date order, conditional update | One hotel-wide lock’s simplicity | One hotel had extreme contention |
| Prepaid payments | Hold, authorize, confirm; a sweeper frees holds | Rooms out of sale during abandoned checkouts | Contention were zero (authorize first) |
| Overbooking | An allowance per type and night, as data | A promise that nobody is ever walked | The hotel sold only non-refundable rates |
| Agency sync | Shared pool, absolute pushes, unconditional delivery | No oversells across channels | Agencies needed zero risk (fixed allotments) |
| Search | A copy: Redis first, then in memory on every server | Always-exact availability | Searches per booking were low (read replicas) |
| Split | By hotel, one home region each | Queries across hotels | One hotel outgrew a primary, which none does |
How to spend 45 minutes
Section titled “How to spend 45 minutes”- 0 to 4. Room or type, searches per booking, payment types, agencies.
- 4 to 9. 8 bookings and 25,000 searches a second, 220 MB of availability, writes mostly rate updates.
- 9 to 20. Build it: counts per night, the stay transaction, search on a copy, holds. A mid-level answer can stop here.
- 20 to 29. The stress tests: the date-range lock, then retries and late payments.
- 29 to 40. The agencies and the oversell race, changes, then ten times the hotels if asked.
- 40 to 45. A primary failing, then your own questions.
Never skip the conditional update over every night, the key in the same transaction, or why search may be stale.
Follow-up questions
Section titled “Follow-up questions”Interviewer asksHow would you sell a specific room, like a named villa?
Each unit becomes its own room type with a total of 1. The table and transaction don’t change. What changes is fragmentation: short bookings leave one-night gaps nobody books, so holiday rental platforms add minimum stays or cheaper gap nights.
Interviewer asksSearch must be exact, or guests will book rooms that are gone. Agree?
No. They can’t book them: the booking transaction refuses. Staleness costs a “sold out” after a click, so measure how often that happens. Exact search would read every primary 25,000 times a second, risking the booking path for a slightly better click rate.
Further reading
Section titled “Further reading”- Hotel booking on this site: the same inventory at the object level, with changes, cancellations and check-in.
- Ticket booking, payment system and partitioning on this site.
- Explicit locking, PostgreSQL docs: row locks, and why acquiring locks in a consistent order is the defense against deadlocks.
- Idempotent requests, Stripe: the idempotency key contract this design copies for bookings.
- Transactional outbox, microservices.io: committing a state change and its event together.
The designbeyondcode digest, every Saturday
New deep dives, interview tips and site news in one short email. Free. Leave with one click.
We never sell or share your email. We only use it to send this newsletter.