Skip to content

System DesignWorked problems

Design a hotel reservation system

Design a hotel reservation system. Start with one service and one database, grow it with counts per night, one transaction for the whole stay, search on a copy and a hold around payment, then stress it with date ranges, late payments, agencies and changed stays, with interview prompts and code.

Mid-level26 min core · 46 min full depth

“Design a hotel reservation system, like the one behind Marriott.com or Booking.com. Guests search a city for some dates, pick a room and book it.”

A platform for 20,000 hotels takes only a few bookings a second, so the difficulty isn’t volume. It hides in three places. A stay is several nights that must all be taken, or none. Search runs a thousand times for every booking. And the same rooms sell on other websites at the same time, from those websites’ own copy of our counts.

So this page starts with one service and one database, and lets the numbers grow it. Then it tests the result against date ranges, late payments, agencies and changed stays.

If you only have a minute, this is the design. The rest of the page explains each part.

  1. Count rooms per room type per night. A guest books “a double”, not room 101. The hotel picks the room at check-in. So inventory is a number: doubles left on the 5th.
  2. Book a stay in one transaction. Lock the nights in date order. Add one to each night only where a room is left. Roll back unless every night changed.
  3. Let search read a stale copy. Search is 1,000 times the bookings, so it never touches the inventory tables. A room shown that just sold is caught by the booking.
  4. Hold the room while the card is checked. The card network never runs inside a transaction. A worker gives back holds that are never paid.
  5. Sync the agencies, and expect them to be late. Push counts within a second. Record their bookings even when a night is full, because their guest is already confirmed. A small overbooking allowance absorbs the rare collision.

A few questions decide what a booking even is.

Interviewerminute 0

Design a hotel reservation system.

What they are testing

Whether you ask whether guests book a room or a room type, how many searches per booking, and about agencies.

These four questions matter most:

  • How many hotels? 20,000, about 100 rooms each: 2 million rooms.
  • Does a guest book a room or a type? A room type for some nights. The room is picked at check-in.
  • How many searches per booking? About 1,000, counting partner and price-comparison traffic.
  • Do other channels sell the same rooms? Yes. About 40% sell through online travel agencies, from one shared pool.

From these answers we make five promises:

  • Never oversold through us. Take any hotel, room type and night. Rooms sold through our own channels never pass the rooms in service plus the hotel’s overbooking allowance. This is a hard promise, under any retry or failure.
  • Booked once. A retried booking never creates a second reservation or a second charge.
  • Always up. Search works 99.99% of the time, booking 99.95%.
  • Fast. Search under 500 ms. Booking under 2 seconds with a card, under 300 ms when paying at the hotel.
  • Fresh enough. Search may be a few seconds old, a minute at worst. Agencies get changes within about 5 seconds.

We leave out four things. Assigning physical rooms (the hotel’s own system does that at check-in). Dynamic pricing (we store the prices hotels send us). Reviews and photos. And the payment provider itself.

Go deeperMore questions, and what each changes1 min
Question We assume If the answer were different
One hotel only? 20,000 hotels One hotel fits one server with a lock in memory: the LLD answer
Pay now or at the hotel? Both: prepaid rates, and pay-at-hotel with a card guarantee Pay-at-hotel only removes holds and payment calls
Can a hotel sell more rooms than it has? Yes, a small allowance per type and night Without it, every agency race is an oversell handled by hand
Fixed allotments per agency? No, one shared pool Allotments remove the race but leave rooms unsold in one channel

Now we work out the numbers. Bookings and searches differ by a factor of a thousand, so we size them apart.

Assumptions
Hotels on the platform
20,000
Rooms per hotel
100
Room types per hotel
5
Occupancy
70%
Nights per stay
3
Bookings later cancelled
35%
Searches per booking
1,000 to 1
Peak over average
3×
Hotels checked per search
100
Rate updates per room type a day
20
Bookings8.3/s720,000 a day

2 million rooms at 70% full, 3 nights a stay, and 35% of bookings later cancelled. About 25 a second at peak.

So we bookings need no distributed database.

Search8,300/s25,000/s at peak

1,000 searches per booking. A city search checks about 100 hotels × 3 nights × 5 room types.

So we search never reads the inventory tables. At 1,500 rows a search, one database answers about 2,000 searches a second.

Inventory37M rows220 MB of availability

20,000 hotels × 5 room types × 365 nights. A count and a lowest rate per row is 6 bytes.

So we we split for writes, never for size. All availability fits in one server's memory.

Writes2,300/srow writes at peak

The heavy writes aren’t bookings. Hotels’ pricing systems change rates about 20 times a day per room type, each covering 30 nights. 90% of row writes are rate and availability updates.

So we one primary carries it, if updates are batched per hotel.

Try the calculator: double the searches per booking, and the cache passes what a 20-node Redis cluster serves. Multiply the hotels by ten, and row writes reach 23,000 a second, the trigger for splitting by hotel.

Change an assumption
Bookings a day, cancellations included
720,000
Bookings per second (average)
8.3/s
A handful a second: write volume from bookings alone is never the problem.
Searches per second at peak
25,000/s
Past the 2,000 searches/s one database can answer: search reads a cache, never the inventory tables.
Cache lookups per second at peak (one per hotel checked)
2,500,000/s
A Redis cluster of about a dozen primaries serves this.
Inventory rows (one per room type per night, 365 nights)
37 million
Small: one database holds all of it, so sharding is for writes, not size.
Availability in memory (6 bytes a row)
220 MB
Fits in every search server’s memory.
Inventory row writes per second at peak
2,300/s
One primary with a synchronous standby takes this.
Reservations stored over 5 years (1 KB each)
1.3 TB
Fits one primary; archive stays more than a year old.

We start with the simplest design that works. Then we let the numbers break it, one at a time. Each step fixes only what broke, adds only the boxes that fix needs, and says what the fix costs. Throughout, follow one scenario: a guest searches Paris, picks a double for three nights, and pays.

The smallest version is one booking service and one Postgres database. Search queries the hotel and room tables directly, and a booking marks a room as taken for each night of the stay.

Hotel reservation · architecture

Stage 1: One service, one database

GuestsBooking servicereserve, cancel500% busyPostgresinventory, bookings1250% busy
Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage 1 of 6

One service, one database

Why this stage

Nothing yet.

This is the smallest design that never sells a night twice.

What we add

A booking service and a Postgres table with one row per hotel, room type and night.

A booking is one transaction that adds to every night of the stay, or to none.

What it costs

Search scans inventory rows in the same database the bookings write to, and one server does everything.

At 25k searches per second, Inventory shards is 1250% busy. Stage 2 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
The numbers behind the busy bars

What one machine can do

  • One app server handles about 2,000 to 5,000 requests a second.
  • One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
  • The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.

How big the shared systems are

  • A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
  • An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
  • The payment provider allows about 1,000 calls a second.

What the writes are

  • Inventory writes are 9% of the search rate.
  • About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
  • From stage 6, search servers cache location queries, so only 30% reach the index.

It works for a few hotels. But two things go wrong as it grows. Search runs about a thousand times for every booking, against the same tables booking writes, so search load slows booking. And we haven’t yet said what “a room is free” means for a stay of several nights.

The inventory table has one row per hotel, room type and night. Each row holds the rooms in service, the overbooking allowance, and how many are reserved (held or confirmed, from every channel). Prices live in a separate table, because they change far more often, and several rate plans (non-refundable, flexible) sell the same doubles.

Why counts, not a calendar per room? Per-room calendars refuse guests while some double is free every night, just not the same double. The hotel assigns rooms at check-in, so it doesn’t need us to.

A booking is one transaction. It inserts the reservation, then adds one to reserved on each night of the stay, only where a room is left. If any night is full, fewer rows change than there are nights, and the transaction rolls back everything. The check and the write are one step, so two guests can never both take the last room.

The race this must survive is two overlapping stays competing for the last double on a shared night. Both lock their nights in date order, so they queue instead of each holding a night the other needs:

Two overlapping stays, one double left on the 5th
Ben, 5th to 8thPostgresBooking serviceAna, 3rd to 6thBen, 5th to 8thPostgresBooking serviceAna, 3rd to 6thBen gets the 5th, then the 6th and 7threserve a doublereserve a doubleAna locks the 3rd, 4th and 5th, in date orderBen locks the 5th first, and waitsAna adds 1 where a room is left: 3 of 3 rows, commitBen adds 1 where a room is left: 2 of 3 rowsBen rolls back, nothing taken201 confirmed409 SOLD_OUT, the 5thBen, 5th to 8thPostgresBooking serviceAna, 3rd to 6thBen, 5th to 8thPostgresBooking serviceAna, 3rd to 6thBen gets the 5th, then the 6th and 7threserve a doublereserve a doubleAna locks the 3rd, 4th and 5th, in date orderBen locks the 5th first, and waitsAna adds 1 where a room is left: 3 of 3 rows, commitBen adds 1 where a room is left: 2 of 3 rowsBen rolls back, nothing taken201 confirmed409 SOLD_OUT, the 5th
Scroll or pinch to zoom, drag to pan
  • Color key
  • Service
  • Database

Search needs location and filter queries, and availability for many hotels at once. Neither needs to be exact, because the booking transaction checks again. So search never touches the inventory tables.

An Elasticsearch index holds the facts that rarely change: location, stars, amenities. A Redis cache holds one hash per hotel. For each room type and night, it stores the rooms left and the lowest rate. A search asks the index for the best 100 matching hotels. It reads their nights from Redis in one round trip. It drops hotels with no type free on every night, sums the rates, and returns 25. Booking servers update the hotel’s hash after every change.

Hotel reservation · architecture

Stage 2: Search apart from booking

GuestsAPI gatewayauth, limitsnewSearch serversfilter, price, ranknew17 servers, 59% busyBooking serversreserve, cancel4 servers, 59% busyHotel indexElasticsearchnew63% busyAvailabilityRedis, per nightnew63% busyPostgres primary+ sync standby23% busy
Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional
Stage

Stage 2 of 6

Search apart from booking

Why this stage

At the 25,000 searches/s peak, each search checks about 100 hotels over 3 nights: far past the 2,000 searches/s one database can answer.

What we add

An API gateway, a fleet of search servers, a hotel index on Elasticsearch for location and filters, and an availability cache in Redis, written after every commit.

Booking servers become a stateless fleet.

What it costs

Search shows availability a few seconds old.

The booking transaction, not the cache, decides whether a room is left.

At 25k searches per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
The numbers behind the busy bars

What one machine can do

  • One app server handles about 2,000 to 5,000 requests a second.
  • One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
  • The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.

How big the shared systems are

  • A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
  • An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
  • The payment provider allows about 1,000 calls a second.

What the writes are

  • Inventory writes are 9% of the search rate.
  • About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
  • From stage 6, search servers cache location queries, so only 30% reach the index.

Follow one search:

Hotel reservation · architecture

Trace a request: Search Paris for 3 nights

GuestsAPI gatewayauth, limitsSearch serversfilter, price, rankBooking serversreserve, cancelHotel indexElasticsearchAvailabilityRedis, per nightPostgres primary+ sync standby
Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Search Paris for 3 nights: step 1 of 5

Guests to API gateway

GET /search with a city, dates and guests. The gateway checks the session or API key and the caller’s rate limit; metasearch partners get their own quota.

15 ms of a 500 ms budget
  1. Guests → API gateway15 ms
  2. API gateway → Search servers1 ms
  3. Search servers → Hotel index25 ms
  4. Search servers → Availability6 ms
  5. Response
Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
Go deeperKeeping the search copy in step1 min

The booking server updates the hotel’s Redis hash right after each commit. If it crashes between the commit and that update, the copy is wrong until something fixes it. So a background job rebuilds each hotel’s hash from Postgres every few minutes, and at once for any hotel that just changed.

A wrong copy can only be wrong in two ways, and they cost different things. Say it shows a room that is gone. The guest clicks, the booking transaction checks the real count, and the guest sees “sold out”. That is annoying, but nothing is sold twice. Now say it shows “sold out” for a room that is free. The hotel loses a sale, and nobody notices. That second case is why the rebuild job exists. It’s also why we watch how often searches show a hotel as full when Postgres says it isn’t.

A card check takes 300 ms to a few seconds, longer with a bank’s identity challenge, so it never runs inside the inventory transaction. A prepaid booking runs in three short steps:

  1. Hold. The booking transaction inserts the reservation as held, with a 15-minute expiry. Held rooms count as reserved, so the room is safe while the guest pays.
  2. Authorize the card, with the reservation ID as the provider’s idempotency key. No transaction is open.
  3. Confirm, only if the reservation is still held, and write an outbox row in the same transaction. A relay sends it to Kafka, and workers take the money, send the confirmation and update the cache.

Pay-at-hotel bookings skip the card call and confirm in the first transaction.

Hotel reservation · architecture

Stage 3: Holds and payments

GuestsAPI gatewayauth, limitsSearch serversfilter, price, rank17 servers, 59% busyBooking servers+ holds, payment4 servers, 59% busyHotel indexElasticsearch63% busyAvailabilityRedis, per night63% busyPaymentsexternal providernew5% busyKafkabooking eventsnew<1% busyPostgres primary+ sync standby23% busyWorkerscapture, expirynew2 servers, 6% busy
Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional
Stage

Stage 3 of 6

Holds and payments

Why this stage

Prepaid rates must take the money before the booking is final, and a card authorization takes about a second, too long to keep inventory rows locked.

What we add

A payment provider called outside any transaction, a 15 minute hold that already counts against inventory, an outbox feeding Kafka, and workers that capture payments, release expired holds and keep the cache fresh.

What it costs

A held room is unsellable until it is paid or released, and holds on counts cannot expire by themselves: a sweeper must give them back.

At 25k searches per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
The numbers behind the busy bars

What one machine can do

  • One app server handles about 2,000 to 5,000 requests a second.
  • One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
  • The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.

How big the shared systems are

  • A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
  • An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
  • The payment provider allows about 1,000 calls a second.

What the writes are

  • Inventory writes are 9% of the search rate.
  • About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
  • From stage 6, search servers cache location queries, so only 30% reach the index.

A count of 37 doesn’t say which of its 37 are holds. So unlike a seat row, an expired hold can’t simply be ignored. A sweeper on each database runs every 30 seconds. It marks expired holds and gives their nights back, in one statement. So a hold is released exactly once.

This is enough for a mid-level answer. Counts per night, one transaction for the stay, search on a copy, and a hold around the card. The stress tests are what make it a senior answer.

Go deeperThe sweeper, in SQL1 min
WITH expired AS (
UPDATE reservations SET status = 'EXPIRED'
WHERE status = 'HELD' AND hold_expires_at < now()
RETURNING hotel_id, room_type_id, check_in, check_out, rooms
), give_back AS ( -- one row per night: two holds on the 5th must give back 2
SELECT hotel_id, room_type_id, n::date AS night, SUM(rooms) AS rooms
FROM expired, generate_series(check_in, check_out - 1, interval '1 day') AS n
GROUP BY 1, 2, 3
)
UPDATE room_type_inventory i
SET reserved = i.reserved - g.rooms
FROM give_back g
WHERE i.hotel_id = g.hotel_id AND i.room_type_id = g.room_type_id AND i.night = g.night;

The status change and the give-back are one statement, so a hold is released exactly once. The give_back step matters. An update that joins one inventory row to two expired holds would apply only one of them. The real version takes a few hundred holds per run. It locks the nights in date order first, like a booking. Abandoned holds keep rooms out of sale for up to 15 minutes and 30 seconds. That’s the price of keeping the room safe through a slow bank.

Go deeperWhy the hold lasts 15 minutes1 min

Most card payments finish in a few seconds. But some banks ask the guest to confirm in their banking app, and that can take a few minutes while they find their phone. A hold that ends too soon would lose the room while the guest is still paying.

Fifteen minutes covers nearly every real payment. The price is that an abandoned checkout keeps a room out of sale for up to 15 minutes and 30 seconds. On a quiet night that costs nothing. On the last few rooms of a busy weekend it can turn a guest away. Some platforms shorten the hold when only a few rooms are left, and show the guest a countdown.

Go deeperThe API2 min
GET /v1/search?area=paris&checkIn=2027-05-03&checkOut=2027-05-06&guests=2&rooms=1
200 OK { "hotels": [ { "id": 81231, "fromPrice": { "amount": 41700, "currency": "EUR" },
"fewLeft": true } ], "asOf": "2027-03-01T10:04:12Z" }
POST /v1/quotes
{ "hotelId": 81231, "roomTypeId": 12, "ratePlanId": 3, "checkIn": "2027-05-03", "checkOut": "2027-05-06" }
201 Created { "quoteId": "q_9f2...", "total": { "amount": 41700 }, "expiresAt": "..." }
POST /v1/reservations
Idempotency-Key: 4c1e7a90-...
{ "quoteId": "q_9f2...", "guest": { ... }, "payment": { "token": "pm_tok_..." } }
201 Created { "reservationId": "R81231-7Q2XK", "status": "CONFIRMED" }
409 Conflict { "error": "SOLD_OUT", "nights": ["2027-05-05"] }

checkOut is the morning the guest leaves, so this stay is the nights of the 3rd, 4th and 5th. A quote is priced from the database when the booking page opens. It fixes the price for 30 minutes. It reserves nothing, so a guest can still lose the last room while typing a card number. That’s cheaper than holding rooms for every page view.

A retry with the same key and body returns the first response. 422 means a key reused with a different body. The reservation ID starts with the hotel ID, so later calls find the right database without a lookup. Agencies and hotel systems use the channel manager instead, often with the travel industry’s standard XML messages.

Go deeperThe tables1 min
CREATE TABLE room_type_inventory (
hotel_id BIGINT NOT NULL,
room_type_id BIGINT NOT NULL,
night DATE NOT NULL,
total_rooms SMALLINT NOT NULL, -- rooms of this type in service that night
overbook_limit SMALLINT NOT NULL DEFAULT 0, -- extra rooms the hotel is willing to sell
reserved SMALLINT NOT NULL DEFAULT 0, -- held plus confirmed, every channel
PRIMARY KEY (hotel_id, room_type_id, night)
);
CREATE TABLE reservations (
hotel_id BIGINT NOT NULL,
reservation_id UUID NOT NULL,
room_type_id BIGINT NOT NULL,
check_in DATE NOT NULL,
check_out DATE NOT NULL, -- nights are check_in up to, not including, check_out
rooms SMALLINT NOT NULL,
status TEXT NOT NULL, -- HELD | CONFIRMED | EXPIRED | CANCELLED | NO_SHOW
hold_expires_at TIMESTAMPTZ,
idem_key TEXT NOT NULL, -- the client's key, or 'agency:their reference'
PRIMARY KEY (hotel_id, reservation_id),
UNIQUE (hotel_id, idem_key)
);

Every table starts with hotel_id, so splitting by hotel later needs no schema change.

Where search could read availability At 25,000 searches a second Verdict
The primary 12 times what it can answer, and it slows bookings Wrong at this ratio
Read replicas About 13 replicas, each still scanning Works, at a high cost per search
A Redis hash per hotel 2.5 million lookups a second, about a dozen nodes The pick for now
Inside the search index Every booking and rate change reindexes a hotel The index is built for reads

The design is complete. Now we test it where hotel booking is hard. Two overlapping stays at once. Retries, and a payment that outlasts its hold. The agencies that sell the same rooms. And a guest who changes their dates. The agencies add one box.

Interviewerminute 20

Show me how a three-night booking avoids double-selling when two overlapping stays arrive at once.

What they are testing

Whether the check and the write are one step, all nights succeed or none do, and lock order is clear.

Approach Correct? Cost here
Read the counts, then write No: two readers both see a room None, which is why it ships by mistake
SERIALIZABLE isolation Yes Aborts under contention; every caller retries
A version number per night Yes Retries on a popular weekend
Lock nights in order + conditional update Yes Two round trips, locks held a few ms
Lock the whole hotel Yes Blocks unrelated room types and stays

The pick locks only the nights it takes, in a fixed order, with no application code between the check and the write. It’s correct under Postgres’s default isolation: a writer waiting on a row checks its condition again against the committed row.

ReservationService.javakey_reused.pyReservationService.csreservation_service.goreservation-service.ts
// Reserving a stay: one transaction takes every night of the range, or none of them.
interface Tx {
int execute(String sql, Object... args); // returns the number of rows changed
int countRows(String sql, Object... args); // runs a SELECT, returns its row count
Reservation findByKey(long hotelId, String idemKey);
}
interface Db {
<T> T inTransaction(Function<Tx, T> work); // commits on return, rolls back on throw
}
record StayRequest(long hotelId, long roomTypeId, LocalDate checkIn, LocalDate checkOut,
int rooms, boolean prepaid, String idemKey, String requestHash) {}
record Reservation(UUID id, String status, String requestHash) {}
class SoldOutException extends RuntimeException {
SoldOutException(String why) { super(why); }
}
class KeyReusedException extends RuntimeException {
KeyReusedException() { super("idempotency key reused for a different request"); }
}
class ReservationService {
// The reservation row goes in first, unique on (hotel_id, idem_key). The hotel is part of
// the key because a unique index is only enforced inside one shard. A concurrent retry
// blocks on this index until we commit, then finds our row instead of booking again.
// Prepaid stays start HELD with a 15 minute expiry; pay-at-hotel stays are final at once.
static final String INSERT_RESERVATION = """
INSERT INTO reservations (reservation_id, hotel_id, room_type_id, check_in, check_out,
rooms, status, hold_expires_at, idem_key, request_hash)
VALUES (?, ?, ?, ?, ?, ?, ?, CASE WHEN ? THEN now() + interval '15 minutes' END, ?, ?)
ON CONFLICT (hotel_id, idem_key) DO NOTHING""";
// Lock the nights in date order. Two overlapping stays (the 3rd to the 6th and the 5th to
// the 8th) then queue on the 5th instead of each locking part of the other's range and
// deadlocking. A plain multi-row UPDATE gives no guarantee about the order it locks rows in.
static final String LOCK_NIGHTS = """
SELECT night FROM room_type_inventory
WHERE hotel_id = ? AND room_type_id = ? AND night >= ? AND night < ?
ORDER BY night FOR UPDATE""";
// Take a room on every night, only where one is left. The limit includes the hotel's
// overbooking allowance for that night, so the policy is data, not code. No SELECT and
// compare in the application: the check and the write are one statement.
static final String TAKE_NIGHTS = """
UPDATE room_type_inventory
SET reserved = reserved + ?, version = version + 1
WHERE hotel_id = ? AND room_type_id = ? AND night >= ? AND night < ?
AND reserved + ? <= total_rooms + overbook_limit""";
private final Db db;
ReservationService(Db db) { this.db = db; }
Reservation reserve(StayRequest r) {
// Half-open range: check-out day is not a night, so back-to-back stays share nothing.
int nights = (int) ChronoUnit.DAYS.between(r.checkIn(), r.checkOut());
UUID id = UUID.randomUUID();
String status = r.prepaid() ? "HELD" : "CONFIRMED";
return db.inTransaction(tx -> {
if (tx.execute(INSERT_RESERVATION, id, r.hotelId(), r.roomTypeId(), r.checkIn(),
r.checkOut(), r.rooms(), status, r.prepaid(), r.idemKey(), r.requestHash()) == 0) {
// A retry. Same key, same request: return the first answer. Same key with a
// different body is a client bug, and silently returning the old booking would
// hide it.
Reservation first = tx.findByKey(r.hotelId(), r.idemKey());
if (!first.requestHash().equals(r.requestHash())) throw new KeyReusedException();
return first;
}
// A missing row means the night is not on sale (beyond the horizon, or closed).
if (tx.countRows(LOCK_NIGHTS, r.hotelId(), r.roomTypeId(), r.checkIn(), r.checkOut()) != nights)
throw new SoldOutException("not on sale for every night");
// All or nothing: if one night is full, the update changes fewer rows than there
// are nights, and the throw rolls back the nights it did take and the reservation
// row. A sold-out answer is not stored, so a retry later may succeed.
if (tx.execute(TAKE_NIGHTS, r.rooms(), r.hotelId(), r.roomTypeId(), r.checkIn(),
r.checkOut(), r.rooms()) != nights)
throw new SoldOutException("no room left on at least one night");
return new Reservation(id, status, r.requestHash());
});
}
}

Why lock order matters. If Ana and Ben locked nights in whatever order the database found them, each could hold a night the other needs. Postgres detects that deadlock after a second and aborts one. A single multi-row update makes no promise about the order it locks rows, so an explicit SELECT ... ORDER BY night FOR UPDATE is what turns date order into a guarantee.

How much contention? Transactions in flight equal the booking rate times how long each holds its locks (Little’s law). That’s 25 a second × 5 ms, or 0.125 across the whole platform. Even a flash sale of 100 bookings a second on one room type is 0.5. Row locks are cheap here, and no waiting room like ticket booking’s is needed.

Retries, and a payment that outlasts its hold

Section titled “Retries, and a payment that outlasts its hold”

The guest clicks Book, the request times out, and they click again. Then a slow bank approves a payment after the hold expired.

A retried click. The page creates the idempotency key when it loads. The key lives in the reservation row, unique per hotel, inserted in the same transaction as the nights. A retry after the commit finds the row. A retry during it waits on the unique index, then finds it. A crash before commit leaves no row, so the retry books normally.

A payment that outlasts its hold. Confirming says “confirmed, only if still held”. The sweeper says “expired, only if still held and past its time”. They lock the same row, so exactly one wins. If the sweeper won, the nights went back on sale. The booking service then tries once more: expired to confirmed, only if still expired, plus the same ordered, conditional update over the nights. At hotel contention levels, the rooms are almost always still there. If a night is now full, it voids the authorization: no money moved.

A slow bank, an expired hold, and a second try
Hold sweeperPayment providerPostgresBookingGuestHold sweeperPayment providerPostgresBookingGuesta 3-D Secure challenge runs past 10:15POST /reservations, prepaidHELD, 3 nights taken, expires 10:15authorize, key = reservation idHELD and expired, so EXPIRED, nights given backapprovedCONFIRMED where still HELD: 0 rowstake the 3 nights again, then CONFIRMED201 confirmedHold sweeperPayment providerPostgresBookingGuestHold sweeperPayment providerPostgresBookingGuesta 3-D Secure challenge runs past 10:15POST /reservations, prepaidHELD, 3 nights taken, expires 10:15authorize, key = reservation idHELD and expired, so EXPIRED, nights given backapprovedCONFIRMED where still HELD: 0 rowstake the 3 nights again, then CONFIRMED201 confirmed
Scroll or pinch to zoom, drag to pan
  • Color key
  • Service
  • Database
Go deeperA key at every step1 min
Step Key What a repeat does
Guest to booking service Idempotency-Key, unique per hotel Returns the first response
Booking service to provider Reservation ID Returns the first authorization, never a second
Agency booking delivery The agency and its booking reference Records nothing new, acknowledges again
Outbox to workers Event ID; each job keyed by reservation Capture, email and fee jobs run once in effect
Channel manager to agency None needed: it sends absolute counts Sets the same count again

“Exactly-once booking” is a phrase to avoid. Networks deliver at most once or at least once. What the system gives is an exactly-once effect: every call may repeat, and a key at each step makes the repeat harmless. Naming the keys is the answer.

Situation Outcome
Approved, hold still valid Confirmed; money taken from the outbox
Approved, hold expired, rooms still free Nights taken again, confirmed
Approved, hold expired, a night now full Void the authorization, 409 SOLD_OUT
Server crash after authorization The guest’s retry re-authorizes under the same key, and confirms

Agencies sell the same rooms we do. What happens when one sells the last double on the 5th, two seconds after our website sold it, and how do we keep that rare?

About 40% of rooms sell through online travel agencies. Most keep their own copy of each hotel’s availability, sell against it, confirm the guest, then send us the booking. A channel manager does three things:

  • Pushes availability and rates to every agency selling a hotel.
  • Records bookings the agencies send, keyed by the agency’s reference. These add to reserved without the limit check, because the guest is already confirmed.
  • Takes rate and room count updates from hotels’ own systems, batched per hotel.
Hotel reservation · architecture

Stage 4: Channel manager

GuestsPartnersOTAs, hotel PMSnewAPI gatewayauth, limitsSearch serversfilter, price, rank17 servers, 59% busyBooking servers+ holds, payment3 servers, 45% busyChannel managerARI out, bookings innew2 servers, 34% busyHotel indexElasticsearch63% busyAvailabilityRedis, per night63% busyPaymentsexternal provider5% busyKafka+ inventory changes<1% busyPostgres primary+ sync standby23% busyWorkerscapture, expiry, fees2 servers, 6% busy
Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional
Stage

Stage 4 of 6

Channel manager

Why this stage

About 40% of rooms are sold by online travel agencies from their own copy of our availability, and hotels push rates and room counts from their own systems.

What we add

A channel manager that pushes availability and rates to every agency within seconds of a change, and records bookings they deliver, which they have already confirmed to the guest.

What it costs

An agency can sell the last room on a copy a few seconds old.

Overbooking allowances and alerts absorb it, because we cannot refuse a booking the agency already confirmed.

At 25k searches per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
The numbers behind the busy bars

What one machine can do

  • One app server handles about 2,000 to 5,000 requests a second.
  • One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
  • The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.

How big the shared systems are

  • A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
  • An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
  • The payment provider allows about 1,000 calls a second.

What the writes are

  • Inventory writes are 9% of the search rate.
  • About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
  • From stage 6, search servers cache location queries, so only 30% reach the index.

Follow one booking all the way to the agencies:

Hotel reservation · architecture

Trace a request: Book 3 nights, prepaid

GuestsPartnersOTAs, hotel PMSAPI gatewayauth, limitsSearch serversfilter, price, rankBooking servers+ holds, paymentChannel managerARI out, bookings inHotel indexElasticsearchAvailabilityRedis, per nightPaymentsexternal providerKafka+ inventory changesPostgres primary+ sync standbyWorkerscapture, expiry, fees
Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Book 3 nights, prepaid: step 1 of 8

Guests to API gateway

POST /reservations with the hotel, room type, dates, rate plan, a quote id and an Idempotency-Key generated when the booking page loaded.

15 ms of a 2000 ms budget
  1. Guests → API gateway15 ms
  2. API gateway → Booking servers1 ms
  3. Booking servers → Inventory shards8 ms
  4. Booking servers → Payments700 ms
  5. Booking servers → Inventory shards5 ms
  6. Booking servers → Kafkaasync
  7. Kafka → Channel managerasync
  8. Response
Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Overbooking is policy, stored as data. Some guests cancel late, and some never arrive. So a hotel that sells exactly its rooms has empty rooms most nights. Revenue managers set an allowance per type and night from history, say 3 extra doubles on a Saturday. A hotel that wants none sets it to 0. When it goes wrong, a guest is “walked” to a similar hotel, at the hotel’s cost.

The agency race. We record the agency’s booking with the same transaction as a direct booking. The one difference is that the add has no condition, and we flag any night past its limit. Refusing it wouldn’t undo the guest’s confirmation.

An agency sells the last double on a stale count
AgencyChannel managerBooking serviceDirect guestAgencyChannel managerBooking serviceDirect guestits copy says 1 double left on the 5thalert the hotel, the allowance absorbs itreserve the last doubleconfirmed, 0 leftinventory changed, through Kafkasells its last double to its own guestset doubles on the 5th to 0deliver booking, with its referencerecord it, unconditionallyrecorded, the 5th is 1 past its limitAgencyChannel managerBooking serviceDirect guestAgencyChannel managerBooking serviceDirect guestits copy says 1 double left on the 5thalert the hotel, the allowance absorbs itreserve the last doubleconfirmed, 0 leftinventory changed, through Kafkasells its last double to its own guestset doubles on the 5th to 0deliver booking, with its referencerecord it, unconditionallyrecorded, the 5th is 1 past its limit
Scroll or pinch to zoom, drag to pan
  • Color key
  • Service

Follow an agency booking through the design:

Hotel reservation · architecture

Trace a request: An agency delivers a booking

GuestsPartnersOTAs, hotel PMSAPI gatewayauth, limitsSearch serversfilter, price, rankBooking servers+ holds, paymentChannel managerARI out, bookings inHotel indexElasticsearchAvailabilityRedis, per nightPaymentsexternal providerKafka+ inventory changesPostgres primary+ sync standbyWorkerscapture, expiry, fees
Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

An agency delivers a booking: step 1 of 6

Partners to Channel manager

An agency has already sold the room to its guest from its copy of our availability, and sends the reservation with its own booking reference.

30 ms of a 1000 ms budget
  1. Partners → Channel manager30 ms
  2. Channel manager → Booking servers1 ms
  3. Booking servers → Inventory shards8 ms
  4. Booking servers → Kafkaasync
  5. Kafka → Channel managerasync
  6. Response
Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Keeping the window short. Oversells are roughly the agency sales that land inside the push delay. So pushes go out within a second, in a shape that’s safe to lose, repeat or reorder: the whole current count, never “one fewer”. Changes are grouped per hotel, and one pusher owns each hotel, so pushes for a hotel don’t race.

AvailabilityPusher.javaavailability_pusher.pyAvailabilityPusher.csavailability_pusher.goavailability-pusher.ts
// The channel manager's push to agencies: coalesce changes per hotel, send absolute counts.
record NightRange(LocalDate from, LocalDate to) { // half-open, like a stay
NightRange union(NightRange o) {
return new NightRange(from.isBefore(o.from) ? from : o.from, to.isAfter(o.to) ? to : o.to);
}
}
record RoomCount(long roomTypeId, LocalDate night, int available, int rateCents) {}
interface Inventory {
// Reads the shard as it is now: the push carries the latest state, not the event's.
List<RoomCount> current(long hotelId, NightRange nights);
}
interface Agency {
String name();
boolean sells(long hotelId);
// "Set availability to n", never "subtract one". A repeated or late push is harmless,
// because applying the same absolute value twice changes nothing.
void setAvailability(long hotelId, List<RoomCount> counts) throws IOException;
}
class AvailabilityPusher {
// Hotels changed since the last flush, with the nights touched. A map, so fifty changes to
// one hotel in a second (a group block, a rate reload) become one push per agency.
private final Map<Long, NightRange> dirty = new HashMap<>();
private final Inventory inventory;
private final List<Agency> agencies;
AvailabilityPusher(Inventory inventory, List<Agency> agencies) {
this.inventory = inventory;
this.agencies = agencies;
}
// Called for every InventoryChanged event from Kafka.
synchronized void onChange(long hotelId, NightRange nights) {
dirty.merge(hotelId, nights, NightRange::union);
}
// Runs every second. Agencies rate-limit pushes per hotel, so batching is required, and
// one second keeps the window in which they sell on stale counts short.
void flush() {
Map<Long, NightRange> batch;
synchronized (this) {
batch = new HashMap<>(dirty);
dirty.clear();
}
batch.forEach((hotelId, nights) -> {
List<RoomCount> counts = inventory.current(hotelId, nights);
for (Agency agency : agencies) {
if (!agency.sells(hotelId)) continue;
try {
agency.setAvailability(hotelId, counts);
} catch (IOException e) {
// Mark the hotel dirty again rather than retrying this payload: the next
// flush reads fresher counts, so a slow agency never receives old ones.
// (A real pusher keeps dirty state per agency, so one slow agency does not
// make the others receive repeats.)
onChange(hotelId, nights);
}
}
});
}
}

Three more habits help. Close out early. At the last one or two rooms of a type, push 0 to the agencies, and keep the rest for direct sale, where the check is exact. Repair drift. Every night, push all 365 nights for every hotel. Watch it. A rising oversell rate for one agency means a slow push or a broken mapping.

Go deeperOther ways to sync agencies1 min
Sync design Wins Loses
Push every change as “one fewer” Simple to send A lost or doubled message corrupts the agency’s count forever
Group per hotel, push absolute counts Repeats and losses are harmless A second of delay before each push
Agencies poll us No push system Minutes of staleness; load grows with every agency
Fixed allotments per agency No race inside an allotment Rooms stuck in one agency; release deadlines to manage

After a rebalance, a late push from the old owner is overwritten by the next one, because every push reads the whole current state. Two agencies selling the same last room both deliver, both are recorded, and the night ends one past its limit: the same alert.

A guest changes their stay from the 3rd–6th to the 4th–8th. What does the system do?

Cancel is one transaction: confirmed to cancelled, only if still confirmed, and subtract its rooms from every night. A second cancel changes no rows and gives nothing back twice. Any fee or refund is an outbox job keyed by reservation ID.

Change locks the old and new nights together, in date order. It adds to the new nights where a room is left, subtracts from the old, and updates the reservation. Shared nights (the 4th and 5th) are untouched, so only the new nights must be free. If one is full, it rolls back, and the guest keeps the original booking.

Go deeperNo-shows, closed rooms and group bookings1 min

No-shows are a morning job. Confirmed stays that started yesterday with no arrival become no-shows. Their remaining nights go back on sale, and the fee is charged.

A flooded room is out of service for a week. The hotel lowers total_rooms on those nights. If a night is now oversold, the update still succeeds. Agencies get the lower counts, and an alert shows which nights need a guest moved. Nothing is cancelled automatically. That’s the hotel’s call.

A tour operator booking 40 doubles uses the same transaction with rooms = 40: all nights or none. Hotels usually want it as a block with a release date, so unsold rooms return to sale on their own. That’s a hold with a long expiry, and its own sweep.

Go deeperTen times the hotels3 min

The platform grows ten times, to 200,000 hotels. Two things break: writes on one side, search on the other.

At ten times, row writes reach 23,000 a second, past one primary with a synchronous standby. Every booking, sweep and rate update touches one hotel, so we split inventory, rates and reservations by hotel, and every transaction stays on one database. Hotels map to 4,096 slots, and slots to databases, so a database splits by moving slots. A guest’s own bookings, the one query without a hotel, move to a table keyed by guest, fed from the outbox.

Hotel reservation · architecture

Stage 5: Shard by hotel

GuestsPartnersOTAs, hotel PMSAPI gatewayauth, limitsSearch serversfilter, price, rank17 servers, 59% busyBooking servers+ holds, payment3 servers, 45% busyChannel managerARI out, bookings in2 servers, 34% busyHotel indexElasticsearch63% busyAvailabilityRedis, per night63% busyPaymentsexternal provider5% busyKafka+ inventory changes<1% busyInventory shardsby hotel id1% busyWorkerscapture, expiry, fees2 servers, 6% busy
Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional
Stage

Stage 5 of 6

Shard by hotel

Why this stage

At 10 times the hotels, inventory takes about 23,000 row writes/s at peak, mostly rate and availability updates: twice what one primary sustains.

What we add

Inventory and reservations sharded by hotel id, so every booking, cancellation and update stays in one shard and one transaction.

What it costs

A guest’s list of bookings spans shards and moves to its own table, fed from the outbox.

One hotel can never use more than one shard, which no hotel comes near.

At 25k searches per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
The numbers behind the busy bars

What one machine can do

  • One app server handles about 2,000 to 5,000 requests a second.
  • One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
  • The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.

How big the shared systems are

  • A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
  • An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
  • The payment provider allows about 1,000 calls a second.

What the writes are

  • Inventory writes are 9% of the search rate.
  • About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
  • From stage 6, search servers cache location queries, so only 30% reach the index.

Search is next. It needs 25 million cache lookups a second, six times what a 20-node Redis cluster handles. But the data is small enough to copy everywhere instead. At 6 bytes a row, all availability is 2.2 GB, even at ten times today’s size. So every search server keeps all availability in its own memory. It loads a snapshot at start, then applies the stream of inventory changes from Kafka. A search reads every candidate hotel’s nights from local arrays, in microseconds. So it can check all 500 hotels in an area, and sort by price exactly. The Redis tier goes away.

Hotel reservation · architecture

Stage 6: Search from memory

Search in every regionSearch servers and theindex run everywhere.Each hotel’s shard hasone home region.GuestsPartnersOTAs, hotel PMSAPI gatewayauth, limitsSearch servers+ RAM availability9 servers, 56% busyBooking servers+ holds, payment3 servers, 45% busyChannel managerARI out, bookings in2 servers, 34% busyHotel index+ replicas6% busyPaymentsexternal provider5% busyKafka+ inventory changes<1% busyInventory shardsby hotel id1% busyWorkerscapture, expiry, fees2 servers, 6% busy
Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional
Stage

Stage 6 of 6

Search from memory

Why this stage

At 10 times the platform, 250,000 searches/s each look up 100 hotels in the availability cache: 25 million lookups/s, about six times what a 20 primary Redis cluster serves.

What we add

Every search server keeps all availability in memory (about 220 MB, 2.2 GB at 10 times), fed by the inventory change stream, and caches location queries, which do not depend on dates.

The search tier runs in every region.

What it costs

Each server is a replica that must load a snapshot at start and apply changes in order.

At the 1M stress mark the hotel index needs more replicas and payments need a second provider.

At 25k searches per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
The numbers behind the busy bars

What one machine can do

  • One app server handles about 2,000 to 5,000 requests a second.
  • One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
  • The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.

How big the shared systems are

  • A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
  • An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
  • The payment provider allows about 1,000 calls a second.

What the writes are

  • Inventory writes are 9% of the search rate.
  • About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
  • From stage 6, search servers cache location queries, so only 30% reach the index.
AvailabilityReplica.javaavailability_replica.pyAvailabilityReplica.csavailability_replica.goavailability-replica.ts
// A search server's copy of all availability: applied from the change stream, read by every search.
// One inventory row after a change. `position` is where the commit sits in its shard's log.
// A hotel lives on one shard and its changes travel on one Kafka partition, so positions for
// one hotel only ever grow.
record InventoryChange(long hotelId, int roomType, int night, int available, int rateCents,
long position) {}
class AvailabilityReplica {
static final int HORIZON = 366; // tonight plus 365 nights on sale
// Primitive arrays per hotel, one slot per room type and night, 6 bytes a row: 20,000
// hotels fit in about 220 MB. A map of small objects would take several times that.
// Nights use a ring: night N and night N + 366 share a slot, and by the time the later one
// goes on sale the earlier one is in the past.
static final class HotelNights {
final short[] available;
final int[] rateCents;
long appliedUpTo = -1; // the last log position applied for this hotel
HotelNights(int roomTypes) {
available = new short[roomTypes * HORIZON];
rateCents = new int[roomTypes * HORIZON];
}
}
private final Map<Long, HotelNights> hotels = new ConcurrentHashMap<>();
private final Function<Long, Integer> roomTypesOf; // from the hotel catalog
AvailabilityReplica(Function<Long, Integer> roomTypesOf) { this.roomTypesOf = roomTypesOf; }
void apply(InventoryChange c) {
HotelNights h = hotels.computeIfAbsent(c.hotelId(), id -> new HotelNights(roomTypesOf.apply(id)));
int i = c.roomType() * HORIZON + Math.floorMod(c.night(), HORIZON);
// A lock per hotel: changes for one hotel arrive a few times a second, and a read
// holds it for microseconds, so nobody waits in practice.
synchronized (h) {
// The stream is at-least-once, and a new server replays it from a point before its
// snapshot was taken, so changes it already has arrive again. One high-water mark
// per hotel makes apply idempotent: anything at or below it is skipped.
if (c.position() <= h.appliedUpTo) return;
h.available[i] = (short) c.available();
h.rateCents[i] = c.rateCents();
h.appliedUpTo = c.position();
}
}
// The cheapest total for `rooms` rooms of one type on every night of [from, to), or -1.
// Nights are days since the epoch; the caller has checked they lie within the horizon.
// At 500 candidate hotels and 3 nights this is a few thousand array reads per search.
long cheapestStay(long hotelId, int from, int to, int rooms) {
HotelNights h = hotels.get(hotelId);
if (h == null) return -1;
long best = -1;
synchronized (h) {
for (int t = 0; t < h.available.length / HORIZON; t++) {
long total = 0;
for (int n = from; n < to && total >= 0; n++) {
int i = t * HORIZON + Math.floorMod(n, HORIZON);
total = h.available[i] >= rooms ? total + (long) h.rateCents[i] * rooms : -1;
}
if (total >= 0 && (best < 0 || total < best)) best = total;
}
}
// May be a second stale. That is fine: the booking transaction checks again, and
// this answer only decides what the guest sees first.
return best;
}
}
Hotel reservation · architecture

Trace a request: Search from memory

Search in every regionSearch servers and theindex run everywhere.Each hotel’s shard hasone home region.GuestsPartnersOTAs, hotel PMSAPI gatewayauth, limitsSearch servers+ RAM availabilityBooking servers+ holds, paymentChannel managerARI out, bookings inHotel index+ replicasPaymentsexternal providerKafka+ inventory changesInventory shardsby hotel idWorkerscapture, expiry, fees
Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Search from memory: step 1 of 4

Guests to API gateway

The same request, served by the nearest region.

15 ms of a 500 ms budget
  1. Guests → API gateway15 ms
  2. API gateway → Search servers1 ms
  3. Search servers → Hotel index3 ms
  4. Response
Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.
Split by Booking on one database Spreads load Problem
Hotel Yes Yes; the largest hotel is tiny A guest’s bookings span databases: a second table
Hotel + room type Yes, for one-type bookings Slightly better A change from a double to a suite spans databases
Night No: a stay spans nights No: next weekend is one hot spot Every booking is a distributed transaction

Each database has a home region near its hotels. A distant guest pays one cross-region round trip, unnoticed next to the card network. Two regions never take writes for one hotel.

At 100 times, bookings reach about 2,500 a second at peak: more databases, nothing new. In-memory availability reaches 22 GB, near the 30 GB point where it’s split by geography. Most search traffic at that scale is machines, so quotas per partner and edge caching matter more than any server change.

Interviewerminute 40

The channel manager to the agencies is down for an hour. What do guests and agencies see?

What they are testing

Whether our own sales carry on, and agency counts are put right afterwards.

The channel manager goes down. Our website and app keep selling, because booking needs only the inventory database. Agencies keep selling too, from their last copy of our counts, which gets staler every minute. Their bookings wait on their side and arrive when we are back. Each one is recorded by the agency’s reference, so a resend is harmless. Then the channel manager pushes fresh, absolute counts to every agency, which corrects their copies in one step. The cost of the hour is a few more oversells on nearly full nights, which the oversell check flags for the hotel.

An inventory database fails. A failure should delay bookings, never corrupt inventory, and search should survive almost anything. Each primary has a standby in another zone that confirms every commit before the primary answers, so no confirmed booking is lost. The standby takes over in about 30 seconds. Meanwhile search keeps working from its copy, and guests and agencies retry with the same keys, so nothing is booked twice.

Click any box to break it and see what guests would notice:

Hotel reservation · architecture

Break it: what fails, who notices, what users see

Search in every regionSearch servers and theindex run everywhere.Each hotel’s shard hasone home region.GuestsPartnersOTAs, hotel PMSAPI gatewayauth, limitsSearch servers+ RAM availabilityBooking servers+ holds, paymentChannel managerARI out, bookings inHotel index+ replicasPaymentsexternal providerKafka+ inventory changesInventory shardsby hotel idWorkerscapture, expiry, fees
Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional
Break

Failure: click any box to break it

Inventory shard primary goes down

What breaks

Bookings, cancellations and rate updates for the hotels on that shard fail.

How we notice

Write error rate and failed health checks on the primary.

What users see

For about 30 seconds, booking those hotels fails with a retry message.

Search keeps working from memory.

Confirmed bookings are not lost: the standby has every commit.

What we do

Promote the synchronous standby.

Clients retry with the same idempotency key.

Agencies retry deliveries until acknowledged, so none is lost.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The quiet failure is worse: a search copy that silently stops updating. Search keeps answering, just wrongly, and guests see more “sold out” after clicking. So the lag of the change stream is an alert, and so is the share of bookings that fail with “sold out”.

Go deeperEvery failure, and what users see1 min
Part Failure What users see What we do
Inventory primary Down Booking its hotels fails ~30 s; search continues Promote the standby; clients and agencies retry with keys
Search server Crash Nothing Load a snapshot, replay the stream, then serve
Hotel index Down Some areas unsearchable; booking fine Replicas; rebuild from change data capture
Payment provider Slow or down Prepaid fails; pay-at-hotel still books Same-key retries, a second provider
Kafka Down Bookings confirm; search and agencies go stale The outbox waits; close out nearly full hotels on agencies
Workers Stopped Expired holds keep rooms out of sale Jobs are safe to repeat and catch up on restart
Channel manager Down Agencies sell on frozen counts Restart; full push for changed hotels
One agency’s API Down That agency shows stale availability Retry; only the latest state is sent
A region Outage Search fails over; its hotels’ booking pauses Promote replicas in another region

Under stress, keep confirmed bookings correct first, then booking, then agency sync, then search freshness.

Operation Guarantee What users can see
Direct booking Exact per night: ordered locks, conditional update Never past the limit
Agency booking Recorded once in effect; may pass the limit A rare oversold night, alerted
Hold expiry Within 30 s, exactly once A room shown sold for up to 30 s after a hold lapsed
Search Usually under 2 s behind A hotel shown available that isn’t: the booking says 409
Go deeperRunning it: alerts, abuse and proof1 min

Wake someone up when:

  • booking success drops below 99.9% (counting “sold out” and “declined” as success)
  • booking takes more than 2 seconds at p99
  • the change stream or agency pushes lag more than 30 seconds
  • oversold nights per hour jump for any agency
  • holds sit past their expiry

Abuse. Most searches are machines: quotas per partner key and short edge caching. Holds never paid: limit open holds per account and card. Card testing through pay-at-hotel bookings: limit per account, device and card.

Proving no oversell. A nightly job recomputes reserved for every row from the held and confirmed stays covering that night. A mismatch is a bug, repaired from the reservations, which are the truth. Oversold nights are then split by cause: agency bookings (expected) or direct ones (a bug).

Here is everything we built, in the order we built it. Use the buttons to step through it.

Hotel reservation · architecture

The full design, stage by stage

Search in every regionSearch servers and theindex run everywhere.Each hotel’s shard hasone home region.GuestsPartnersOTAs, hotel PMSAPI gatewayauth, limitsSearch servers+ RAM availability9 servers, 56% busyBooking servers+ holds, payment3 servers, 45% busyChannel managerARI out, bookings in2 servers, 34% busyHotel index+ replicas6% busyPaymentsexternal provider5% busyKafka+ inventory changes<1% busyInventory shardsby hotel id1% busyWorkerscapture, expiry, fees2 servers, 6% busy
Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional
Stage

Stage 6 of 6

Search from memory

Why this stage

At 10 times the platform, 250,000 searches/s each look up 100 hotels in the availability cache: 25 million lookups/s, about six times what a 20 primary Redis cluster serves.

What we add

Every search server keeps all availability in memory (about 220 MB, 2.2 GB at 10 times), fed by the inventory change stream, and caches location queries, which do not depend on dates.

The search tier runs in every region.

What it costs

Each server is a replica that must load a snapshot at start and apply changes in order.

At the 1M stress mark the hotel index needs more replicas and payments need a second provider.

At 25k searches per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.The whole design. Step through it: one database, search on a copy, holds and payments, the channel manager, a split by hotel, then search from memory.
The numbers behind the busy bars

What one machine can do

  • One app server handles about 2,000 to 5,000 requests a second.
  • One Postgres primary manages about 2,000 searches a second if it has to answer them, because each reads about 1,500 inventory rows.
  • The same primary does about 10,000 point reads a second, and about 10,000 row writes with a synchronous standby.

How big the shared systems are

  • A Redis cluster of 20 primaries answers about 4 million lookups a second. At 100 hotels a search, that is 40,000 searches.
  • An Elasticsearch cluster answers about 40,000 geo and filter queries a second, and 120,000 with more replicas.
  • The payment provider allows about 1,000 calls a second.

What the writes are

  • Inventory writes are 9% of the search rate.
  • About 90% of them are hotels updating rates and availability. About 10% are bookings, holds and cancellations.
  • From stage 6, search servers cache location queries, so only 30% reach the index.
Decision We chose We gave up We’d choose differently if
Inventory Counts per room type per night Booking a specific room Each unit were unique (villas): a calendar each
Concurrency Nights locked in date order, conditional update One hotel-wide lock’s simplicity One hotel had extreme contention
Prepaid payments Hold, authorize, confirm; a sweeper frees holds Rooms out of sale during abandoned checkouts Contention were zero (authorize first)
Overbooking An allowance per type and night, as data A promise that nobody is ever walked The hotel sold only non-refundable rates
Agency sync Shared pool, absolute pushes, unconditional delivery No oversells across channels Agencies needed zero risk (fixed allotments)
Search A copy: Redis first, then in memory on every server Always-exact availability Searches per booking were low (read replicas)
Split By hotel, one home region each Queries across hotels One hotel outgrew a primary, which none does
  • 0 to 4. Room or type, searches per booking, payment types, agencies.
  • 4 to 9. 8 bookings and 25,000 searches a second, 220 MB of availability, writes mostly rate updates.
  • 9 to 20. Build it: counts per night, the stay transaction, search on a copy, holds. A mid-level answer can stop here.
  • 20 to 29. The stress tests: the date-range lock, then retries and late payments.
  • 29 to 40. The agencies and the oversell race, changes, then ten times the hotels if asked.
  • 40 to 45. A primary failing, then your own questions.

Never skip the conditional update over every night, the key in the same transaction, or why search may be stale.

Interviewer asksHow would you sell a specific room, like a named villa?

Each unit becomes its own room type with a total of 1. The table and transaction don’t change. What changes is fragmentation: short bookings leave one-night gaps nobody books, so holiday rental platforms add minimum stays or cheaper gap nights.

Interviewer asksSearch must be exact, or guests will book rooms that are gone. Agree?

No. They can’t book them: the booking transaction refuses. Staleness costs a “sold out” after a click, so measure how often that happens. Exact search would read every primary 25,000 times a second, risking the booking path for a slightly better click rate.