Design a ride hailing service
Design a ride hailing service like Uber or Lyft. Start with one server and one table, grow it with positions in memory in H3 cells, an atomic claim on each driver, sockets to phones and a split by city, then stress it with every race, choosing the driver, and surge and payments, with code.
- Read first
- EstimationConsistency models
- Scaling patterns
- Idempotency and exactly-once effects
“Design Uber.” The rider taps a button, and a car shows up.
It sounds like a “nearest driver” query plus a database row. The trap is treating it as one problem. Two workloads hide in this prompt. A million phones report positions that are worthless 4 seconds later. And a few hundred requests a second must each end with exactly one driver. The design keeps them apart: positions in memory, assignments in a database.
So this page starts with one server and one table, and lets the numbers grow it. Then it tests the result against the races, the choice of driver, and surge pricing.
The answer in one minute
Section titled “The answer in one minute”If you only have a minute, this is the design. The rest of the page explains each part.
- Keep driver positions in memory, per city. Only the newest position matters, and the next ping replaces a lost one. So 250,000 pings a second never touch a database.
- Claim a driver with a conditional update. “Set this driver to offered, only if they’re still available.” The database refuses the second claim, however many dispatchers race.
- Offer to one driver at a time, for 15 seconds. Pick the best road ETA, not the shortest straight line. If the driver says no, or nothing, try the next.
- Push over open sockets. Offers reach drivers in tens of milliseconds, and riders watch the car move. Events flow to Kafka for surge pricing and payments.
- Split by city, with a standby per city. Every dispatch question is local. When a city’s dispatcher dies, a standby takes over in seconds, and the database still refuses any double claim.
Scope it
Section titled “Scope it”A few questions decide where positions live and how drivers are chosen.
Design a ride hailing service like Uber or Lyft.
Whether you separate the location stream from matching right away, and pin down how many drivers report how often, because that number decides where positions live. About three minutes. Asking about payments and ratings before drivers and pings reads as missing the hard part.
These four questions matter most:
- How many drivers online? 1 million at the global peak, 400,000 on average.
- How often does a driver report? Every 4 seconds while online.
- How many trips? 20 million a day.
- How is a driver chosen? The trip is offered to one driver at a time, who has 15 seconds to accept.
From these answers we make five promises:
- Never double booked. A driver is on at most one trip, and a trip has at most one driver. This holds through retries, timeouts and failovers. It’s the one promise with no “usually”.
- Always matching. Requesting and matching work 99.99% of the time in every city. A rider who can’t get a car opens a competitor’s app.
- Fast. A price quote in under 300 ms. The offer leaves our servers within 1 second of the request. A driver’s position reaches the watching rider within 2 seconds.
- Fresh. Matching only uses positions under 15 seconds old.
- Charged once. Every finished trip is charged exactly once.
We leave out pooled rides, building the payment system itself (we hand trips to it), driver onboarding, ratings, and the inside of the routing engine.
Go deeperMore questions, and what each changes1 min
| Question | We assume | If the answer were different |
|---|---|---|
| Shared or scheduled rides? | Not in v1: one rider, one car, now | Pooling turns matching into route insertion |
| How fresh is the rider’s car view? | Within a couple of seconds | Minute-level freshness would allow polling, and drop the push layer |
| Is the price fixed at request time? | Yes, a quote locked for 2 minutes | Metered fares move pricing to the end of the trip |
| Do we build payments? | No: hand finished trips to a payment system | See payment system |
| Do we keep location history? | Yes, for receipts, disputes and safety | Without it, the location stream needs no durable copy at all |
The numbers that decide it
Section titled “The numbers that decide it”Now we work out the numbers. The most important is how pings compare to requests.
- Drivers online at peak
- 1 million
- Seconds between pings
- 4
- Average load as a share of peak
- 40%
- Trips a day
- 20 million
- Peak trips over average
- 2×
- Minutes per trip, pickup included
- 20
- Bytes per ping
- 100 bytes
- Busiest city’s share of drivers
- 5%
1 million drivers at peak ÷ 4 s. 100,000 a second on average.
So we positions can't go in a database: one primary takes about 5,000 writes a second, and each position is replaced 4 seconds later anyway.
20 million trips a day ÷ 86,400 s.
So we dispatch is tiny. It's hard because it must be exact, not because it's busy. About 6 writes per trip is 2,800 a second at peak, which one database primary takes.
460 a second × 20 minutes, pickup included. Each watching rider gets a position every 4 seconds.
So we push over open sockets; polling would be mostly empty requests.
1 million drivers × about 200 bytes. The busiest city has about 5% of drivers: 13,000 pings a second.
So we memory never forces a split. When we split, it's for write rate and to contain failures.
Raw pings at the average rate. Trip rows at 2 KB are about 15 TB a year.
So we stream pings to object storage, and keep 90 days of trips in the trips database.
Try the calculator: double the drivers, or ping every 2 seconds, and pings pass what one index handles, so the split by city comes on day one. Ten times the trips passes one database primary.
- Location pings, peak
- 250,000/s
- Past the 5,000 writes/s one database primary takes: positions live in memory, not in a table.
- Busiest city’s pings, peak
- 13,000/s
- One city fits one geo index shard with room to spare.
- Ping ingest bandwidth, peak
- 25 MB/s
- Not a design driver. A TLS handshake per ping would be: keep one socket open per phone.
- Trip requests, average
- 230/s
- Trip requests, peak
- 460/s
- Low. Dispatch is hard because it must be exact, not because it is busy.
- Trip transactions, peak (6 per trip)
- 2,800/s
- One relational primary takes this, with conditional updates for every claim.
- Trips in progress, peak
- 560,000
- Drivers busy at peak (percent)
- 56
- About half the fleet carries riders at peak, a plausible rush hour.
- Live position pushes to riders, peak
- 140,000/s
- One message per ping per watching rider: push it over open sockets, never by polling.
- Geo index memory, all drivers
- 200 MB
- Tiny. Memory never forces sharding here; write rate and failure domains do.
- Raw location history a day
- 860 GB
- Append-only and rarely read: stream it to object storage, never into the trips database.
- Trip records a year (2 KB each)
- 15 TB
- Too much for one primary within months: keep 90 days hot, archive the rest, partition by city.
Build the design
Section titled “Build the design”We start with the simplest design that is correct. Then we let the numbers break it, one at a time. Each step fixes only what broke, adds only the boxes that fix needs, and says what the fix costs. Throughout, follow one scenario: a driver is online, and a rider taps Request.
Step 1. One server, one table
Section titled “Step 1. One server, one table”The simplest version updates each driver’s row every 4 seconds. A request selects available drivers near the pickup, picks the nearest, and marks them busy.
Stage 1: One server, one table
Stage 1 of 7
One server, one table
Why this stage
Nothing yet.
This is the smallest design that puts a rider in a car.
What we add
An app server that writes each driver’s latest position to a Postgres row, finds the nearest free driver with a bounding box query, and marks that driver busy.
What it costs
250,000 row updates a second at peak, a check-then-set race on every match, and riders who poll for news.
At 250k location pings per second, Trip service is 5000% busy. Stage 2 is the fix.
The numbers behind the busy bars
What one machine can do
- One Postgres primary takes about 5,000 small write transactions a second.
- One in-memory geo index node takes about 400,000 location updates a second when location servers batch them.
- One location server handles about 20,000 pings a second.
- One gateway handles about 25,000 messages a second over about 100,000 sockets.
What one ride request costs
- Each ride request makes about 6 trip transactions and 1.5 offers.
- It also makes 7 routing calls: 5 price quotes, 1 candidate matrix and 1 route.
- Once sharded, there are 20 city shards of each.
Why Kafka has no bar
- 250,000 batched pings a second is about 25 MB/s. That is a few percent of a 6-broker cluster, which takes several hundred MB/s.
- It grows by adding brokers and partitions per city.
It works for one small city, with three flaws. At peak it’s 250,000 row updates a second, 50 times what one database takes. Two requests can read the same “available” driver before either marks them busy. And riders poll for news.
Step 2. Positions in memory
Section titled “Step 2. Positions in memory”A position has three properties that decide where it lives. Only the newest value matters. It’s rewritten every 4 seconds. And if one is lost, the next ping restores it. This is soft state: data that’s fine to lose, because it rebuilds itself.
So a location service keeps an in-memory index per city: a map from driver to position, and a set of drivers per map cell. The cells are H3 cells, Uber’s grid of hexagons. At resolution 9 each hexagon’s edge is about 174 m, and all six neighbors are the same distance away, so “two rings around the pickup” is a fair circle. When a driver crosses into a new cell, they move between two sets, a constant-time update.
// The location path on a city's geo index shard: about 13,000 pings a second in the busiest city.// Positions are soft state. Nothing here is written to disk, because the next ping, 4 seconds// later, replaces whatever a crash loses.record Ping(long driverId, long seq, long atMillis, double lat, double lng, double accuracyM) {}
interface Cells { long cellOf(double lat, double lng); } // H3 resolution 9 in productioninterface TripFeed { void publish(long tripId, Ping p); } // the trip's pub/sub channelinterface Trail { void append(Ping p); } // batched to Kafka, never awaited
class GeoIndex { static final double MAX_SPEED_MPS = 70; // 250 km/h: faster is a GPS jump or a spoofer static final double MAX_ACCURACY_M = 100; // a worse fix would move the car onto the wrong street static final long LIVE_MS = 15_000; // no ping for 15 s: not offered trips, kept for a while static final double M_PER_DEG = 111_320;
static final class Entry { long seq, atMillis, seenAt; double lat, lng; long cell; boolean available; // a hint from dispatch; the database row is the truth Long tripId; // set while on a trip, so pings reach the rider }
private final Map<Long, Entry> drivers = new HashMap<>(); private final Map<Long, Set<Long>> byCell = new HashMap<>(); private final Cells cells; private final TripFeed feed; private final Trail trail; private final LongSupplier clock;
GeoIndex(Cells cells, TripFeed feed, Trail trail, LongSupplier clock) { this.cells = cells; this.feed = feed; this.trail = trail; this.clock = clock; }
void onPing(Ping p) { Long tripId; // One lock per city shard is enough at 13,000 updates a second: each update is a few // map operations. Publishing happens outside it, so a slow channel never stalls pings. synchronized (this) { Entry e = drivers.get(p.driverId()); // Going online is an explicit, durable action. A ping alone never makes a driver // dispatchable, so a stale app cannot put a driver back on the map. if (e == null) return; // Phones batch and resend pings after a dead zone, so order is not guaranteed. // The sequence number, not arrival time, decides what is newest. if (p.seq() <= e.seq) return; e.seq = p.seq(); e.seenAt = clock.getAsLong(); // server clock: liveness never trusts the phone if (p.accuracyM() > MAX_ACCURACY_M || impliedSpeed(e, p) > MAX_SPEED_MPS) return; // alive, not moved long cell = cells.cellOf(p.lat(), p.lng()); if (cell != e.cell) { // Moving between cell sets is what keeps "drivers near here" a lookup of a few // cells instead of a scan of the whole city. Set<Long> old = byCell.get(e.cell); if (old != null) old.remove(p.driverId()); byCell.computeIfAbsent(cell, k -> new HashSet<>()).add(p.driverId()); e.cell = cell; } e.lat = p.lat(); e.lng = p.lng(); e.atMillis = p.atMillis(); tripId = e.tripId; } if (tripId != null) feed.publish(tripId, p); // the rider's map moves trail.append(p); // receipts, disputes, surge supply }
// What dispatch reads: available, recently seen drivers in the pickup cell and its rings. synchronized List<Long> nearby(Collection<Long> cellIds) { long now = clock.getAsLong(); List<Long> out = new ArrayList<>(); for (long cell : cellIds) for (long id : byCell.getOrDefault(cell, Set.of())) { Entry e = drivers.get(id); if (e.available && now - e.seenAt <= LIVE_MS) out.add(id); } return out; // a candidate list, not a promise: the claim in the database decides }
// Pings are 4 s apart, so a flat-earth distance is accurate to well under 1%. private static double impliedSpeed(Entry e, Ping p) { if (e.atMillis == 0 || p.atMillis() <= e.atMillis) return 0; double dy = (p.lat() - e.lat) * M_PER_DEG; double dx = (p.lng() - e.lng) * M_PER_DEG * Math.cos(Math.toRadians(p.lat())); return Math.hypot(dx, dy) / ((p.atMillis() - e.atMillis) / 1000.0); }}# The location path on a city's geo index shard: about 13,000 pings a second in the busiest city.# Positions are soft state. Nothing here is written to disk, because the next ping, 4 seconds# later, replaces whatever a crash loses.@dataclass(frozen=True)class Ping: driver_id: int seq: int at_millis: int lat: float lng: float accuracy_m: float
class Cells(Protocol): def cell_of(self, lat: float, lng: float) -> int: ... # H3 resolution 9 in production
class TripFeed(Protocol): def publish(self, trip_id: int, p: Ping) -> None: ... # the trip's pub/sub channel
class Trail(Protocol): def append(self, p: Ping) -> None: ... # batched to Kafka, never awaited
MAX_SPEED_MPS = 70 # 250 km/h: faster is a GPS jump or a spooferMAX_ACCURACY_M = 100 # a worse fix would move the car onto the wrong streetLIVE_SECONDS = 15 # no ping for 15 s: not offered trips, kept for a whileM_PER_DEG = 111_320
@dataclassclass Entry: seq: int = 0 at_millis: int = 0 seen_at: float = 0.0 lat: float = 0.0 lng: float = 0.0 cell: int | None = None available: bool = False # a hint from dispatch; the database row is the truth trip_id: int | None = None # set while on a trip, so pings reach the rider
class GeoIndex: def __init__(self, cells: Cells, feed: TripFeed, trail: Trail, clock=time.monotonic): self._cells, self._feed, self._trail, self._clock = cells, feed, trail, clock self._drivers: dict[int, Entry] = {} self._by_cell: dict[int, set[int]] = defaultdict(set) # One lock per city shard is enough at 13,000 updates a second: each update is a few # dict operations. Publishing happens outside it, so a slow channel never stalls pings. self._lock = threading.Lock()
def on_ping(self, p: Ping) -> None: with self._lock: e = self._drivers.get(p.driver_id) # Going online is an explicit, durable action. A ping alone never makes a driver # dispatchable, so a stale app cannot put a driver back on the map. if e is None: return # Phones batch and resend pings after a dead zone, so order is not guaranteed. # The sequence number, not arrival time, decides what is newest. if p.seq <= e.seq: return e.seq = p.seq e.seen_at = self._clock() # server clock: liveness never trusts the phone if p.accuracy_m > MAX_ACCURACY_M or _implied_speed(e, p) > MAX_SPEED_MPS: return # alive, not moved cell = self._cells.cell_of(p.lat, p.lng) if cell != e.cell: # Moving between cell sets is what keeps "drivers near here" a lookup of a few # cells instead of a scan of the whole city. if e.cell is not None: self._by_cell[e.cell].discard(p.driver_id) self._by_cell[cell].add(p.driver_id) e.cell = cell e.lat, e.lng, e.at_millis = p.lat, p.lng, p.at_millis trip_id = e.trip_id if trip_id is not None: self._feed.publish(trip_id, p) # the rider's map moves self._trail.append(p) # receipts, disputes, surge supply
# What dispatch reads: available, recently seen drivers in the pickup cell and its rings. def nearby(self, cell_ids: Iterable[int]) -> list[int]: with self._lock: now = self._clock() return [ # a candidate list, not a promise: the claim in the database decides d for c in cell_ids for d in self._by_cell.get(c, ()) if self._drivers[d].available and now - self._drivers[d].seen_at <= LIVE_SECONDS ]
def _implied_speed(e: Entry, p: Ping) -> float: # Pings are 4 s apart, so a flat-earth distance is accurate to well under 1%. if e.at_millis == 0 or p.at_millis <= e.at_millis: return 0.0 dy = (p.lat - e.lat) * M_PER_DEG dx = (p.lng - e.lng) * M_PER_DEG * math.cos(math.radians(p.lat)) return math.hypot(dx, dy) / ((p.at_millis - e.at_millis) / 1000)// The location path on a city's geo index shard: about 13,000 pings a second in the busiest city.// Positions are soft state. Nothing here is written to disk, because the next ping, 4 seconds// later, replaces whatever a crash loses.record Ping(long DriverId, long Seq, long AtMillis, double Lat, double Lng, double AccuracyM);
interface ICells { long CellOf(double lat, double lng); } // H3 resolution 9 in productioninterface ITripFeed { void Publish(long tripId, Ping p); } // the trip's pub/sub channelinterface ITrail { void Append(Ping p); } // batched to Kafka, never awaited
class GeoIndex(ICells cells, ITripFeed feed, ITrail trail, TimeProvider clock){ const double MaxSpeedMps = 70; // 250 km/h: faster is a GPS jump or a spoofer const double MaxAccuracyM = 100; // a worse fix would move the car onto the wrong street static readonly TimeSpan Live = TimeSpan.FromSeconds(15); // no ping for 15 s: not offered trips const double MPerDeg = 111_320;
sealed class Entry { public long Seq, AtMillis, Cell; public DateTimeOffset SeenAt; public double Lat, Lng; public bool Available; // a hint from dispatch; the database row is the truth public long? TripId; // set while on a trip, so pings reach the rider }
readonly Dictionary<long, Entry> _drivers = new(); readonly Dictionary<long, HashSet<long>> _byCell = new(); // One lock per city shard is enough at 13,000 updates a second: each update is a few // dictionary operations. Publishing happens outside it, so a slow channel never stalls pings. readonly object _gate = new();
public void OnPing(Ping p) { long? tripId; lock (_gate) { // Going online is an explicit, durable action. A ping alone never makes a driver // dispatchable, so a stale app cannot put a driver back on the map. if (!_drivers.TryGetValue(p.DriverId, out var e)) return; // Phones batch and resend pings after a dead zone, so order is not guaranteed. // The sequence number, not arrival time, decides what is newest. if (p.Seq <= e.Seq) return; e.Seq = p.Seq; e.SeenAt = clock.GetUtcNow(); // server clock: liveness never trusts the phone if (p.AccuracyM > MaxAccuracyM || ImpliedSpeed(e, p) > MaxSpeedMps) return; // alive, not moved long cell = cells.CellOf(p.Lat, p.Lng); if (cell != e.Cell) { // Moving between cell sets is what keeps "drivers near here" a lookup of a few // cells instead of a scan of the whole city. if (_byCell.TryGetValue(e.Cell, out var old)) old.Remove(p.DriverId); if (!_byCell.TryGetValue(cell, out var now)) _byCell[cell] = now = new HashSet<long>(); now.Add(p.DriverId); e.Cell = cell; } (e.Lat, e.Lng, e.AtMillis) = (p.Lat, p.Lng, p.AtMillis); tripId = e.TripId; } if (tripId is long trip) feed.Publish(trip, p); // the rider's map moves trail.Append(p); // receipts, disputes, surge supply }
// What dispatch reads: available, recently seen drivers in the pickup cell and its rings. public List<long> Nearby(IEnumerable<long> cellIds) { lock (_gate) { var now = clock.GetUtcNow(); var found = new List<long>(); foreach (long cell in cellIds) if (_byCell.TryGetValue(cell, out var ids)) foreach (long id in ids) if (_drivers[id].Available && now - _drivers[id].SeenAt <= Live) found.Add(id); return found; // a candidate list, not a promise: the claim in the database decides } }
// Pings are 4 s apart, so a flat-earth distance is accurate to well under 1%. static double ImpliedSpeed(Entry e, Ping p) { if (e.AtMillis == 0 || p.AtMillis <= e.AtMillis) return 0; double dy = (p.Lat - e.Lat) * MPerDeg; double dx = (p.Lng - e.Lng) * MPerDeg * Math.Cos(p.Lat * Math.PI / 180); return Math.Sqrt(dx * dx + dy * dy) / ((p.AtMillis - e.AtMillis) / 1000.0); }}// The location path on a city's geo index shard: about 13,000 pings a second in the busiest city.// Positions are soft state. Nothing here is written to disk, because the next ping, 4 seconds// later, replaces whatever a crash loses.type Ping struct { DriverID, Seq, AtMillis int64 Lat, Lng, AccuracyM float64}
type Cells interface{ CellOf(lat, lng float64) uint64 } // H3 resolution 9 in productiontype TripFeed interface{ Publish(tripID int64, p Ping) } // the trip's pub/sub channeltype Trail interface{ Append(p Ping) } // batched to Kafka, never awaited
const ( maxSpeedMps = 70 // 250 km/h: faster is a GPS jump or a spoofer maxAccuracyM = 100 // a worse fix would move the car onto the wrong street live = 15 * time.Second // no ping for 15 s: not offered trips, kept for a while mPerDeg = 111_320)
type entry struct { seq, atMillis int64 seenAt time.Time lat, lng float64 cell uint64 available bool // a hint from dispatch; the database row is the truth tripID *int64 // set while on a trip, so pings reach the rider}
type GeoIndex struct { cells Cells feed TripFeed trail Trail now func() time.Time
// One lock per city shard is enough at 13,000 updates a second: each update is a few map // operations. Publishing happens outside it, so a slow channel never stalls pings. mu sync.Mutex drivers map[int64]*entry byCell map[uint64]map[int64]struct{}}
func (g *GeoIndex) OnPing(p Ping) { g.mu.Lock() e, ok := g.drivers[p.DriverID] // Going online is an explicit, durable action. A ping alone never makes a driver // dispatchable, so a stale app cannot put a driver back on the map. // Phones batch and resend pings after a dead zone, so order is not guaranteed: the // sequence number, not arrival time, decides what is newest. if !ok || p.Seq <= e.seq { g.mu.Unlock() return } e.seq = p.Seq e.seenAt = g.now() // server clock: liveness never trusts the phone if p.AccuracyM > maxAccuracyM || impliedSpeed(e, p) > maxSpeedMps { g.mu.Unlock() return // alive, not moved } if cell := g.cells.CellOf(p.Lat, p.Lng); cell != e.cell { // Moving between cell sets is what keeps "drivers near here" a lookup of a few cells // instead of a scan of the whole city. delete(g.byCell[e.cell], p.DriverID) if g.byCell[cell] == nil { g.byCell[cell] = map[int64]struct{}{} } g.byCell[cell][p.DriverID] = struct{}{} e.cell = cell } e.lat, e.lng, e.atMillis = p.Lat, p.Lng, p.AtMillis tripID := e.tripID g.mu.Unlock()
if tripID != nil { g.feed.Publish(*tripID, p) // the rider's map moves } g.trail.Append(p) // receipts, disputes, surge supply}
// Nearby is what dispatch reads: available, recently seen drivers in the pickup cell and its rings.func (g *GeoIndex) Nearby(cells []uint64) []int64 { g.mu.Lock() defer g.mu.Unlock() now := g.now() var found []int64 for _, c := range cells { for id := range g.byCell[c] { if e := g.drivers[id]; e.available && now.Sub(e.seenAt) <= live { found = append(found, id) } } } return found // a candidate list, not a promise: the claim in the database decides}
// Pings are 4 s apart, so a flat-earth distance is accurate to well under 1%.func impliedSpeed(e *entry, p Ping) float64 { if e.atMillis == 0 || p.AtMillis <= e.atMillis { return 0 } dy := (p.Lat - e.lat) * mPerDeg dx := (p.Lng - e.lng) * mPerDeg * math.Cos(p.Lat*math.Pi/180) return math.Hypot(dx, dy) / (float64(p.AtMillis-e.atMillis) / 1000)}// The location path on a city's geo index shard: about 13,000 pings a second in the busiest city.// Positions are soft state. Nothing here is written to disk, because the next ping, 4 seconds// later, replaces whatever a crash loses.interface Ping { driverId: number; seq: number; atMillis: number; lat: number; lng: number; accuracyM: number;}
type CellOf = (lat: number, lng: number) => string; // H3 resolution 9 in productiontype Publish = (tripId: number, p: Ping) => void; // the trip's pub/sub channeltype Append = (p: Ping) => void; // batched to Kafka, never awaited
const MAX_SPEED_MPS = 70; // 250 km/h: faster is a GPS jump or a spooferconst MAX_ACCURACY_M = 100; // a worse fix would move the car onto the wrong streetconst LIVE_MS = 15_000; // no ping for 15 s: not offered trips, kept for a whileconst M_PER_DEG = 111_320;
interface Entry { seq: number; atMillis: number; seenAt: number; lat: number; lng: number; cell?: string; available: boolean; // a hint from dispatch; the database row is the truth tripId?: number; // set while on a trip, so pings reach the rider}
// Node runs this on one thread per shard, so no lock: each handler finishes before the next// ping is read. The other languages need one lock per city shard, which is plenty at 13,000/s.class GeoIndex { private readonly drivers = new Map<number, Entry>(); private readonly byCell = new Map<string, Set<number>>();
constructor( private readonly cellOf: CellOf, private readonly publish: Publish, private readonly append: Append, private readonly now: () => number = Date.now, ) {}
onPing(p: Ping): void { const e = this.drivers.get(p.driverId); // Going online is an explicit, durable action. A ping alone never makes a driver // dispatchable, so a stale app cannot put a driver back on the map. if (!e) return; // Phones batch and resend pings after a dead zone, so order is not guaranteed. // The sequence number, not arrival time, decides what is newest. if (p.seq <= e.seq) return; e.seq = p.seq; e.seenAt = this.now(); // server clock: liveness never trusts the phone if (p.accuracyM > MAX_ACCURACY_M || impliedSpeed(e, p) > MAX_SPEED_MPS) return; // alive, not moved const cell = this.cellOf(p.lat, p.lng); if (cell !== e.cell) { // Moving between cell sets is what keeps "drivers near here" a lookup of a few cells // instead of a scan of the whole city. if (e.cell !== undefined) this.byCell.get(e.cell)?.delete(p.driverId); let set = this.byCell.get(cell); if (!set) this.byCell.set(cell, (set = new Set())); set.add(p.driverId); e.cell = cell; } Object.assign(e, { lat: p.lat, lng: p.lng, atMillis: p.atMillis }); if (e.tripId !== undefined) this.publish(e.tripId, p); // the rider's map moves this.append(p); // receipts, disputes, surge supply }
// What dispatch reads: available, recently seen drivers in the pickup cell and its rings. nearby(cells: Iterable<string>): number[] { const now = this.now(); const found: number[] = []; for (const c of cells) for (const id of this.byCell.get(c) ?? []) { const e = this.drivers.get(id)!; if (e.available && now - e.seenAt <= LIVE_MS) found.push(id); } return found; // a candidate list, not a promise: the claim in the database decides }}
// Pings are 4 s apart, so a flat-earth distance is accurate to well under 1%.function impliedSpeed(e: Entry, p: Ping): number { if (e.atMillis === 0 || p.atMillis <= e.atMillis) return 0; const dy = (p.lat - e.lat) * M_PER_DEG; const dx = (p.lng - e.lng) * M_PER_DEG * Math.cos((p.lat * Math.PI) / 180); return Math.hypot(dx, dy) / ((p.atMillis - e.atMillis) / 1000);}Three rules in that handler matter more than the data structure:
- Order comes from the phone’s sequence number. A phone leaving a tunnel resends a batch of old pings, and they must not overwrite newer ones.
- “Last seen” uses the server’s clock. A phone with a wrong clock can’t make itself look fresh.
- A ping never makes a driver available. Going online is an explicit, durable call. So an old app or a stale socket can’t put a driver back on the map.
If the index crashes, it reloads the city’s online drivers from the database, and the next round of pings fills in positions 4 seconds later.
Stage 2: Locations in memory
Stage 2 of 7
Locations in memory
Why this stage
At the 250,000 pings/s peak, Postgres would be 50 times past its write capacity, for data that is stale 4 seconds later anyway.
What we add
Stateless location servers that validate pings, and an in-memory geo index of drivers by H3 cell.
Postgres keeps only trips.
What it costs
Positions are soft state.
A lost index forgets every driver until their next ping, 4 seconds later.
At 250k location pings per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One Postgres primary takes about 5,000 small write transactions a second.
- One in-memory geo index node takes about 400,000 location updates a second when location servers batch them.
- One location server handles about 20,000 pings a second.
- One gateway handles about 25,000 messages a second over about 100,000 sockets.
What one ride request costs
- Each ride request makes about 6 trip transactions and 1.5 offers.
- It also makes 7 routing calls: 5 price quotes, 1 candidate matrix and 1 route.
- Once sharded, there are 20 city shards of each.
Why Kafka has no bar
- 250,000 batched pings a second is about 25 MB/s. That is a few percent of a 6-broker cluster, which takes several hundred MB/s.
- It grows by adding brokers and partitions per city.
Positions are now cheap. But two requests can still read the same “available” driver before either marks them busy.
Step 3. Claim a driver safely
Section titled “Step 3. Claim a driver safely”Now the part that must be exact. A dispatcher per city reads about 10 nearby free drivers from the index. It asks a routing service for their road ETAs, because a driver 300 m away across a river can be 9 minutes out. Then it offers the trip to the best one, for 15 seconds.
The offer is a compare-and-set in the database: a write that only succeeds if the current value is what you expect. One transaction makes two conditional updates:
- the driver goes from
availabletooffered, and - the trip goes from
requestedtooffering,
both stamped with a new offer ID and an expiry 15 seconds out, on the database’s clock. If either update changes 0 rows, someone else got there first, and the transaction rolls back. No read comes first. Reading “available” and then writing is exactly the race that broke the one-table version.
// Dispatch claims: every change to who holds a driver is one conditional UPDATE in the city's// trips database. The WHERE clause is the compare, the SET is the swap, and the row lock makes// the pair atomic. now() is the database clock, so servers whose clocks drift never disagree// about when an offer ended.interface Tx { int execute(String sql, Object... args); // returns the number of rows changed boolean exists(String sql, Object... args);}
interface Db { <T> T inTransaction(Function<Tx, T> work); // commits on return, rolls back on throw}
enum Offer { OFFERED, DRIVER_BUSY, TRIP_GONE }enum Accept { ACCEPTED, EXPIRED }
class Rollback extends RuntimeException { final Object result; Rollback(Object result) { super(null, null, false, false); this.result = result; }}
class DispatchClaims { // Free, or held by an offer whose 15 seconds ran out: a dead dispatcher's offer frees // itself, with no sweeper on the correctness path. static final String CLAIM_DRIVER = """ UPDATE drivers SET state = 'offered', trip_id = ?, offer_id = ?, offer_expires_at = now() + interval '15 seconds' WHERE driver_id = ? AND (state = 'available' OR (state = 'offered' AND offer_expires_at < now()))"""; // A trip has at most one live offer. This is what stops two dispatchers (say, an old // leader that has not noticed its lease is gone) from offering one trip to two drivers. static final String CLAIM_TRIP = """ UPDATE trips SET state = 'offering', driver_id = ?, offer_id = ?, offer_expires_at = now() + interval '15 seconds' WHERE trip_id = ? AND (state = 'requested' OR (state = 'offering' AND offer_expires_at < now()))"""; // Accept only the exact offer, and only before it expires. static final String ACCEPT_DRIVER = """ UPDATE drivers SET state = 'on_trip', offer_expires_at = NULL WHERE driver_id = ? AND state = 'offered' AND offer_id = ? AND offer_expires_at >= now()"""; static final String ACCEPT_TRIP = """ UPDATE trips SET state = 'accepted', accepted_at = now() WHERE trip_id = ? AND state = 'offering' AND offer_id = ?"""; static final String ALREADY_ACCEPTED = """ SELECT 1 FROM trips WHERE trip_id = ? AND offer_id = ? AND state <> 'offering' AND state <> 'cancelled'"""; // Timeout and decline: release only if this offer still holds the row. 0 rows means the // accept won the race, and releasing nothing is the correct outcome. static final String RELEASE_DRIVER = """ UPDATE drivers SET state = 'available', trip_id = NULL, offer_id = NULL, offer_expires_at = NULL WHERE driver_id = ? AND state = 'offered' AND offer_id = ?"""; static final String RELEASE_TRIP = """ UPDATE trips SET state = 'requested', driver_id = NULL, offer_id = NULL WHERE trip_id = ? AND state = 'offering' AND offer_id = ?""";
private final Db db;
DispatchClaims(Db db) { this.db = db; }
Offer offer(long tripId, long driverId, UUID offerId) { try { return db.inTransaction(tx -> { // No SELECT first: reading "available" and then writing is the race. if (tx.execute(CLAIM_DRIVER, tripId, offerId, driverId) == 0) return Offer.DRIVER_BUSY; // the index hint was stale: next driver if (tx.execute(CLAIM_TRIP, driverId, offerId, tripId) == 0) throw new Rollback(Offer.TRIP_GONE); // cancelled or offered elsewhere: undo the driver claim return Offer.OFFERED; // only now send the offer to the phone }); } catch (Rollback r) { return (Offer) r.result; } }
Accept accept(long tripId, long driverId, UUID offerId) { try { return db.inTransaction(tx -> { if (tx.execute(ACCEPT_DRIVER, driverId, offerId) == 0) // A retried accept whose first response was lost must not see "expired". return tx.exists(ALREADY_ACCEPTED, tripId, offerId) ? Accept.ACCEPTED : Accept.EXPIRED; if (tx.execute(ACCEPT_TRIP, tripId, offerId) == 0) throw new Rollback(Accept.EXPIRED); // the rider cancelled meanwhile return Accept.ACCEPTED; }); } catch (Rollback r) { return (Accept) r.result; } }
// Called when the 15 second timer fires or the driver declines. Safe to call twice. void release(long tripId, long driverId, UUID offerId) { db.inTransaction(tx -> { tx.execute(RELEASE_DRIVER, driverId, offerId); tx.execute(RELEASE_TRIP, tripId, offerId); return null; }); }}# Dispatch claims: every change to who holds a driver is one conditional UPDATE in the city's# trips database. The WHERE clause is the compare, the SET is the swap, and the row lock makes# the pair atomic. now() is the database clock, so servers whose clocks drift never disagree# about when an offer ended.class Tx(Protocol): def execute(self, sql: str, *args) -> int: ... # returns the number of rows changed def exists(self, sql: str, *args) -> bool: ...
class Db(Protocol): def transaction(self) -> ContextManager[Tx]: ... # commits on exit, rolls back on raise
class Offer(Enum): OFFERED = "offered" DRIVER_BUSY = "driver_busy" TRIP_GONE = "trip_gone"
class Accept(Enum): ACCEPTED = "accepted" EXPIRED = "expired"
class _Rollback(Exception): def __init__(self, result): self.result = result
# Free, or held by an offer whose 15 seconds ran out: a dead dispatcher's offer frees itself,# with no sweeper on the correctness path.CLAIM_DRIVER = """ UPDATE drivers SET state = 'offered', trip_id = %s, offer_id = %s, offer_expires_at = now() + interval '15 seconds' WHERE driver_id = %s AND (state = 'available' OR (state = 'offered' AND offer_expires_at < now()))"""# A trip has at most one live offer. This is what stops two dispatchers (say, an old leader# that has not noticed its lease is gone) from offering one trip to two drivers.CLAIM_TRIP = """ UPDATE trips SET state = 'offering', driver_id = %s, offer_id = %s, offer_expires_at = now() + interval '15 seconds' WHERE trip_id = %s AND (state = 'requested' OR (state = 'offering' AND offer_expires_at < now()))"""# Accept only the exact offer, and only before it expires.ACCEPT_DRIVER = """ UPDATE drivers SET state = 'on_trip', offer_expires_at = NULL WHERE driver_id = %s AND state = 'offered' AND offer_id = %s AND offer_expires_at >= now()"""ACCEPT_TRIP = """ UPDATE trips SET state = 'accepted', accepted_at = now() WHERE trip_id = %s AND state = 'offering' AND offer_id = %s"""ALREADY_ACCEPTED = """ SELECT 1 FROM trips WHERE trip_id = %s AND offer_id = %s AND state <> 'offering' AND state <> 'cancelled'"""# Timeout and decline: release only if this offer still holds the row. 0 rows means the accept# won the race, and releasing nothing is the correct outcome.RELEASE_DRIVER = """ UPDATE drivers SET state = 'available', trip_id = NULL, offer_id = NULL, offer_expires_at = NULL WHERE driver_id = %s AND state = 'offered' AND offer_id = %s"""RELEASE_TRIP = """ UPDATE trips SET state = 'requested', driver_id = NULL, offer_id = NULL WHERE trip_id = %s AND state = 'offering' AND offer_id = %s"""
class DispatchClaims: def __init__(self, db: Db): self._db = db
def offer(self, trip_id: int, driver_id: int, offer_id: UUID) -> Offer: try: with self._db.transaction() as tx: # No SELECT first: reading "available" and then writing is the race. if tx.execute(CLAIM_DRIVER, trip_id, offer_id, driver_id) == 0: return Offer.DRIVER_BUSY # the index hint was stale: next driver if tx.execute(CLAIM_TRIP, driver_id, offer_id, trip_id) == 0: # Cancelled or offered elsewhere: raising rolls back the driver claim. raise _Rollback(Offer.TRIP_GONE) return Offer.OFFERED # only now send the offer to the phone except _Rollback as r: return r.result
def accept(self, trip_id: int, driver_id: int, offer_id: UUID) -> Accept: try: with self._db.transaction() as tx: if tx.execute(ACCEPT_DRIVER, driver_id, offer_id) == 0: # A retried accept whose first response was lost must not see "expired". if tx.exists(ALREADY_ACCEPTED, trip_id, offer_id): return Accept.ACCEPTED return Accept.EXPIRED if tx.execute(ACCEPT_TRIP, trip_id, offer_id) == 0: raise _Rollback(Accept.EXPIRED) # the rider cancelled meanwhile return Accept.ACCEPTED except _Rollback as r: return r.result
# Called when the 15 second timer fires or the driver declines. Safe to call twice. def release(self, trip_id: int, driver_id: int, offer_id: UUID) -> None: with self._db.transaction() as tx: tx.execute(RELEASE_DRIVER, driver_id, offer_id) tx.execute(RELEASE_TRIP, trip_id, offer_id)// Dispatch claims: every change to who holds a driver is one conditional UPDATE in the city's// trips database. The WHERE clause is the compare, the SET is the swap, and the row lock makes// the pair atomic. now() is the database clock, so servers whose clocks drift never disagree// about when an offer ended.interface ITx{ int Execute(string sql, params object[] args); // returns the number of rows changed bool Exists(string sql, params object[] args);}
interface IDb{ T InTransaction<T>(Func<ITx, T> work); // commits on return, rolls back on throw}
enum Offer { Offered, DriverBusy, TripGone }enum Accept { Accepted, Expired }
class RollbackException(object result) : Exception{ public object Result { get; } = result;}
class DispatchClaims(IDb db){ // Free, or held by an offer whose 15 seconds ran out: a dead dispatcher's offer frees // itself, with no sweeper on the correctness path. const string ClaimDriver = """ UPDATE drivers SET state = 'offered', trip_id = @p0, offer_id = @p1, offer_expires_at = now() + interval '15 seconds' WHERE driver_id = @p2 AND (state = 'available' OR (state = 'offered' AND offer_expires_at < now())) """; // A trip has at most one live offer. This is what stops two dispatchers (say, an old // leader that has not noticed its lease is gone) from offering one trip to two drivers. const string ClaimTrip = """ UPDATE trips SET state = 'offering', driver_id = @p0, offer_id = @p1, offer_expires_at = now() + interval '15 seconds' WHERE trip_id = @p2 AND (state = 'requested' OR (state = 'offering' AND offer_expires_at < now())) """; // Accept only the exact offer, and only before it expires. const string AcceptDriver = """ UPDATE drivers SET state = 'on_trip', offer_expires_at = NULL WHERE driver_id = @p0 AND state = 'offered' AND offer_id = @p1 AND offer_expires_at >= now() """; const string AcceptTrip = """ UPDATE trips SET state = 'accepted', accepted_at = now() WHERE trip_id = @p0 AND state = 'offering' AND offer_id = @p1 """; const string AlreadyAccepted = """ SELECT 1 FROM trips WHERE trip_id = @p0 AND offer_id = @p1 AND state <> 'offering' AND state <> 'cancelled' """; // Timeout and decline: release only if this offer still holds the row. 0 rows means the // accept won the race, and releasing nothing is the correct outcome. const string ReleaseDriver = """ UPDATE drivers SET state = 'available', trip_id = NULL, offer_id = NULL, offer_expires_at = NULL WHERE driver_id = @p0 AND state = 'offered' AND offer_id = @p1 """; const string ReleaseTrip = """ UPDATE trips SET state = 'requested', driver_id = NULL, offer_id = NULL WHERE trip_id = @p0 AND state = 'offering' AND offer_id = @p1 """;
public Offer TryOffer(long tripId, long driverId, Guid offerId) { try { return db.InTransaction(tx => { // No SELECT first: reading "available" and then writing is the race. if (tx.Execute(ClaimDriver, tripId, offerId, driverId) == 0) return Offer.DriverBusy; // the index hint was stale: next driver if (tx.Execute(ClaimTrip, driverId, offerId, tripId) == 0) throw new RollbackException(Offer.TripGone); // cancelled or offered elsewhere: undo the driver claim return Offer.Offered; // only now send the offer to the phone }); } catch (RollbackException r) { return (Offer)r.Result; } }
public Accept TryAccept(long tripId, long driverId, Guid offerId) { try { return db.InTransaction(tx => { if (tx.Execute(AcceptDriver, driverId, offerId) == 0) // A retried accept whose first response was lost must not see "expired". return tx.Exists(AlreadyAccepted, tripId, offerId) ? Accept.Accepted : Accept.Expired; if (tx.Execute(AcceptTrip, tripId, offerId) == 0) throw new RollbackException(Accept.Expired); // the rider cancelled meanwhile return Accept.Accepted; }); } catch (RollbackException r) { return (Accept)r.Result; } }
// Called when the 15 second timer fires or the driver declines. Safe to call twice. public void Release(long tripId, long driverId, Guid offerId) => db.InTransaction(tx => { tx.Execute(ReleaseDriver, driverId, offerId); tx.Execute(ReleaseTrip, tripId, offerId); return 0; });}// Dispatch claims: every change to who holds a driver is one conditional UPDATE in the city's// trips database. The WHERE clause is the compare, the SET is the swap, and the row lock makes// the pair atomic. now() is the database clock, so servers whose clocks drift never disagree// about when an offer ended.
// Tx is one open database transaction.type Tx interface { Exec(sql string, args ...any) (int64, error) // returns the number of rows changed Exists(sql string, args ...any) (bool, error)}
// DB commits when fn returns nil and rolls back when it returns an error.type DB interface { InTx(fn func(Tx) error) error}
type Offer int
const ( Offered Offer = iota DriverBusy TripGone)
var errRollback = errors.New("rollback") // undoes the transaction; the outcome is already set
const ( // Free, or held by an offer whose 15 seconds ran out: a dead dispatcher's offer frees // itself, with no sweeper on the correctness path. claimDriver = `UPDATE drivers SET state = 'offered', trip_id = $1, offer_id = $2, offer_expires_at = now() + interval '15 seconds'WHERE driver_id = $3 AND (state = 'available' OR (state = 'offered' AND offer_expires_at < now()))` // A trip has at most one live offer. This is what stops two dispatchers (say, an old // leader that has not noticed its lease is gone) from offering one trip to two drivers. claimTrip = `UPDATE trips SET state = 'offering', driver_id = $1, offer_id = $2, offer_expires_at = now() + interval '15 seconds'WHERE trip_id = $3 AND (state = 'requested' OR (state = 'offering' AND offer_expires_at < now()))` // Accept only the exact offer, and only before it expires. acceptDriver = `UPDATE drivers SET state = 'on_trip', offer_expires_at = NULLWHERE driver_id = $1 AND state = 'offered' AND offer_id = $2 AND offer_expires_at >= now()` acceptTrip = `UPDATE trips SET state = 'accepted', accepted_at = now()WHERE trip_id = $1 AND state = 'offering' AND offer_id = $2` alreadyAccepted = `SELECT 1 FROM trips WHERE trip_id = $1 AND offer_id = $2 AND state <> 'offering' AND state <> 'cancelled'` // Timeout and decline: release only if this offer still holds the row. 0 rows means the // accept won the race, and releasing nothing is the correct outcome. releaseDriver = `UPDATE drivers SET state = 'available', trip_id = NULL, offer_id = NULL, offer_expires_at = NULLWHERE driver_id = $1 AND state = 'offered' AND offer_id = $2` releaseTrip = `UPDATE trips SET state = 'requested', driver_id = NULL, offer_id = NULLWHERE trip_id = $1 AND state = 'offering' AND offer_id = $2`)
type DispatchClaims struct{ db DB }
func (c *DispatchClaims) Offer(tripID, driverID int64, offerID string) (Offer, error) { result := Offered err := c.db.InTx(func(tx Tx) error { // No SELECT first: reading "available" and then writing is the race. n, err := tx.Exec(claimDriver, tripID, offerID, driverID) if err != nil { return err } if n == 0 { result = DriverBusy // the index hint was stale: next driver return nil } if n, err = tx.Exec(claimTrip, driverID, offerID, tripID); err != nil { return err } if n == 0 { result = TripGone // cancelled or offered elsewhere: undo the driver claim return errRollback } return nil // only now send the offer to the phone }) if errors.Is(err, errRollback) { err = nil } return result, err}
// Accept reports whether the driver now has the trip.func (c *DispatchClaims) Accept(tripID, driverID int64, offerID string) (bool, error) { accepted := false err := c.db.InTx(func(tx Tx) error { n, err := tx.Exec(acceptDriver, driverID, offerID) if err != nil { return err } if n == 0 { // A retried accept whose first response was lost must not see "expired". accepted, err = tx.Exists(alreadyAccepted, tripID, offerID) return err } if n, err = tx.Exec(acceptTrip, tripID, offerID); err != nil { return err } if n == 0 { return errRollback // the rider cancelled meanwhile } accepted = true return nil }) if errors.Is(err, errRollback) { err = nil } return accepted, err}
// Release runs when the 15 second timer fires or the driver declines. Safe to call twice.func (c *DispatchClaims) Release(tripID, driverID int64, offerID string) error { return c.db.InTx(func(tx Tx) error { if _, err := tx.Exec(releaseDriver, driverID, offerID); err != nil { return err } _, err := tx.Exec(releaseTrip, tripID, offerID) return err })}// Dispatch claims: every change to who holds a driver is one conditional UPDATE in the city's// trips database. The WHERE clause is the compare, the SET is the swap, and the row lock makes// the pair atomic. now() is the database clock, so servers whose clocks drift never disagree// about when an offer ended.interface Tx { execute(sql: string, ...args: unknown[]): Promise<number>; // rows changed exists(sql: string, ...args: unknown[]): Promise<boolean>;}
interface Db { inTransaction<T>(work: (tx: Tx) => Promise<T>): Promise<T>; // commits, or rolls back on throw}
type Offer = 'offered' | 'driverBusy' | 'tripGone';type Accept = 'accepted' | 'expired';
class Rollback<T> extends Error { constructor(readonly result: T) { super('rollback'); }}
// Free, or held by an offer whose 15 seconds ran out: a dead dispatcher's offer frees itself,// with no sweeper on the correctness path.const CLAIM_DRIVER = ` UPDATE drivers SET state = 'offered', trip_id = $1, offer_id = $2, offer_expires_at = now() + interval '15 seconds' WHERE driver_id = $3 AND (state = 'available' OR (state = 'offered' AND offer_expires_at < now()))`;// A trip has at most one live offer. This is what stops two dispatchers (say, an old leader// that has not noticed its lease is gone) from offering one trip to two drivers.const CLAIM_TRIP = ` UPDATE trips SET state = 'offering', driver_id = $1, offer_id = $2, offer_expires_at = now() + interval '15 seconds' WHERE trip_id = $3 AND (state = 'requested' OR (state = 'offering' AND offer_expires_at < now()))`;// Accept only the exact offer, and only before it expires.const ACCEPT_DRIVER = ` UPDATE drivers SET state = 'on_trip', offer_expires_at = NULL WHERE driver_id = $1 AND state = 'offered' AND offer_id = $2 AND offer_expires_at >= now()`;const ACCEPT_TRIP = ` UPDATE trips SET state = 'accepted', accepted_at = now() WHERE trip_id = $1 AND state = 'offering' AND offer_id = $2`;const ALREADY_ACCEPTED = ` SELECT 1 FROM trips WHERE trip_id = $1 AND offer_id = $2 AND state <> 'offering' AND state <> 'cancelled'`;// Timeout and decline: release only if this offer still holds the row. 0 rows means the accept// won the race, and releasing nothing is the correct outcome.const RELEASE_DRIVER = ` UPDATE drivers SET state = 'available', trip_id = NULL, offer_id = NULL, offer_expires_at = NULL WHERE driver_id = $1 AND state = 'offered' AND offer_id = $2`;const RELEASE_TRIP = ` UPDATE trips SET state = 'requested', driver_id = NULL, offer_id = NULL WHERE trip_id = $1 AND state = 'offering' AND offer_id = $2`;
class DispatchClaims { constructor(private readonly db: Db) {}
async offer(tripId: number, driverId: number, offerId: string): Promise<Offer> { try { return await this.db.inTransaction(async (tx) => { // No SELECT first: reading "available" and then writing is the race. if ((await tx.execute(CLAIM_DRIVER, tripId, offerId, driverId)) === 0) return 'driverBusy'; // the index hint was stale: next driver if ((await tx.execute(CLAIM_TRIP, driverId, offerId, tripId)) === 0) throw new Rollback<Offer>('tripGone'); // cancelled or offered elsewhere: undo the driver claim return 'offered'; // only now send the offer to the phone }); } catch (e) { if (e instanceof Rollback) return e.result as Offer; throw e; } }
async accept(tripId: number, driverId: number, offerId: string): Promise<Accept> { try { return await this.db.inTransaction(async (tx) => { if ((await tx.execute(ACCEPT_DRIVER, driverId, offerId)) === 0) // A retried accept whose first response was lost must not see "expired". return (await tx.exists(ALREADY_ACCEPTED, tripId, offerId)) ? 'accepted' : 'expired'; if ((await tx.execute(ACCEPT_TRIP, tripId, offerId)) === 0) throw new Rollback<Accept>('expired'); // the rider cancelled meanwhile return 'accepted'; }); } catch (e) { if (e instanceof Rollback) return e.result as Accept; throw e; } }
// Called when the 15 second timer fires or the driver declines. Safe to call twice. async release(tripId: number, driverId: number, offerId: string): Promise<void> { await this.db.inTransaction(async (tx) => { await tx.execute(RELEASE_DRIVER, driverId, offerId); await tx.execute(RELEASE_TRIP, tripId, offerId); }); }}The driver row is the truth about who is free. The available flag in the memory index is only a
hint, copied from the database. So the dispatcher sometimes ranks a driver who was claimed a moment
ago. That claim fails, and it tries the next one.
Stage 3: Dispatch with atomic claims
Stage 3 of 7
Dispatch with atomic claims
Why this stage
A requirement, not load: two riders must never get the same driver, and a driver who ignores an offer must not strand the rider.
What we add
A dispatch service that ranks nearby drivers by road ETA and offers the trip to one driver at a time for 15 seconds, claiming each with a conditional update in the trips database.
What it costs
A database round trip per offer, and up to 15 seconds lost per driver who does not answer.
At 250k location pings per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One Postgres primary takes about 5,000 small write transactions a second.
- One in-memory geo index node takes about 400,000 location updates a second when location servers batch them.
- One location server handles about 20,000 pings a second.
- One gateway handles about 25,000 messages a second over about 100,000 sockets.
What one ride request costs
- Each ride request makes about 6 trip transactions and 1.5 offers.
- It also makes 7 routing calls: 5 price quotes, 1 candidate matrix and 1 route.
- Once sharded, there are 20 city shards of each.
Why Kafka has no bar
- 250,000 batched pings a second is about 25 MB/s. That is a few percent of a 6-broker cluster, which takes several hundred MB/s.
- It grows by adding brokers and partitions per city.
Go deeperWhy the claim lives in the database, not in Redis1 min
Redis is fast, and a Redis script can also check and set a driver’s state in one atomic step. So why pay for a database transaction on every offer?
Because a claim is hard state: if it is lost, a driver can get two trips. A Redis primary confirms a write before its replica has it. If the primary dies just then, the replica takes over without that write, and the driver looks free again. A database that copies each commit to a second zone before confirming it doesn’t have that gap.
The numbers make this affordable. Positions arrive 250,000 times a second, so they stay in memory. Claims happen about 700 times a second at peak, 1.5 offers for each of about 460 requests. One database primary handles that with room to spare. Each kind of data goes where its rate and its cost of loss say it should.
Step 4. Get news to phones
Section titled “Step 4. Get news to phones”An offer must reach the driver within a second, and 560,000 riders watch a car move. Polling every second would be 1.5 million mostly empty requests a second. So phones keep a socket open to an edge gateway, and services publish to channels: one per driver for offers, one per trip for the rider. The gateway holding a phone’s socket subscribes to that phone’s channels, so no service needs to know which gateway that is.
Messages are fire and forget, because the truth is always in the database or the next ping. When the app is in the background, offers go by mobile push instead. When the socket is quiet, the app polls its trip every few seconds.
Stage 4: Push to phones
Stage 4 of 7
Push to phones
Why this stage
A requirement: an offer must reach the driver within a second, and the rider watches the car approach.
Polling every second from 1.5 million phones is 1.5 million mostly empty requests a second.
What we add
Edge gateways that hold one socket per phone, and a pub/sub layer with a channel per trip and per driver.
What it costs
Millions of long-lived connections to drain on every deploy, and a fallback to mobile push when a socket is down.
At 250k location pings per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One Postgres primary takes about 5,000 small write transactions a second.
- One in-memory geo index node takes about 400,000 location updates a second when location servers batch them.
- One location server handles about 20,000 pings a second.
- One gateway handles about 25,000 messages a second over about 100,000 sockets.
What one ride request costs
- Each ride request makes about 6 trip transactions and 1.5 offers.
- It also makes 7 routing calls: 5 price quotes, 1 candidate matrix and 1 route.
- Once sharded, there are 20 city shards of each.
Why Kafka has no bar
- 250,000 batched pings a second is about 25 MB/s. That is a few percent of a 6-broker cluster, which takes several hundred MB/s.
- It grows by adding brokers and partitions per city.
Follow a request, then an accept, through the design:
Trace a request: Request a ride
Request a ride: step 1 of 10
Riders to Edge gateways
The request arrives on the rider’s open socket with an idempotency key and the quote id. The gateway checks the session token and forwards by city.
- Riders → Edge gateways2 ms
- Edge gateways → Trip service1 ms
- Trip service → Trips DB5 ms
- Trip service → Dispatch1 ms
- Dispatch → Geo index2 ms
- Dispatch → ETA + pricing30 ms
- Dispatch → Trips DB5 ms
- Dispatch → Pub/sub2 ms
- Pub/sub → Edge gateways2 ms
- Response
This is enough for a mid-level answer. Positions in memory, a conditional claim in the database, one offer at a time, and sockets. Step 5 and the stress tests are what make it a senior answer.
Go deeperOther ways to reach phones1 min
| Option | Delay | Cost |
|---|---|---|
| Polling every few seconds | Up to the interval | Mostly empty requests; drains the battery |
| WebSocket or gRPC stream | Tens of ms | Gateways hold sockets; deploys must drain them |
| Mobile push (APNs, FCM) | Seconds, no guarantee | Works in the background; rate limited |
One socket per phone carries both pings and offers, so there’s no TLS handshake or HTTP headers per ping. The cost is gateways holding millions of long-lived connections. A lost “driver assigned” message leaves a rider on a spinner until the app’s poll catches up, which is why the app polls while its socket is quiet.
Go deeperThe API and the two tables2 min
POST /v1/quotes { "pickup": {...}, "dropoff": {...}, "product": "standard" }200 OK { "quoteId": "q_8f2c...", "fare": {"amount": 2350, "currency": "USD"}, "surge": 1.4, "pickupEtaSeconds": 240, "expiresAt": "..." }
POST /v1/trips Idempotency-Key: 3b9e7d10-... { "quoteId": "q_8f2c..." }201 Created { "tripId": 90471123, "state": "requested" }
DRIVER SOCKET up {"type":"ping","seq":5512,"lat":37.7801,"lng":-122.4120,"heading":84} down {"type":"offer","offerId":"o_51ac...","tripId":90471123,"expiresAt":"..."}
POST /v1/offers/{offerId}/accept 200 {tripId} | 409 {"reason":"expired"}POST /v1/trips/{tripId}/events {"type":"arrived" | "started" | "completed"}A quote locks the price for 2 minutes, so surge can’t change between what the rider saw and what they requested. The idempotency key makes a retry after a lost response return the same trip. The accept names the offer, not just the trip, because only the current offer may be accepted.
CREATE TABLE drivers ( -- the truth about who is free driver_id BIGINT PRIMARY KEY, city_id INT NOT NULL, state TEXT NOT NULL, -- offline | available | offered | on_trip trip_id BIGINT, offer_id UUID, -- which offer holds the driver right now offer_expires_at TIMESTAMPTZ);
CREATE TABLE trips ( trip_id BIGINT PRIMARY KEY, city_id INT NOT NULL, rider_id BIGINT NOT NULL, driver_id BIGINT, state TEXT NOT NULL, -- requested | offering | accepted | arrived | in_progress | ... offer_id UUID, quote_id UUID NOT NULL, -- the locked price idem_key UUID NOT NULL);CREATE UNIQUE INDEX one_active_trip ON trips (rider_id) WHERE state IN ('requested', 'offering', 'accepted', 'arrived', 'in_progress');The partial unique index allows one active trip per rider, so a double tap on Request can’t create two trips, even on two app servers.
Go deeperOther places positions and claims could live1 min
| Positions in | Wins | Loses |
|---|---|---|
| Postgres with PostGIS | Exact queries, one store | 50 times past write capacity; index churn on every move |
Redis GEOADD |
Built in, fast, a good start | One set per area gets hot; replication can lose writes |
| In-memory index per city | Millions of updates a second | We run the service, and a restart starts empty |
| Claim made by | Verdict |
|---|---|
| The dispatcher’s memory | Correct until there are two processes, which failover guarantees one day |
| A Redis lock per driver | A paused holder, or a failover, can give two dispatchers the same driver |
| A ZooKeeper or etcd lock | Safe with fencing, but a consensus write per offer, and it still needs a fence |
| A conditional update in the trips database | The store refuses the second claim, and the claim is durable |
Redis is the wrong store for claims even without locks: it copies writes to replicas after answering, so a failover can lose a claim it already confirmed. DynamoDB with conditions, or Spanner, would also work.
Why not a quadtree, since drivers crowd downtown? Every move in a quadtree is a delete and an insert into a tree that may split or merge. That’s a poor fit for 250,000 moves a second. Fixed cells make a move two set operations. Density is handled at search time: downtown, one ring already holds enough drivers, and the search stops early.
Step 5. Split by city
Section titled “Step 5. Split by city”The last thing to break is size: ten times the drivers, in ten times the cities.
At ten times the drivers, pings are six times what one index takes, and trip writes five times one database primary. Every dispatch question is local: drivers near this pickup, in this city. So the position index, the dispatcher and the trips database are all split by city, and a request never leaves its city’s shard. A city map says which shard owns each city. Every gateway caches it, and routes a ping by the driver’s city, and a request by its pickup.
Stage 5: Shard by city
Stage 5 of 7
Shard by city
Why this stage
At 10 times the load, 2.5 million pings/s is 6 times what one geo index takes, and trip transactions are 5 times what one primary takes.
What we add
The geo index, the dispatcher and the trips database partitioned by city, and a city map that routes every ping and request to its city’s shard.
What it costs
Trips that start near a city line need a rule, and the biggest city is still one shard.
At 250k location pings per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One Postgres primary takes about 5,000 small write transactions a second.
- One in-memory geo index node takes about 400,000 location updates a second when location servers batch them.
- One location server handles about 20,000 pings a second.
- One gateway handles about 25,000 messages a second over about 100,000 sockets.
What one ride request costs
- Each ride request makes about 6 trip transactions and 1.5 offers.
- It also makes 7 routing calls: 5 price quotes, 1 candidate matrix and 1 route.
- Once sharded, there are 20 city shards of each.
Why Kafka has no bar
- 250,000 batched pings a second is about 25 MB/s. That is a few percent of a 6-broker cluster, which takes several hundred MB/s.
- It grows by adding brokers and partitions per city.
| Split by | Nearby search | Claims | Problem |
|---|---|---|---|
| City | One shard | One shard, one transaction | The biggest city is one shard; trips near a city line |
| H3 cell, hashed | 7 to 19 cells, many shards | A driver moves between shards | Every search asks many shards |
| Driver ID | Every shard | One shard | Every search asks every shard |
The biggest city, at ten times today, is 125,000 pings a second: a third of one shard. Near a city line, like an airport just outside the limits, the pickup decides the owner. Its dispatcher also reads the neighbor’s index for the outer ring of cells. Shards are uneven (São Paulo isn’t Boise), so they’re sized per group of cities, not evenly.
Stress it
Section titled “Stress it”The design is complete. Now we test it where ride hailing is hard: every race that could give a driver two trips, how to choose the driver, and surge pricing with payments. The last one adds a few boxes, because surge and payments need events that leave the dispatch path.
Never two riders, never two drivers
Section titled “Never two riders, never two drivers”List every way a driver could end up with two trips, or a trip with two drivers. Show me each one is closed.
Whether you can list the races one by one, including retries and cancels, and show each is closed.
The hardest race is an accept that arrives just as the 15 seconds run out. Both the accept and the timeout are conditional updates on the same offer ID. Whichever commits first wins, and the other changes 0 rows. There’s one outcome, judged by one clock, the database’s.
- Color key
- Service
- Database
Every other race closes the same way:
| Race | What closes it |
|---|---|
| Two requests pick the same driver | The driver update needs available: one changes 1 row, the other 0 and moves on |
| Two dispatchers offer one trip | The trip allows one live offer; the second transaction rolls back |
| Accept retried after a lost response | The retry finds the trip accepted with its own offer ID, and returns 200 |
| Rider cancels while an offer is out | Cancel releases the driver held by that offer; a racing accept needs offering |
| Rider double taps Request | The idempotency key, and the unique index on active trips |
| Dispatcher dies holding offers | Its offers expire after 15 seconds, and anyone can claim those drivers again |
The promise, stated exactly. At every committed point, a driver references at most one trip, and a trip at most one driver. This holds whatever leases, clocks, retries and network splits do, because every write that could break it is conditional, in one store, with one clock. We don’t promise an offer never fails on accept, or that matching is never a few seconds late.
The trip itself moves through a small state machine, and every arrow is a compare-and-set:
- Color key
- State
- Ends well
Go deeperHow exact each answer is1 min
| Operation | Guarantee | What users can see |
|---|---|---|
| Driver claim, accept, release | Exact per row: conditional updates on one primary | Nothing odd; a late accept gets a clear “expired” |
| Trip state changes | Exact per trip, only along the state machine | A cancel and an accept never both succeed |
| Positions in the index | Newest wins by sequence number, at most 15 s old | A car on the map 4 seconds behind reality |
available in the index |
A hint that catches up with the database | Sometimes a claim fails; riders never see it |
| Surge multiplier | About a minute behind | Price lags a sudden crowd |
| Fare charged | Exactly once, through the outbox and idempotency key | The charge appears minutes after the ride |
“The system is eventually consistent”, said of the whole design, is wrong. It invites two riders sharing a car for a few seconds. Positions and surge may lag, with stated limits. Claims and trip states are exact. Charges happen once through idempotency, not through a transaction across systems.
Choosing the driver
Section titled “Choosing the driver”Why not just pick the nearest driver? And why offer to one driver instead of five?
We already rank by road ETA. The open question is how many drivers to ask, and when.
| Strategy | How it works | Wins | Loses |
|---|---|---|---|
| Broadcast to several | Offer to the 5 best, first accept wins | Fast acceptance | 4 drivers lose a trip they tapped; they stop trusting offers |
| Greedy, one at a time | Best ETA first, 15 s, then the next | Fair to drivers, simple to get right | A slow decline costs the rider 15 s |
| Batched matching | Collect 1 to 2 s of requests, solve them together | Lower total wait across the city | Adds a second or two; more complex |
We pick one at a time, with batching for dense cities. In a quiet area, requests rarely compete for drivers, and taking the best driver now is close to the best overall. Downtown at rush hour, it can take the only driver near the next rider. There, solving the last second or two of requests together, as a minimum-cost assignment, cuts the average pickup time. Each chosen pair still goes through the same claim, so the correctness argument doesn’t change.
Surge, receipts and payments
Section titled “Surge, receipts and payments”How do prices go up on a Friday night, and how does the rider get charged?
Three consumers want the same events, and none may slow dispatch. So the location service batches pings into Kafka, split by city. The trip service writes each event to an outbox table in the same transaction as the trip change, and a relay ships them to Kafka. An outbox means the event can’t be lost between the database and Kafka, because it’s committed with the change.
Stage 6: Events, surge and payments
Stage 6 of 7
Events, surge and payments
Why this stage
A requirement: prices follow supply and demand per area, every fare is charged exactly once, and trip trails are kept for receipts and disputes, none of it on the dispatch path.
What we add
Kafka carrying pings and trip events, a stream job that computes a surge multiplier per cell every minute, and an outbox that hands finished trips to the payment system.
What it costs
Surge lags demand by about a minute, and every consumer must tolerate duplicates.
At 250k location pings per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One Postgres primary takes about 5,000 small write transactions a second.
- One in-memory geo index node takes about 400,000 location updates a second when location servers batch them.
- One location server handles about 20,000 pings a second.
- One gateway handles about 25,000 messages a second over about 100,000 sockets.
What one ride request costs
- Each ride request makes about 6 trip transactions and 1.5 offers.
- It also makes 7 routing calls: 5 price quotes, 1 candidate matrix and 1 route.
- Once sharded, there are 20 city shards of each.
Why Kafka has no bar
- 250,000 batched pings a second is about 25 MB/s. That is a few percent of a 6-broker cluster, which takes several hundred MB/s.
- It grows by adding brokers and partitions per city.
- Trail. Pings go to object storage, by day and city, for receipts, disputes and ETA training.
- Surge. Every minute, a stream job compares demand with free drivers in each larger H3 cell (about 5 km²). Demand counts quotes too, which catches riders who looked and left. The ratio maps to a capped multiplier, smoothed across neighboring minutes and cells so it doesn’t flicker.
- Payments. The fare and an outbox row commit together when a trip ends. The relay hands the trip to the payment system, with the trip ID as the idempotency key, so a retry never charges twice.
A payment outage delays charges; it never stops a ride. The cost is that surge lags demand by about a minute, and every consumer must handle duplicates.
Follow one ping into the event stream:
Trace a request: A driver’s ping on a trip
A driver’s ping on a trip: step 1 of 7
Drivers to Edge gateways
A 60 byte frame on the open socket: sequence number, time, position, heading, speed and GPS accuracy. No TLS handshake, no HTTP headers.
- Drivers → Edge gateways1 ms
- Edge gateways → Location service1 ms
- Location service → Geo index1 ms
- Location service → Pub/sub1 ms
- Pub/sub → Edge gateways1 ms
- Location service → Kafkaasync
- Response
Go deeperETA and price, in more detail1 min
ETA. A routing service holds each region’s road map in memory, with travel times updated from recent trails. With a precomputed shortcut graph (contraction hierarchies), one route takes a few milliseconds, and a one-to-ten table about 30 ms. A model corrects the result, because map ETAs are too hopeful about pickups.
Price. A quote is (base + per_km × km + per_min × minutes) × surge, with a minimum fare. It’s
stored and locked for 2 minutes. The trip carries its quote ID, so what the rider saw is what they
pay, unless the route changes a lot.
| Choice | We chose | Alternative | Why |
|---|---|---|---|
| Surge cell size | About 5 km² | About 0.1 km² | Small cells have too few events to be stable |
| Surge update | Every minute, smoothed | Every request | Prices that change every second feel random and invite gaming |
| Where ETA comes from | One table call per dispatch | A precomputed cell-to-cell table | Tables go stale in traffic; the table is the fallback |
If the surge job stalls, pricing slowly brings stale multipliers back toward 1.
When things break
Section titled “When things break”The machine running New York’s dispatcher dies at 6 pm on a Friday. What happens, and how do you make sure the replacement never double dispatches?
Whether failover is fast, and whether correctness holds even when two dispatchers act at once.
A city’s dispatcher owns its in-flight matching, so its failure stops matching in that city. The fix has two halves: make the takeover fast, and make it safe even when it isn’t clean.
Fast. Each city runs a hot standby dispatcher. Its own position index reads the same pings from Kafka, at most a second behind. Ownership is a lease, a claim that’s valid for 5 seconds and renewed every second. When renewals stop, the standby takes the lease with a higher number, called an epoch. It reloads driver states from the database, and frees any offer past its expiry. Riders who just tapped Request wait 5 to 10 seconds longer. Matched trips never notice.
Safe. Sometimes two dispatchers will both act for a city: a long garbage-collection pause, a network split, a slow machine. The design makes that harmless, not impossible. Every offer is the same conditional update, so the old dispatcher’s writes fail like anyone else’s that lost a race. The epoch only tells it to step down when it next checks. Correctness never depends on it.
- Color key
- Database
- Service
Offers the dead dispatcher sent need no cleanup. They expire after 15 seconds. A driver who accepts one in time still gets the trip, because accepts go to the trip service and the database.
These add the last boxes to the drawing: a standby dispatcher and a standby database for each city.
Stage 7: Fail over a city
Stage 7 of 7
Fail over a city
Why this stage
A city shard is a failure domain: if its dispatcher dies, nobody in that city gets a car.
Failover must never let two dispatchers assign the same driver or the same trip.
What we add
A hot standby per city that tails the same ping stream, ownership leases with a rising epoch, and every claim fenced by the database.
What it costs
Twice the geo index memory, and 5 to 10 seconds without new matches in that city during a failover.
At 250k location pings per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One Postgres primary takes about 5,000 small write transactions a second.
- One in-memory geo index node takes about 400,000 location updates a second when location servers batch them.
- One location server handles about 20,000 pings a second.
- One gateway handles about 25,000 messages a second over about 100,000 sockets.
What one ride request costs
- Each ride request makes about 6 trip transactions and 1.5 offers.
- It also makes 7 routing calls: 5 price quotes, 1 candidate matrix and 1 route.
- Once sharded, there are 20 city shards of each.
Why Kafka has no bar
- 250,000 batched pings a second is about 25 MB/s. That is a few percent of a 6-broker cluster, which takes several hundred MB/s.
- It grows by adding brokers and partitions per city.
Click any box to break it and see what riders and drivers would notice:
Break it: what fails, who notices, what users see
Failure: click any box to break it
A city’s dispatcher goes down
What breaks
New requests in that city are not matched; offers in flight are not followed up.
How we notice
Its lease is not renewed within 5 seconds.
What users see
Riders who just requested wait 5 to 10 seconds longer.
Trips already matched are untouched.
What we do
The standby takes the lease with a higher epoch, reloads driver states and the requested and offering trips from the database, and releases expired offers.
Every claim is a conditional update, so a paused old dispatcher that wakes up has its writes refused.
Positions are soft state and claims are hard state, so the failure story splits the same way. Losing a position index costs seconds and heals itself. Losing a claim would be a correctness bug, so claims only live in a database that copies each write to another zone before confirming it.
Go deeperEvery failure, and what users see1 min
| Part | What users see | What we do |
|---|---|---|
| Edge gateway | 100,000 phones reconnect within seconds | Spread-out reconnects; pings deduplicated by sequence |
| Pub/sub | Maps freeze; offers time out | Mobile push for offers, polling for trips |
| Position index | A second of slower matching | Promote the standby; pings refill any gap in 4 s |
| Dispatcher | New requests wait 5 to 10 s longer | Standby takes the lease; claims fenced by the database |
| Trips database | Requests and accepts fail for about 30 s | Synchronous replica, safe retries |
| ETA and pricing | Slightly worse matches | Straight-line time × a speed factor; a cached ETA table |
| Kafka | Nothing at once; surge and receipts lag | The outbox holds trip events; surge drifts toward 1 |
| Whole region | Its cities show “no cars” for minutes | Promote replicas and leases in the standby region |
Under stress, shed in this order: the “cars near you” map for riders who haven’t requested, then quotes (serve cached surge), then position pushes to riders (every 8 seconds). Requests, offers and accepts go last.
A whole region failing is rarer. Each city’s database has a synchronous replica in another zone, and an asynchronous one in a standby region. So promoting the standby region can lose the last second of claims. Matching there resumes after driver apps reconnect and report the trip they’re on, which repairs those rows.
Go deeperRunning it: alerts, abuse and deploys1 min
Wake someone up, per city, when: the share of requests matched within 60 seconds drops below normal; request-to-offer time passes 1 second at p99; the share of online drivers seen in the last 15 seconds drops; the offer acceptance rate falls suddenly (offers aren’t reaching phones); or claims that change 0 rows spike (stale hints, or two active dispatchers).
Abuse and safety. Spoofing apps fake positions to jump airport queues or inflate fares. Reject impossible speeds, cross-check cell tower and Wi-Fi hints, and flag trails that jump. Keep exact trails only as long as disputes and law require.
Deploys. Gateways drain: stop taking sockets, then close old ones a few thousand at a time, so phones don’t all reconnect at once. Dispatchers update the standby first, then hand over the lease. Moving a city to another shard is the same handover.
Wrap up
Section titled “Wrap up”The whole design
Section titled “The whole design”Here is everything we built, in the order we built it. Use the buttons to step through it.
The full design, stage by stage
Stage 7 of 7
Fail over a city
Why this stage
A city shard is a failure domain: if its dispatcher dies, nobody in that city gets a car.
Failover must never let two dispatchers assign the same driver or the same trip.
What we add
A hot standby per city that tails the same ping stream, ownership leases with a rising epoch, and every claim fenced by the database.
What it costs
Twice the geo index memory, and 5 to 10 seconds without new matches in that city during a failover.
At 250k location pings per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One Postgres primary takes about 5,000 small write transactions a second.
- One in-memory geo index node takes about 400,000 location updates a second when location servers batch them.
- One location server handles about 20,000 pings a second.
- One gateway handles about 25,000 messages a second over about 100,000 sockets.
What one ride request costs
- Each ride request makes about 6 trip transactions and 1.5 offers.
- It also makes 7 routing calls: 5 price quotes, 1 candidate matrix and 1 route.
- Once sharded, there are 20 city shards of each.
Why Kafka has no bar
- 250,000 batched pings a second is about 25 MB/s. That is a few percent of a 6-broker cluster, which takes several hundred MB/s.
- It grows by adding brokers and partitions per city.
The decisions, and what each one costs
Section titled “The decisions, and what each one costs”| Decision | We chose | We gave up | We’d choose differently if |
|---|---|---|---|
| Positions | In-memory index per city, soft state | Durable positions | Positions were billed or audited in real time |
| Spatial index | H3 cells with a set of drivers per cell | Exact nearest-neighbor structures | Points barely moved (see proximity service) |
| Claims | Conditional updates in the trips database | A few ms per offer | One process per city, with no failover, were fine |
| Offers | One driver at a time, batched when dense | Fastest possible acceptance | Drivers didn’t mind losing trips they tapped |
| Split | By city | Even shard sizes | One city outgrew a shard (split it into regions) |
| Failover | Hot standby and leases, fenced by the database | Memory for a second index per city | Seconds without matching were acceptable |
| Surge, payments | Asynchronous, from Kafka and an outbox | Real-time prices, instant charges | Prices had to react within seconds |
How to spend 45 minutes
Section titled “How to spend 45 minutes”- 0 to 4. Drivers, pings, trips, offers. State “never double booked” as the one hard promise.
- 4 to 9. 250,000 pings against 460 requests a second, and a 200 MB index.
- 9 to 20. Build it: positions in memory, the conditional claim, sockets. A mid-level answer can stop here.
- 20 to 24. The split by city.
- 24 to 38. The stress tests: the race table first, always. Then matching, then surge and payments.
- 38 to 45. Failover, then your own questions.
Never skip the claim, its race with the timeout, or why positions are soft state.
Follow-up questions
Section titled “Follow-up questions”Interviewer asksAdd pooled rides. What changes?
Matching becomes route insertion: for each car already carrying a rider, compute the detour of adding the new pickup and dropoff, and keep it within a limit for both riders. A driver can now be on several trips, so the driver row holds seats left and a version number, and the claim updates it only if the version matches. The race table still applies, with “seats left” in place of “available”.
Interviewer asksThe trip completes, but the payment hand-off times out. Could the rider be charged twice?
No. The fare and an outbox row were committed together. The relay retries with the trip ID as the idempotency key, and the payment system returns the original result for a key it has seen. A trip that’s never acknowledged stays in the outbox, and pages someone.
Go deeperMore follow-up questions1 min
Riders who haven’t requested see little cars moving near them. That’s a million riders refreshing. How? Never search the index per rider. Each city publishes, every few seconds, a rough snapshot per cell: a few approximate positions and headings. Gateways cache it and send each rider their cell’s and its neighbors’. It’s decoration, and it hides exact positions.
What changes at 100 times the scale? The biggest cities split into regions of cells, with dispatch reading across region lines. The trips database moves to a store that splits ranges on its own, such as Spanner or CockroachDB, still keyed by city so a claim stays in one range.
A rider says the driver took a long detour. How do you settle it? From the trail in object storage: the driven route against the route the quote assumed. A large detour triggers an automatic refund. This is why on-trip pings are kept.
Further reading
Section titled “Further reading”- The proximity service problem for geohash, H3 and neighbor cells in depth, the payment system for the hand-off, and partitioning, consensus and message queues on this site.
- H3: Uber’s Hexagonal Hierarchical Spatial Index, Uber engineering: why hexagons, and how the grid is used for dispatch and pricing. The H3 documentation has the cell size tables.
- How to do distributed locking, Martin Kleppmann: why a lease needs a fence, the argument behind stage 7.
The designbeyondcode digest, every Saturday
New deep dives, interview tips and site news in one short email. Free. Leave with one click.
We never sell or share your email. We only use it to send this newsletter.