Design a real-time leaderboard
Design a real-time game leaderboard. Start with one Redis sorted set, let the numbers add replicas, trusted scores, windows, a split by player and ranks from a histogram, then stress it with ties, the top page, rank 5 million and cheaters, with interview prompts and code.
- Read first
- EstimationCaching
- Related
- Top-k heavy hitters
“Design a leaderboard for our mobile game. 25 million people play every day, and the moment a match ends they want to see where they stand.”
Everyone knows the answer starts with a Redis sorted set: a set where each member has a score,
kept in order. ZINCRBY adds points, ZREVRANGE gives the top page, ZREVRANK gives one player’s
rank. The rest of the problem is what a sorted set doesn’t give you. It lives in memory. ZINCRBY
counts a retry twice. Ties fall in an order nobody chose. And one sorted set lives on one machine,
so outgrowing it makes “the top 100” a merge, and “my rank” a question for every machine.
So this page starts with that one sorted set, and lets the numbers break it. Then it tests the finished design against ties, the top page, a rank far down the board, and cheaters.
The answer in one minute
Section titled “The answer in one minute”If you only have a minute, this is the design. The rest of the page explains each part.
- Only game servers report scores. Never the player’s phone. Each match result is recorded once, in a database, keyed by match and player.
- The sorted set is a copy, not the record. Scores are safe in the database first. Redis is rebuilt from it whenever needed.
- Write absolute totals with
ZADD GT. “Set this player to 2,210, only if that’s higher.” A repeat or a late message changes nothing. So delivery can be at least once. - Encode ties into the score. Points in the high bits, an inverted timestamp in the low bits, so whoever reached the score first ranks higher.
- Exact at the top, estimated below. Split boards by player across 16 machines. The top 10,000 are merged exactly every second. Rank 5 million is estimated within about 1%, and labelled so.
Scope it
Section titled “Scope it”Four questions decide the design: what a score is, who reports it, which boards exist, and how exact a rank must be.
Design a real-time leaderboard for our game.
Whether you ask what a score is, who reports it, and how exact ranks must be.
These four questions matter most:
- What is a score? Points summed over matches in a time window. Totals only go up.
- Who reports it? The game server that ran the match. Never the client.
- Which boards? Season (13 weeks), weekly and daily, all global, plus a friends board.
- How exact must ranks be? Exact for the top 10,000. Below that, “about 5,120,000th, top 5%”.
From these answers we make five promises:
- Fresh. A new score shows in ranks within 1 second, at p99.
- Fast. Reads under 30 ms of server time. Accepting a result under 100 ms.
- Never lost, never doubled. An accepted result is never lost and never counted twice. A lost result is a player who won and got nothing.
- Always up. Reads work 99.95% of the time, accepting results 99.99%.
- Fair. Exact ranks for the top 10,000, the same tie rule everywhere, and no player can set their own score.
We leave out matchmaking ratings, paying out rewards, cheat-detection models (we provide the hooks), and per-country boards.
Go deeperMore questions, and what each changes1 min
| Question | We assume | If the answer were different |
|---|---|---|
| How many players? | 25 million a day; 100 million ranked a season | Ten times more makes estimated ranks a day-one need |
| Scores that go down? | No, totals only go up | Ratings like Elo break ZADD GT (a follow-up) |
| How real-time? | Within 1 second | Minutes would allow a batch job and no Redis at all |
| Client-reported scores? | No | They can be signed and checked, but never fully trusted |
| How are ties broken? | Whoever reached the score first ranks higher | Shared ranks (1, 2, 2, 4) are cheaper |
The numbers that decide it
Section titled “The numbers that decide it”Now we work out the numbers. Count Redis commands, not requests, and multiply writes by windows.
- Daily active players
- 25 million
- Matches per player a day
- 10
- Leaderboard reads per match
- 3.5 to 1
- Peak over average
- 10×
- Players on the season board
- 100 million
- Bytes per sorted-set member
- 100 bytes
25 million players × 10 matches a day. Evenings, weekly resets and live events stack up, so plan for ten times.
So we recording each in a database transaction passes one Postgres primary at peak, so the score database is split by player from day one.
About 3.5 reads per match: your rank, the top page, the friends tab. Half are the top page, which is the same for everyone.
So we serve the top page from memory, and send rank reads to replicas.
On 100 million members, each sorted-set command follows a few dozen pointers through memory.
Budget about 100,000 a second per node, well below the figure quoted for a simple GET.
So we count commands, not requests.
Every score event updates three windows: 87% of one Redis primary, with no headroom.
So we split every board by player before the windows launch.
About 100 bytes per member, and 175 million members across the season, week and day.
So we memory doesn't force a split, and each window leaves Redis once its final standings are saved.
Writes, not memory, force the split. Try the calculator: at 20 million daily players, peak writes fall under 70% of one primary. At five times the players, memory forces a split on its own.
- Score events, average
- 2,900/s
- Score events, peak
- 29,000/s
- Past one Postgres primary: shard the score DB by player from day one.
- Sorted-set writes at peak (3 windows)
- 87,000/s
- Past 70% of one Redis primary, with no room for a failover replay, and one sorted set cannot span nodes: shard every board by player.
- Leaderboard reads, average
- 10,000/s
- Leaderboard reads, peak
- 100,000/s
- Past what one node serves next to the writes: rank reads go to replicas, and the top page is cached.
- Redis memory, season + week + day
- 18 GB
- One Redis node holds every board with room to spare.
- Match results a year
- 9.1 TB
- At about 100 bytes a row. Keep 90 days in the score DB and archive the rest.
Build the design
Section titled “Build the design”We start with the simplest design that works. Then we let the numbers break it, one at a time. Each step fixes only what broke, adds only the boxes that fix needs, and says what the fix costs.
Step 1. One sorted set
Section titled “Step 1. One sorted set”The simple version: when a match ends, the client calls our service, which runs ZINCRBY on one
sorted set in Redis. The top page and a player’s rank are one command each.
Stage 1: One sorted set
Stage 1 of 6
One sorted set
Why this stage
Nothing yet.
This is the smallest design that ranks players.
What we add
A leaderboard service and one Redis sorted set: ZINCRBY when a match ends, ZREVRANGE for the top page, ZREVRANK for one player’s rank.
What it costs
Clients report their own scores, the only copy of every score lives in memory, and one server carries every request.
At 100k leaderboard reads per second, API servers is 1000% busy. Stage 2 is the fix.
The numbers behind the busy bars
What one machine can do
- One API server handles about 10,000 requests a second.
- One Redis node runs about 100,000 sorted-set commands a second on a set of 100 million members. Each command follows a few dozen skiplist pointers, and most of them miss the CPU cache.
- One Postgres shard takes about 5,000 write transactions a second.
What the reads are
- Half of reads are the top page. From stage 2, each API server holds it for 1 second.
- The other half are a player's rank with neighbors. That takes 2 commands, then 32 across 16 shards in stage 5, then 1 in stage 6.
What the writes are
- Each score event is one transaction and one Kafka message. From stage 4 it is also 3 ZADDs.
It has four problems. Clients report their own scores, so anyone can post nine million. The
only copy is in memory. Redis copies writes to its replica after answering, so a failover can lose
a write with no record of it. And ZINCRBY isn’t safe to repeat: a retry after a timeout adds the
points twice. Before we fix those, the load breaks it too.
At the 100,000-a-second peak, this version breaks twice. One API server takes about 10,000 requests a second. And Redis would run 150,000 commands a second, because a rank with neighbors is two commands. Stateless API servers behind a load balancer, and two Redis read replicas, fix the reads.
Stage 2: A fleet and read replicas
Stage 2 of 6
A fleet and read replicas
Why this stage
One server handles about 10,000 requests a second and the peak is 100,000.
One Redis node would run about 150,000 commands a second at peak, past the roughly 100,000 it takes on a set this big.
What we add
A load balancer, stateless API servers that keep the top page in memory for one second, and two Redis replicas that answer rank reads.
Writes stay on the primary.
What it costs
A rank read from a replica can trail the primary by a few milliseconds.
At 100k leaderboard reads per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One API server handles about 10,000 requests a second.
- One Redis node runs about 100,000 sorted-set commands a second on a set of 100 million members. Each command follows a few dozen skiplist pointers, and most of them miss the CPU cache.
- One Postgres shard takes about 5,000 write transactions a second.
What the reads are
- Half of reads are the top page. From stage 2, each API server holds it for 1 second.
- The other half are a player's rank with neighbors. That takes 2 commands, then 32 across 16 shards in stage 5, then 1 in stage 6.
What the writes are
- Each score event is one transaction and one Kafka message. From stage 4 it is also 3 ZADDs.
Writes stay on the primary, which is where the real limit sits. And the first four problems are still there: untrusted scores, one copy in memory, lost writes on a failover, and double counting. Step 2 fixes all four.
Go deeperThe first version, in Redis commands1 min
When a match ends, ZINCRBY lb points player adds the points. The top page is
ZREVRANGE lb 0 99 WITHSCORES, and a player’s rank is ZREVRANK lb player plus one. Rank, top page
and score lookups each take time that grows with the logarithm of the board size. That’s because a
sorted set is a skiplist: linked lists stacked in levels, where each link stores how many
members it skips.
Step 2. Trusted, durable scores
Section titled “Step 2. Trusted, durable scores”Game servers submit each finished match, with service credentials. The match ID is the idempotency key. One result is one database transaction, on the player’s own shard of the score database:
- insert the result, keyed by player and match, so a retry finds it and changes nothing.
- add the points to the player’s totals for each window (season, week, day).
- write an outbox row with the new totals, in the same transaction.
The window comes from when the match ended, not when the result arrived. So a match ending at 23:59:58 counts for that day, even if it lands after midnight. The response carries the new totals, so the match-end screen can show them at once.
Stage 3: Trusted, durable scores
Stage 3 of 6
Trusted, durable scores
Why this stage
Not load: two requirements.
Scores must come from game servers, never from clients, and an accepted score must never be lost.
Redis replicates asynchronously, so a failover can drop acknowledged writes.
What we add
Game servers submit match results.
A score DB, sharded by player, records each result once and keeps every player’s totals.
An outbox feeds Kafka, and updater workers write absolute totals into Redis with ZADD GT.
What it costs
A new score reaches the board about 100 ms after it is accepted, not inside the same request.
Two more systems to run.
At 100k leaderboard reads per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One API server handles about 10,000 requests a second.
- One Redis node runs about 100,000 sorted-set commands a second on a set of 100 million members. Each command follows a few dozen skiplist pointers, and most of them miss the CPU cache.
- One Postgres shard takes about 5,000 write transactions a second.
What the reads are
- Half of reads are the top page. From stage 2, each API server holds it for 1 second.
- The other half are a player's rank with neighbors. That takes 2 commands, then 32 across 16 shards in stage 5, then 1 in stage 6.
What the writes are
- Each score event is one transaction and one Kafka message. From stage 4 it is also 3 ZADDs.
Absolute totals, written with GT
Section titled “Absolute totals, written with GT”A relay publishes outbox rows to Kafka, keyed by player. Updaters write each player’s new
absolute total into Redis with ZADD key GT score player.
GT updates an existing member only if the new score is greater. That’s safe only because each
message carries a total, not a change. A repeat has the same score, and an older message a lower
one. So both change nothing, and the newest wins in any order. Kafka can deliver twice. An updater
can replay a minute after a crash. And the board still converges.
- Color key
- Cache
- Service
- Queue or stream
- Database
The updater, with the score encoding from the deep dives:
// The bit layout every board shares. Redis stores scores as IEEE 754 doubles, which hold// integers exactly up to 2^53, so points and the tie-breaker must fit in 53 bits together.final class ScoreCodec { static final int TIME_BITS = 24; // 2^24 s = 194 days, longer than a season static final long TIME_MASK = (1L << TIME_BITS) - 1; static final long MAX_POINTS = (1L << (53 - TIME_BITS)) - 1; // about 536 million
// More points always rank higher. On equal points, whoever got there first ranks higher, // so the time part is inverted: an earlier second gives a larger number. static double encode(long points, long secondsIntoWindow) { if (points < 0 || points > MAX_POINTS) throw new IllegalArgumentException("points out of range"); // never wrap silently if (secondsIntoWindow < 0 || secondsIntoWindow > TIME_MASK) throw new IllegalArgumentException("time outside the window"); return (double) ((points << TIME_BITS) | (TIME_MASK - secondsIntoWindow)); }
static long points(double score) { return ((long) score) >>> TIME_BITS; // exact: the double holds an integer }}
// A player's new totals after one match, as the score DB committed them. Absolute values,// never a delta: that is what makes applying the same change twice harmless.record WindowTotal(String window, long points, long reachedAt, long windowStart, long windowEnd) {}record ScoreChange(String playerId, java.util.List<WindowTotal> totals) {}
interface SortedSets { void zaddGt(String key, String member, double score); // ZADD key GT score member, pipelined void expireAt(String key, long unixSeconds);}
// Runs in the updater workers, the only writers of the boards. Kafka delivers at least once,// and after a Redis failover we rewind and replay on purpose, so apply must be idempotent.final class BoardUpdater { private final SortedSets redis; // Window keys this process has already given a TTL: a few new keys a day, so it stays small. private final java.util.Set<String> withTtl = java.util.concurrent.ConcurrentHashMap.newKeySet();
BoardUpdater(SortedSets redis) { this.redis = redis; }
void apply(ScoreChange change) { for (WindowTotal t : change.totals()) { String key = "lb:" + t.window(); // stage 5 adds a shard hash tag here double score = ScoreCodec.encode(t.points(), t.reachedAt() - t.windowStart()); // GT only raises a score. A duplicate scores the same and an older change scores // lower, so neither does anything, in any order. ZINCRBY would count both twice. redis.zaddGt(key, change.playerId(), score); // Old windows delete themselves a week after they close, once rewards are paid. // Repeating this after a restart is harmless. if (withTtl.add(key)) redis.expireAt(key, t.windowEnd() + 7 * 86_400); } }}# The bit layout every board shares. Redis stores scores as IEEE 754 doubles, which hold# integers exactly up to 2^53, so points and the tie-breaker must fit in 53 bits together.TIME_BITS = 24 # 2^24 s = 194 days, longer than a seasonTIME_MASK = (1 << TIME_BITS) - 1MAX_POINTS = (1 << (53 - TIME_BITS)) - 1 # about 536 million
# More points always rank higher. On equal points, whoever got there first ranks higher,# so the time part is inverted: an earlier second gives a larger number.def encode(points: int, seconds_into_window: int) -> float: if not 0 <= points <= MAX_POINTS: raise ValueError("points out of range") # never wrap silently if not 0 <= seconds_into_window <= TIME_MASK: raise ValueError("time outside the window") return float((points << TIME_BITS) | (TIME_MASK - seconds_into_window))
def points_of(score: float) -> int: return int(score) >> TIME_BITS # exact: the float holds an integer
@dataclass(frozen=True)class WindowTotal: window: str # "season:2026-S4", "week:2026-W44", "day:2026-10-28" points: int reached_at: int window_start: int window_end: int
# A player's new totals after one match, as the score DB committed them. Absolute values,# never a delta: that is what makes applying the same change twice harmless.@dataclass(frozen=True)class ScoreChange: player_id: str totals: list[WindowTotal]
class SortedSets(Protocol): def zadd_gt(self, key: str, member: str, score: float) -> None: ... # ZADD key GT, pipelined def expire_at(self, key: str, unix_seconds: int) -> None: ...
# Runs in the updater workers, the only writers of the boards. Kafka delivers at least once,# and after a Redis failover we rewind and replay on purpose, so apply must be idempotent.class BoardUpdater: def __init__(self, redis: SortedSets): self._redis = redis # Window keys this process has already given a TTL: a few new keys a day, so it stays small. self._with_ttl: set[str] = set()
def apply(self, change: ScoreChange) -> None: for t in change.totals: key = f"lb:{t.window}" # stage 5 adds a shard hash tag here score = encode(t.points, t.reached_at - t.window_start) # GT only raises a score. A duplicate scores the same and an older change scores # lower, so neither does anything, in any order. ZINCRBY would count both twice. self._redis.zadd_gt(key, change.player_id, score) # Old windows delete themselves a week after they close, once rewards are paid. # Repeating this after a restart is harmless. if key not in self._with_ttl: self._with_ttl.add(key) self._redis.expire_at(key, t.window_end + 7 * 86_400)// The bit layout every board shares. Redis stores scores as IEEE 754 doubles, which hold// integers exactly up to 2^53, so points and the tie-breaker must fit in 53 bits together.static class ScoreCodec{ public const int TimeBits = 24; // 2^24 s = 194 days, longer than a season public const long TimeMask = (1L << TimeBits) - 1; public const long MaxPoints = (1L << (53 - TimeBits)) - 1; // about 536 million
// More points always rank higher. On equal points, whoever got there first ranks higher, // so the time part is inverted: an earlier second gives a larger number. public static double Encode(long points, long secondsIntoWindow) { ArgumentOutOfRangeException.ThrowIfNegative(points); // never wrap silently ArgumentOutOfRangeException.ThrowIfGreaterThan(points, MaxPoints); ArgumentOutOfRangeException.ThrowIfNegative(secondsIntoWindow); ArgumentOutOfRangeException.ThrowIfGreaterThan(secondsIntoWindow, TimeMask); return (points << TimeBits) | (TimeMask - secondsIntoWindow); }
public static long Points(double score) => (long)score >>> TimeBits; // exact: an integer}
// A player's new totals after one match, as the score DB committed them. Absolute values,// never a delta: that is what makes applying the same change twice harmless.record WindowTotal(string Window, long Points, long ReachedAt, long WindowStart, long WindowEnd);record ScoreChange(string PlayerId, IReadOnlyList<WindowTotal> Totals);
interface ISortedSets{ void ZAddGt(string key, string member, double score); // ZADD key GT score member, pipelined void ExpireAt(string key, long unixSeconds);}
// Runs in the updater workers, the only writers of the boards. Kafka delivers at least once,// and after a Redis failover we rewind and replay on purpose, so Apply must be idempotent.class BoardUpdater(ISortedSets redis){ // Window keys this process has already given a TTL: a few new keys a day, so it stays small. private readonly ConcurrentDictionary<string, bool> _withTtl = new();
public void Apply(ScoreChange change) { foreach (var t in change.Totals) { string key = "lb:" + t.Window; // stage 5 adds a shard hash tag here double score = ScoreCodec.Encode(t.Points, t.ReachedAt - t.WindowStart); // GT only raises a score. A duplicate scores the same and an older change scores // lower, so neither does anything, in any order. ZINCRBY would count both twice. redis.ZAddGt(key, change.PlayerId, score); // Old windows delete themselves a week after they close, once rewards are paid. // Repeating this after a restart is harmless. if (_withTtl.TryAdd(key, true)) redis.ExpireAt(key, t.WindowEnd + 7 * 86_400); } }}// The bit layout every board shares. Redis stores scores as IEEE 754 doubles, which hold// integers exactly up to 2^53, so points and the tie-breaker must fit in 53 bits together.const ( timeBits = 24 // 2^24 s = 194 days, longer than a season timeMask = 1<<timeBits - 1 maxPoints = 1<<(53-timeBits) - 1 // about 536 million)
// Encode ranks more points higher. On equal points, whoever got there first ranks higher,// so the time part is inverted: an earlier second gives a larger number.func Encode(points, secondsIntoWindow int64) (float64, error) { if points < 0 || points > maxPoints { return 0, errors.New("points out of range") // never wrap silently } if secondsIntoWindow < 0 || secondsIntoWindow > timeMask { return 0, errors.New("time outside the window") } return float64(points<<timeBits | (timeMask - secondsIntoWindow)), nil}
// Points is exact: the float64 holds an integer below 2^53.func Points(score float64) int64 { return int64(score) >> timeBits }
type WindowTotal struct { Window string // "season:2026-S4", "week:2026-W44", "day:2026-10-28" Points, ReachedAt int64 WindowStart, WindowEnd int64}
// ScoreChange is a player's new totals after one match, as the score DB committed them.// Absolute values, never a delta: that is what makes applying the same change twice harmless.type ScoreChange struct { PlayerID string Totals []WindowTotal}
type SortedSets interface { ZAddGT(key, member string, score float64) error // ZADD key GT score member, pipelined ExpireAt(key string, unixSeconds int64) error}
// BoardUpdater runs in the updater workers, the only writers of the boards. Kafka delivers at// least once, and after a Redis failover we rewind and replay on purpose, so Apply must be// idempotent.type BoardUpdater struct { redis SortedSets withTTL sync.Map // window keys already given a TTL: a few new keys a day}
func (u *BoardUpdater) Apply(c ScoreChange) error { for _, t := range c.Totals { key := "lb:" + t.Window // stage 5 adds a shard hash tag here score, err := Encode(t.Points, t.ReachedAt-t.WindowStart) if err != nil { return err // a bad total is a bug upstream: stop, don't write a wrong rank } // GT only raises a score. A duplicate scores the same and an older change scores // lower, so neither does anything, in any order. ZINCRBY would count both twice. if err := u.redis.ZAddGT(key, c.PlayerID, score); err != nil { return err // the offset is not committed, so Kafka redelivers and we retry } // Old windows delete themselves a week after they close, once rewards are paid. // Repeating this after a restart is harmless. if _, seen := u.withTTL.LoadOrStore(key, true); !seen { if err := u.redis.ExpireAt(key, t.WindowEnd+7*86_400); err != nil { u.withTTL.Delete(key) // try again on the next change } } } return nil}// The bit layout every board shares. Redis stores scores as IEEE 754 doubles, which hold// integers exactly up to 2^53, so points and the tie-breaker must fit in 53 bits together.// JavaScript numbers are the same doubles, so plain arithmetic is exact below 2^53;// bitwise operators are not (they truncate to 32 bits), so this multiplies instead of shifting.const TIME_BITS = 24; // 2^24 s = 194 days, longer than a seasonconst TIME_SPAN = 2 ** TIME_BITS;const MAX_POINTS = 2 ** (53 - TIME_BITS) - 1; // about 536 million
// More points always rank higher. On equal points, whoever got there first ranks higher,// so the time part is inverted: an earlier second gives a larger number.export function encode(points: number, secondsIntoWindow: number): number { if (!Number.isInteger(points) || points < 0 || points > MAX_POINTS) throw new RangeError('points out of range'); // never wrap silently if ( !Number.isInteger(secondsIntoWindow) || secondsIntoWindow < 0 || secondsIntoWindow >= TIME_SPAN ) throw new RangeError('time outside the window'); return points * TIME_SPAN + (TIME_SPAN - 1 - secondsIntoWindow);}
export const pointsOf = (score: number) => Math.floor(score / TIME_SPAN); // exact below 2^53
interface WindowTotal { window: string; // "season:2026-S4", "week:2026-W44", "day:2026-10-28" points: number; reachedAt: number; windowStart: number; windowEnd: number;}
// A player's new totals after one match, as the score DB committed them. Absolute values,// never a delta: that is what makes applying the same change twice harmless.interface ScoreChange { playerId: string; totals: WindowTotal[];}
interface SortedSets { zaddGt(key: string, member: string, score: number): Promise<void>; // ZADD key GT, pipelined expireAt(key: string, unixSeconds: number): Promise<void>;}
// Runs in the updater workers, the only writers of the boards. Kafka delivers at least once,// and after a Redis failover we rewind and replay on purpose, so apply must be idempotent.export class BoardUpdater { // Window keys this process has already given a TTL: a few new keys a day, so it stays small. private readonly withTtl = new Set<string>();
constructor(private readonly redis: SortedSets) {}
async apply(change: ScoreChange): Promise<void> { for (const t of change.totals) { const key = `lb:${t.window}`; // stage 5 adds a shard hash tag here const score = encode(t.points, t.reachedAt - t.windowStart); // GT only raises a score. A duplicate scores the same and an older change scores // lower, so neither does anything, in any order. ZINCRBY would count both twice. await this.redis.zaddGt(key, change.playerId, score); // Old windows delete themselves a week after they close, once rewards are paid. // Repeating this after a restart is harmless. if (!this.withTtl.has(key)) { this.withTtl.add(key); await this.redis.expireAt(key, t.windowEnd + 7 * 86_400); } } }}A new score reaches Redis 70 to 100 ms after it’s accepted. The match-end screen shows the totals from the response at once, and fetches the rank during the animation.
Go deeperWhy an outbox, and not two writes?1 min
The API could save the score in the database, and then publish it to Kafka. But a crash between the two loses the message. The score is saved, and the board never hears about it. Publishing first is worse: the board shows points that may never be saved.
The outbox fixes this. The message is written as a row in the same transaction as the score, so
both happen or neither does. A small relay reads new outbox rows, publishes them to Kafka, and then
marks them sent. If the relay crashes after publishing but before marking, it publishes them again
when it restarts. That is harmless, because updaters write absolute totals with GT.
Some teams skip the relay and read the database’s own change log instead, with a tool such as Debezium. This is called change data capture. It gives the same guarantee without the extra table.
Step 3. Daily, weekly and friends boards
Section titled “Step 3. Daily, weekly and friends boards”Each window is its own key, named by its period: lb:week:2026-W44, lb:day:2026-10-28. A
reset is just the clock moving to a new key name. Each key expires a week after its window closes,
which leaves time to freeze the final standings and pay rewards.
Friends are scored on read. Writing every score into each of 500 friends’ personal boards would
turn 29,000 events a second into millions of writes. Instead, the API fetches every friend’s score
with one ZMSCORE and sorts 500 numbers itself.
Stage 4: Daily, weekly and friends boards
Stage 4 of 6
Daily, weekly and friends boards
Why this stage
Not load: a requirement.
Product wants daily and weekly boards that reset on their own, and a board of just your friends.
What we add
One sorted set per window, named by its date and expiring a week after it closes, and a friends service whose friend lists the API scores with one ZMSCORE.
What it costs
Every score event now writes three sorted sets, so the Redis primary runs at about 87% at peak.
At 100k leaderboard reads per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One API server handles about 10,000 requests a second.
- One Redis node runs about 100,000 sorted-set commands a second on a set of 100 million members. Each command follows a few dozen skiplist pointers, and most of them miss the CPU cache.
- One Postgres shard takes about 5,000 write transactions a second.
What the reads are
- Half of reads are the top page. From stage 2, each API server holds it for 1 second.
- The other half are a player's rank with neighbors. That takes 2 commands, then 32 across 16 shards in stage 5, then 1 in stage 6.
What the writes are
- Each score event is one transaction and one Kafka message. From stage 4 it is also 3 ZADDs.
Reads go to Redis replicas, and API servers keep the top page of each board in memory for one second. So 50,000 top-page reads a second become one Redis read per board per server per second.
Follow a rank read and a submitted result:
Trace a request: My rank, with neighbors
My rank, with neighbors: step 1 of 4
Players to Load balancer
GET /v1/boards/season/players/me?neighbors=2 from the match-end screen. The load balancer terminates TLS and picks any API server.
- Players → Load balancer2 ms
- Load balancer → API servers1 ms
- API servers → Redis replicas1 ms
- Response
This is enough for a mid-level answer. Trusted scores in a database, absolute totals with
ZADD GT, a key per window, and replicas for reads. Splitting the board, ranks from a histogram,
and the stress tests are what make it a senior answer.
Go deeperThe API2 min
POST /v1/matches/m_81f2/resultsAuthorization: mTLS, game-server identity
{ "mode": "ranked-duo", "endedAt": "2026-10-28T19:04:11Z", "results": [ { "playerId": "p_5521", "points": 140 }, { "playerId": "p_0193", "points": 95 } ] }
200 OK{ "players": [ { "playerId": "p_5521", "season": 18240, "week": 2210, "day": 410 }, ... ] }
GET /v1/boards/week?offset=0&limit=100200 OK Cache-Control: public, max-age=1{ "window": "2026-W44", "asOf": "2026-10-28T19:04:12Z", "entries": [ { "rank": 1, "playerId": "p_0042", "name": "Vex", "points": 9120 }, ... ] }
GET /v1/boards/week/players/p_5521?neighbors=2200 OK { "rank": 48213, "exact": true, "points": 2210, "neighbors": [ ... ] } { "rank": 5120000, "exact": false, "topPercent": 5, "points": 310 }A retried submit returns the same totals. 409 means the same match ID with different points.
422 means points that are impossible for the mode. Pages go down to rank 10,000. Below that,
players jump to themselves. The second rank response is outside the top 10,000. It’s an estimate,
flagged, with no neighbors.
The top page is polled, not pushed. It’s the same for everyone, so a one-second cache answers almost all of it. A live tournament final is where pushing earns its connections.
Go deeperThe tables, and where ranks could live1 min
-- Split by player_id: everything one result touches for one player is on one shard.CREATE TABLE match_results ( player_id BIGINT NOT NULL, match_id BIGINT NOT NULL, points INT NOT NULL CHECK (points >= 0), ended_at TIMESTAMPTZ NOT NULL, PRIMARY KEY (player_id, match_id) -- a retried submit can't count twice);
CREATE TABLE window_totals ( player_id BIGINT NOT NULL, window_id TEXT NOT NULL, -- 'season:2026-S4', 'week:2026-W44', 'day:2026-10-28' points BIGINT NOT NULL, reached_at TIMESTAMPTZ NOT NULL, -- when points last went up: the tie-breaker PRIMARY KEY (player_id, window_id));The totals update keeps reached_at as the latest time points went up. So results arriving out of
order still give the right tie-breaker. And a zero-point match leaves it alone.
| Option | Top 100 | Rank of player 5 million |
|---|---|---|
| SQL with an index on score | Fast | Count every row above: 5 million index reads |
| Redis sorted set | Fast | Fast: each link in its skiplist stores how many members it skips |
| DynamoDB or Cassandra | Needs a scatter | No rank operation at all |
“Put the scores in SQL and ORDER BY score” works for the top page. A player’s rank is
SELECT COUNT(*) WHERE score > :mine, which reads 5 million index entries for rank 5 million, on
every request.
| How a result reaches the board | Problem |
|---|---|
ZINCRBY from the API |
Retries double-count; failover loses writes silently |
| Database, then Redis, in the API | A crash between the two loses the board update |
| A dedupe set in Redis | 250 million keys a day, and still lost on failover |
Database + outbox → Kafka → ZADD GT |
About 100 ms between accepted and visible; two more systems |
Step 4. Split the board across machines
Section titled “Step 4. Split the board across machines”The last number to break is writes. At ten times the traffic, one Redis primary can’t keep up.
At ten times, the primary needs about 870,000 writes a second, against one thread that takes about 100,000. Replicas fix reads, never writes, and a bigger machine doesn’t add threads to Redis.
There are two real ways to split one ranking. By score range keeps the top page on one machine. But most players crowd into the low ranges, and the boundaries are wrong by week two. And a player crossing one needs a delete on one machine and an insert on another, with no transaction between. By a hash of the player ID keeps writes even, whatever the scores do. We pick that.
So each board becomes 16 keys, lb:{s00}:week:2026-W44 to lb:{s15}:week:2026-W44. The part in
braces is a hash tag: Redis Cluster places a key by hashing only that part. A player’s three
windows share a tag, so one round trip updates them all. We place the 16 tags on 16 different
primaries.
Stage 5: Shard by player
Stage 5 of 6
Shard by player
Why this stage
Headroom now, load at 10 times.
At peak the primary already runs at 87%, past the 70% that leaves room for a failover replay.
At the 10x mark about 870,000 window writes a second would land on it.
A sorted set cannot be split across nodes, and Redis runs commands on one thread.
What we add
Redis split into 16 shards by a hash of the player ID, each holding a slice of every board under one hash tag, and the score DB grown to 80 shards.
Top pages and exact ranks become scatter-gather reads.
What it costs
An exact global rank now asks every shard: 16 ZCOUNTs, plus 16 more reads for the neighbors.
At 100k leaderboard reads per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One API server handles about 10,000 requests a second.
- One Redis node runs about 100,000 sorted-set commands a second on a set of 100 million members. Each command follows a few dozen skiplist pointers, and most of them miss the CPU cache.
- One Postgres shard takes about 5,000 write transactions a second.
What the reads are
- Half of reads are the top page. From stage 2, each API server holds it for 1 second.
- The other half are a player's rank with neighbors. That takes 2 commands, then 32 across 16 shards in stage 5, then 1 in stage 6.
What the writes are
- Each score event is one transaction and one Kafka message. From stage 4 it is also 3 ZADDs.
Reads now gather from every shard. An exact rank is one plus the players above you on each shard: 16
ZCOUNTs, summed, plus more for neighbors. About 32 commands per rank read instead of 2. That’s
fine at today’s peak, and too much at ten times. The next step fixes it for ranks far down the
board.
Because the boards are only a copy, changing the shard count is a rebuild from the score database, not a data move.
Go deeperScore ranges against player hash, in detail1 min
| Split by | Writes | Top page | Rank of any player | Problem |
|---|---|---|---|---|
| Score range | Uneven: most players sit low | One shard, the highest | Count the shards above, plus a local rank | Players cross boundaries; ranges drift |
| Hash of player ID | Even | Merge each shard’s top | Sum one count per shard, or estimate | Reads go to every shard |
The trigger to split is peak writes above 70% of one primary, or memory above about 50 GB. Three windows put peak writes at 87%, so the split ships with the windows, sized for ten times the traffic. The score database grows to 80 shards behind its own map of which players live where.
Go deeperSizing the split1 min
At ten times today’s traffic, the boards take about 870,000 writes a second. Spread over 16 primaries, that is about 54,000 each, a little over half of what one Redis thread can do. That keeps every primary under the 70% line, which leaves room for the replay after a failover that the failure section describes.
Each primary has one replica in another zone, which serves reads and takes over on a failure. If writes grow again, going from 16 shards to 32 is the same rebuild from the score database, not a move of live data. That is the payoff of treating Redis as a copy.
Step 5. Ranks from a histogram
Section titled “Step 5. Ranks from a histogram”The last thing to break is a rank far down the board. The exact answer is 16 ZCOUNTs. That’s cheap
once, and expensive half a million times a second. So below the top 10,000, we estimate the rank
from a histogram: a count of how many players have at least each of about 1,000 scores.
Every five seconds, an aggregator counts how many players on each shard have at least each of those
boundary scores, and sums them. A rank read is then one ZSCORE on the player’s own shard, plus a
lookup in memory. The rank is the count above the player’s bucket, plus a share of the bucket. The
top 10,000 stay exact.
Stage 6: Ranks from a histogram
Stage 6 of 6
Ranks from a histogram
Why this stage
Load at the 10x mark: rank reads fanning out to 16 shards would cost about 16 million replica commands a second, five times what 32 replicas take.
At peak they run at 50%.
What we add
A rank aggregator that every second merges each shard’s top 10,000 into one exact list and sums per-shard score histograms.
API servers rank the top 10,000 exactly from that snapshot and estimate everyone else in memory.
What it costs
Ranks below 10,000 are estimates, labelled as such, and up to a few seconds old.
At 100k leaderboard reads per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One API server handles about 10,000 requests a second.
- One Redis node runs about 100,000 sorted-set commands a second on a set of 100 million members. Each command follows a few dozen skiplist pointers, and most of them miss the CPU cache.
- One Postgres shard takes about 5,000 write transactions a second.
What the reads are
- Half of reads are the top page. From stage 2, each API server holds it for 1 second.
- The other half are a player's rank with neighbors. That takes 2 commands, then 32 across 16 shards in stage 5, then 1 in stage 6.
What the writes are
- Each score event is one transaction and one Kafka message. From stage 4 it is also 3 ZADDs.
Follow a rank read for a player far down the board:
Trace a request: Rank of player 5 million
Rank of player 5 million: step 1 of 4
Players to Load balancer
The same request as before, from a player far down the board.
- Players → Load balancer2 ms
- Load balancer → API servers1 ms
- API servers → Redis replicas1 ms
- Response
Stress it
Section titled “Stress it”The design is complete. Now we test it against what makes leaderboards hard. Ties. The top page across 16 machines. The rank of a player far down the board. And cheaters. No new boxes appear here.
Ties, and the score encoding
Section titled “Ties, and the score encoding”Two players both have 2,210 points. Who’s ranked higher, and how does Redis know?
Whether you define who wins a tie, and encode it into the score so Redis sorts it for free.
A sorted set orders equal scores by member name. So by default, of two players on 2,210, the one whose ID sorts later ranks higher: consistent, and meaningless to a player. Our rule: whoever reached the score first ranks higher.
Redis scores are 64-bit floating-point numbers (doubles). A double represents every whole number up to 2⁵³ exactly. So we build the score as one whole number. The high 29 bits are points. The low 24 bits are the time, inverted, so that earlier is larger.
score = points × 2²⁴ + (2²⁴ − 1 − seconds into the window)
2²⁴ seconds is 194 days, longer than a 13-week season. 2²⁹ is about 536 million points. The time part never reaches one point’s worth, so more points always win, and on equal points the earlier time wins.
For 2,210 points reached four days into the week (345,600 seconds): 2,210 × 16,777,216 + (16,777,215 − 345,600) = 37,094,078,975. The encode function in the updater is exactly this, with range checks, because a value that overflows its bits silently corrupts the order around it.
Go deeperOther tie rules1 min
| Approach | Wins | Loses |
|---|---|---|
| Redis default (by player ID) | Nothing to build | Arbitrary to players |
| A fraction for time | One number | Large points leave no room for the fraction |
| Whole number in 53 bits | One number, one write, exact | Points below 536 million, windows under 194 days |
Two players who reach the same total in the same second still tie, and Redis falls back to member order: rare, consistent, harmless. For shared ranks instead (1, 2, 2, 4), skip the time part: a rank is one plus the count of players with more points. JavaScript numbers are the same doubles, so the TypeScript version multiplies instead of shifting bits, because its bit operators work on only 32 bits.
The top page across 16 shards
Section titled “The top page across 16 shards”The board is now split across 16 machines. How do we show the top 100?
Every entry of the global top 100 is in its own shard’s top 100. That’s because at most 99 players in the whole board, and so in any one shard, rank above it. So asking each shard for its top 100, and merging the 16 sorted lists, is exact. The merge must use the same order Redis uses inside a shard: score descending, then player ID. With any other tie rule, two pages can disagree about who is 41st.
record Entry(String playerId, double score) {}
interface Shard { // ZREVRANGE lb:{sNN}:week:2026-W44 0 k-1 WITHSCORES: highest first, and among equal // scores, player IDs in descending byte order. That is Redis's order, so we reuse it. CompletableFuture<List<Entry>> topK(int k);}
final class TopK { // The one global order: higher score first, then the same tie rule Redis uses inside // a shard. Merging with any other rule would rank tied players differently per page. static final Comparator<Entry> ORDER = Comparator.comparingDouble(Entry::score).reversed() .thenComparing(Entry::playerId, Comparator.reverseOrder());
private record Head(int shard, int pos, Entry entry) {}
static List<Entry> merge(List<Shard> shards, int k) { // Ask every shard for k, not k / 16: the whole global top k may sit on one shard. // All requests go out at once, so the merge costs one round trip, not sixteen. List<CompletableFuture<List<Entry>>> calls = shards.stream().map(s -> s.topK(k)).toList(); // join() throws if any shard failed. The aggregator then keeps its last snapshot // rather than publishing a top k that silently skips a sixteenth of the players. List<List<Entry>> lists = calls.stream().map(CompletableFuture::join).toList();
// k-way merge: the heap holds each shard's best remaining entry, so producing k // results costs k log 16 comparisons instead of sorting 16 k entries. PriorityQueue<Head> heap = new PriorityQueue<>((a, b) -> ORDER.compare(a.entry(), b.entry())); for (int i = 0; i < lists.size(); i++) if (!lists.get(i).isEmpty()) heap.add(new Head(i, 0, lists.get(i).get(0)));
List<Entry> top = new ArrayList<>(k); while (top.size() < k && !heap.isEmpty()) { Head h = heap.poll(); top.add(h.entry()); List<Entry> from = lists.get(h.shard()); if (h.pos() + 1 < from.size()) heap.add(new Head(h.shard(), h.pos() + 1, from.get(h.pos() + 1))); } return top; // ranks are positions: top.get(0) is rank 1 }}@dataclass(frozen=True)class Entry: player_id: str score: float
class Shard(Protocol): # ZREVRANGE lb:{sNN}:week:2026-W44 0 k-1 WITHSCORES: highest first, and among equal # scores, player IDs in descending byte order. That is Redis's order, so we reuse it. async def top_k(self, k: int) -> list[Entry]: ...
def order_key(e: Entry) -> tuple[float, str]: # The one global order: higher score first, then the same tie rule Redis uses inside a # shard. Merging with any other rule would rank tied players differently per page. # (Python compares str by code point, which matches bytes for ASCII player IDs.) return (e.score, e.player_id)
async def merge_top_k(shards: list[Shard], k: int) -> list[Entry]: # Ask every shard for k, not k / 16: the whole global top k may sit on one shard. # All requests go out at once, so the merge costs one round trip, not sixteen. # gather raises if any shard failed. The aggregator then keeps its last snapshot rather # than publishing a top k that silently skips a sixteenth of the players. lists = await asyncio.gather(*(s.top_k(k) for s in shards))
# k-way merge: heapq.merge keeps one head per shard in a heap, so producing k results # costs k log 16 comparisons instead of sorting 16 k entries. Each input is already in # descending order, which is what reverse=True expects. merged = heapq.merge(*lists, key=order_key, reverse=True) return list(itertools.islice(merged, k)) # ranks are positions: index 0 is rank 1record Entry(string PlayerId, double Score);
interface IShard{ // ZREVRANGE lb:{sNN}:week:2026-W44 0 k-1 WITHSCORES: highest first, and among equal // scores, player IDs in descending byte order. That is Redis's order, so we reuse it. Task<IReadOnlyList<Entry>> TopKAsync(int k);}
static class TopK{ // The one global order: higher score first, then the same tie rule Redis uses inside // a shard. Merging with any other rule would rank tied players differently per page. // Ordinal comparison matches Redis's byte order; culture-aware comparison would not. static readonly Comparer<Entry> Order = Comparer<Entry>.Create((a, b) => { int byScore = b.Score.CompareTo(a.Score); return byScore != 0 ? byScore : string.CompareOrdinal(b.PlayerId, a.PlayerId); });
public static async Task<List<Entry>> MergeAsync(IReadOnlyList<IShard> shards, int k) { // Ask every shard for k, not k / 16: the whole global top k may sit on one shard. // All requests go out at once, so the merge costs one round trip, not sixteen. // WhenAll throws if any shard failed. The aggregator then keeps its last snapshot // rather than publishing a top k that silently skips a sixteenth of the players. var lists = await Task.WhenAll(shards.Select(s => s.TopKAsync(k)));
// k-way merge: the queue holds each shard's best remaining entry, so producing k // results costs k log 16 comparisons instead of sorting 16 k entries. var heap = new PriorityQueue<(int Shard, int Pos), Entry>(Order); for (int i = 0; i < lists.Length; i++) if (lists[i].Count > 0) heap.Enqueue((i, 0), lists[i][0]);
var top = new List<Entry>(k); while (top.Count < k && heap.TryDequeue(out var head, out var entry)) { top.Add(entry); var from = lists[head.Shard]; if (head.Pos + 1 < from.Count) heap.Enqueue((head.Shard, head.Pos + 1), from[head.Pos + 1]); } return top; // ranks are positions: top[0] is rank 1 }}type Entry struct { PlayerID string Score float64}
// Shard runs ZREVRANGE lb:{sNN}:week:2026-W44 0 k-1 WITHSCORES: highest first, and among// equal scores, player IDs in descending byte order. That is Redis's order, so we reuse it.type Shard interface { TopK(ctx context.Context, k int) ([]Entry, error)}
// before is the one global order: higher score first, then the same tie rule Redis uses// inside a shard. Merging with any other rule would rank tied players differently per page.// Go compares strings byte by byte, which matches Redis.func before(a, b Entry) bool { if a.Score != b.Score { return a.Score > b.Score } return a.PlayerID > b.PlayerID}
func MergeTopK(ctx context.Context, shards []Shard, k int) ([]Entry, error) { // Ask every shard for k, not k / 16: the whole global top k may sit on one shard. // All requests go out at once, so the merge costs one round trip, not sixteen. lists := make([][]Entry, len(shards)) errs := make([]error, len(shards)) var wg sync.WaitGroup for i, s := range shards { wg.Add(1) go func() { defer wg.Done() lists[i], errs[i] = s.TopK(ctx, k) }() } wg.Wait() // Any shard failing fails the merge. The aggregator then keeps its last snapshot rather // than publishing a top k that silently skips a sixteenth of the players. if err := errors.Join(errs...); err != nil { return nil, err }
// k-way merge over 16 sorted lists. With this few shards, scanning the 16 heads for the // best one is as fast as a heap and simpler: 16 k comparisons for k results. pos := make([]int, len(lists)) top := make([]Entry, 0, k) for len(top) < k { best := -1 for i, l := range lists { if pos[i] < len(l) && (best < 0 || before(l[pos[i]], lists[best][pos[best]])) { best = i } } if best < 0 { break // every shard is exhausted: fewer than k players in total } top = append(top, lists[best][pos[best]]) pos[best]++ } return top, nil // ranks are positions: top[0] is rank 1}interface Entry { playerId: string; score: number;}
interface Shard { // ZREVRANGE lb:{sNN}:week:2026-W44 0 k-1 WITHSCORES: highest first, and among equal // scores, player IDs in descending byte order. That is Redis's order, so we reuse it. topK(k: number): Promise<Entry[]>;}
// The one global order: higher score first, then the same tie rule Redis uses inside a// shard. Merging with any other rule would rank tied players differently per page.// Plain < on strings compares UTF-16 code units, which matches bytes for ASCII player IDs;// localeCompare would not.function before(a: Entry, b: Entry): boolean { if (a.score !== b.score) return a.score > b.score; return a.playerId > b.playerId;}
export async function mergeTopK(shards: Shard[], k: number): Promise<Entry[]> { // Ask every shard for k, not k / 16: the whole global top k may sit on one shard. // All requests go out at once, so the merge costs one round trip, not sixteen. // Promise.all rejects if any shard failed. The aggregator then keeps its last snapshot // rather than publishing a top k that silently skips a sixteenth of the players. const lists = await Promise.all(shards.map((s) => s.topK(k)));
// k-way merge over 16 sorted lists. With this few shards, scanning the 16 heads for the // best one is as fast as a heap and simpler: 16 k comparisons for k results. const pos = lists.map(() => 0); const top: Entry[] = []; while (top.length < k) { let best = -1; lists.forEach((l, i) => { if (pos[i] < l.length && (best < 0 || before(l[pos[i]], lists[best][pos[best]]))) best = i; }); if (best < 0) break; // every shard is exhausted: fewer than k players in total top.push(lists[best][pos[best]++]); } return top; // ranks are positions: top[0] is rank 1}Deep pages are the cost. Page 50 needs the top 5,000 from every shard: 80,000 entries fetched for 100 shown. So the API never merges per request. A rank aggregator merges the top 10,000 once a second, and every top-page read slices that snapshot in memory.
- Color key
- Service
- Cache
- Database
- Background worker
The rank of player 5 million
Section titled “The rank of player 5 million”A player ranked about 5 millionth opens the app. What do we show, and how close is it?
The histogram from Step 5 gives the answer in one command. The accuracy is all in where the boundaries are. Equal-width buckets fail: most players crowd into low scores, so one bucket can hold 30 million players. Equal-population buckets of 100,000 are fine for player 5 million, and useless for player 20,000. So the buckets are shaped by rank: each spans about 1% of the rank at its top. From rank 10,000 to 100 million, that takes about 930 boundaries.
For player 5 million, the answer is “about 5,120,000th, top 5%”: within about 50,000 ranks, and up to five seconds old. The top 10,000 stay exact, because that’s where players compare closely and rewards are decided.
Go deeperThe options compared, and why not a sketch1 min
| Approach | Cost per read | Error | How old |
|---|---|---|---|
| Exact, from every shard | 16 commands | None | Fresh |
| Sort everyone periodically | 1 lookup | None | Minutes |
| Histogram with rank-shaped buckets | 1 command + memory lookup | Under 1% of the rank | Up to 5 seconds |
The counts at the boundaries are exact when taken. The rank between them is not. The 16 shards are counted a few milliseconds apart, which is noise far below the error. Each minute, the aggregator moves the boundaries back toward their target ranks.
A standard quantile sketch such as KLL wouldn’t do this. Its error is a share of all players: ±100,000 ranks at 0.1% of 100 million. Fine at rank 5 million, absurd at rank 20,000.
If one shard is slow when the aggregator merges, never publish a top 10,000 without it. Reuse that shard’s list from the previous cycle if it’s under ten seconds old, and stamp the snapshot with the oldest age.
Cheaters, and closing a window
Section titled “Cheaters, and closing a window”Someone finds an exploit and posts nine million points. The weekly prizes go out tomorrow.
Four layers each catch something different. Only game servers can submit. Each result is checked on write: points within the mode’s maximum, a plausible match length, a cap on results per player per hour. A background job flags sudden jumps and impossible win streaks. And replays plus human review check the top 1,000 before any prize.
Removal has an ordering trap. Say we ZREM a cheater, and an updater then replays a change for
them. ZADD GT adds them back, because GT never blocks a new member. So the ban comes first: a
flag in the score database, and a ban list that updaters check. Only then is the player removed from
every window. Totals stay in the database, for appeals.
Closing a window. Results keep arriving for matches that ended just before midnight. So the window freezes ten minutes after it closes. Later results still count for the season, not that week. At the freeze, a job copies the final top 10,000 to the database, and the full standings to object storage. Rewards come from that copy after review, never from the live board.
When things break
Section titled “When things break”A Redis primary dies at peak. What do players see, and what do you lose?
Whether you can say exactly what’s lost (some board writes) and what isn’t (any score), and how the board catches up.
No accepted result is ever lost, because the score database has it before the game server hears yes. Kafka, the updaters and Redis can all fail, and be replayed. A Redis failover is the interesting case, because the board does lose writes: the replica was a few hundred messages behind when it was promoted.
- Color key
- Queue or stream
- Database
- Cache
- Service
The updaters rewind every Kafka partition by 60 seconds and apply everything again. GT makes the
already-applied changes do nothing, so a generous rewind closes the gap for a few seconds of extra
writes. Keeping each primary under 70% of its write ceiling is what leaves room for this replay.
Click any box to break it and see what players would notice:
Break it: what fails, who notices, what users see
Failure: click any box to break it
A Redis primary goes down
What breaks
Writes to its shard stop until a replica is promoted, and the last few hundred milliseconds of writes can be lost, because replication is asynchronous.
How we notice
Sentinel or cluster failure detection, a few seconds.
Updater write errors.
What users see
Ranks on that shard freeze for a few seconds.
No score is lost: the score DB has it.
What we do
After promotion, updaters rewind their Kafka offsets by 60 seconds and replay.
ZADD GT makes the replay harmless, and it restores any writes the failover dropped.
Rebuilding a whole board, after losing a shard and its replica, or to change the encoding. Note the current Kafka positions. Scan the totals table on every database shard in parallel, and load each shard’s keys under a new name. Then replay Kafka from the noted positions, and switch reads. 175 million members across 16 shards load in under two minutes. Changing the shard count is the same procedure, practiced every quarter.
Go deeperEvery failure, and what users see1 min
| Part | Failure | What players see | What we do |
|---|---|---|---|
| Redis primary | Down | Ranks on one shard freeze for seconds | Promote the replica; updaters rewind 60 s and replay |
| Redis replica | Down | A few slow reads | Reads move to the other replica |
| Score database | Down | Nothing visible; results wait on game servers | Failover in about 30 s; retries are safe |
| Kafka | Down | Scores accepted, ranks stop moving | The outbox holds every change and drains in order |
| Updater | Crash | Some ranks lag 10 to 30 s | Resume from the last committed position |
| Rank aggregator | Down | Top page and far-down ranks go stale | A standby takes over; fall back to a live merge |
| Friends service | Down | The friends tab uses a cached list | Degrades alone |
Under stress, protect in this order: accepting results, my rank, the top page (stale is fine), friends boards, far-down estimates.
| Operation | Guarantee | What a player can see |
|---|---|---|
| Recording a result | Exactly once per player and match | Nothing: retries return the same totals |
| A player on the board | Under 1 s at p99; never goes backwards | Rank can trail the new score by about 100 ms |
| Top page | A consistent snapshot, up to 1 to 2 s old | Your own row may lag your rank by a second |
| Far-down rank | Within 1% of the rank, up to 5 s old | “Top 5%” can stay the same after a small climb |
Kafka’s “exactly-once” doesn’t make scores count once here: it covers Kafka-to-Kafka writes. An
updater can write to Redis and crash before recording its position. This design is at-least-once
delivery with effects that are safe to repeat at both ends: the key on player and match, and GT.
Go deeperRunning it: alerts, abuse and cost1 min
Wake someone up when:
- Time from commit to the board passes 5 seconds at p99.
- Result acceptance drops below 99.99%, or the outbox is older than 30 seconds.
- Read p99 passes 50 ms.
- Updater lag or snapshot age grows.
And every minute, compare 1,000 random players’ totals in the database with Redis. A steady mismatch is a write-path bug no other metric shows.
Abuse. Cheating is covered above. Also limit reads per player against scraping, and scope game server credentials so a compromised server can only submit its own matches.
Deploys. Updaters can roll at any time, because their writes are safe to repeat. Any change to the encoding, window naming or shard count is a new set of keys, built from the database. It’s compared side by side, then switched.
Cost is mostly memory: 18 GB of boards on 16 shards of three nodes each. The score database’s 9 TB of results a year is the line that keeps growing.
Wrap up
Section titled “Wrap up”The whole design
Section titled “The whole design”Here is everything we built, in the order we built it. Use the buttons to step through it.
The full design, stage by stage
Stage 6 of 6
Ranks from a histogram
Why this stage
Load at the 10x mark: rank reads fanning out to 16 shards would cost about 16 million replica commands a second, five times what 32 replicas take.
At peak they run at 50%.
What we add
A rank aggregator that every second merges each shard’s top 10,000 into one exact list and sums per-shard score histograms.
API servers rank the top 10,000 exactly from that snapshot and estimate everyone else in memory.
What it costs
Ranks below 10,000 are estimates, labelled as such, and up to a few seconds old.
At 100k leaderboard reads per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one machine can do
- One API server handles about 10,000 requests a second.
- One Redis node runs about 100,000 sorted-set commands a second on a set of 100 million members. Each command follows a few dozen skiplist pointers, and most of them miss the CPU cache.
- One Postgres shard takes about 5,000 write transactions a second.
What the reads are
- Half of reads are the top page. From stage 2, each API server holds it for 1 second.
- The other half are a player's rank with neighbors. That takes 2 commands, then 32 across 16 shards in stage 5, then 1 in stage 6.
What the writes are
- Each score event is one transaction and one Kafka message. From stage 4 it is also 3 ZADDs.
The decisions, and what each one costs
Section titled “The decisions, and what each one costs”| Decision | We chose | We gave up | We’d choose differently if |
|---|---|---|---|
| Rank store | Redis sorted sets as a copy | One store; rank in the same request | The board were small enough for SQL |
| Score source | Game servers, checked on write | The simplicity of client-reported scores | Never, for a board with prizes |
| Write path | Database + outbox → Kafka → ZADD GT |
About 100 ms of freshness | Ranks had to update in the same request |
| Ties | Whole number: points, then earlier time | Points above 536 million; windows over 194 days | Shared ranks were acceptable |
| Windows | A key per window with an expiry | Memory for every live window | Windows were rolling (the last 7 days) |
| Splitting | By player, 16 shards, sized for ten times | A top page on one machine | Only the top page mattered |
| Far-down ranks | A histogram with rank-shaped buckets | Exact ranks below 10,000 | Exact ranks were a product requirement |
How to spend 45 minutes
Section titled “How to spend 45 minutes”- 0 to 4. What a score is, who reports it, which windows, how exact ranks must be.
- 4 to 8. 29,000 events and 87,000 sorted-set writes a second at peak, 18 GB.
- 8 to 20. Build it: one sorted set, replicas, scores in a database, the outbox,
ZADD GT, a key per window. A mid-level answer can stop here. - 20 to 26. Splitting the board by player, then ranks from a histogram.
- 26 to 40. The stress tests: ties, the top page merge, rank 5 million, then cheaters.
- 40 to 45. A Redis failover and the replay, then your own questions.
Never skip repeat-safe writes, or ties.
Follow-up questions
Section titled “Follow-up questions”Interviewer asksScores can now go down, like trophies lost in a defeat. Does GT still work?
No. A lower total is a real change, and GT would ignore it. Give each player’s totals a version, raised in the database transaction. Apply with a small Lua script, which writes the score only if the version is newer than the one stored beside the sorted set. That’s safe to repeat and to reorder again, at about 50 extra bytes per player.
Interviewer asksThe board should rank each player's best single match, not total points.
Simpler. ZADD GT with the match score itself keeps the maximum, safe to repeat and to reorder,
with no totals. The tie-breaker is the time the best score was set.
Go deeperMore follow-up questions1 min
The game launches in five more regions. Players in Asia see 200 ms reads. Keep one global board, with its writes in a home region. Put Redis replicas of every shard in each region, for reads. A second of copying delay fits the freshness promise. Game servers submit to the nearest API, which forwards to the home region’s database.
Product wants a board per country. Each country is one more key per window, written by the same updater. That’s one more sorted-set write per score event. Most countries are small enough for one shard, so only the largest few are split. A country board can live in the region nearest its players.
A live final: 100,000 viewers want the top 10 every second. That board is small, one sorted set on one machine. Its top 10 is pushed each second to a service holding the viewers’ open connections.
Further reading
Section titled “Further reading”- Caching, partitioning and message queues on this site, and the distributed rate limiter, which uses the same Redis Cluster hash tags.
- Redis sorted sets, Redis docs: the commands, their complexity, and the leaderboard example.
- ZADD, Redis docs: the GT, LT, NX and XX flags this design depends on.
- Transactional outbox, microservices.io: publishing changes in the same transaction that makes them.
The designbeyondcode digest, every Saturday
New deep dives, interview tips and site news in one short email. Free. Leave with one click.
We never sell or share your email. We only use it to send this newsletter.