Design a notification system

Design a notification system for push, SMS and email. Start by sending inside the API call, let the numbers force a queue, two stores, lanes and broadcast jobs, then stress it with duplicates, failing providers, quiet hours and delivery receipts, with interview prompts and code.

Foundation29 min core · 46 min full depth

Read first: Estimation Queues and streams
Scaling patterns: Idempotency and exactly-once effects Fan-out on write vs fan-out on read Timeouts, retries and circuit breakers Multi-region and active-active
Building blocks: Queues and streams Rate limiting Caching Partitioning Choosing a database Observability
Related: Notification service News feed Distributed job scheduler

“Design a notification system. Our services need to send push notifications, text messages and emails to users.” Then, ten minutes in: “Marketing wants one push to all 50 million users at kickoff. And the login codes still have to arrive.”

Every team that sends notifications thinks this is simple: call an API, and a phone buzzes. It isn’t, for three reasons. One request can mean one message or fifty million. The last step belongs to Apple, Google, phone carriers and email providers, who fail and slow down on their own schedule. And users notice a missing login code, and a duplicate one, equally.

So this page starts with the simplest version, a provider call inside the request, and lets the numbers break it. Then it tests the finished pipeline against duplicates, failing providers, quiet hours and delivery receipts. The classes for channels, templates and retries are in the LLD notification service. This page is the distributed system around them.

The answer in one minute

If you only have a minute, this is the design. The rest of the page explains each part.

Accept, then send later. The API checks the request, saves it durably in Kafka, and answers “accepted” (202). A slow provider never slows the service that asked.
Every request carries an idempotency key. A retried request gets the same notification back, and every later step can spot a repeat. So messages are never lost and rarely doubled.
One lane per channel and priority. Login codes, order updates and marketing each have their own queue, workers and provider connections. A blast or a slow SMS provider stays in its lane.
Broadcasts are one object, not 50 million calls. The audience is split into shards, sent at a controlled pace, saved as it goes, and can be paused within a second.
Each category chooses how to fail. When a send’s outcome is unknown, a login code is retried, and a marketing push is dropped. When we can’t tell whether a user opted out, we don’t send them marketing.

Scope it

A few questions decide which kinds of traffic we carry, and how fast each must move.

Interviewerminute 0

Design a notification system for our app: push, SMS and email.

What they are testing

Whether you separate one message from fifty million, and name isolation as the hard part.

These five questions matter most:

One user, or millions? Both. Most messages go to one user, like a login code. But marketing sends broadcasts of up to 50 million users, within 10 minutes. Those two can’t share a path.
How many users? 100 million a day, 300 million a month.
How fast must a login code go out? Handed to the provider within 1 second, for 99% of codes.
Can messages be lost or doubled? Never lost once accepted. Duplicates should be rare. Nobody can promise exactly once, and it’s worth saying so.
Do users have preferences? Yes: opt-outs per category and channel, quiet hours, and limits on how many marketing messages a day. Some of these are legal duties.

From these answers we make four promises:

Login works. The API is up 99.99% of the time, because login depends on it.
Fast where it matters. The API answers in under 50 ms. Login codes reach the provider within 1 second, other single-user messages within 10 seconds, a 50 million broadcast within 10 minutes.
Never lost, rarely twice. After a 202, a message survives a server dying. Fewer than 1 in 100,000 sends reaches a user twice.
Respect opt-outs. An opt-out stops marketing within a minute. An unknown preference never gets marketing.

The hard one is isolation. A campaign can make 50 million sends, and a provider can slow to a crawl. Neither may delay a login code.

We leave out an in-app inbox (that’s a feed), designing campaigns and audiences, and running our own SMS or mail servers.

Go deeperMore questions, and what each changes1 min

Question	We assume	If the answer were different
Who calls it?	About 50 internal services	Outside customers add tenants, quotas and billing
Track deliveries?	Sent, delivered, opened, 90 days	No tracking removes 300,000 writes a second
Order per user?	Best effort	Strict order, as in chat, needs per-user sequencing and no parallel retries
Ten times fewer users?	No	One region, no cells, Postgres for everything

The numbers that decide it

Now we work out the numbers. Each one should change a decision.

Assumptions

Daily active users: 100 million
Transactional deliveries per user a day: 5
Transactional peak over average: 3×
Campaign deliveries a day: 500 million
Largest broadcast audience: 50 million
Minutes to finish that broadcast: 10
Status writes per delivery: 3
Bytes per delivery record: 300 bytes
Days of delivery history kept: 90

Single-user5,800/s17,000/s at peak

100 million daily users × 5 messages a day = 500 million ≈ 5,800 a second, 17,000 at the daily peak.

So we a handful of API servers is enough.

Broadcasts100,000/sset by one broadcast

The largest broadcast is 50 million users in 10 minutes: 50,000,000 ÷ 600 s ≈ 83,000 a second. So one broadcast sets the peak, about 100,000 a second, eight times the daily average.

So we size senders for 100,000 a second, and raise provider quotas well ahead of time.

SMS bill$20Ma year

Only 0.5% of messages are SMS, mostly login codes. But that’s 1.8 billion texts a year, at about a cent each in the US and several cents elsewhere.

So we send SMS only for critical messages, and use push whenever the user has the app.

Status writes300,000/sat peak

Sent, delivered and sometimes opened: about 3 writes per message.

So we one relational database (about 5,000 writes a second) can't hold the delivery log.

Storage27 TB80 TB replicated

1 billion messages a day × 300 bytes × 90 days ≈ 27 TB.

So we the store must expire old rows by itself, not by a nightly delete over billions of rows.

User data is small: 300 million users × about 600 bytes is 180 GB. But it’s read once per message, 100,000 times a second, so it needs a cache. Try the calculator: shrink the broadcast window to 2 minutes, and the peak passes 400,000 a second, where provider quotas run out.

Change an assumption

Daily active users100 millionTransactional deliveries per user a day5Transactional peak over average3×Campaign deliveries a day500 millionLargest broadcast audience50 millionMinutes to finish that broadcast10Status writes per delivery3Bytes per delivery record300 bytesDays of delivery history kept90

Transactional deliveries, average: 5,800/s
Transactional deliveries, peak: 17,000/s
One broadcast at its deadline: 83,000/s; One broadcast, not the daily curve, sets the peak.
Deliveries, peak: 100,000/s; About 170 senders at 1,000 sends/s each, run at 60%; provider quotas and one Redis cluster still fit.
Deliveries a day: 1 billion
Deliveries, average: 12,000/s; The average sizes nothing; the fleet is sized for the peak and autoscales down.
Status writes, peak: 300,000/s; Past one Postgres primary: the delivery log needs its own write-optimized, partitioned store.
Delivery history stored: 27 TB; Tens of TB: a horizontally partitioned store with TTL expiry, not one database with a nightly delete job.
Idempotency keys, 24 hours: 50 GB; Keys of about 100 bytes, one per transactional request at most. Fits one Redis node with a replica.
Users and devices (MAU about 3 × DAU): 180 GB; About 600 bytes per user with two devices. Fits one Postgres; the hot set fits a Redis cluster.

Build the design

We start with the simplest design that works. Then we let the numbers break it, one at a time. Each step fixes only what broke, adds only the boxes that fix needs, and says what the fix costs.

Step 1. Send inside the API call

The simplest notification service calls the provider inside the request. The order service asks us to send a push, we call Apple or Google, and we answer when they answer.

Notification system · architecture

Stage 1: Send inline

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Deliveries per second: 100,000

Stage 1 of 6

Send inline

Why this stage

Nothing yet.

This is the smallest design that meets every functional requirement.

What we add

One notification service that checks preferences, renders a template and calls the provider while the producer waits, and one Postgres for users, preferences, devices and the delivery log.

What it costs

Producers wait on APNs, SMS and email providers; a provider outage becomes a producer outage; one server and one database carry everything.

At 100k deliveries per second, Ingest API is 10000% busy. Stage 2 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One server sending inline manages about 1,000 deliveries a second: 200 ms provider calls, 200 in flight.
One ingest server handles about 5,000 requests a second. A router handles about 5,000 deliveries, and a sender about 1,000 sends.
One Postgres does about 10,000 point reads a second and 5,000 writes.
One Redis node does about 100,000 operations a second, or 200,000 pipelined.

How big the shared systems are

A Cassandra node takes about 50,000 small writes a second, so a 12-node ring takes 600,000.
The Kafka cluster takes about 2 million messages a second.
Our provider accounts accept about 400,000 sends a second combined, mostly push.

Per delivery

Each delivery makes 3 status writes.
The preference cache answers about 97% of lookups.

This works until a provider has a bad day. Then the caller waits on Apple or a phone carrier, and a provider outage becomes the caller’s outage. A slow SMS provider would make the login service time out. And if our server crashes mid-call, the message is simply lost.

Step 2. Accept, queue, send later

So the API does only what takes milliseconds. It checks the caller and the template, checks the idempotency key, saves the message to Kafka on at least two machines, and answers 202 Accepted. Workers do the slow part later.

Notification system · architecture

Stage 2: Accept, queue, send later

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Deliveries per second: 100,000

Stage 2 of 6

Accept, queue, send later

Why this stage

At the 100,000/s peak one server making 200 ms provider calls is 100 times over, and a slow SMS provider would hold the login service hostage.

What we add

An ingest API that validates, dedups on an idempotency key and returns 202, a Kafka events topic, and a fleet of send workers.

What it costs

Delivery is now asynchronous, so producers need a status API; and the database still takes a read and three writes per delivery.

At 100k deliveries per second, User data is 6000% busy. Stage 3 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One server sending inline manages about 1,000 deliveries a second: 200 ms provider calls, 200 in flight.
One ingest server handles about 5,000 requests a second. A router handles about 5,000 deliveries, and a sender about 1,000 sends.
One Postgres does about 10,000 point reads a second and 5,000 writes.
One Redis node does about 100,000 operations a second, or 200,000 pipelined.

How big the shared systems are

A Cassandra node takes about 50,000 small writes a second, so a 12-node ring takes 600,000.
The Kafka cluster takes about 2 million messages a second.
Our provider accounts accept about 400,000 sends a second combined, mostly push.

Per delivery

Each delivery makes 3 status writes.
The preference cache answers about 97% of lookups.

POST /v1/notifications
Idempotency-Key: login-attempt-8f3a1c

{
  "userId": "u_4821",
  "category": "security.otp",
  "priority": "critical",
  "template": "otp_code",
  "params": { "code": "481 207" },
  "ttlSeconds": 300
}

202 Accepted
{ "notificationId": "a91f0c3e7d2b44e1b0c5f6a8d9e2b371" }

The category’s rules pick the channels, like “push, else email” for an order update. ttlSeconds lets the system drop a message that can no longer help, like a login code after 5 minutes.

Spotting a retried request

Callers retry when a request times out. So the same request can arrive twice. The idempotency key is how we spot it: the caller sends the same key with every retry of one attempt.

The API saves the key in Redis with “only if absent”. A new key goes through. A known key gets the same notification ID back. The notification ID is also a hash of the caller and the key. So even if a message lands in Kafka twice, both copies share one ID, and the next step drops the second.

A retried send returns the first notification

Scroll or pinch to zoom, drag to pan

Color key
Queue or stream
Cache
Service

IngestService.javaingest_service.pyIngestService.csingest_service.goingest-service.ts

// The ingest API's accept step. A producer that times out retries with the same
// Idempotency-Key, and must get the first notification back, never a second message.
interface KeyStore {
    String putIfAbsent(String key, String value, Duration ttl); // SET NX EX: null if we set it
    void put(String key, String value, Duration ttl);
    void delete(String key);
}

interface EventLog {
    // Returns once the record is on the in-sync replicas (acks=all), so a 202 means durable.
    void append(String topic, String partitionKey, String notificationId, String body);
}

enum Outcome { ACCEPTED, DUPLICATE, IN_PROGRESS, KEY_REUSED }

record Result(Outcome outcome, String notificationId) {}

class IngestService {
    // Covers any producer's retry window, and small enough to keep every key in memory.
    static final Duration DONE_TTL = Duration.ofHours(24);
    // A crash between claiming the key and appending must not block it for a day.
    static final Duration PENDING_TTL = Duration.ofSeconds(30);

    private final KeyStore keys;
    private final EventLog log;

    IngestService(KeyStore keys, EventLog log) {
        this.keys = keys;
        this.log = log;
    }

    Result accept(String producer, String idemKey, String userId, String lane, String body) {
        // The ID comes from the key, not from a random generator. If an append times out
        // but did land, and the retry appends again, both copies carry the same ID and the
        // sender's claim drops the second one.
        String id = sha256(producer + "\n" + idemKey).substring(0, 32);
        String slot = "idem:" + id;
        String fingerprint = sha256(body);

        String prior = keys.putIfAbsent(slot, "pending:" + fingerprint, PENDING_TTL);
        if (prior != null) {
            // Same key, different body: a producer bug, not a retry. Refuse it (409).
            if (!prior.endsWith(fingerprint)) return new Result(Outcome.KEY_REUSED, id);
            // The first attempt is still appending. A 202 now could promise a message that
            // then fails, so the producer is told to retry shortly (409 with Retry-After).
            if (prior.startsWith("pending:")) return new Result(Outcome.IN_PROGRESS, id);
            return new Result(Outcome.DUPLICATE, id); // 202 with the first ID; nothing queued
        }
        try {
            // One topic per lane (critical, high, bulk). Partitioning by user keeps one
            // user's messages in order within a lane.
            log.append("events." + lane, userId, id, body);
        } catch (RuntimeException e) {
            keys.delete(slot); // as far as we know nothing was queued: let the retry run
            throw e;           // 503, and the producer retries with the same key
        }
        keys.put(slot, "done:" + fingerprint, DONE_TTL);
        return new Result(Outcome.ACCEPTED, id);
    }

    static String sha256(String s) {
        try {
            byte[] d = MessageDigest.getInstance("SHA-256").digest(s.getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(d);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException(e);
        }
    }
}

Step 3. Two kinds of data, two stores

Every message needs two things: a read of the user’s settings, and about three status writes. They have nothing in common except the user ID, so they live apart.

User settings (contacts, opt-outs, quiet hours, devices) are small and read constantly. They stay in Postgres, with a Redis copy of each user in front. About 97% of reads hit the copy. When a user changes a setting, Postgres commits first, then the copy is deleted, retried until it succeeds.
The delivery log is huge, write-heavy and expires. It goes to Cassandra with a 90-day expiry built in. Each status (sent, delivered, opened) is its own column, so a “delivered” receipt that arrives before “sent” is harmless. Each just sets its own column.

Notification system · architecture

Stage 3: Split the data by access pattern

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Deliveries per second: 100,000

Stage 3 of 6

Split the data by access pattern

Why this stage

At the peak Postgres would serve 100,000 preference reads and 300,000 status writes a second, against roughly 10,000 and 5,000.

Tracking also needs provider callbacks.

What we add

A Redis cache for preferences and devices, a Cassandra delivery log with a 90 day TTL, and a callback receiver for provider receipts.

What it costs

An opt-out reaches the routers through cache invalidation, a second or two late, and there are three stores to run.

At 100k deliveries per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One server sending inline manages about 1,000 deliveries a second: 200 ms provider calls, 200 in flight.
One ingest server handles about 5,000 requests a second. A router handles about 5,000 deliveries, and a sender about 1,000 sends.
One Postgres does about 10,000 point reads a second and 5,000 writes.
One Redis node does about 100,000 operations a second, or 200,000 pipelined.

How big the shared systems are

A Cassandra node takes about 50,000 small writes a second, so a 12-node ring takes 600,000.
The Kafka cluster takes about 2 million messages a second.
Our provider accounts accept about 400,000 sends a second combined, mostly push.

Per delivery

Each delivery makes 3 status writes.
The preference cache answers about 97% of lookups.

Go deeperSizing the two stores1 min

The delivery log. At a broadcast peak, about 300,000 status writes arrive each second. One Cassandra node takes roughly 20,000 small writes a second while staying healthy, and every write is stored on 3 nodes. So the log needs about 300,000 × 3 ÷ 20,000 = 45 nodes at peak. 80 TB of replicated data spread over 45 nodes is under 2 TB each, which is comfortable. Rows expire after 90 days on their own, so nothing has to run a huge nightly delete. Each row is found by its notification ID, so a status update is one write to one place. A support page that asks “what did we send this user today?” reads a second table, keyed by user and day, that the sender fills at the same time.

User settings. 180 GB is small enough for one Postgres primary with read copies. The Redis copy in front answers about 97% of the 100,000 reads a second, so Postgres sees about 3,000 a second. That is a normal load for one database.

The point of the split shows up here. If both kinds of data lived in one store, it would have to be sized for 300,000 writes a second and for 100,000 reads a second at once. Apart, each store is sized for one job.

Step 4. Lanes, not priorities

With one queue, a login code waits behind 20 million marketing pushes: four minutes at 83,000 a second. And if the SMS provider slows from 200 ms to 5 seconds a call, every worker ends up stuck on an SMS call, and pushes stop too. Both problems are called head-of-line blocking: one slow thing at the front holds up everything behind it.

So the workers split in two:

Routers read the user’s settings, apply opt-outs, quiet hours and limits, pick the channels, fill in the template, and write one job per channel into a lane.
A lane is a topic for one channel and priority, like sms.critical or push.bulk. Senders drain each lane with their own workers and their own provider connections.

Notification system · architecture

Stage 4: Channels and priorities

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Deliveries per second: 100,000

Stage 4 of 6

Channels and priorities

Why this stage

Not load: at the peak nothing is red.

A marketing blast queued ahead of an OTP, or one slow SMS provider holding every worker, breaks the product.

What we add

Routers that apply preferences, quiet hours and caps, then one queue per channel and priority, each drained by its own sender pool.

What it costs

More topics and pools to size, and capacity reserved for critical lanes that sits idle most of the day.

At 100k deliveries per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One server sending inline manages about 1,000 deliveries a second: 200 ms provider calls, 200 in flight.
One ingest server handles about 5,000 requests a second. A router handles about 5,000 deliveries, and a sender about 1,000 sends.
One Postgres does about 10,000 point reads a second and 5,000 writes.
One Redis node does about 100,000 operations a second, or 200,000 pipelined.

How big the shared systems are

A Cassandra node takes about 50,000 small writes a second, so a 12-node ring takes 600,000.
The Kafka cluster takes about 2 million messages a second.
Our provider accounts accept about 400,000 sends a second combined, mostly push.

Per delivery

Each delivery makes 3 status writes.
The preference cache answers about 97% of lookups.

Critical lanes get their own workers, sized for the worst hour plus 100%, and never scaled to zero. Bulk lanes are throttled per campaign, so a blast can only slow other blasts. Follow a login code to the SMS provider, then its receipt coming back:

Notification system · architecture

Trace a request: An OTP by SMS

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

An OTP by SMS: step 1 of 10

Producers to Ingest API

The auth service posts category security.otp, priority critical, a 5 minute TTL, and the login attempt ID as the Idempotency-Key.

2 ms of a 1000 ms budget

Producers → Ingest API2 ms
Ingest API → Idempotency1 ms
Ingest API → Events8 ms
Events → Routers10 ms
Routers → Prefs cache2 ms
Routers → Channel queues8 ms
Channel queues → Senders10 ms
Senders → Providers250 ms
Senders → Delivery logasync
Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

This is enough for a mid-level answer. Accept and queue, an idempotency key, two stores, and lanes per channel. Step 5 and the stress tests are what make it a senior answer.

Go deeperOther ways to isolate traffic1 min

Approach	Login code behind a blast?	Slow SMS stops push?	Cost
One queue with a priority field	Yes: the blast is already ahead	Yes	Nothing
A broker with built-in priorities	Mostly no, but busy workers hold it	Yes	One broker’s throughput
A lane per channel and priority	No: critical lanes have reserved capacity	No	About 12 lanes to size

Only separate lanes keep a login code clear of both a queued blast and a slow provider.

Go deeperWhy Kafka, and the rest of the API2 min

Option	Strength here	Weakness here
Kafka	Millions a second, replay after a bug, order per user	No per-message delay: retries need delay topics
Amazon SQS	Per-message delays and a built-in dead-letter queue	No replay; per-request cost at 100,000 a second
RabbitMQ	Per-message acks, priorities	Throughput per queue and operations at this scale

Kafka wins on throughput and replay: a template bug is fixed by replaying from a position. SQS is an equally good answer on AWS.

Why not let callers write to Kafka directly? Then 50 services each check templates, quotas and keys, and one bad deploy writes garbage into a shared topic. The API is thin, but it’s the one place those rules are enforced.

The rest of the API:

POST   /v1/campaigns                      { segmentId, template, sendAt, deadlineMinutes }
POST   /v1/campaigns/{id}/pause           the kill switch
PUT    /v1/users/{id}/devices/{deviceId}  { platform, token } on every app start
DELETE /v1/users/{id}/devices/{deviceId}  on logout
PUT    /v1/users/{id}/preferences         { category, channel, enabled } and quiet hours
GET    /v1/notifications/{id}             per channel: sent, delivered, opened, failed

Step 5. Broadcasts as batch jobs

One request breaks everything we have built so far. Marketing wants one push to all 50 million users at kickoff, done in ten minutes.

Sending a broadcast through the normal API means 50 million calls, each with its own key, from a marketing tool that can’t resume if it crashes halfway. So a broadcast becomes one object instead.

The campaign service stores the plan. At send time, the audience (a file of user IDs in S3) is split into 1,000 shards of 50,000. Fan-out workers take shards, read the user IDs, and write batches of 500 into the bulk lane, at a pace that spreads the campaign over its 10 minutes. The same routers expand each batch, so broadcasts get the same opt-outs, quiet hours and templates.

Notification system · architecture

Stage 5: Broadcast fan-out

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Deliveries per second: 100,000

Stage 5 of 6

Broadcast fan-out

Why this stage

A 50 million user campaign sent as 50 million API calls needs 34 ingest servers and 50 million idempotency keys for one decision, and must finish in 10 minutes without starving OTPs.

What we add

A campaign service, audience snapshots in S3 split into 1,000 shards, and fan-out workers that stream user IDs into a throttled bulk lane.

What it costs

A second ingestion path, and a campaign that must be resumable and pausable mid-send.

At 100k deliveries per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One server sending inline manages about 1,000 deliveries a second: 200 ms provider calls, 200 in flight.
One ingest server handles about 5,000 requests a second. A router handles about 5,000 deliveries, and a sender about 1,000 sends.
One Postgres does about 10,000 point reads a second and 5,000 writes.
One Redis node does about 100,000 operations a second, or 200,000 pipelined.

How big the shared systems are

A Cassandra node takes about 50,000 small writes a second, so a 12-node ring takes 600,000.
The Kafka cluster takes about 2 million messages a second.
Our provider accounts accept about 400,000 sends a second combined, mostly push.

Per delivery

Each delivery makes 3 status writes.
The preference cache answers about 97% of lookups.

Stress it

The design is complete. Now we test it against what goes wrong with real notifications: duplicates, 50 million users in 10 minutes, a provider that fails, quiet hours, and knowing whether anything arrived. No new boxes appear here.

Never lost, rarely twice

Interviewerminute 20

A user just got the same push three times. Where can duplicates come from, and what stops each?

What they are testing

Whether you admit that exactly once is impossible at the provider, and choose per category which way to fail.

Source	Defense
The caller retries after a timeout	Idempotency key; ID made from the key
Kafka delivers a message again after a crash	The sender claims (notification ID, channel) in Redis before calling out
A broadcast replays a batch after a crash	Each recipient’s ID is a hash of campaign and user, so the same claim catches it
The sender called the provider, then died	Can’t be fully solved: collapse IDs on the phone, and a choice per category

The last row is the honest part. Say the connection to Apple drops mid-request. Apple may or may not have accepted the push, and there’s no way to ask. Retrying risks a duplicate. Not retrying risks a loss. So each category chooses. A login code retries, because two codes annoy and no code locks the user out. A marketing push doesn’t.

A sender dies after the provider accepted the push

Scroll or pinch to zoom, drag to pan

Color key
Service
Cache
Queue or stream

The phone helps too. Apple’s apns-collapse-id and Android’s collapse_key make a second copy replace the first, instead of showing beside it. So even the unavoidable duplicate is usually invisible. SMS has nothing like this, which is one more reason SMS is kept for a few message types.

50 million users in 10 minutes

Kickoff arrives. Fifty million users in ten minutes is about 83,000 sends a second, on top of normal traffic. The fan-out workers pace each shard so the whole campaign spreads evenly over its ten minutes. The bulk lane fills at that pace and no faster, so login codes in the critical lane never notice.

FanoutWorker.javafanout_worker.pyFanoutWorker.csfanout_worker.gofanout-worker.ts

// A fan-out worker leases one audience shard (about 50,000 user IDs) at a time and turns it
// into batches on the bulk lane. A broadcast is 1,000 of these running in parallel.
interface Audience { List<String> read(String campaignId, int shard, long offset, int limit); }

interface Checkpoints {
    long load(String campaignId, int shard);
    void save(String campaignId, int shard, long offset);
}

interface Throttle { void acquire(int permits) throws InterruptedException; }

interface Campaigns { boolean paused(String campaignId); }

interface BulkLane { void append(String partitionKey, String body); }

class FanoutWorker {
    static final int BATCH = 500;

    private final Audience audience;
    private final Checkpoints checkpoints;
    private final Throttle throttle;
    private final Campaigns campaigns;
    private final BulkLane lane;

    FanoutWorker(Audience audience, Checkpoints checkpoints, Throttle throttle,
                 Campaigns campaigns, BulkLane lane) {
        this.audience = audience;
        this.checkpoints = checkpoints;
        this.throttle = throttle;
        this.campaigns = campaigns;
        this.lane = lane;
    }

    void runShard(String campaignId, int shard) throws InterruptedException {
        long offset = checkpoints.load(campaignId, shard); // resume where a dead worker stopped
        while (true) {
            // The kill switch is read every batch, so "stop this campaign" takes effect
            // within a second, not after 50 million sends.
            if (campaigns.paused(campaignId)) return;
            List<String> users = audience.read(campaignId, shard, offset, BATCH);
            if (users.isEmpty()) return; // shard finished
            // The campaign's rate (audience / deadline) split across its running shards,
            // so the blast takes its 10 minutes instead of everything at once.
            throttle.acquire(users.size());
            // One record per 500 users: the router renders once per locale per batch,
            // and the bulk lane carries 1/500 of the records.
            lane.append(campaignId + ":" + shard, campaignId + "|" + String.join(",", users));
            offset += users.size();
            // Save after the append, never before. A crash in between replays this batch,
            // which the senders drop by notification ID, instead of skipping 500 users.
            checkpoints.save(campaignId, shard, offset);
        }
    }

    // Every recipient's ID is derived, not random, so a replayed batch yields the same IDs.
    static String notificationId(String campaignId, String userId) {
        // SHA-256, cut to 32 hex characters, like the ingest path's IDs.
        try {
            byte[] d = MessageDigest.getInstance("SHA-256")
                    .digest((campaignId + "\n" + userId).getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(d).substring(0, 32);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException(e);
        }
    }
}

The worker saves its place after writing a batch, never before. A worker that dies in between replays up to 500 users, and the senders drop the repeats, because each recipient’s ID is a hash of campaign and user. Saving first would skip those users silently.

Notification system · architecture

Trace a request: One broadcast recipient

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

One broadcast recipient: step 1 of 9

Campaigns to Fan-out

At send time the campaign is split into 1,000 audience shards of 50,000 users. A fan-out worker leases one shard.

50 ms of a 1000 ms budget

Campaigns → Fan-out50 ms
Fan-out → Audiences100 ms
Fan-out → Events20 ms
Events → Routers50 ms
Routers → Prefs cache3 ms
Routers → Channel queues10 ms
Channel queues → Senders20 ms
Senders → Providers100 ms
Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The kill switch. Say the campaign has a typo, and it’s 30 seconds in. The marketer presses pause. Fan-out workers check the flag before every batch, so new batches stop within a second. Routers check the same flag, so what’s already in the lane is dropped too.

Go deeperWhy not the provider's own topics?1 min

Android’s push service can send one message to every device subscribed to a topic. One call replaces millions. But it gives up preferences, quiet hours, languages and tracking per user. It suits identical breaking news, not marketing.

Approach	Respects settings?	Tracking per user	Cost
The caller loops over the API	Yes	Yes	50 million calls; no resume or pace
Shards, batches, shared routers	Yes	Yes	A second way in to keep consistent
Provider topics	No	Totals only	Almost free

Users who already got a wrong message can’t be un-sent. A correction goes to exactly the users the delivery log shows as sent.

A provider that fails

Now the SMS provider starts failing 30% of requests. Here are the next five minutes.

The first decision is whether to retry at all:

Answer	Meaning	Action
Apple `410 Unregistered`, Android `UNREGISTERED`	Token is dead	Delete the token, never retry
Invalid phone number, email hard bounce	Permanent	Mark the address bad, never retry
`429`, quota exceeded	Throttled	Retry no sooner than `Retry-After`
`5xx`, connection refused	Not sent	Retry with backoff, maybe on another provider
Timeout after the request was sent	Unknown	Retry if critical, give up if bulk

Retries never sleep inside a worker. A failed job goes to a delay topic (retry.10s, retry.1m, retry.10m) that holds it until it’s due. After eight tries, or once its time limit passes, it goes to a dead-letter topic: a holding area with the last error, where an alert counts arrivals and an operator can replay them after a fix.

Sender.javasender.pySender.cssender.gosender.ts

// A sender handles one delivery job from a channel queue. Kafka redelivers after a crash
// or a rebalance, so handling the same job twice must not send it twice.
enum SendStatus {
    OK,       // the provider accepted it
    REJECTED, // permanent: bad device token, invalid number. Retrying cannot help.
    FAILED,   // definitely not sent: connection refused, 429, a 5xx before acceptance
    UNKNOWN   // timed out after the request left: the provider may have sent it
}

record SendResult(SendStatus status, String providerId, Duration retryAfter, boolean tokenInvalid) {}

record Job(String notificationId, String channel, String userId, String address, String payload,
           int attempt, Instant expiresAt, boolean critical) {
    Job nextAttempt() {
        return new Job(notificationId, channel, userId, address, payload, attempt + 1, expiresAt, critical);
    }
}

interface Provider { SendResult send(Job job); }

interface Claims {
    boolean claim(String key, Duration lease); // SET key "sending" NX PX lease
    void markSent(String key, Duration keep);
    void release(String key);
}

interface Outbox {
    void retryLater(Job job, Duration delay); // a delay topic: Kafka has no per-message delay
    void deadLetter(Job job, String reason);
    void status(Job job, String status, String detail);
    void removeToken(String userId, String token);
}

class Sender {
    static final int MAX_ATTEMPTS = 8;
    static final Duration BASE = Duration.ofSeconds(1);
    static final Duration CAP = Duration.ofMinutes(10);
    static final Duration KEEP = Duration.ofDays(7); // longer than any redelivery window

    private final Provider provider;
    private final Claims claims;
    private final Outbox out;

    Sender(Provider provider, Claims claims, Outbox out) {
        this.provider = provider;
        this.claims = claims;
        this.out = out;
    }

    void handle(Job job, Instant now) {
        // A login code that arrives after it expired is worse than none.
        if (now.isAfter(job.expiresAt())) {
            out.status(job, "expired", null);
            return;
        }
        String key = "claim:" + job.notificationId() + ":" + job.channel();
        // A lease, not a lock: if this worker dies mid-call the claim expires, and a
        // redelivered copy may try again. False means sent already or in flight elsewhere.
        if (!claims.claim(key, Duration.ofMinutes(2))) return;

        SendResult r = provider.send(job);
        switch (r.status()) {
            case OK -> {
                claims.markSent(key, KEEP);
                out.status(job, "sent", r.providerId());
            }
            case REJECTED -> {
                claims.markSent(key, KEEP); // final: never retried
                if (r.tokenInvalid()) out.removeToken(job.userId(), job.address());
                out.status(job, "failed", "rejected");
            }
            case UNKNOWN -> {
                // Retrying risks a duplicate; not retrying risks a loss. Critical
                // messages accept the duplicate, marketing accepts the loss.
                if (job.critical()) {
                    retry(job, key, r);
                } else {
                    claims.markSent(key, KEEP);
                    out.status(job, "unknown", null);
                }
            }
            case FAILED -> retry(job, key, r);
        }
    }

    private void retry(Job job, String key, SendResult r) {
        claims.release(key); // the retry has to be able to claim it again
        if (job.attempt() + 1 >= MAX_ATTEMPTS) {
            out.deadLetter(job, r.status().name()); // kept for inspection and replay
            out.status(job, "failed", "retries exhausted");
            return;
        }
        out.retryLater(job.nextAttempt(), backoff(job.attempt(), r.retryAfter()));
    }

    // Full jitter: a random delay up to an exponential bound. Without the randomness every
    // job that failed in the same provider blip retries in the same second and causes the next one.
    static Duration backoff(int attempt, Duration retryAfter) {
        long bound = Math.min(CAP.toMillis(), BASE.toMillis() << Math.min(attempt, 20));
        long delay = ThreadLocalRandom.current().nextLong(bound + 1);
        // The provider's Retry-After is a floor: retrying sooner only earns another 429.
        long floor = retryAfter == null ? 0 : retryAfter.toMillis();
        return Duration.ofMillis(Math.max(delay, floor));
    }
}

The randomness, called jitter, is the important line. Without it, every job that failed during a 10-second blip retries at exactly the same moments, in waves that hit the provider just as it recovers. A random delay spreads them out.

Failover. Senders track errors per provider. When errors pass 20% over 30 seconds, a circuit breaker opens: the sender stops calling that provider and sends to the backup.

SMS failover when the primary provider degrades

Scroll or pinch to zoom, drag to pan

Color key
Service

For SMS and email there are real alternatives. We keep two providers per channel, both warm with a 90/10 split, because a provider used for the first time in an emergency fails in new ways. Push has no alternative: Apple is the only road to an iPhone. When Apple struggles, a login code falls back to SMS, and everything else waits in its lane until its time limit.

Go deeper2 million messages in the dead-letter topic1 min

Say a template bug sent 2 million messages to the dead-letter topic. Fix or roll back the template first, then decide per category instead of replaying everything. Expired login codes are dropped. Order updates are replayed slowly with their original IDs, so anything that did go out is skipped. Marketing is usually dropped: a campaign hours late is worse than none.

Quiet hours and limits

A user in Tokyo has quiet hours from 10 p.m. to 8 a.m. The kickoff campaign fires at 3 a.m. their time. What happens?

A router runs the same checks for every recipient, all from the one cached copy of the user:

What a router checks for every recipient, in order

1. Hard filters

Opted out of this category and channel? Address unverified or bounced? No recorded consent for marketing SMS? Any yes drops that channel, with a reason support tools can show.

2. Quiet hours

Local time comes from the time zone name, like Asia/Tokyo, never a stored offset, which is wrong half the year. Critical messages go anyway. Everything else waits until the window ends, plus a random few minutes, so 8:00 a.m. isn’t a spike.

3. Daily limits

A Redis counter per user and kind, such as 3 marketing pushes a day, reset at the user’s local midnight. Over the limit, marketing is dropped.

4. Channel and template

The category’s channel order, filtered by what survived, then the template in the user’s language, falling back from pt-BR to pt to en.

Go deeperWhere delayed messages wait, and why limits can be loose1 min

Delayed messages go to a schedule: a Redis sorted set by send time for the next hour, and a Cassandra table beyond that. A poller moves due messages back into their lanes. A campaign meant to land “at 9 a.m. local” is split by time zone into about 30 waves, instead of being delayed user by user.

What “delivered” means

How do we know a push was delivered? And how do we keep 600 million device tokens from going stale?

“Delivered” means something different on each channel, and the log records only what’s true. A 200 from Apple means Apple accepted the push, nothing more. If the phone is offline, Apple keeps only the latest message per app. So iOS delivery is known only if our own app reports it.

Tokens go stale constantly, so they are removed on the provider’s say-so, on logout, and after 60 days unused.

Go deeperHow tokens die, and what each channel can tell you3 min

A push token identifies one app install. It changes when the app is reinstalled or moved to a new phone. So the app sends its current token on every start. Tokens die in four ways:

The provider says so. Apple’s 410 or Android’s UNREGISTERED: delete it at once.
The user logs out. The app deletes its token. Skipping this is a privacy bug: the next person on a shared tablet gets the previous user’s messages.
Another account claims it. Each token has one owner, so it moves to the new user.
Age. A daily job removes tokens the app hasn’t refreshed in 60 days.

Channel	“Sent” means	“Delivered” comes from	“Opened” comes from
iOS push	Apple accepted it	Only our app, reporting back	Our app, on tap
Android push	Google accepted it	Google’s summary reports, or our app	Our app, on tap
SMS	Provider accepted it	The carrier’s report, via the provider	Not available; a tracked link at best
Email	Provider accepted it	The receiving mail server took it	A tracking pixel, inflated by prefetch

Provider callbacks are answered 200 at once and processed from a queue. Providers retry slow callbacks and back off, which delays every later receipt.

Email has its own rules. Since 2024, Gmail and Yahoo require bulk senders to prove who they are (SPF, DKIM and DMARC), offer one-click unsubscribe, and keep complaints low. So transactional and marketing mail use separate servers and domains, and a bad campaign can’t hurt password resets.

When things break

Interviewerminute 40

The push provider stops answering at kickoff. What happens to the messages?

What they are testing

Whether you know messages wait safely in the queue, retries back off, and urgent ones have another way out.

The push provider stops answering. Nothing is lost, because every message was saved before anyone tried to send it. Push messages wait in their lane, and retries back off with jitter, so we don’t hammer the provider while it recovers. The circuit breaker opens, so senders stop wasting time on calls that will fail. Urgent messages, like a login code, can go out by SMS instead, if the user allowed it. Marketing simply waits, and is dropped if its time limit passes first.

The preference cache goes down at kickoff. Login codes survive almost anything. Marketing is the first thing turned off. Under stress, lanes are shed in a fixed order: marketing, then social, then order updates, never critical. Bulk lanes pause by themselves when the oldest login code waits more than 2 seconds. If the preference cache is down, bulk pauses, critical messages read Postgres directly, and marketing never guesses.

Click any box to break it and see what users would notice:

Notification system · architecture

Break it: what fails, who notices, what users see

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Break

Failure: click any box to break it

An SMS provider goes down

What breaks

Timeouts and 5xx on every call, or silently missing delivery reports.

How we notice

Error rate and p99 per provider, and delivered-to-sent ratio per country.

What users see

Seconds of delay while the circuit breaker opens, then texts flow through the second provider.

What we do

Weighted routing across two SMS and two email providers.

APNs and FCM have no substitute, so push queues and retries with backoff.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

At a million messages a second, the cache, the user database, the log and the provider accounts all run out. Each could grow on its own, but one bigger shared system also means one bad deploy reaches all 300 million users. So we split users into cells. A hash of the user ID picks one of 8 cells, and each cell runs the whole pipeline, with its own Redis and Cassandra, for an eighth of the users. Four cells run in each of two regions, each with a standby copy of its user data in the other region. A region outage moves four cells, not everything.

Notification system · architecture

Stage 6: Cells and regions

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Deliveries per second: 100,000

Stage 6 of 6

Cells and regions

Why this stage

At 1 million deliveries/s the preference cache, the delivery log and our provider quotas are all past 100%, and one region is one failure domain for OTPs.

What we add

Eight user cells, each running the whole pipeline for one slice of users, four per region, with more provider accounts per channel.

What it costs

Cell routing at ingest, cross-cell broadcasts, and eight copies of everything to deploy and watch.

At 100k deliveries per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One server sending inline manages about 1,000 deliveries a second: 200 ms provider calls, 200 in flight.
One ingest server handles about 5,000 requests a second. A router handles about 5,000 deliveries, and a sender about 1,000 sends.
One Postgres does about 10,000 point reads a second and 5,000 writes.
One Redis node does about 100,000 operations a second, or 200,000 pipelined.

How big the shared systems are

A Cassandra node takes about 50,000 small writes a second, so a 12-node ring takes 600,000.
The Kafka cluster takes about 2 million messages a second.
Our provider accounts accept about 400,000 sends a second combined, mostly push.

Per delivery

Each delivery makes 3 status writes.
The preference cache answers about 97% of lookups.

Go deeperEvery failure, and what users see2 min

Part	Failure	What users see	What we do
API server	Crash	Nothing; callers retry	No state on it; keys make retries safe
Idempotency Redis	Failover loses recent keys	A rare duplicate	Derived IDs and sender claims catch it
Kafka	Broker down	Seconds of delay	Three copies, writes wait for two
Router or sender	Crash	Seconds of delay	Others take over; claims drop the repeats
User database	Primary down	Settings fail for 30 s; sends continue	Failover; the cache serves reads
SMS or email provider	Errors or slowness	Seconds of delay	Circuit breaker, switch to the warm backup
Apple or Google push	Degraded	Push delayed	Back off within the time limit; codes go by SMS
Delivery log	Node down	Status lags; sends unaffected	Three copies, buffered status writes
Fan-out worker	Crash	Part of a campaign pauses briefly	Resume from the saved place; repeats dropped
Cell or region	Outage	An eighth of users delayed in failover	Promote the cell’s standby in the other region

A late opt-out. A marketing message routed a second before an opt-out is already in its lane. Senders re-check a short list of recent marketing opt-outs just before sending, which closes most of that gap.

Go deeperRunning it: alerts and abuse2 min

Wake someone up when: login codes take more than 2 seconds to reach the provider for 5 minutes; the API’s success rate drops below 99.95%; the receipt rate for any provider and country drops 20% below its weekly normal; a lane falls behind its target; or the dead-letter topic fills faster than usual, which almost always means a bad template.

The most dangerous failure is silent. A provider that accepts everything and delivers nothing shows no errors at all. The signal is the receipt rate per provider and country. A drop below normal pages someone like an outage. Each part of the system also sends a test notification to real test phones every 10 seconds.

SMS pumping. Attackers trigger login codes to premium-rate numbers they profit from. So limit codes per phone, address and country, block countries the product doesn’t serve, and alert on SMS spend per country per hour.

Caller bugs are the most common incident: a loop that messages every user. Quotas per caller, volume alarms per category, and a kill switch per category stop it before it reaches millions.

Wrap up

The whole design

Here is everything we built, in the order we built it. Use the buttons to step through it.

Notification system · architecture

The full design, stage by stage

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Deliveries per second: 100,000

Stage 6 of 6

Cells and regions

Why this stage

At 1 million deliveries/s the preference cache, the delivery log and our provider quotas are all past 100%, and one region is one failure domain for OTPs.

What we add

Eight user cells, each running the whole pipeline for one slice of users, four per region, with more provider accounts per channel.

What it costs

Cell routing at ingest, cross-cell broadcasts, and eight copies of everything to deploy and watch.

At 100k deliveries per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.The whole design. Step through it: sending inline, a queue, two stores, lanes, broadcasts, then cells and regions.

The numbers behind the busy bars

What one machine can do

One server sending inline manages about 1,000 deliveries a second: 200 ms provider calls, 200 in flight.
One ingest server handles about 5,000 requests a second. A router handles about 5,000 deliveries, and a sender about 1,000 sends.
One Postgres does about 10,000 point reads a second and 5,000 writes.
One Redis node does about 100,000 operations a second, or 200,000 pipelined.

How big the shared systems are

A Cassandra node takes about 50,000 small writes a second, so a 12-node ring takes 600,000.
The Kafka cluster takes about 2 million messages a second.
Our provider accounts accept about 400,000 sends a second combined, mostly push.

Per delivery

Each delivery makes 3 status writes.
The preference cache answers about 97% of lookups.

The decisions, and what each one costs

Decision	We chose	We gave up	We’d choose differently if
Accepting	`202` after a durable write, idempotency keys	An answer that says “sent”	Callers truly needed the provider’s answer
Isolation	A lane per channel and priority	Idle reserved capacity	One channel and no broadcasts
User data	Postgres plus a Redis copy, deleted on change	An opt-out can lag by about a second	We were ten times smaller (no cache)
Delivery log	Cassandra, a column per status, 90 days	Ad hoc queries	We were far smaller (Postgres)
Broadcasts	Sharded, paced, saved-as-it-goes fan-out	A second way in	Alerts were identical for everyone (topics)
Unknown outcome	Retry critical, give up on bulk	Some double codes, some lost promos	Neither were acceptable (confirm another way)
Growth	Cells by user ID, from about 300,000 a second	One place to look	Load stayed below that

How to spend 45 minutes

0 to 4. One user versus millions, login code speed, duplicates, preferences. Name isolation as the hard part.
4 to 9. 100,000 a second at a broadcast peak, 300,000 status writes, 27 TB. Price SMS in one sentence.
9 to 20. Build it: sending inline, then 202 and the key, two stores, lanes. Trace a login code. A mid-level answer can stop here.
20 to 24. The broadcast as a job.
24 to 40. The stress tests: duplicates, the kickoff with its kill switch, a failing provider, then quiet hours or tokens, whichever the interviewer leans toward.
40 to 45. The push provider stopping, a cache failure at kickoff, the shedding order, then your own questions.

Never skip the lanes, the idempotency key, or the honest answer to “exactly once”.

Follow-up questions

Interviewer asksSMS costs are up 40% this quarter. Where do you look?

Spend per country first: a jump in a few countries with no new sign-ups is SMS pumping. Then resends per login, since a broken flow that resends codes doubles the bill. Then channel choice: users with the app could get the code by push.

Interviewer asksA user opts out and still gets a marketing push 10 seconds later. Bug?

Expected if the message was routed before the opt-out. It was already in its lane, which is why senders re-check recent opt-outs. Minutes later is a bug, most likely a lost cache delete, which the cache’s 10-minute expiry limits.

Go deeperMore follow-up questions2 min

We want to offer this to outside customers. What changes? Every key, quota and lane gains a customer ID. Each customer gets its own limits and its own share of the bulk lanes, so one customer’s blast can’t delay another’s login codes. Templates, preferences and provider accounts become per customer, and usage is billed from the status stream.

Users say we send too many notifications. What would you build? Digests: hold non-urgent messages of one kind for a while and send one summary, like “12 people liked your photo”. It reuses the quiet-hours path: the router puts the message in a per-user digest instead of a lane, and the scheduler sends the summary when the window closes.

Two messages for one user arrive in the wrong order. Can you promise order? Not across lanes, because critical traffic is meant to overtake bulk, and a retry can land after a newer message. Where order matters, like “cancelled” after “confirmed”, give both a sequence number and the same collapse ID. The newer one replaces the older on the phone, and a late older one is dropped.

How do you test this without spamming real users? Callers get a sandbox mode that runs the whole pipeline and stops at a fake provider. In production, test users with real phones get a message every 10 seconds per cell, and every campaign goes to a list of internal accounts first.

Design a notification system

The answer in one minute

Scope it

The numbers that decide it

Build the design

Step 1. Send inside the API call

Stage 1: Send inline

Why this stage

What we add

What it costs

Step 2. Accept, queue, send later

Stage 2: Accept, queue, send later

Why this stage

What we add

What it costs

Spotting a retried request

Step 3. Two kinds of data, two stores

Stage 3: Split the data by access pattern

Why this stage

What we add

What it costs

Step 4. Lanes, not priorities

Stage 4: Channels and priorities

Why this stage

What we add

What it costs

Trace a request: An OTP by SMS

Step 5. Broadcasts as batch jobs

Stage 5: Broadcast fan-out

Why this stage

What we add

What it costs

Stress it

Never lost, rarely twice

50 million users in 10 minutes

Trace a request: One broadcast recipient

A provider that fails

Quiet hours and limits

What “delivered” means

When things break

Break it: what fails, who notices, what users see

What breaks

How we notice

What users see

What we do

Stage 6: Cells and regions

Why this stage

What we add

What it costs

Wrap up

The whole design

The full design, stage by stage

Why this stage

What we add

What it costs

The decisions, and what each one costs

How to spend 45 minutes

Follow-up questions

Further reading

Keep reading

The designbeyondcode digest, every Saturday