Design a ChatGPT-style assistant

Design a ChatGPT-style chat assistant that serves a large language model to 10 million daily users, step by step. Size the GPU fleet from tokens, stream answers, keep each conversation's memory where its next message will land, limit users in tokens by tier, send easy messages to a small model, and plan for the day the GPUs run out, with code in Java, Python, C#, Go and TypeScript.

Senior35 min core · 58 min full depth

Read first: How LLM inference works Evals and guardrails
Patterns: Backpressure and load shedding Multi-region and active-active
Building blocks: How LLM inference works Evals and guardrails Choosing an approach Rate limiting Load balancing
Related: Code completion Support agent Chat (WhatsApp)

“Design ChatGPT. Millions of people type a message and get an answer from a large language model. Focus on serving the model, not on training it.”

The product looks like a text box. Behind it, every message takes several seconds of work on a GPU that costs a few dollars an hour. While the answer streams, it also holds a gigabyte of GPU memory.

A normal web service runs out of CPU or database capacity first. This one runs out of GPU time and GPU memory first. Almost every decision on this page is about spending those two well. We start with one model server, and grow the design only when a number forces us to.

The answer in one minute

If you only have a minute, this is the design. The rest of the page explains each part.

Answers stream word by word, so the first words appear in under a second instead of after 13 seconds.
Each GPU server answers about 100 people at once. It writes one word for all of them in each step. This is called batching, and it is what makes the cost work.
Each conversation goes back to the same server. That server still remembers the conversation, so it only has to read the new message, not the whole history again.
Limits count tokens, not messages. A huge prompt costs fifty times a short one, so it uses up fifty times more of the user’s limit.
Easy messages go to a small model. About 70% of messages don’t need the big model. This halves the number of GPUs.
When GPUs run out, free users give way first, in a fixed order planned in advance.

Scope it

A few questions change the design a lot. For the rest, we assume an answer and move on.

Interviewerminute 0

Design ChatGPT. Focus on serving the model.

What they are testing

Whether you ask the questions that size the GPU fleet: users and messages, prompt and answer length, whether you run the model yourself, and the tiers. About three minutes. Asking about the UI first reads badly.

These five questions matter most:

How many users? 10 million a day, 10 messages each. Below about 100,000 users a day, paying a provider’s API would be cheaper, and most of this page would go away.
Our own model, or a provider’s? Our own open model, on our own GPUs. With a provider’s API, the GPU work becomes their problem.
Which model size? A large model (70 billion parameters, “70B”) for quality, and a small one (“8B”) for easy messages.
How long are conversations? About 3,000 tokens of prompt per message on average. A token is a piece of a word, about three quarters of a word on average.
Tiers? Free and Plus. Free is 60% of messages and pays nothing.

We leave out training the model, tools and web browsing (see the agent platform), images, voice and billing.

Go deeperThe full requirements, and more questions3 min

Question	We assume	If the answer were different
Do answers stream?	Yes, token by token	Without streaming, a 400-token answer means a 13 second spinner
Regions?	Users worldwide. Start in one, grow to five	Strict data residency pins each user to one region
Paid context length?	Up to 32,000 tokens for Plus	Book-length contexts make one conversation fill a whole GPU
Train on conversations?	Only from users who opt in	Training on everything changes consent and deletion

Must do. Send a message and get a streamed answer. Save conversations and continue them on any device. Stop an answer (which frees the GPU at once) or regenerate it. Refuse harmful requests.

Speed. The first words appear within 1 second for 95% of Plus messages, and within 2 seconds for Free. Answers stream at 20 to 30 tokens a second, faster than people read.

Availability. 99.9% of messages succeed. Under overload, Plus keeps 99.9%, and Free may get a fast “busy” message instead.

Cost. Under $1.50 of GPU time per 1,000 messages at the planned peak.

Quality. A new model must win at least half of side-by-side comparisons against the current one, with no category below 45%. At least 99% of harmful requests are refused, and at most 2% of harmless ones are refused by mistake.

How much AI it needs

Interviewerminute 3

Do we really need a 70B model for every message?

What they are testing

Whether you treat the model as the product but still look for cheaper paths. “A smaller model is fine” without saying how you would know ignores quality.

Here the model is the product. So the usual cheap options, like rules, search or a fixed set of answers, can’t replace it. The real questions are which model, where it runs, and whether every message needs the biggest one.

Which model answers, and what it costs

Scroll or pinch to zoom, drag to pan

Color key
The message
Router
Model
GPU cost

We pick our own 70B model, plus a router that sends easy messages to an 8B model. At our size, this costs about $1.10 per 1,000 messages, against about $8 on a hosted API. The cost is a GPU fleet to run, and a router that sometimes sends a hard question to the small model. A quality test decides how often that may happen. Below about 100,000 users a day, the API would win.

Fine-tuning doesn’t help here. It changes a model’s tone and format. It doesn’t make an 8B model reason like a 70B one.

The numbers that decide it

Interviewerminute 6

How many GPUs does this need?

What they are testing

Whether you size from tokens, not requests, and separate reading the prompt from writing the answer. Dividing requests per second by “requests per GPU” with no tokens in sight is the red flag.

First, one idea we need for the numbers. A model does two different kinds of work for each message:

The two kinds of work in one message

Scroll or pinch to zoom, drag to pan

Color key
Text in and out
Model work
The KV cache

Prefill reads the whole prompt at once. It is heavy math, but it happens in one go. Decode writes the answer one token at a time, and each token needs a full pass through the model. While it reads the prompt, the model builds a memory of the conversation, called the KV cache. Decode uses that memory for every token it writes. (The inference page explains why.)

Assumptions

Daily active users: 10 million
Messages per user per day: 10
Peak over average: 3×
Prompt tokens per message (system prompt, history, new message): 3000
Answer tokens per message: 400
Share of the history found in the prefix cache: 90%
Messages routed to the small model: 0%

Messages1,200/s3,500/s at peak

10 million users × 10 messages ÷ 86,400 seconds is about 1,160 a second. Evenings in the biggest time zone push it to about three times that.

So we plan for 3,500 messages a second. Every one is seconds of GPU work.

Answers streaming at once46,000at the peak

Each answer is about 400 tokens at 30 tokens a second, so it streams for about 13 seconds. 3,500 new messages a second × 13 seconds is about 46,000.

So we tens of thousands of open connections, and a KV cache for each one, at the same time.

GPU time0.67 GPU-sper message on the 70B model

Writing a 400-token answer takes about 0.53 GPU-seconds. Reading the prompt takes 0.14, but only because the server already remembers the old part of the conversation. Without that memory, reading all 3,000 tokens again would take 1.05 GPU-seconds, more than writing the answer.

So we writing the answer is the main cost. Reading the prompt is cheap only if we reuse the memory.

Fleet3,000 GPUs1,500 with a router

3,500 messages × 0.67 seconds is about 2,350 busy GPUs. Add 30% spare for failures and spikes, and it is about 3,000. With 70% of messages on the 8B model, it is about 1,500.

So we send easy messages to the small model. It halves the fleet.

Cost$1.10per 1,000 messages, with the router

1,500 GPUs × $3 an hour × 24 hours, plus the team, over 100 million messages a day. On the 70B model alone, it is $2.20. A hosted API would be about $8.

So we run our own fleet at this size. Below 100,000 users a day, use a provider.

Look at the GPU time again. Remembering the conversation between messages is what keeps prefill cheap. Without it, prefill alone would need about 3,600 GPUs. So the memory isn’t an optimization. It is a requirement. Step 3 is built around it.

Go deeperThe calculator, memory per conversation, and the math behind 100 answers3 min

Change an assumption

Daily active users10 millionMessages per user per day10Peak over average3×Prompt tokens per message (system prompt, history, new message)3000Answer tokens per message400Share of the history found in the prefix cache90%Messages routed to the small model0%

Messages, average: 1,200/s
Messages, peak: 3,500/s; Every one of these is seconds of GPU work, not milliseconds of CPU.
Answers streaming at once, at the peak: 46,000; Each answer streams at 30 tokens a second and holds a connection and a KV cache until it ends.
New tokens to prefill per message: 390; The new message, plus the part of the history that was not in the cache.
GPU-seconds per message, 70B model: 0.67
GPU-seconds per message, 8B model: 0.18
GPUs busy with prefill at the peak: 470; Small next to decode, because most of each prompt is already cached.
GPUs in the fleet (peak + 30%): 3,000; More than one region’s quota of about 3,000 GPUs: route easy messages to a small model, then add regions.
Self-hosted cost per 1,000 messages, in dollars: 2.2
Hosted API cost per 1,000 messages, in dollars: 8; Assumes $3 per million input tokens, $0.30 cached and $15 output, for a large hosted model.
Self-hosting as a share of the API bill, in percent: 28; Self-hosting is cheaper at this volume, if you can run a GPU fleet.
KV cache per conversation, 70B model: 1.1 GB; A 4-GPU replica streams about 100 answers at once: about 110 GB of cache beside 140 GB of weights, in 320 GB.
KV cache of conversations between turns, at the peak: 230 TB; Users read and type for about a minute between messages. This does not fit in GPU memory: keep it in host RAM.
Conversation writes, peak: 6,900/s; Past what one relational primary sustains: use a store partitioned by user.
Conversation storage per year: 91 TB; About 2.5 KB per message with its metadata. Text is small next to the GPU bill.

Memory per conversation. For our 70B model, the KV cache takes about 320 KB per token. A 3,400 token conversation needs about 1.1 GB. Users read and type for about a minute between messages, so at the peak about 210,000 conversations are waiting for their next turn. Their memory adds up to 230 TB, far more than all the GPUs hold. It does fit in the servers’ normal memory (RAM), at about 300 GB per server.

Why one server streams about 100 answers. Decode is limited by how fast a GPU can read its memory, not by math. Each step reads all the model’s weights (140 GB, split over 4 GPUs) plus every conversation’s KV cache (100 × 1.1 GB). At 3.35 TB a second per GPU, one step takes about 30 ms. One token every 30 ms is about 33 tokens a second for each answer, just above our 30 target. Twice the answers would double the memory read, and drop everyone below reading speed.

History storage. Each message stores a question and an answer, about 2.5 KB. That is 7,000 writes a second at the peak, and 91 TB a year. Text is cheap next to GPUs.

Build the design

We start with one model server. Then each step fixes the biggest problem left. The drawing grows with each step.

Step 1. One GPU server

Interviewerminute 12

Start simple. A user sends a message. What happens?

What they are testing

Whether you get a working version in two minutes and say what is naive about it. Drawing a multi-region GPU fleet before saying what one request does reads as memorized.

An API server receives the message. It sends the conversation to a model server, which runs the 70B model on four GPUs. It waits for the whole answer, then returns it.

Chat assistant · architecture

Stage 1: One GPU server

Color keyServiceDatabaseCacheBackground workerModelNew in this stageAsync or occasional

Messages per second: 3,500

Stage 1 of 6

One GPU server

Why this stage

Nothing yet.

This is the smallest design that answers a message.

What we add

An API server.

It sends the conversation to one model server (a 70B model on 4 GPUs) and returns the answer when it is complete.

What it costs

Users wait about 13 seconds to see anything.

History lives only in the app.

One server handles about 2 messages a second.

At 4k messages per second, Large model pool is 175000% busy. Stage 2 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one message costs

A large-model message costs about 0.67 GPU-seconds: 0.53 to write the answer and 0.14 to read the prompt, with a warm prefix cache.
A small-model message costs about 0.18 GPU-seconds.

What one region can do

One region can get about 3,000 GPUs.
As one large pool, that serves about 4,500 messages a second.
Split into 2,000 large and 1,000 small GPUs, it serves about 3,000 large and 5,700 small messages a second.

What one machine can do

One replica with no batching and no caching serves only about 2 messages a second.
A gateway instance holds about 10,000 open streams. At 13 seconds a stream, that is about 750 new messages a second.

This works, and it is far too simple. The user waits 13 seconds before seeing any text. History lives only in the app. And one server handles only about 2 messages a second.

Step 2. Stream the answer, and keep the history

Interviewerminute 14

Users say it feels broken: nothing happens for ten seconds. And they want their chats on their laptop too.

What they are testing

Whether you stream tokens, store history on the server, and manage the prompt size on purpose. “Send the whole history every time” without a word about the limit is the red flag.

Three changes:

A chat gateway streams the answer. It holds each user’s connection open and passes on each token as the model writes it. The first words now arrive in well under a second. This uses server-sent events: one HTTP response that stays open and sends small updates.
A chat service stores every message in a conversation store, so any device can continue the chat.
The chat service decides what goes into the prompt. A conversation keeps growing, but a model can only read so many tokens at once. This limit is called the context window.

The third change matters more than it looks. In our local-model lab, a 12-message conversation of about 5,000 tokens was sent to a model with a 4,096-token window. The server quietly cut off the oldest part, and the model forgot a code word from the first message. Nothing showed an error.

So the chat service builds each prompt on purpose: the instructions first, then a short summary of old messages, then as many recent messages as fit, with room left for the answer. A background worker folds older messages into the summary.

Chat assistant · architecture

Stage 2: Streaming and conversations

Color keyServiceDatabaseCacheBackground workerModelNew in this stageAsync or occasional

Stage

Messages per second: 3,500

Stage 2 of 6

Streaming and conversations

Why this stage

An answer takes about 13 seconds to write, but users need the first words in under a second.

Their history must follow them to every device.

What we add

A chat gateway that streams each token as it is written.

A chat service that builds each prompt to fit the model.

A conversation store, and a worker that summarizes old messages.

What it costs

Many long open connections to manage.

A summary can lose a detail the user cared about.

At 4k messages per second, Large model pool is 175000% busy. Stage 3 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one message costs

A large-model message costs about 0.67 GPU-seconds: 0.53 to write the answer and 0.14 to read the prompt, with a warm prefix cache.
A small-model message costs about 0.18 GPU-seconds.

What one region can do

One region can get about 3,000 GPUs.
As one large pool, that serves about 4,500 messages a second.
Split into 2,000 large and 1,000 small GPUs, it serves about 3,000 large and 5,700 small messages a second.

What one machine can do

One replica with no batching and no caching serves only about 2 messages a second.
A gateway instance holds about 10,000 open streams. At 13 seconds a stream, that is about 750 new messages a second.

One message, streamed

Scroll or pinch to zoom, drag to pan

Color key
Service
Entity

Go deeperThe context builder code, and the API3 min

ContextBuilder.javacontext_builder.pyContextBuilder.cscontext_builder.gocontext-builder.ts

// tokens is counted once, when the turn is saved, so building a prompt never re-tokenizes.
record Turn(String role, String text, int tokens) {}

// dropped: older turns left out; the summarizer folds them into the summary later.
record Prompt(List<Turn> turns, int dropped) {}

// Fits a conversation into the model's window on purpose. If we sent too much, the model
// server would cut the oldest tokens silently, system prompt first, and the answer would
// simply forget things.
final class ContextBuilder {
    private final Turn system;
    private final int window;     // the tier's budget: 8,000 tokens free, 32,000 paid
    private final int maxAnswer;  // reserved up front, or the answer is cut off mid-sentence

    ContextBuilder(Turn system, int window, int maxAnswer) {
        this.system = system;
        this.window = window;
        this.maxAnswer = maxAnswer;
    }

    Prompt build(Optional<Turn> summary, List<Turn> history, Turn next) {
        // The gateway already rejected messages too long for the tier, so the fixed part fits.
        List<Turn> fixed = new ArrayList<>(List.of(system));
        summary.ifPresent(fixed::add);
        int budget = window - maxAnswer - next.tokens()
                - fixed.stream().mapToInt(Turn::tokens).sum();
        Deque<Turn> kept = new ArrayDeque<>();
        // Newest first: the user is asking about what was just said, not about turn one.
        for (int i = history.size() - 1; i >= 0; i--) {
            Turn turn = history.get(i);
            if (turn.tokens() > budget) break;  // skipping it to fit an older turn leaves a hole
            kept.addFirst(turn);
            budget -= turn.tokens();
        }
        // Stable parts first. The system prompt and the summary change rarely, so the replica's
        // prefix cache already holds them, and only the tail of the prompt needs a prefill.
        List<Turn> turns = new ArrayList<>(fixed);
        turns.addAll(kept);
        turns.add(next);
        return new Prompt(turns, history.size() - kept.size());
    }
}

The summary changes the start of the prompt. As we’ll see in step 3, that empties the server’s memory of the conversation. So the summary is only refreshed every few thousand tokens, not every message.

The API. One endpoint does almost all the work.

POST /v1/conversations/c_8f2e/messages
Authorization: Bearer <session>
Idempotency-Key: m_01J9X4          # the app makes the message ID, so a retry never runs twice
Accept: text/event-stream

{ "content": "Why is decode memory-bound?", "parentId": "m_01J9X1", "model": "auto" }

200 OK, then events:
id: 1   event: meta    data: {"messageId":"m_01J9X4","model":"70b"}
id: 2   event: delta   data: {"text":"Each decode step reads"}
id: 57  event: done    data: {"finish":"stop","usage":{"input":3010,"cached":2610,"output":412}}

GET  /v1/messages/m_01J9X4/stream     Last-Event-ID: 41   → replays from event 42, then live
POST /v1/messages/m_01J9X4/stop       → frees the GPU slot and the memory at once

The answer keeps being written even if the connection drops. If a phone switches networks, the app reconnects with the last event ID it saw, and the gateway replays the rest. Stop matters for cost. An answer nobody is reading still holds a slot on the GPU and a gigabyte of memory.

The data model. Messages live in a wide-column store, like Cassandra or DynamoDB, split by conversation. The busy read is “the summary plus the newest messages of one conversation”, 3,500 times a second. Each message stores its token count, so building a prompt never has to count tokens again.

conversations (user_id, updated_at DESC, conversation_id)
  title, model, summary, summary_upto_seq, total_tokens

messages (conversation_id, seq ASC)
  parent_seq, role, content, tokens, model, finish, created_at

Step 3. A GPU fleet, and a smart scheduler

Interviewerminute 17

We’re at 3,500 messages a second. How do you serve that?

What they are testing

Whether you know what keeps a GPU busy (batching), and that the load balancer in front of a model fleet is not round-robin. “Put the servers behind a load balancer” with no word about the KV cache is the red flag.

One server answering one message at a time does 2 messages a second. The peak needs about 1,750 times that. Two ideas close the gap.

Batching inside each server. Each decode step reads the model’s weights from memory, and that read is the slow part. So the server writes one token for about 100 answers in the same step, for almost the same cost as one. When an answer finishes, a waiting one takes its place on the next step. This is called continuous batching.

A scheduler in front of the servers. It sends each conversation back to the server that answered its last message. That server still has the conversation’s KV cache, first in GPU memory and later in normal RAM. So it only reads about 390 new tokens instead of all 3,000. Remember the numbers: that saving is the difference between about 470 GPUs of prefill and about 3,600.

Where a conversation's memory waits between messages

Scroll or pinch to zoom, drag to pan

Color key
A step
In GPU memory
In normal RAM

Chat assistant · architecture

Stage 3: A GPU fleet with a scheduler

Color keyServiceDatabaseCacheBackground workerModelNew in this stageAsync or occasional

Stage

Messages per second: 3,500

Stage 3 of 6

A GPU fleet with a scheduler

Why this stage

The peak is 3,500 messages a second, and one server handles about 2.

Reading every prompt from scratch would need about 3,600 GPUs for that alone.

What we add

Many model servers, each answering about 100 people at once.

A scheduler that sends each conversation back to the server that still remembers it.

What it costs

About 3,000 GPUs, all one region can supply.

The scheduler must balance reusing memory against long queues.

At 4k messages per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one message costs

A large-model message costs about 0.67 GPU-seconds: 0.53 to write the answer and 0.14 to read the prompt, with a warm prefix cache.
A small-model message costs about 0.18 GPU-seconds.

What one region can do

One region can get about 3,000 GPUs.
As one large pool, that serves about 4,500 messages a second.
Split into 2,000 large and 1,000 small GPUs, it serves about 3,000 large and 5,700 small messages a second.

What one machine can do

One replica with no batching and no caching serves only about 2 messages a second.
A gateway instance holds about 10,000 open streams. At 13 seconds a stream, that is about 750 new messages a second.

The scheduler’s choice has one twist. Always going back to the same server could pile work onto it. Always picking the least busy server would throw the memory away. So the scheduler goes elsewhere only when the wait at the usual server would cost more than reading the whole prompt again.

ReplicaPicker.javareplica.pyReplicaPicker.csreplica.goreplica.ts

// queuedTokens: prompt tokens waiting for prefill on this replica.
// freeKvTokens: KV cache room left. Both are reported every few hundred milliseconds.
record Replica(String id, int queuedTokens, int freeKvTokens) {}

// Sends each conversation back to the replica that holds its KV cache, unless waiting there
// would cost more than starting cold somewhere else.
final class ReplicaPicker {
    // FNV-1a, 32 bits: fast, and stable across processes and languages.
    private static int score(String key) {
        int h = 0x811C9DC5;
        for (byte b : key.getBytes(StandardCharsets.UTF_8)) {
            h = (h ^ (b & 0xFF)) * 0x01000193;  // int overflow wraps, which is what FNV wants
        }
        return h;
    }

    Replica pick(String conversationId, int promptTokens, List<Replica> replicas) {
        // Rendezvous hashing: every conversation has a stable favourite replica. When a replica
        // joins or leaves, only the conversations that favoured it move, so the rest keep
        // their cache. Round-robin would throw the prefix cache away on almost every turn.
        Replica favourite = Collections.max(replicas, Comparator.comparingLong(
                (Replica r) -> Integer.toUnsignedLong(score(conversationId + "/" + r.id()))));
        Replica idlest = Collections.min(replicas, Comparator.comparingInt(Replica::queuedTokens));
        // Elsewhere, the whole prompt is prefilled again. Pay that only when the extra queue on
        // the favourite is more work than the prefill it saves.
        int extraWait = favourite.queuedTokens() - idlest.queuedTokens();
        if (favourite.freeKvTokens() >= promptTokens && extraWait <= promptTokens) {
            return favourite;
        }
        return idlest;
    }
}

Our local lab measured how big this saving is. The same 2,744-token prompt took 194.5 seconds to the first token the first time, and 0.2 seconds the second time, because the memory was reused. A real GPU is hundreds of times faster than a laptop, but the ratio is the point.

Step 4. Limits and safety by tier

Interviewerminute 21

A free user runs a script that sends the longest prompts it can, all day. Another asks how to make a weapon. What stops them?

What they are testing

Whether limits count tokens, not requests, and differ by tier, and whether safety checks sit outside the model. “100 requests a minute per user” is the red flag.

A limit of “100 messages an hour” doesn’t work here. One 30,000-token prompt costs the GPUs as much as fifty short ones. So the limit must count what a message costs, in tokens.

Each user gets a token bucket: a budget that slowly refills over time. It is kept in Redis, so every gateway sees the same one. A message spends tokens from it, weighted by GPU cost:

New prompt tokens cost 1.
Prompt tokens the server already remembers cost 0.1.
Answer tokens cost 4, because each one is a full pass through the model.

A typical message costs about 2,250. Free users get 25,000, refilled every three hours, which is about 11 typical messages. Plus gets ten times that.

There is one catch. We don’t know an answer’s length until it is done. So before the message runs, the gateway reserves the worst case. After the answer, it charges the real cost and gives the rest back. Without the reservation, a script could send 30 messages at once, and they would all pass before any of them was charged.

Safety checks sit on both sides of the model. A small classifier reads each message before it is queued. Another reads the answer sentence by sentence as it streams, before each sentence is sent. They are small models on their own GPUs, so they add milliseconds, not seconds.

Chat assistant · architecture

Stage 4: Limits and safety by tier

Color keyServiceDatabaseCacheBackground workerModelNew in this stageAsync or occasional

Stage

Messages per second: 3,500

Stage 4 of 6

Limits and safety by tier

Why this stage

Free users send 60% of messages and pay nothing.

One script with huge prompts can fill a whole server.

Harmful requests must be caught.

What we add

A token budget per user in Redis, sized by tier and counted in tokens, not messages.

Small safety checks on the message and on the answer as it streams.

What it costs

About 30 ms more before the first token.

Refusals must be watched as closely as errors.

At 4k messages per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one message costs

A large-model message costs about 0.67 GPU-seconds: 0.53 to write the answer and 0.14 to read the prompt, with a warm prefix cache.
A small-model message costs about 0.18 GPU-seconds.

What one region can do

One region can get about 3,000 GPUs.
As one large pool, that serves about 4,500 messages a second.
Split into 2,000 large and 1,000 small GPUs, it serves about 3,000 large and 5,700 small messages a second.

What one machine can do

One replica with no batching and no caching serves only about 2 messages a second.
A gateway instance holds about 10,000 open streams. At 13 seconds a stream, that is about 750 new messages a second.

TokenLimiter.javatoken_limiter.pyTokenLimiter.cstoken_limiter.gotoken-limiter.ts

enum Tier { FREE, PLUS }

// capacity: the most a user can spend in one burst.
record Limit(double capacity, double refillPerSecond) {}

record Reservation(boolean ok, double amount, double retryAfterSeconds) {}

// One token bucket per user. In production this is a Lua script in Redis, so every gateway
// sees the same bucket in one atomic step; the arithmetic is the same.
final class TokenLimiter {
    // Both refill completely in three hours. A typical message costs about 2,250 units,
    // so free is about 11 messages per three hours, and Plus ten times that.
    static final Map<Tier, Limit> LIMITS = Map.of(
            Tier.FREE, new Limit(25_000, 25_000 / 10_800.0),
            Tier.PLUS, new Limit(250_000, 250_000 / 10_800.0));

    // Weighted by GPU time, not by count: an output token is a whole decode pass, and an input
    // token found in the prefix cache costs almost nothing.
    static double cost(int uncachedIn, int cachedIn, int out) {
        return uncachedIn + 0.1 * cachedIn + 4.0 * out;
    }

    private record Bucket(double level, double at) {}

    private final DoubleSupplier clock;  // seconds; injected so tests control time
    private final Map<String, Bucket> buckets = new HashMap<>();

    TokenLimiter(DoubleSupplier clock) {
        this.clock = clock;
    }

    private double level(String user, Limit limit) {
        double now = clock.getAsDouble();
        Bucket b = buckets.getOrDefault(user, new Bucket(limit.capacity(), now));
        return Math.min(limit.capacity(), b.level() + (now - b.at()) * limit.refillPerSecond());
    }

    synchronized Reservation reserve(String user, Tier tier, int promptTokens, int maxAnswer) {
        Limit limit = LIMITS.get(tier);
        // Before the call we know neither the cache hit nor the answer length, so take the worst
        // case. Taking the average would let a burst of 30 parallel requests all pass on a
        // bucket that can only afford a few of them.
        double worst = cost(promptTokens, 0, maxAnswer);
        double level = level(user, limit);
        if (level < worst) {
            buckets.put(user, new Bucket(level, clock.getAsDouble()));
            return new Reservation(false, 0, (worst - level) / limit.refillPerSecond());
        }
        buckets.put(user, new Bucket(level - worst, clock.getAsDouble()));
        return new Reservation(true, worst, 0);
    }

    // Give back what the message did not use. A short answer from cache should not cost a
    // free user their next message.
    synchronized void settle(String user, Tier tier, double reserved, double actual) {
        Limit limit = LIMITS.get(tier);
        double level = Math.min(limit.capacity(), level(user, limit) + reserved - actual);
        buckets.put(user, new Bucket(level, clock.getAsDouble()));
    }
}

This is enough for a mid-level answer. Streaming, history with a prompt budget, a fleet with batching and a scheduler, and token limits by tier. The rest of the page is what makes it a senior answer: quality, ten times the users, and the day the GPUs run out.

Go deeperFollow a message, the reservation step by step, and the classes4 min

Follow one message through the design. Use Next hop to move it along. The bar is the budget for the first token.

Chat assistant · architecture

Trace a request: Send a message

Color keyServiceDatabaseCacheBackground workerModelNew in this stageAsync or occasionalHop in motionHops doneResponse

Send a message: step 1 of 11

Users to Chat gateway

The app POSTs the message and the gateway answers with a server-sent event stream. TLS ends here; the session token says who the user is and which tier they pay for.

20 ms of a 1000 ms budget

Users → Chat gateway20 ms
Chat gateway → Token buckets2 ms
Chat gateway → Safety classifiers30 ms
Chat gateway → Chat service1 ms
Chat service → Conversation store8 ms
Chat service → Inference scheduler2 ms
Inference scheduler → Large model pool50 ms
Large model pool → Prefix cache35 ms
Chat service → Conversation storeasync
Chat gateway → Token bucketsasync
Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Reserve, then settle

Scroll or pinch to zoom, drag to pan

Color key
Service
Cache

Our ship-it lab measured the burst case. One key with 6,000 tokens a minute sent 30 requests of about 900 tokens at once. 6 were allowed and 24 refused, and the refused caller was told to retry after about 3 seconds.

The chat service's classes after step 4

Scroll or pinch to zoom, drag to pan

Color key
Facade
Entity
Value object
Enum
Interface
Implementation
New in this step

The model sits behind InferenceClient, and the choice of model behind ModelPicker. Step 5 replaces the fixed model with a router, without touching ChatService.

Prove it works

Interviewerminute 25

The research team has a new model version. How do you decide whether to ship it?

What they are testing

Whether there is a gate with numbers before users see it, and a check after. “We’d try it on some prompts” is the red flag. So is one overall score that hides a broken category.

A chat assistant has no single right answer. So we measure quality three ways.

Before release, on fixed test sets. 2,000 real prompts from users who opted in, in 12 categories. A judge model, a second model asked to grade, compares each new answer with the current model’s answer. The new model must win at least half the time, with no category below 45%. We swap the order of the two answers and average, because judges tend to prefer whichever answer they see first. Separate sets check that harmful requests are refused, and harmless ones are not.

During release, step by step. The new model first runs in shadow mode: it answers real messages, the answers are judged, but users never see them. Then it goes to 1%, 10% and 50% of conversations. We watch thumbs down and how often people regenerate an answer. A conversation stays on the model it started with, so the tone doesn’t change mid-chat.

The router has its own test. Every day, 1% of small-model messages are also answered by the big model, and a judge compares them. The small model may lose at most 5% of the time.

Our ship-it lab shows why the gate matters. A model one sixth the size answered 15 of 20 questions right, against 19 of 20. Every wrong answer sounded confident, and the gate failed the build.

Go deeperAll the release metrics2 min

Metric	How we measure it	Target	Blocks a release?
Wins against the live model	2,000 prompts, a judge, order swapped	At least 50%. No category below 45%	Yes
Judge agrees with people	300 pairs graded by people each quarter	At least 85%	The judge isn’t trusted below it
Harmful requests refused	1,000 attack prompts	At least 99%	Yes
Harmless requests refused	1,000 edgy but harmless prompts	At most 2%	Yes
Factual accuracy	500 questions with known answers	No drop over 1 point	Yes
Router loss rate	Small-model answers judged against the big model’s	Small loses at most 5%	Yes, for router changes
Serving speed	Load test with real prompt lengths at the planned peak	First token under 1 s, 25+ tokens/s	Yes

Stress it

The design works at today’s size. Now we push it through three hard situations: ten times the users, one huge prompt, and the day the GPUs run out.

Ten times the users

Interviewerminute 29

We grow ten times, to 100 million daily users. What breaks first?

What they are testing

Whether you go back to the numbers. 30,000 GPUs on the large model alone is not a plan. Name the cost lever (smaller models) and the supply limit (GPUs per region) before drawing regions.

At ten times the users, the 70B model alone would need about 30,000 GPUs. One cloud region can only supply so many. We assume about 3,000. Two steps fix it, cheapest first.

Step 5: send easy messages to a small model. Most messages don’t need a 70B model: greetings, rewording an email, a short fact. A router is a small classifier that guesses whether the 8B model’s answer would be judged worse than the 70B’s. If not, the message goes to the 8B model. We set the line on the test set, at the point where the small model loses at most 5% of the time. That sends about 70% of messages to the small model, and halves the fleet.

Chat assistant · architecture

Stage 5: Send easy messages to a small model

Color keyServiceDatabaseCacheBackground workerModelNew in this stageAsync or occasional

Stage

Messages per second: 3,500

Stage 5 of 6

Send easy messages to a small model

Why this stage

With only the 70B model, it costs $2.20 per 1,000 messages.

At ten times the users it would need about 30,000 GPUs.

What we add

A router that sends about 70% of messages to an 8B model, and a pool of small-model servers.

What it costs

A second model to test and release.

A slightly worse answer when the router guesses wrong.

At 4k messages per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one message costs

A large-model message costs about 0.67 GPU-seconds: 0.53 to write the answer and 0.14 to read the prompt, with a warm prefix cache.
A small-model message costs about 0.18 GPU-seconds.

What one region can do

One region can get about 3,000 GPUs.
As one large pool, that serves about 4,500 messages a second.
Split into 2,000 large and 1,000 small GPUs, it serves about 3,000 large and 5,700 small messages a second.

What one machine can do

One replica with no batching and no caching serves only about 2 messages a second.
A gateway instance holds about 10,000 open streams. At 13 seconds a stream, that is about 750 new messages a second.

Step 6: five regions. Even with the router, ten times the users needs about 15,000 GPUs, or five regions. Each user has a home region, where their history lives. When a region is full, new conversations spill over to a region with room. A conversation that moves pays the full prefill once, because its memory stayed behind.

Chat assistant · architecture

Stage 6: Regions and a shedding ladder

Color keyServiceDatabaseCacheBackground workerModelNew in this stageAsync or occasional

Stage

Messages per second: 3,500

Stage 6 of 6

Regions and a shedding ladder

Why this stage

At ten times the users, we need about 15,000 GPUs, five regions.

Losing one region must not take the product down.

What we add

The whole design in five regions.

Users go to the nearest one, and new conversations spill to a region with room.

A planned order of what to give up, protecting paid users first.

What it costs

A conversation that moves region must read its whole prompt again.

Free users sometimes get a smaller model or a busy message.

At 4k messages per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one message costs

A large-model message costs about 0.67 GPU-seconds: 0.53 to write the answer and 0.14 to read the prompt, with a warm prefix cache.
A small-model message costs about 0.18 GPU-seconds.

What one region can do

One region can get about 3,000 GPUs.
As one large pool, that serves about 4,500 messages a second.
Split into 2,000 large and 1,000 small GPUs, it serves about 3,000 large and 5,700 small messages a second.

What one machine can do

One replica with no batching and no caching serves only about 2 messages a second.
A gateway instance holds about 10,000 open streams. At 13 seconds a stream, that is about 750 new messages a second.

Go deeperThe router code, and other ways to save GPUs2 min

DifficultyRouter.javadifficulty_router.pyDifficultyRouter.csdifficulty_router.godifficulty-router.ts

enum Model { SMALL, LARGE }

// Picks the model for one message. The score comes from a small classifier: the chance that
// the 8B model's answer would be judged worse than the 70B model's for this message.
final class DifficultyRouter {
    // Tuned offline on the eval set, not by feel: the highest threshold at which messages sent
    // to the small model lose to the large model at most 5% of the time.
    private final double threshold;

    DifficultyRouter(double threshold) {
        this.threshold = threshold;
    }

    Model pick(OptionalDouble score, Tier tier, Optional<Model> chosen, int tokens, boolean hasCode) {
        if (chosen.isPresent() && tier == Tier.PLUS) {
            return chosen.get();  // a paid user who picked a model gets it; that is what they pay for
        }
        if (score.isEmpty()) {
            // The classifier is down or slow. A crude rule keeps the split near 70/30 instead of
            // sending everything to the large model, which would need twice the GPUs at peak.
            return tokens < 200 && !hasCode ? Model.SMALL : Model.LARGE;
        }
        return score.getAsDouble() < threshold ? Model.SMALL : Model.LARGE;
    }
}

Lever	Saving	Cost
Router to an 8B model	Fleet from about 3,000 to 1,500 GPUs	5% of routed answers a bit worse
8-bit weights	Half the memory, nearly twice the decode	A quality check per model
Speculative decoding	2 to 3 times faster decode on easy text	A small draft model on every server
Night work on idle GPUs	Tests and summaries for free at night	A scheduler that can pause that work

Speculative decoding means a small draft model guesses a few tokens ahead, and the big model checks them all in one pass. When the guesses are right, decode is much faster, with the same output.

If the router’s classifier is down, a simple rule takes over: short messages go to the small model.

One huge prompt

Interviewerminute 33

A user pastes a 30,000-token document. What happens to the other 99 people streaming on that server?

What they are testing

Whether you know the step loop: who is in the batch, how a long prompt is split, and what waits when memory runs out. “The GPU handles it” is the red flag.

Each server runs a loop, about every 30 ms. Pick the work for one pass, run it, send the new tokens, repeat. If one pass had to read the whole 30,000-token document, every other answer on that server would freeze for seconds.

So each pass has a token budget, about 512 tokens. First, every running answer gets its next token. Then waiting requests are let in, paid users first, while memory lasts. Whatever budget is left goes to reading the long prompt, a piece at a time. This is called chunked prefill.

The result: the 30,000-token document is read in about 70 small pieces. The other 99 answers slow from 30 to about 20 tokens a second while that happens, but they never stop. The document’s own first token comes after about 3.5 seconds. That is the price of not freezing everyone else.

Go deeperThe step loop code, and other ways to schedule3 min

StepScheduler.javastep_scheduler.pyStepScheduler.csstep_scheduler.gostep-scheduler.ts

// priority: 0 for paid, 1 for free; lower goes first.
final class Seq {
    final String id;
    final int priority;
    final double arrived;
    int promptLeft;  // prompt tokens not yet prefilled
    int blocks;      // KV cache blocks this sequence holds

    Seq(String id, int priority, double arrived, int promptTokens) {
        this.id = id;
        this.priority = priority;
        this.arrived = arrived;
        this.promptLeft = promptTokens;
    }
}

record Work(Seq seq, int tokens) {}

// Chooses the work for every forward pass of one replica, about every 30 ms. This is
// continuous batching: who is in the batch is decided each step, so a finished answer leaves
// at once and a waiting request joins on the next step.
final class StepScheduler {
    static final int BLOCK = 16;  // tokens per KV cache block: memory is handed out in pages
    private final int stepTokens;
    private int freeBlocks;
    private final List<Seq> running = new ArrayList<>();
    private final PriorityQueue<Seq> waiting = new PriorityQueue<>(
            Comparator.comparingInt((Seq s) -> s.priority).thenComparingDouble(s -> s.arrived));

    // The token budget per step caps how long one step takes: 512 prompt tokens
    // add about 45 ms on four GPUs. A 30,000-token prompt is fed in chunks across many
    // steps, so it never freezes everyone else's stream (chunked prefill).
    StepScheduler(int freeBlocks, int stepTokens) {
        this.freeBlocks = freeBlocks;
        this.stepTokens = stepTokens;
    }

    void submit(Seq s) {
        waiting.add(s);
    }

    List<Work> nextStep() {
        int budget = stepTokens;
        List<Work> work = new ArrayList<>();
        // 1. Everyone mid-answer gets their next token first. Those are the streams people are
        //    reading; starving them to admit newcomers is the stutter users notice.
        for (Seq s : running) {
            if (s.promptLeft == 0) {
                work.add(new Work(s, 1));
                budget--;
            }
        }
        // 2. Admit waiting requests, paid before free, while memory lasts. If the head of the
        //    queue does not fit, stop: letting smaller free requests jump past it would starve
        //    a paid request with a long prompt.
        while (!waiting.isEmpty() && budget > 0) {
            Seq s = waiting.peek();
            int need = (s.promptLeft + BLOCK - 1) / BLOCK;
            if (need > freeBlocks) break;  // waiting beats evicting someone halfway through an answer
            waiting.poll();
            freeBlocks -= need;
            s.blocks += need;
            running.add(s);
        }
        // 3. Spend what is left of the budget on prefill, oldest admission first.
        for (Seq s : running) {
            if (s.promptLeft > 0 && budget > 0) {
                int chunk = Math.min(s.promptLeft, budget);
                work.add(new Work(s, chunk));
                s.promptLeft -= chunk;
                budget -= chunk;
            }
        }
        return work;
    }

    // The answer ended: its blocks go back at once, and the next step can admit someone.
    // (A real server keeps them as prefix cache until it needs the room.)
    void finish(Seq s) {
        running.remove(s);
        freeBlocks += s.blocks;
    }
}

Option	How it works	Wins	Loses
Fixed batches	Collect a batch, run it until every answer ends	Simple	Short answers wait for long ones
Continuous batching	Pick the batch again every step	Full batches	A long prompt freezes every answer for that step
Continuous + chunked prefill	A token budget per step. Long prompts in pieces	Steady streams	A long prompt’s own first token comes later
Separate prefill and decode GPUs	Read prompts on one pool, write answers on another	Each pool sized for its job	Moving a gigabyte of memory per request between machines

The ideas come from three papers in further reading: Orca (continuous batching), vLLM’s PagedAttention (the memory layout that makes “while memory lasts” cheap to check), and Sarathi-Serve (chunked prefill).

Long conversations get two more limits. Each tier has a prompt budget (8,000 tokens for Free, 32,000 for Plus), and old messages become a summary. A bigger window is never free. A 128,000-token conversation holds about 41 GB of memory, a quarter of a server’s spare room. Often, retrieval can put the 5,000 tokens that matter into a normal prompt instead.

The day the GPUs run out

Interviewerminute 37

It’s launch day. Traffic is twice the plan and you can’t get more GPUs this week. What do users see?

What they are testing

Whether you have a planned order of what to give up, that protects paying users and fails fast instead of slowly. “Autoscale” is the red flag: there is nothing to scale onto.

When demand passes capacity, a queue grows until requests time out. Then every user waits 10 seconds, many get an error, and GPU time is wasted on answers whose users already left.

The fix is to decide in advance what to give up, in order. We call it a shedding ladder:

Paid first. The queue is ordered by tier, so paid messages never wait behind free ones.
Free messages go to the small model, even hard ones. That uses about a quarter of the GPU time.
Shorter free answers, capped at 300 tokens.
Refuse at the door. A free message that can’t start within 2 seconds gets an instant “busy, try again soon”, instead of a 10 second wait that ends in an error.

Try it. One region has 1,500 GPUs, planned for 3,500 messages a second. Push the traffic to twice that with every rung off. Then turn the rungs on one at a time.

Traffic: messages a secondLose a third of the GPUs (a zone outage or a bad deploy)

The shedding ladder1. Paid messages go first in the queue2. Free messages go to the small model3. Free answers capped at 300 tokens4. Refuse free messages that cannot start within 2 s

GPU work asked for, as a share of the fleet

With every rung off, 34% of all messages time out, paid as often as free. Paid-first alone saves the paid tier, but most free messages still fail slowly. Rungs 2 and 3 bring demand back under capacity. Rung 4 refuses about 15% of free messages at once, so everyone else starts within 2 seconds.

The ladder moves one rung every 10 seconds, based on how busy the big-model GPUs are. It climbs at 85%, 92% and 97% busy. It comes down a rung only after load drops 7 points below where it went up, so it doesn’t flip on and off while load hovers near a line.

Ladder.javaladder.pyLadder.csladder.goladder.ts

// model is empty when the message is refused with a "busy, try again soon" message.
record Decision(Optional<Model> model, int maxAnswer) {}

// What the large pool gives up as it fills, one rung at a time, free tier first.
// 0: normal. 1: free messages go to the small model. 2: free answers are capped at
// 300 tokens. 3: free messages that cannot start within 2 seconds are refused.
final class Ladder {
    private static final double[] UP = {0.85, 0.92, 0.97};  // busy share that moves us to rung 1, 2, 3
    private static final double GAP = 0.07;  // come down a rung only well below the line we went up at
    private int rung = 0;

    int rung() {
        return rung;
    }

    // Called every 10 seconds with the pool's busy share. One rung per call, so each step has
    // time to take effect before the next. The gap stops the ladder flipping up and down every
    // few seconds while the load hovers near a line.
    synchronized void observe(double busy) {
        if (rung < 3 && busy >= UP[rung]) rung++;
        else if (rung > 0 && busy < UP[rung - 1] - GAP) rung--;
    }

    synchronized Decision decide(Tier tier, Model routed, double expectedWaitSeconds) {
        if (tier == Tier.PLUS) {
            return new Decision(Optional.of(routed), 2000);  // protected; it also goes first in the queue
        }
        if (rung >= 3 && expectedWaitSeconds > 2.0) {
            // Refuse at the door. A request that would wait 10 seconds and then time out has
            // used GPU time and the user's patience for nothing. Refusing only the ones that
            // cannot start in time sheds just the excess, because the wait falls as we refuse.
            return new Decision(Optional.empty(), 0);
        }
        Model model = rung >= 1 ? Model.SMALL : routed;
        return new Decision(Optional.of(model), rung >= 2 ? 300 : 1000);
    }
}

When things break

Interviewerminute 42

A GPU dies halfway through someone’s answer. What do they see?

What they are testing

Whether the answer can continue somewhere else, and whether you cover AI failures (made-up answers, a worse model) as well as boxes going down.

The answer doesn’t depend on the connection, or on one server. The gateway keeps every token it has sent. If a server dies, the scheduler sends each of its answers to another server, with the tokens already sent as part of the prompt. The new server reads them and carries on. The user sees a pause of about 2.5 seconds, not an error.

A server dies mid-answer

Scroll or pinch to zoom, drag to pan

Color key
Service
Background worker

Click any box to break it and see what users would notice:

Chat assistant · architecture

Break it: what fails, who notices, what users see

Color keyServiceDatabaseCacheBackground workerModelNew in this stageAsync or occasional

Break

Failure: click any box to break it

A large-model replica goes down

What breaks

A GPU fails or the server crashes.

About 100 answers stop mid-stream, and their KV cache is gone.

How we notice

The scheduler misses two heartbeats (about 2 seconds), and token output from that replica stops.

What users see

A pause of two or three seconds in the middle of the answer, then it continues.

No one has to resend.

What we do

The scheduler re-queues each interrupted request on another replica with the tokens already sent appended to the prompt, so the answer picks up where it stopped.

A replacement loads weights on a warm spare in minutes.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Go deeperEvery failure, and what users see2 min

Part or failure	What happens	What we do
A model server	About 100 answers stop mid-stream	Continue elsewhere with the sent tokens. Warm spares
All memory lost at once	Prefill jumps from about 180 to 1,400 GPUs	Roll out changes slowly. The ladder absorbs the spike
A gateway	10,000 streams cut	Apps reconnect with the last event ID. The answers never stopped
The conversation store	Some users can’t send for about 30 s	Switch to a copy
A whole region	Its users move. Free may be shed	DNS sends users to another region
Made-up answers	Confident, wrong facts	Test factual accuracy. Use RAG for facts
A worse model	Quality drops in one category	Gradual rollout and rollback. Conversations stay on one version
A runaway answer	It repeats until the window is full	A maximum answer length per tier. Stop on repetition

One word changed in the system prompt empties the memory of every conversation, because every prompt starts with it. Prefill jumps toward its full cost for a while. So prompt changes are rolled out slowly, like model changes, and off peak.

Wrap up

The whole design

Here is everything we built, in the order we built it. Use the buttons to step through it.

Chat assistant · architecture

The full design, stage by stage

Color keyServiceDatabaseCacheBackground workerModelNew in this stageAsync or occasional

Stage

Messages per second: 3,500

Stage 6 of 6

Regions and a shedding ladder

Why this stage

At ten times the users, we need about 15,000 GPUs, five regions.

Losing one region must not take the product down.

What we add

The whole design in five regions.

Users go to the nearest one, and new conversations spill to a region with room.

A planned order of what to give up, protecting paid users first.

What it costs

A conversation that moves region must read its whole prompt again.

Free users sometimes get a smaller model or a busy message.

At 4k messages per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.The whole design. Step back through the stages to rebuild it from memory.

The numbers behind the busy bars

What one message costs

A large-model message costs about 0.67 GPU-seconds: 0.53 to write the answer and 0.14 to read the prompt, with a warm prefix cache.
A small-model message costs about 0.18 GPU-seconds.

What one region can do

One region can get about 3,000 GPUs.
As one large pool, that serves about 4,500 messages a second.
Split into 2,000 large and 1,000 small GPUs, it serves about 3,000 large and 5,700 small messages a second.

What one machine can do

One replica with no batching and no caching serves only about 2 messages a second.
A gateway instance holds about 10,000 open streams. At 13 seconds a stream, that is about 750 new messages a second.

The decisions, and what each one costs

Decision	We chose	We gave up	We’d choose differently if
Where the model runs	Our own fleet	Hosted frontier models	Under about 100,000 users a day
Models	70B plus a routed 8B	One model to test and ship	Easy messages also needed the 70B
Prompt size	Summary plus newest messages	Every word of long conversations	Users needed exact recall of long documents
Which server	Back to the server that remembers	Perfectly even load	Messages shared little history
Batching	Continuous, with a token budget per step	The fastest first token for huge prompts	Very long prompts were common
Limits	Weighted tokens, reserve then settle	Limits users can count in messages	The product sold message bundles
Overload	A ladder that sheds the free tier first	Equal treatment of every user	There were no free tier

How to spend 45 minutes

0 to 6. Users, tokens, hosting and tiers. Say early that a router to a small model is the main lever.
6 to 12. The numbers: 3,500 messages a second, 46,000 streams, 0.67 GPU-seconds a message, 1,500 to 3,000 GPUs. Point out that prefill without memory would cost more than decode.
12 to 25. Build it: one server, streaming and history, the fleet with batching and the scheduler, token limits. A mid-level answer can stop here.
25 to 29. The test gate and the rollout.
29 to 42. Stress it: ten times the users, one huge prompt, and the shedding ladder. This is where senior answers win or lose.
42 to 45. A server dying mid-answer, then your own questions.

Never skip the token math, the conversation memory, or what happens when GPUs run out.

Follow-up questions

Interviewer asksTime to first token jumped from 0.3 s to 4 s, but tokens per second is normal. Where do you look?

The wait is before the first token, so it is queueing or prefill. Check queue length and memory use, how often the conversation memory is reused (a change to the start of every prompt empties it), and prompt length. Writing speed is fine, so more decode capacity won’t help.

Go deeperMore follow-up questions2 min

Plus users can pick the 70B model for every message. Does the router still matter? Yes, for the free tier and for Plus users on “auto”, which most never change. If many Plus users pin the big model, price it separately, because each pinned message costs about four times a routed one.

Users want a 1 million token context. What would you say? One such conversation would hold about 320 GB of memory, as much as a whole 4-GPU server, and reading it would take minutes of GPU time. It would be its own product, with its own GPUs and price. For most uses, retrieval puts the relevant 5,000 tokens into a normal prompt.

Finance asks why GPUs sit idle at 4 am. The fleet is sized for the evening peak, so nights are cheap capacity. Run work that can be paused: test runs, summaries of long conversations, other teams’ batch jobs. It gets the lowest priority, and is paused when chat traffic rises.

The safety check flags a sentence halfway through an answer. What does the user see? Each sentence is checked before it is sent, so the flagged one is never shown. The stream stops, and the app shows a refusal in place of the rest.

Design a ChatGPT-style assistant

The answer in one minute

Scope it

How much AI it needs

The numbers that decide it

Build the design

Step 1. One GPU server

Stage 1: One GPU server

Why this stage

What we add

What it costs

Step 2. Stream the answer, and keep the history

Stage 2: Streaming and conversations

Why this stage

What we add

What it costs

Step 3. A GPU fleet, and a smart scheduler

Stage 3: A GPU fleet with a scheduler

Why this stage

What we add

What it costs

Step 4. Limits and safety by tier

Stage 4: Limits and safety by tier

Why this stage

What we add

What it costs

Trace a request: Send a message

Prove it works

Stress it

Ten times the users

Stage 5: Send easy messages to a small model

Why this stage

What we add

What it costs

Stage 6: Regions and a shedding ladder

Why this stage

What we add

What it costs

One huge prompt

The day the GPUs run out

When things break

Break it: what fails, who notices, what users see

What breaks

How we notice

What users see

What we do

Wrap up

The whole design

The full design, stage by stage

Why this stage

What we add

What it costs

The decisions, and what each one costs

How to spend 45 minutes

Follow-up questions

Further reading

Keep reading

The designbeyondcode digest, every Saturday