How LLM inference works

A language model writes one token at a time. Every token needs one pass through the whole network. Almost every number in an AI system follows from how that loop runs on a GPU. That includes the seconds a user waits for the first word, and the price per million tokens. Once you can reason about the loop, capacity planning for a chat product stops being guesswork.

The problem it solves

A normal API request takes 50 ms of CPU and a few kilobytes of memory. An LLM request is different in every way:

• It runs for seconds. A 500-token answer at 50 tokens a second takes 10 seconds. The connection stays open the whole time.
• It holds gigabytes. The model’s weights sit in GPU memory: 16 GB for an 8 billion parameter model in 16-bit. Every active conversation also keeps a growing cache next to them.
• It costs real money per request. A GPU costs a few dollars an hour. So the question is how many tokens you get out of it per hour, and how many users share them.

Serving well means keeping the GPU busy with as many users as fit, without any one user waiting too long. The rest of this page shows how that’s done, and where it breaks.

How it works

Step 1: Text becomes tokens

A model doesn’t read characters or words. A tokenizer splits text into tokens. Common words are one token. Rare words are several. Spaces and punctuation are folded in. For English, a token is about 4 characters, or three quarters of a word. Code and non-English text use more tokens per word.

Tokens are the unit of everything that follows:

Thing	Measured in tokens
Price	Dollars per million input tokens, and per million output tokens
Context window	The most tokens the prompt and the answer may hold together
Speed	Tokens per second, for reading the prompt and for writing the answer
Memory	Bytes of cache per token of conversation

So when someone says “the prompt is 2,000 tokens”, that is roughly 1,500 words, or three pages.

Step 2: One pass through the model per token

Generation is a loop:

1. The model reads every token so far.
2. It gives a probability for every possible next token.
3. The runtime picks one: the most likely one, or a random draw shaped by the temperature setting.
4. It adds that token to the text and runs the model again.

So a 500-token answer is 500 passes.

Each pass has to read every weight in the model from GPU memory. So the loop is limited by memory bandwidth (how fast memory can be read), not by how fast the GPU can do math:

• An 8B model in 16-bit is 16 GB of weights.
• An NVIDIA H100 reads memory at about 3.35 TB a second.
• So one pass takes at least 16 / 3,350 seconds, about 5 ms. That caps one conversation at around 200 tokens a second, however fast the GPU’s math units are.

The math units sit mostly idle during this. Keep that in mind. It is the reason batching works so well (Step 5).

Step 3: Prefill, then decode

A request has two phases, and they behave in opposite ways.

Prefill reads the prompt. All prompt tokens are known up front, so the model handles them in one pass, in parallel. That keeps the math units fully busy, so prefill is compute-bound. A 2,000-token prompt on an 8B model is about 32 trillion operations (2 per parameter per token). That takes roughly 50 to 100 ms on an H100.

Decode writes the answer, one token per pass. Each pass reads all the weights. So decode is memory-bound, as Step 2 showed.

The two phases set the two numbers users feel:

Metric	What it is	Set by	Users feel it as
Time to first token (TTFT)	Wait from sending the request to the first word	Queueing plus prefill, so it grows with prompt length	“Is it doing anything?”
Time per output token (TPOT), or inter-token latency	Gap between words while the answer streams	Decode speed, and how many others share the batch	Reading speed: below about 10 tokens a second, users wait for the text

Streaming the answer doesn’t make it finish sooner. It moves the wait from “the whole answer” to “the first token”. That is why every chat product streams.

Step 4: The KV cache

Inside each layer, attention compares the newest token with every earlier token. It uses two vectors per token per layer: a key and a value.

Without a cache, writing token 501 would recompute the keys and values of all 500 earlier tokens. Every step would get slower than the last. So the runtime keeps them in the KV cache. Each new token computes its own key and value once, and adds them to the cache.

The cache is what makes decode fast. It is also what fills the GPU. Its size per token is:

bytes per token = 2 (key and value) x layers x KV heads x head dimension x bytes per number

Model	Layers	KV heads x dimension	KV cache per token (16-bit)	An 8,000-token conversation
Llama 3.1 8B	32	8 x 128	128 KB	1 GB
Llama 3.1 70B	80	8 x 128	320 KB	2.5 GB

These models use grouped-query attention. Their 32 or 64 query heads share just 8 key-value heads. That cuts the cache four to eight times, compared with one key-value head per query head. Without it, the 70B model would need 2.5 MB per token.

Now the memory budget of one GPU is concrete. An H100 has 80 GB. The 8B model takes 16 GB. The other 64 GB holds about 500,000 tokens of KV cache. That is 60 conversations of 8,000 tokens, or 250 of 2,000.

The 70B model doesn’t fit on one H100 at 16-bit: its weights alone are 140 GB. So it is split across two to eight GPUs, and the leftover memory is shared out the same way.

This is why long context is expensive. A 128,000-token conversation on the 8B model holds 16 GB of cache by itself. That is as much as the weights, and it pushes out dozens of shorter conversations.

Step 5: Batching

Each decode pass reads all the weights anyway. So the GPU can move many conversations forward in the same pass, for almost the same cost. It reads the weights once and applies them to 64 sequences instead of 1.

This holds until the math units become the limit, at batch sizes in the low hundreds for most models. Until then, a bigger batch gives nearly proportional throughput, and each user’s speed drops only a little.

TradeoffBigger batches

We gain

Far more tokens per GPU-second, so a lower cost per token. Going from a batch of 1 to a batch of 64 can raise throughput tenfold.

We give up

Each user’s tokens come a little slower. More memory goes to the KV cache, and requests may queue for a slot. Batch size is the dial between cost and latency.

How requests join and leave the batch matters as much as its size:

• Static batching collects a batch and runs it until every request in it has finished. Only then does the next batch start. So a 30-token answer waits for a 1,000-token answer in the same batch, and new arrivals wait for the whole batch. Older servers also reserved KV cache for the longest possible answer up front, and most of it was never used.
• Continuous batching decides who is in the batch at every decode step. It is also called iteration-level scheduling, from the Orca paper. A finished request leaves at once, and a waiting one takes its place on the next step.
• Paged KV cache (vLLM’s PagedAttention) stores the cache in small fixed-size blocks, like the pages of virtual memory. Blocks are added as a conversation grows. The vLLM paper measured older systems wasting 60 to 80% of KV cache memory on reservations and gaps. Paging cut that to under 4%, and raised throughput two to four times at the same latency.

Try both below. The GPU has 24 GB: 16 GB for an 8B model’s weights, and 8 GB (about 64,000 tokens) for the KV cache. Watch what static batching does to the queue. Then switch to continuous batching. Then add a 30,000-token request and watch it take half the cache.

Static batchesContinuous batching

PauseAdd a 30K-token requestReset

GPU memory, 24 GB0 of 8 GB KV cache in use

weights 16 GB

KV cache in usereserved, still emptyfree

In the batch

0

Waiting

0

Output tokens/s, total

0

Tokens/s, each user

0

Time to first token, median

-

Continuous batching: a request joins the batch at the next step if its tokens fit, and leaves the moment it finishes.

Two things to notice:

• Continuous batching keeps the batch full. Total throughput rises and the queue drains, while each user’s tokens a second barely change.
• When a request joins, its prefill is added to that step. So everyone already decoding stutters for a moment. Long prompts make that stutter longer. Chunked prefill (Step 6) fixes this.

Step 6: Making each token cheaper

Once batching is in place, the remaining levers cut how much memory and work each token needs.

Quantization stores weights in fewer bits. At 8-bit, the 8B model is 8 GB instead of 16. That leaves twice the room for KV cache. Each decode pass also reads half the bytes, so it is close to twice as fast. At 8-bit, the quality loss is usually too small to measure.

At 4-bit (GPTQ, AWQ, or the q4 files Ollama downloads), the weights are a quarter of the size. Now the quality loss is measurable, more so for small models. The KV cache can be quantized too.

Prefix caching reuses the KV cache for the start of a prompt that many requests share. Examples are a long system prompt, a document several questions are asked about, or the earlier turns of a conversation. The shared part skips prefill entirely. Hosted APIs pass the saving on: cached input tokens cost a fraction of the normal input price. So put the stable parts of a prompt first and the changing parts last.

Speculative decoding uses a small, fast draft model to guess the next few tokens. The large model then checks all the guesses in one pass. Checking several tokens costs about the same as writing one, because the pass is memory-bound anyway. When most guesses are right, the large model emits several tokens per pass. Published results show decode two to three times faster. The output is exactly what the large model alone would produce.

Chunked prefill splits a long prompt’s prefill into pieces, and mixes them into the decode steps. So a 30,000-token prompt doesn’t freeze every other user for its whole prefill.

A smaller model is often the biggest saving of all. A router can send easy requests to a small model and hard ones to a large model. That can cut cost several times, if you can measure which requests are easy (see evaluation).

Go deeperWhy output tokens cost more than input tokens1 min

Hosted APIs usually charge three to five times more per output token than per input token. Here is why:

• Input tokens are handled together, in one compute-bound prefill pass that uses the GPU fully.
• Output tokens each need their own memory-bound decode pass. They share it only with whatever else is in the batch. And they hold KV cache memory the whole time the answer is written.

So a request with a 10,000-token prompt and a 100-token answer is cheap to serve. One with a 100-token prompt and a 10,000-token answer is not.

Options and tradeoffs

Where the model runs

Option	Examples	Good for	You give up
Hosted API	Anthropic, OpenAI, Google, and cloud model catalogs	Starting fast, the strongest models, spiky traffic	Per-token price at high volume, data leaving your network, rate limits you don’t control
Self-hosted on GPUs	vLLM, SGLang, TensorRT-LLM, Hugging Face TGI	Steady high volume, open models, data that must stay in house	Running a GPU fleet: capacity, upgrades, failover, and the people to do it
On the device	llama.cpp, Ollama, Apple and Android on-device models	Privacy, offline use, zero marginal cost	Small models only, and every user’s hardware is different

TradeoffA hosted API first, self-hosting when the numbers say so

We gain

No fleet to run, and the best models from day one. Most products never reach the volume where a GPU fleet is cheaper.

We give up

Cost grows in step with usage. Latency and limits are the provider’s. And switching later means running every eval again against the new model.

The break-even point is simple arithmetic. Say a GPU costs $3 an hour, and it sustains 2,000 output tokens a second at the latency you need. It produces 7.2 million tokens an hour, which is about $0.42 per million.

But that only holds at full use, around the clock. At 20% average use, it is $2.08 per million, plus the engineers. Compare that with the API price for a model of the same quality, not with the cheapest one.

Latency or throughput

Every serving system has the same dial. Batch more for cheaper tokens, or batch less for faster ones.

• Interactive chat wants a decent time to first token (under a second or two), and decode at reading speed.
• A nightly job that summarizes 2 million documents cares about neither. It wants the cheapest tokens. So it runs with huge batches, or through a provider’s batch API at a discount.

How it fails

The KV cache runs out

When the cache is full, the server has three choices:

• Queue new requests, which raises their time to first token.
• Preempt a running request: free its cache now, and recompute it later.
• Reject the request.

All three show up as latency spikes that CPU and GPU usage graphs don’t explain. The signal to watch is KV cache use, and the queue in front of it.

Long prompts hold everyone up

A single 100,000-token prompt can take seconds of prefill. Without chunked prefill, every other user’s stream pauses while it runs. With chunked prefill, the long request’s own time to first token gets longer instead. Either way, cap prompt length per request and per user.

Cold starts

A new GPU server has to load the weights before it can serve anything. Loading 16 GB from a local disk at 2 to 3 GB a second takes several seconds. A 140 GB model pulled over the network can take minutes.

So LLM fleets can’t add servers on demand the way stateless web servers do. Keep warm spare capacity. Scale on queue depth and KV cache use, not on CPU. And pre-load weights on standby nodes.

Runaway outputs

A model that repeats itself, or a prompt that asks for “everything”, can write until the context window is full. That holds a batch slot and memory for minutes. Always set a maximum number of output tokens per request.

Common mistake

Quoting one tokens-per-second number

“Our server does 2,000 tokens a second” means nothing on its own. It could be the total across a batch of 100 users, each getting 20 tokens a second. Or it could be one user getting 2,000. Always say which, and at what batch size and prompt length. A server can have a great number for one and a bad number for the other.

At scale

Capacity planning for a chat product starts from the user, not the GPU.

1. Peak concurrency. 1 million daily users. 5% are active in the peak hour, and each has a conversation in flight a third of the time. That is about 17,000 streams at once.
2. Per-stream speed. 30 tokens a second is comfortably faster than reading. That is 500,000 output tokens a second at peak.
3. Per-GPU throughput at that speed. Measure it with your model, prompts and server. Suppose one GPU sustains 2,500 output tokens a second, while still giving each stream 30.
4. GPUs. 500,000 / 2,500 = 200 GPUs at peak. Add 30 to 50% headroom for failures and spikes: about 280.
5. Memory check. 2,500 tokens a second at 30 per stream is 83 streams per GPU. At 3,000 tokens of context each and 128 KB per token, that is 32 GB of KV cache. It fits next to an 8B model on an 80 GB GPU. It does not fit next to a 70B model on two.

Beyond one model server, three ideas matter at scale:

• Route by prefix. Requests that share a prefix (the same system prompt, the same conversation) should land on the server that already holds its KV cache. Plain round-robin throws prefix caching away.
• Split prefill and decode. Some fleets run prefill and decode on separate GPU pools (DistServe, Splitwise), and ship the KV cache between them. Then long prompts never stall decoding, and each pool is sized for its own bottleneck.
• Treat models as versions. A new model or a new quantization changes quality, speed and cost all at once. Roll it out like code, behind an eval gate, with traffic shifted over gradually.

In the interview

Interviewer asksUsers say the chatbot takes 6 seconds to start answering, but then streams fast. What's going on?

Time to first token is queueing plus prefill. Decode speed is fine, so the problem is before the first token. I’d check three things. First, the queue: is the KV cache full, so requests wait for a slot? Second, prompt length: has the system prompt or the retrieved context grown to tens of thousands of tokens? Third, cold starts: are requests landing on servers still loading weights? The fixes differ. More capacity or a smaller context for the first. Prefix caching and less retrieved text for the second. Warm spares for the third.

Interviewer asksWhy can one GPU serve 100 users almost as fast as it serves one?

Decode is memory-bound. Each step reads every weight once, and reading them is the slow part. So applying them to 100 sequences instead of 1 adds little time. That holds until one of two limits arrives. Either the batch gets big enough that arithmetic becomes the limit, or the KV cache for all those conversations no longer fits in memory.

Interviewer asksWe want to offer a 1 million token context window. What does that do to serving?

The KV cache grows in step with context. At 128 KB per token, one conversation on an 8B model needs 128 GB. That is more than a whole GPU, and bigger models need more. Prefill of a million tokens takes a long time, and attention cost grows faster than the length. So I’d split the cache across GPUs and chunk the prefill. I’d cache the long prefix, so follow-up questions don’t pay for it again, and price long requests differently. And I’d ask whether retrieval could give the model the 5,000 tokens it actually needs instead.

Mid-level covers

Explains tokens and streaming, and that longer prompts raise time to first token. Knows that batching raises throughput, and that GPU memory limits how many users fit.

Senior adds

Separates prefill from decode and says which is compute-bound. Sizes the KV cache from the model’s shape. Explains continuous batching and paged attention. Plans capacity from concurrency and per-stream speed. Picks quantization, prefix caching and speculative decoding for stated reasons. Scales on queue depth and KV cache use instead of CPU.

Try it on your laptop

Step 1: Run a local model →

Serve a 3B model on your laptop, then measure what this page describes: time to first token growing with prompt length, and what happens to each user’s speed when four users share the model.

Recap

The loop. One pass through the model per output token. Each pass reads every weight, so decode is limited by memory bandwidth: about 5 ms a pass for 16 GB of weights on an H100.

Two phases. Prefill reads the prompt in parallel. It is compute-bound and sets time to first token. Decode writes one token per pass. It is memory-bound and sets the streaming speed.

The KV cache. Keys and values for every token so far, per layer. 128 KB per token for Llama 3.1 8B, 320 KB for 70B. It decides how many conversations fit next to the weights.

Batching. Many sequences per pass, for nearly the cost of one. Continuous batching swaps requests in and out every step. A paged KV cache stops memory going to waste.

Cheaper tokens. Quantization, prefix caching, speculative decoding, chunked prefill, and routing easy requests to smaller models.

Biggest risks. A full KV cache that shows up as unexplained latency. Long prompts that stall everyone. Cold starts that defeat autoscaling. And outputs with no length limit.

Further reading

• Efficient Memory Management for Large Language Model Serving with PagedAttention: the vLLM paper, with the measurements of wasted KV cache memory.
• Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022: iteration-level scheduling, the idea behind continuous batching.
• Fast Inference from Transformers via Speculative Decoding: draft and verify, with the proof that the output distribution is unchanged.
• DistServe: disaggregating prefill and decode onto separate GPUs.
• Retrieval-augmented generation for keeping prompts short, and evaluation for checking that a cheaper model is still good enough.