Design a code completion assistant

“Design a code completion assistant like GitHub Copilot. As a developer types, it shows the next few lines in grey, and Tab accepts them.”

It sounds like “call a model on every keystroke”. Two numbers make it hard.

First, the suggestion must appear before the developer types the next character. That gives the whole system about 300 ms. Second, the model can read only a few thousand tokens, but the repo holds millions. Almost every decision on this page follows from those two numbers. We start with a plugin that sends code to one GPU, and add parts only when a number forces us to.

The framework

1. 01Answer
2. 02Scope
3. 03How much AI
4. 04Numbers
5. 05Build
6. 06Quality
7. 07Stress
8. 08Failures
9. 09Wrap up

The answer in one minute

If you only have a minute, this is the design. The rest of the page explains each part.

1. A small, fast code model, because 300 ms leaves the model only about 65 ms. A big model can’t answer in time, and a late suggestion is never seen.
2. Ask less often, and cancel fast. The plugin waits for a short pause in typing before it asks. It cancels a request as soon as the text changes, all the way down to the GPU.
3. Pick the right 2,000 tokens on the laptop. The code around the cursor, plus the definitions and similar code it needs, chosen from a small index of the repo. The repo itself never leaves.
4. Send each file to the same GPU. That GPU still remembers most of the last prompt, so it does a third of the work.
5. Check every suggestion before showing it, and measure success by what developers keep. Nothing on the server stores code.

Scope it

A few questions change the design a lot. For the rest, we assume an answer and move on.

Interviewerminute 0

Design an AI code completion assistant, like Copilot’s grey ghost text.

What they are testing

Whether you pin down speed and privacy before you pick a model. Those two decide how big the model can be and where it runs. “The biggest model with RAG over the repo”, before asking how fast it must be, is the red flag.

These five questions matter most:

• How fast must a suggestion appear? Within 300 ms of the last keystroke, for 95% of requests. With 1 second, a bigger model would fit. With 100 ms, the model would have to run on the laptop.
• Can code leave the laptop? Yes, to our servers in the customer’s region. It is never stored and never used for training.
• How many developers? 1 million a day, each making about 4,000 edits.
• How big are the repos? Up to 100,000 files.
• Inline completion only, or chat too? Inline first. Chat comes later, on its own GPUs.

We promise three things: suggestions inside 300 ms, no code kept on our side, and under $2 of GPU per developer a month. We leave out training the base model, the chat assistant itself (see the chat assistant problem) and code review agents.

Go deeperThe full requirements and quality targets3 min

Must do. Show a suggestion in grey at the cursor. That is the rest of the line, or a block of up to about 10 lines after a line that opens a block. Tab accepts it. Typing anything else dismisses it. Paths the customer excludes never leave the laptop. Suggestions that copy public code are blocked for customers who ask.

Speed. Half of suggestions within 200 ms, and 95% within 300 ms, measured in the plugin. At most 1 ms of plugin work per keystroke on the editor’s main thread, so typing never lags.

Scale. 35,000 requests a second at peak, about 1 billion a day.

Availability. 99.9% of requests end in a suggestion or a clean skip, never an error.

Quality.

• At least 25% of shown suggestions are accepted. A new model may lose at most 1 point.
• At least 70% of accepted characters are still in the file 5 minutes later.
• At least 98% of suggestions are valid code where they are placed.
• Zero secrets and zero blocked public code shown on our attack test set.

More questions	We assume	If the answer were different
Languages and editors?	The top 15 languages. VS Code, JetBrains, Neovim	One language in one editor could lean on its tools more
Avoid copying public code?	Business customers can block it	Without the rule, the public code check goes away
Air-gapped customers?	Not at first	They need a deployment in their own network

How much AI it needs

Interviewerminute 3

Do you need a large language model for this at all?

What they are testing

Whether you start from the cheapest tool. Editors already complete names. A strong answer splits the product by speed: the model for the next line isn’t the model for chat. One big model for every keystroke is the red flag.

Editors already complete names using a language server, a tool that understands the code’s structure. It is exact and instant, but it can’t write a line of logic. So we split the work by how fast it must be:

Three tools, each for the job it can do in time

Scroll or pinch to zoom, drag to pan

• Color key
• What the developer does
• Instant, no model
• Small fast model
• Large model, slower

We pick a small code model, about 3 billion parameters (“3B”), trained for fill-in-the-middle. That means it can fill a gap between the code above the cursor and the code below. It is fast enough to fit in 300 ms on a GPU.

A model ten times bigger would need about ten times the work for each prompt. It alone would use the whole 300 ms. The small model is sometimes wrong where a big one would be right. We accept that, because a late suggestion is never seen at all.

Go deeperAll the options, and why each falls short2 min

Option	Would it work?	Why it falls short, or why we pick it
Language server completion	For names only	Exact and instant, but can’t write logic. We keep it, and its names feed our prompt
Paste the most similar code from the repo	Sometimes	Can’t adapt names and types to this spot. We use it as context, not as the answer
A small fill-in-the-middle model	Yes	Fast enough for 300 ms. The pick
A large model on every keystroke	Quality yes, speed no	Ten times the work per prompt, and ten times the bill
The model on the developer’s laptop	Only on fast laptops	Nothing leaves the machine, but laptops differ ten times in speed
Fine-tuning on each customer’s code	Later, opt-in only	Our privacy promise rules it out by default. Context gives most of the benefit

The numbers that decide it

Interviewerminute 6

Roughly how many GPUs is this, and what does one suggestion cost?

What they are testing

Whether you go from keystrokes to requests to tokens to GPUs, and notice that prompt tokens are the cost, not answer tokens. Multiplying an API price by a request count is the red flag.

One idea first. A model does two kinds of work. Prefill reads the prompt. Decode writes the answer, one token at a time. A token is a piece of a word or of code. For chat, decode is the main cost. For code completion, it is the opposite. The prompt is about 2,000 tokens, and the answer is about 24. So reading the prompt is the cost.

Assumptions

Daily active developers

1 million

Completion requests per developer a day, after debouncing

1,000

Peak over average

3×

Keystrokes per request, without debouncing

4×

Requests cancelled before they finish

45%

Prompt tokens per request

2,000

Prompt tokens served from the prefix cache

70%

Output tokens per finished suggestion

24

Prefill tokens per GPU-second

60,000

Output tokens per GPU-second, at interactive speed

8,000

GPU price per hour, USD

3

GPU busy at peak (latency headroom)

60%

Requests12,000/s35,000/s at peak

1 million developers × 1,000 requests a day ÷ 86,400 seconds. The peak is about 3 times that, because a region’s developers work the same hours. The 1,000 assumes one request per about 4 keystrokes.

So we ask less often. Asking on every keystroke would mean 140,000 a second.

Prompt2,000 tokens2 trillion a day

The code above and below the cursor, plus snippets from the repo. The answer is about 24 tokens, so the prompt is about 100 times bigger.

So we treat reading the prompt as the cost, and reuse as much of it as we can.

GPU time12 msper request, 35 ms with no reuse

A GPU reads about 60,000 prompt tokens a second with our 3B model. If 70% of the prompt is reused from the last request, only about 600 tokens are new: 10 ms. Writing the answer adds about 2 ms.

So we send each file's requests to the GPU that already holds most of its prompt.

GPUsabout 700at peak, all regions

35,000 requests a second × 12 ms is 420 GPUs fully busy. Queues eat into the 300 ms, so we run GPUs only about 60% busy: 700.

So we run our own GPUs in each region, where we control reuse, region and cancelling.

Cost$1.50per developer a month

700 GPUs at $3 an hour cost about $50,000 a day. That is about $0.10 per 1,000 suggestions shown, and well under our $2 target.

So we cost doesn't limit the model size. Speed does.

Speed limits the model, not cost. Here is where the 300 ms goes:

Where the 300 ms go

Scroll or pinch to zoom, drag to pan

• Color key
• On the laptop
• Network and gateway
• The model
• The total

A round trip across an ocean alone takes 150 ms or more. So we need GPUs in every major region.

Go deeperThe calculator, and what happens when you change an assumption2 min

Change an assumptionReset to the page's numbers

Daily active developers

1 million

Completion requests per developer a day, after debouncing

1,000

Peak over average

3×

Keystrokes per request, without debouncing

4×

Requests cancelled before they finish

45%

Prompt tokens per request

2,000

Prompt tokens served from the prefix cache

70%

Output tokens per finished suggestion

24

Prefill tokens per GPU-second

60,000

Output tokens per GPU-second, at interactive speed

8,000

GPU price per hour, USD

3

GPU busy at peak (latency headroom)

60%

Requests, average

12,000/s

Requests, peak

35,000/s

Every one of these needs a GPU answer within about 200 ms.

Requests at peak if every keystroke asked

140,000/s

Debouncing and typed-ahead reuse remove most of these before they leave the laptop.

Prompt tokens a day

2 trillion

Prompt tokens outnumber output tokens about 100 to 1: prefill, not decode, is the cost.

GPUs at peak

700

Self-hosted pools, split across regions.

GPU cost a day, USD, fleet sized for peak

50,000

GPU cost per 1,000 suggestions shown, USD

0.1

GPU cost per developer a month, USD

1.5

Cheap enough to include in a per-seat price.

Telemetry a day

600 GB

About 3 events per request at about 200 bytes. Ids, counts and timings only, never code.

Set the prompt reuse to 0%, and the fleet triples to about 2,100 GPUs. Double the prompt to 4,000 tokens, and it grows to about 1,300.

Telemetry adds about 600 GB a day: 3 events per request, 200 bytes each, and no code. A stream and a column store handle that easily.

Build the design

We start with one GPU. Then each step fixes the biggest problem left. The drawing grows with each step.

Step 1. Send the code to one model

Interviewerminute 12

Start simple. The developer is typing. What happens?

What they are testing

Whether you get a working loop in two minutes (plugin, model, grey text) and say what is naive about it. A retrieval pipeline before one suggestion has appeared reads as memorized.

On every keystroke, the editor plugin sends the code above the cursor to a model server on one GPU. The model continues the text, and the plugin draws the answer in grey. Tab inserts it.

Code completion assistant · architecture

Stage 1: The prefix to one model

Color keyServiceDatabaseCacheQueue or streamBackground workerModelNew in this stageAsync or occasional

Completion requests per second (after debouncing):

35,000

12k average35k peak350k (10×)

Stage 1 of 7

The prefix to one model

Why this stage

Nothing yet.

This is the smallest design that puts ghost text in an editor.

What we add

An IDE plugin that sends the code above the cursor on every keystroke to one GPU server running a code model, and draws the answer as grey ghost text.

What it costs

A request per keystroke, answers that arrive after the developer has typed on, no code below the cursor, nothing from other files, and one region.

At 35k completion requests per second (after debouncing), Regional GPU pools is 350000% busy. Stage 2 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What the slider counts

• The slider counts completion requests after debouncing, which means waiting for a short pause in typing.
• Stage 1 has no debounce. It sends one request per keystroke, about 4 for every debounced one, and its single GPU serves about 40 a second.

What one GPU can do

• With a 1,500-token prefix (stage 2): about 37 requests a second.
• With a 2,000-token prompt (stage 3): about 28.
• Once the prefix cache supplies 70% of each prompt (stage 4): about 83.
• With 8-bit weights (stage 6): about 130.

Headroom and filtering

• GPU pools are kept near 60% busy, because waiting in a queue eats the latency budget.
• In stage 6 the trigger model skips about 30% of requests.
• About 55% of requests finish and reach the filters.

It works, with four problems:

• It asks on every keystroke, so the GPU is overloaded at any real load.
• Most answers arrive after the text has already changed.
• The model can’t see the code below the cursor, so it writes a closing bracket that is already there.
• A developer in Europe pays a 150 ms round trip to a US GPU before the model even starts.

Step 2. Wait, cancel, stream

Interviewerminute 14

A developer types five keys a second. What does your server see?

What they are testing

Whether you see that most requests are out of date before they finish, and that a cancel must reach the GPU, not just the plugin. “Rate limit the user” is the red flag. The user did nothing wrong.

A suggestion for for u is useless once the developer has typed for us. Four things stop the system from working on text that no longer exists:

1. Wait for a pause. Ask only after 75 ms with no key. A burst of keys becomes one request. This is called debouncing.
2. Reuse what still fits. If the developer types the start of the grey text, the suggestion is still right. Trim it, and send nothing.
3. Cancel. Any other edit cancels the request in flight. A late answer is never drawn.
4. Stream, and stop on the GPU. The answer streams back token by token. When the plugin closes the stream, the model server drops the request at its next step and frees its memory.

Code completion assistant · architecture

Stage 2: Debounce, cancel, stream

Grow itTrace a request

Color keyServiceDatabaseCacheQueue or streamBackground workerModelNew in this stageAsync or occasional

Play

PreviousNext

Stage

12

Completion requests per second (after debouncing):

35,000

12k average35k peak350k (10×)

Stage 2 of 7

Debounce, cancel, stream

Why this stage

At about 4 keystrokes per useful request, sending every key costs 4 times the GPUs, and most answers arrive for text that no longer exists.

What we add

A 75 ms debounce and typed-ahead reuse in the plugin, one cancel for any stale request, and a gateway that streams tokens and turns a closed stream into an abort on the GPU.

What it costs

About 45% of requests are still cancelled after prefill has started, and the plugin now holds state per editor.

At 35k completion requests per second (after debouncing), every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What the slider counts

• The slider counts completion requests after debouncing, which means waiting for a short pause in typing.
• Stage 1 has no debounce. It sends one request per keystroke, about 4 for every debounced one, and its single GPU serves about 40 a second.

What one GPU can do

• With a 1,500-token prefix (stage 2): about 37 requests a second.
• With a 2,000-token prompt (stage 3): about 28.
• Once the prefix cache supplies 70% of each prompt (stage 4): about 83.
• With 8-bit weights (stage 6): about 130.

Headroom and filtering

• GPU pools are kept near 60% busy, because waiting in a queue eats the latency budget.
• In stage 6 the trigger model skips about 30% of requests.
• About 55% of requests finish and reach the filters.

Wait, reuse and cancel

Scroll or pinch to zoom, drag to pan

• Color key
• Service

JCompletionTrigger.javaPycompletion_trigger.pyC#CompletionTrigger.csGocompletion_trigger.goTScompletion-trigger.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

// The plugin side of a completion: wait for a pause, keep at most one request in flight, and
// reuse the ghost text while the developer types exactly what it predicted.
class CompletionTrigger {
    // Long enough to skip requests in the middle of a fast burst of keys, short enough to be a
    // small slice of the 300 ms budget.
    private static final long DEBOUNCE_MS = 75;

    private record Shown(String prefix, String text) {}

    private final ScheduledExecutorService timer = Executors.newSingleThreadScheduledExecutor();
    // Streams from the gateway. Cancelling the future resets the HTTP/2 stream, and the gateway
    // turns that into an abort on the GPU, so nobody computes an answer nobody will see.
    private final Function<String, CompletableFuture<String>> fetch;
    private final Consumer<String> show;   // "" clears the ghost text
    private ScheduledFuture<?> pending;
    private CompletableFuture<String> inFlight;
    private Shown shown;
    private long generation;               // which edit an answer belongs to

    CompletionTrigger(Function<String, CompletableFuture<String>> fetch, Consumer<String> show) {
        this.fetch = fetch;
        this.show = show;
    }

    synchronized void onEdit(String prefix) {
        // Typed-ahead reuse: if the developer typed the start of the ghost text, the suggestion
        // is still right. Trim it and send nothing.
        if (shown != null && prefix.startsWith(shown.prefix())) {
            String typed = prefix.substring(shown.prefix().length());
            if (shown.text().startsWith(typed) && typed.length() < shown.text().length()) {
                show.accept(shown.text().substring(typed.length()));
                return;
            }
        }
        shown = null;
        show.accept("");
        // Any answer for an older prefix is now useless: stop the wait and the request.
        generation++;
        if (pending != null) pending.cancel(false);
        if (inFlight != null) inFlight.cancel(true);
        inFlight = null;
        long gen = generation;
        pending = timer.schedule(() -> request(prefix, gen), DEBOUNCE_MS, TimeUnit.MILLISECONDS);
    }

    private synchronized void request(String prefix, long gen) {
        if (gen != generation) return;      // a newer edit arrived while the timer fired
        inFlight = fetch.apply(prefix);
        // Failures complete the future exceptionally and skip this callback: a failed
        // suggestion is silent, because an error popup would not help anyone.
        inFlight.thenAccept(text -> onAnswer(prefix, text, gen));
    }

    private synchronized void onAnswer(String prefix, String text, long gen) {
        // A late answer for an old prefix must never be drawn: it would insert stale code.
        if (gen != generation || text.isEmpty()) return;
        shown = new Shown(prefix, text);
        show.accept(text);
    }
}

Now the plugin asks about once per 4 keystrokes. Still, about 45% of requests are cancelled. A typing pause is often longer than 75 ms but shorter than the answer. That is fine. Asking early is what makes suggestions appear fast. What matters is that a cancelled request is cheap.

Common mistake

Cancelling only in the plugin

If the plugin just ignores late answers, the GPU keeps writing up to 64 tokens nobody will see. With 45% of requests cancelled, that wastes about a quarter of the fleet. The cancel must reach the model server, which can drop a request between two steps.

Go deeperFollow a cancel, and what each kind of cancel costs2 min

Code completion assistant · architecture

Trace a request: The developer types on

Color keyServiceDatabaseCacheQueue or streamBackground workerModelNew in this stageAsync or occasionalHop in motionHops doneResponse

Play

PreviousNext

The developer types on: step 1 of 7

Developers to IDE plugin

The developer pauses after `for u`. After 75 ms of quiet the plugin sends request 41.

75 ms of a 300 ms budget

1. Developers → IDE plugin75 ms
2. IDE plugin → Completion gateway20 ms
3. Completion gateway → Regional GPU pools30 ms
4. Developers → IDE plugin0 ms
5. IDE plugin → Completion gateway20 ms
6. Completion gateway → Regional GPU pools1 ms
7. Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Where the cancel stops	GPU work per cancelled request	Note
In the plugin only	Prompt plus up to 64 tokens	Easy, and expensive
At the gateway	The same	The gateway stops passing it on. The GPU doesn’t
At the model server	Prompt plus about 8 tokens	Freed at the next step
Before it starts: a deadline	Nothing	For requests that waited too long in a queue

Every request carries a deadline. If it is still waiting when the plugin would stop waiting, the gateway drops it before it reaches a GPU.

Step 3. Context from the repo

Interviewerminute 17

The suggestions call functions that don’t exist. How do you fix that?

What they are testing

Whether you reach for context before a bigger model: the code below the cursor, definitions of names, similar code, all inside a fixed budget. “Put the whole repo in a long-context model” is the red flag. It breaks both speed and cost.

A model that sees only the code above the cursor has to guess names. Two changes fix that.

Send the code below too. Code models are trained with special markers to fill a gap. Given the code before and after, they write the middle. This is called fill-in-the-middle. Now the model knows the block already closes, and which names come next.

Add what it needs from other files. The plugin keeps a small index of the repo on the laptop: the names defined in every file, and short windows of code. For each request, a context packer picks the most useful pieces and fits them into a 2,048-token budget.

What goes into the 2,048 tokens

Scroll or pinch to zoom, drag to pan

• Color key
• Code around the cursor
• The packer
• Code from elsewhere
• What the model gets

The rule is simple. The 400 tokens above the cursor always go in, then the code below. Then the other pieces, ranked by value per token, so one big file can’t crowd out five small, useful definitions. This is retrieval-augmented generation, where the code is both the question and the documents.

Code completion assistant · architecture

Stage 3: Context from the repo

Grow itTrace a request

Color keyServiceDatabaseCacheQueue or streamBackground workerModelNew in this stageAsync or occasional

Play

PreviousNext

Stage

123

Completion requests per second (after debouncing):

35,000

12k average35k peak350k (10×)

Stage 3 of 7

Context from the repo

Why this stage

With only the prefix, the model invents function names and closes blocks that the code below already closes.

Quality, not load, forces this stage.

What we add

Fill-in-the-middle prompts with the code below the cursor, and a local index of symbols and snippets that a context packer fits into a 2,048-token budget.

What it costs

Prompts grow to about 2,000 tokens, so each request needs about a third more GPU time.

At 35k completion requests per second (after debouncing), every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What the slider counts

• The slider counts completion requests after debouncing, which means waiting for a short pause in typing.
• Stage 1 has no debounce. It sends one request per keystroke, about 4 for every debounced one, and its single GPU serves about 40 a second.

What one GPU can do

• With a 1,500-token prefix (stage 2): about 37 requests a second.
• With a 2,000-token prompt (stage 3): about 28.
• Once the prefix cache supplies 70% of each prompt (stage 4): about 83.
• With 8-bit weights (stage 6): about 130.

Headroom and filtering

• GPU pools are kept near 60% busy, because waiting in a queue eats the latency budget.
• In stage 6 the trigger model skips about 30% of requests.
• About 55% of requests finish and reach the filters.

JContextPacker.javaPysnippet.pyC#ContextPacker.csGosnippet.goTSsnippet.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

record Snippet(String source, String path, String text, double score) {}   // score: 0 to 1

class ContextPacker {
    static final int PREFIX_MIN = 400;  // the lines just above the cursor predict best: never cut
    static final int SUFFIX_MAX = 250;  // enough to see where the block ends, so the model stops

    // About 4 characters per token for code. The plugin ships the model's real tokenizer; an
    // estimate that runs a little high only wastes a few tokens of budget.
    static int tokens(String text) { return text.length() / 4 + 1; }

    static int sum(List<String> lines) { return lines.stream().mapToInt(ContextPacker::tokens).sum(); }

    // Whole lines only. Taken from the end of the prefix, the window's start moves only when a
    // whole line falls off, not on every keystroke, so the prompt start stays the same.
    static List<String> fit(List<String> lines, int maxTokens) {
        List<String> out = new ArrayList<>();
        int used = 0;
        for (String line : lines) {
            if (used + tokens(line) > maxTokens) break;
            out.add(line);
            used += tokens(line);
        }
        return out;
    }

    static String pack(List<String> prefix, List<String> suffix, List<Snippet> candidates, int budget) {
        List<String> reversed = new ArrayList<>(prefix);
        Collections.reverse(reversed);
        int near = sum(fit(reversed, PREFIX_MIN));
        List<String> after = fit(suffix, SUFFIX_MAX);   // the first lines below the cursor
        int left = budget - near - sum(after);

        // Everything else competes on value per token, so one big file cannot crowd out five
        // small, relevant signatures.
        List<Snippet> byDensity = new ArrayList<>(candidates);
        byDensity.sort(Comparator.comparingDouble((Snippet s) -> s.score() / tokens(s.text())).reversed());
        List<Snippet> chosen = new ArrayList<>();
        for (Snippet s : byDensity) {
            int cost = tokens(s.text()) + tokens(s.path());   // the path goes in a header line
            if (cost <= left) {
                chosen.add(s);
                left -= cost;
            }
        }

        // Budget nobody claimed buys more of the current file, which is never irrelevant.
        List<String> before = fit(reversed, near + left);
        Collections.reverse(before);

        // Stable order, not score order: if the same snippets win again on the next keystroke,
        // the prompt starts with the same bytes and the server's prefix cache skips their prefill.
        chosen.sort(Comparator.comparing(Snippet::source).thenComparing(Snippet::path));
        StringBuilder p = new StringBuilder();
        for (Snippet s : chosen) p.append("# ").append(s.path()).append('\n').append(s.text()).append('\n');
        // Suffix before prefix: typing only appends to the end of the prompt, so the prefix
        // cache covers everything up to the cursor.
        return p + "<fim_suffix>" + String.join("\n", after)
                 + "<fim_prefix>" + String.join("\n", before) + "<fim_middle>";
    }
}

The prompt grows from about 1,500 to 2,000 tokens. That is a third more GPU work per request. Step 4 wins it back.

Go deeperThe packer's classes, and why similar names beat embeddings2 min

The context packer in the plugin

Scroll or pinch to zoom, drag to pan

• Color key
• Facade
• Entity
• Value object
• Enum
• Interface
• Implementation
• New in this step

Why similar names, and not embeddings? For the next line of code, the strongest signal is shared names. Matching names finds the other loop over invoices that calls the same helpers. It costs nothing on a laptop and needs no model. Embeddings shine when the question is in plain English, which is chat’s job (see vector search).

Packing on the laptop has a cost. The prompt format lives in four editor plugins, and changing it needs a plugin release. So each request carries its format version, and the gateway accepts the last few.

Step 4. Serve it fast

Interviewerminute 20

Walk me through the 300 milliseconds. Where does the time go?

What they are testing

Whether every hop gets a number, and whether you see that reusing a prompt only helps if the request goes back to the same GPU. “Buy faster GPUs” is the red flag.

The budget has no slack. Three changes make the model’s share fit.

GPUs in every region. A gateway and a pool of GPUs in the US, Europe and Asia keep the round trip near 40 ms. They also give each customer a home region their code never leaves.

Send each file to the same GPU. Two requests a second apart, from the same file, share almost their whole prompt. A model server can keep its memory of recent prompts. This is called the prefix cache. Then the second request only reads what changed. But that works only on the same GPU. So the gateway picks the GPU from a hash of the customer and the file. If that GPU is too busy, it uses a second choice.

Guess ahead. A one-line suggestion is about 16 tokens, about 100 ms at one token per step. With speculative decoding, a fast helper guesses several tokens, and the model checks them all in one step. Code repeats a lot, so the helper can often copy names straight from the prompt.

Code completion assistant · architecture

Stage 4: Serve for latency

Grow itTrace a request

Color keyServiceDatabaseCacheQueue or streamBackground workerModelNew in this stageAsync or occasional

Play

PreviousNext

Stage

1234

Completion requests per second (after debouncing):

35,000

12k average35k peak350k (10×)

Stage 4 of 7

Serve for latency

Why this stage

The p95 target is 300 ms from the last keystroke.

A full prefill of 2,000 tokens plus a cross-ocean round trip leaves almost nothing for decoding.

What we add

GPU pools in every major region, routing that sends each file to the server already holding its prompt in the prefix cache, and speculative decoding.

What it costs

A routing layer that must balance cache hits against queueing, and pools that must be sized per region.

At 35k completion requests per second (after debouncing), every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What the slider counts

• The slider counts completion requests after debouncing, which means waiting for a short pause in typing.
• Stage 1 has no debounce. It sends one request per keystroke, about 4 for every debounced one, and its single GPU serves about 40 a second.

What one GPU can do

• With a 1,500-token prefix (stage 2): about 37 requests a second.
• With a 2,000-token prompt (stage 3): about 28.
• Once the prefix cache supplies 70% of each prompt (stage 4): about 83.
• With 8-bit weights (stage 6): about 130.

Headroom and filtering

• GPU pools are kept near 60% busy, because waiting in a queue eats the latency budget.
• In stage 6 the trigger model skips about 30% of requests.
• About 55% of requests finish and reach the filters.

JSessionRouter.javaPymodel_server.pyC#SessionRouter.csGomodel_server.goTSmodel-server.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

class ModelServer {
    final String id;
    final AtomicInteger inFlight = new AtomicInteger();
    final int maxInFlight;   // above this its queue, not its cache, decides the latency

    ModelServer(String id, int maxInFlight) { this.id = id; this.maxInFlight = maxInFlight; }
}

class SessionRouter {
    // A fixed hash, not String.hashCode() of a mixed key: every gateway must rank servers alike.
    static long fnv1a(String text) {
        int h = 0x811C9DC5;
        for (byte b : text.getBytes(StandardCharsets.UTF_8)) h = (h ^ (b & 0xFF)) * 0x01000193;
        return Integer.toUnsignedLong(h);
    }

    static Optional<ModelServer> route(String tenant, String filePath, List<ModelServer> servers) {
        // The tenant is part of the key, and the server's cache is keyed the same way, so two
        // companies never share cached prompt state, even for identical code.
        String key = tenant + "|" + filePath;
        // Highest random weight: each gateway computes the same order with no shared state,
        // and removing one server moves only the sessions that were on it.
        List<ModelServer> ranked = new ArrayList<>(servers);
        ranked.sort(Comparator.comparingLong((ModelServer s) -> fnv1a(key + "|" + s.id)).reversed());
        for (ModelServer s : ranked) {
            // Bounded load: a warm cache saves about 20 ms; a long queue costs more than that.
            // A busy first choice hands the session to its second choice, which is also stable.
            if (s.inFlight.get() < s.maxInFlight) return Optional.of(s);
        }
        return Optional.empty();   // every server is full: skip this suggestion, don't queue it
    }
}

With 70% of each prompt reused, GPU time per request falls from 35 ms to 12 ms. The fleet falls from about 2,100 GPUs to about 700.

There is one catch. The prefix cache reuses a prompt only up to the first character that changed. If the code above the cursor came first, every keystroke would change the start, and nothing could be reused. So the packer puts the most stable parts first:

Stable first, so typing only changes the end

Scroll or pinch to zoom, drag to pan

• Color key
• Changes rarely
• Changes on every key
• Reused
• Read again

Fill-in-the-middle models learn both orders, so this costs no quality. Try it. Change the budget, choose what the packer may use, and turn the stable order off:

Prompt budget 2,048 tokensKeep the prefix cache warm (stable snippet order, suffix before prefix)

What the packer may use

Code just above the cursor last 40 lines, 400 tokensCode below the cursor next 20 lines, 250 tokensSignatures of symbols used here from the local index, 200 tokensSimilar code from open tabs 2 snippets, 250 tokensSimilar code from the repo index 2 snippets, 200 tokensThe rest of the file above up to 70 more lines, 700 tokensWhole neighboring files 3 files, 3,000 tokensREADME and docs top of the repo, 1,500 tokens

Time to ghost text

Acceptance (illustrative)of suggestions shown

GPU time per requestvs the design on this page

Past about 2,000 tokens, each extra piece helps little and costs time. And with the stable order off, the same prompt costs almost three times the GPU time.

Common mistake

Round-robin in front of model servers

Round-robin spreads the load evenly, and throws the prefix cache away. Each request lands on a GPU that has never seen its file, and reads all 2,000 tokens again. For this workload, it triples the fleet.

Try it on your laptop

Step 1: Run a local model →

Send the same long prompt twice to a 3B model on a laptop. In our run, the first token took 194 s the first time and 0.2 s the second, because the prompt was reused.

Step 5. Check suggestions, and measure them

Interviewerminute 23

How do you know a suggestion is safe to show, and how do you know any of this is working?

What they are testing

Whether you treat model output as untrusted (secrets, copied licensed code, broken blocks), and measure quality from what developers do. Logging prompts to measure quality is the red flag. It breaks the privacy promise.

The model learned from public code. Some of it has leaked passwords, or a license the customer can’t accept. So the gateway checks each line before showing it:

1. Trim to the block. Stop at a line that is less indented than the cursor. The model has left the block it was asked to fill.
2. Drop secrets. Anything shaped like an access key or a private key.
3. Check for public code. For long suggestions, look up each 30-token window in an index of public code. Customers who block matches get nothing. Others see the source.

JPostProcessor.javaPyexample.pyC#PostProcessor.csGomain.goTSexample.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

class PostProcessor {
    // Shapes of real credentials. A model trained on public code has seen leaked keys and can
    // reproduce one; a suggestion must never put a working secret into someone's file.
    static final Pattern SECRETS = Pattern.compile(
        "AKIA[0-9A-Z]{16}"                                  // AWS access key id
        + "|-----BEGIN [A-Z ]*PRIVATE KEY-----"
        + "|(?i:(?:api|secret)_?key\\s*[:=]\\s*['\"][A-Za-z0-9_\\-]{20,}['\"])");
    static final Pattern WORDS = Pattern.compile("\\w+|[^\\w\\s]");
    static final int WINDOW = 30;   // code tokens; shorter matches are idioms, not copied code

    static int indent(String line) { return line.length() - line.stripLeading().length(); }

    static Optional<String> process(String text, int cursorIndent, Set<Integer> publicIndex) {
        // 1. Stop where the block ends. A line indented less than the cursor's line means the
        //    model has left the block it was asked to fill; the suffix already holds the rest.
        String[] lines = text.split("\n", -1);
        StringBuilder kept = new StringBuilder(lines[0]);
        for (int i = 1; i < lines.length; i++) {
            if (!lines[i].isBlank() && indent(lines[i]) < cursorIndent) break;
            kept.append('\n').append(lines[i]);
        }
        String out = kept.toString().stripTrailing();
        if (out.isEmpty()) return Optional.empty();

        // 2. Drop anything shaped like a credential. Dropping beats masking: a masked key is
        //    still a broken line of code.
        if (SECRETS.matcher(out).find()) return Optional.empty();

        // 3. Long suggestions are checked against hashes of public code. Words, not characters,
        //    so a copy with different spacing still matches.
        List<String> words = WORDS.matcher(out).results().map(MatchResult::group).toList();
        for (int i = 0; i + WINDOW <= words.size(); i++) {
            // This tenant blocks matches; another may get a license note instead.
            if (publicIndex.contains(hashWindow(words.subList(i, i + WINDOW)))) return Optional.empty();
        }
        return Optional.of(out);
    }

    // A fixed hash (FNV-1a), so the plugin, the gateway and the index builder all agree.
    static int hashWindow(List<String> words) {
        int h = 0x811C9DC5;
        for (byte b : String.join(" ", words).getBytes(StandardCharsets.UTF_8)) h = (h ^ (b & 0xFF)) * 0x01000193;
        return h;
    }
}

Then we measure. The plugin reports each suggestion as shown, accepted or rejected. For accepted ones, it checks after 5 minutes how much of the text is still in the file, and sends only that number. Code that was accepted and then deleted didn’t help.

Code completion assistant · architecture

Stage 5: Filters and feedback

Grow itTrace a request

Color keyServiceDatabaseCacheQueue or streamBackground workerModelNew in this stageAsync or occasional

Play

PreviousNext

Stage

12345

Completion requests per second (after debouncing):

35,000

12k average35k peak350k (10×)

Stage 5 of 7

Filters and feedback

Why this stage

A suggestion that leaks a key, copies licensed code, or breaks the block is worse than none.

And without telemetry nobody knows if suggestions help.

What we add

Post-processing that trims to the block and drops secrets and public-code matches, and an event pipeline that measures acceptance and how much accepted code survives.

What it costs

A few milliseconds per suggestion, an index of public code to keep fresh, and 600 GB of events a day.

At 35k completion requests per second (after debouncing), every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What the slider counts

• The slider counts completion requests after debouncing, which means waiting for a short pause in typing.
• Stage 1 has no debounce. It sends one request per keystroke, about 4 for every debounced one, and its single GPU serves about 40 a second.

What one GPU can do

• With a 1,500-token prefix (stage 2): about 37 requests a second.
• With a 2,000-token prompt (stage 3): about 28.
• Once the prefix cache supplies 70% of each prompt (stage 4): about 83.
• With 8-bit weights (stage 6): about 130.

Headroom and filtering

• GPU pools are kept near 60% busy, because waiting in a queue eats the latency budget.
• In stage 6 the trigger model skips about 30% of requests.
• About 55% of requests finish and reach the filters.

This is enough for a mid-level answer. A small model, a plugin that waits and cancels, context packed on the laptop, GPUs in each region with prompt reuse, and checks on the output. The rest of the page is what makes it a senior answer: proving quality, ten times the developers, and privacy.

Go deeperThe API, the data, follow a suggestion, and the classes5 min

The API. The plugin keeps one HTTP/2 connection open to its region’s gateway. Each request is one stream on it, so a keystroke never pays for a new connection.

POST /v1/completions
Authorization: Bearer <short-lived token from the editor sign-in>

{
  "requestId": "r-41",
  "promptFormat": 7,                // the packer's version. The gateway accepts the last few
  "language": "python",
  "path": "billing/invoice.py",     // routing key with the customer. Hashed before any log
  "prompt": "<snippets><fim_suffix>...<fim_prefix>...<fim_middle>",
  "maxTokens": 64,
  "deadlineMs": 260                 // after this, the plugin stops waiting
}

200 OK
Content-Type: text/event-stream

data: {"delta": "user in users:"}
data: {"done": true, "suggestionId": "sg-77a1", "finish": "stop", "cachedTokens": 1408}

There is no cancel endpoint. Closing the stream is the cancel, so a cancel can never arrive before its request. Events go to POST /v1/events every 30 seconds. They carry IDs, lengths and timings, never code.

The data. The prompt is never stored. It lives in memory on the laptop and on one GPU, for one request. The repo index is a small SQLite database in the plugin’s cache folder. Delete it and it is rebuilt from the files.

CREATE TABLE files   (path TEXT PRIMARY KEY, lang TEXT, mtime INTEGER, excluded INTEGER);
CREATE TABLE symbols (name TEXT, kind TEXT, path TEXT, line INTEGER, signature TEXT);
CREATE INDEX symbols_by_name ON symbols (name);
-- 20-line windows of code, with hashed words for "similar code"
CREATE TABLE windows (path TEXT, start_line INTEGER, shingles BLOB);

On the server there are only three things. Each customer’s policy (a small Postgres table, cached in every gateway). The events (a column store, by day). And the public code index, held in memory in each region.

Follow one suggestion against its budget. Use Next hop to move along:

Code completion assistant · architecture

Trace a request: One inline completion

One inline completionThe developer types on

Color keyServiceDatabaseCacheQueue or streamBackground workerModelNew in this stageAsync or occasionalHop in motionHops doneResponse

Play

PreviousNext

One inline completion: step 1 of 9

Developers to IDE plugin

The developer types `for user in` and stops. After 75 ms with no new key, the plugin decides this pause is worth a request.

75 ms of a 300 ms budget

1. Developers → IDE plugin75 ms
2. IDE plugin → Local repo index10 ms
3. IDE plugin → Completion gateway25 ms
4. Completion gateway → Regional GPU pools15 ms
5. Completion gateway → Regional GPU pools50 ms
6. Completion gateway → Post-processing5 ms
7. Post-processing → Public code index2 ms
8. Completion gateway → Event streamasync
9. Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Code completion: the core classes

Scroll or pinch to zoom, drag to pan

• Color key
• Facade
• Entity
• Value object
• Enum
• Interface
• Implementation

The trigger is the only class whose state changes on every keystroke, and it lives in the plugin. The checks are a list behind one interface, so each customer’s policy picks which ones run. Nothing on the server holds code after a request ends.

Prove it works

Interviewerminute 26

You have a new model. How do you decide to ship it?

What they are testing

Whether you test offline on held-out repos with tests you can run, then run an A/B test on what developers accept and keep. A judge model as the main metric is the red flag. Code can be run.

Code has an advantage most AI products lack. The right answer is free. Hide a piece of a real file, and ask the model to fill it. The hidden piece is the right answer, and the repo’s own tests can check the result.

Before release, offline. Three test sets:

• 20,000 hidden pieces from new repos, created after the model was trained, so it can’t have memorized them. Cursor positions are where developers really ask for suggestions.
• Tasks with unit tests. Did the filled code pass?
• Attack prompts built to coax out passwords and famous licensed code.

The new model must be no worse than today’s in each top language, and show zero secrets.

During release, online. Our own developers first, for a week. Then an A/B test at 1%, 10% and 50% of developers. We watch three numbers together:

• Acceptance: the share of shown suggestions that are accepted.
• Kept code: the share of accepted characters still there after 5 minutes.
• Accepted characters per hour.

One number alone can fool you. A model that suggests only ) is accepted often and helps nobody. Characters per hour catches that. A model whose code is accepted and then deleted fails the kept code check. Rolling back is a routing change.

Go deeperAll the release metrics2 min

Metric	How we measure it	Target	Blocks a release?
Exact match and similarity	Offline, hidden pieces, per language	No worse than today’s model	Yes, per top language
Unit tests passed	Offline, function and repo tasks	No worse than today’s model	Yes
Valid code	Offline, parse the file with the suggestion in it	At least 98%	Yes
Secrets and blocked code shown	Offline, attack set, checks on	Zero	Yes
Speed (95% of requests)	Load test at real prompt sizes, 60% busy	At most 300 ms	Yes
Acceptance	Online: accepted ÷ shown	At least 25%, at most 1 point below	Yes, in the A/B test
Kept code	Online: accepted characters still there after 5 min	At least 70%	Yes, in the A/B test
Accepted characters per hour	Online	Up, or flat	Watched

A study of GitHub’s users (Ziegler et al.) found acceptance matched how productive developers felt better than other measures. A judge model plays a small role, only for what can’t be run, such as readability.

Common mistake

Shipping on a public benchmark score

Public benchmarks have leaked into training data. And writing a function from its description is not filling the middle of a real file. A model can climb a leaderboard and lose acceptance. Test on hidden pieces of new repos.

Stress it

The design works at today’s size. Now we push it through three hard situations: ten times the developers, a slow region, and a bank asking where its code goes.

Ten times the developers

Interviewerminute 30

You’re at ten times the developers. The GPU bill is about to grow ten times. Where do you cut first?

What they are testing

Whether you cut requests and tokens before buying hardware, and check every cut against a quality metric. Shrinking the model first, with no test, is the red flag.

At ten times, the fleet heads for about 7,000 GPUs, about $500,000 a day. Gateways and checks scale by adding servers. The GPUs are the bill. Two cuts come first.

Step 6: skip requests nobody will accept. Many requests come from places where developers almost never take a suggestion: mid-word, inside a comment, right after rejecting the last one. A tiny trigger model in the plugin, a few kilobytes, scores each request from cheap signals, like the character before the cursor and the kind of line. Below a line, the request is skipped. It learns from telemetry, never from code. We assume it skips about 30%.

Run the model in 8-bit numbers. Current GPUs do 8-bit math about twice as fast as 16-bit, and reading the prompt is math-heavy. We expect about 1.5 times more requests per GPU, once the 8-bit model passes the offline tests.

Together, today’s peak needs about 315 GPUs, and ten times needs about 3,100.

Code completion assistant · architecture

Stage 6: Fewer, cheaper requests

Grow itTrace a request

Color keyServiceDatabaseCacheQueue or streamBackground workerModelNew in this stageAsync or occasional

Play

PreviousNext

Stage

123456

Completion requests per second (after debouncing):

35,000

12k average35k peak350k (10×)

Stage 6 of 7

Fewer, cheaper requests

Why this stage

At 10 times the developers the fleet heads for 7,000 GPUs.

Many requests ask in places where nobody accepts a suggestion anyway.

What we add

A tiny trigger model in the plugin that skips requests unlikely to be accepted, and 8-bit weights on the GPU pools.

What it costs

A few suggestions that would have been accepted are never requested, and every quantized model must pass the eval gate again.

At 35k completion requests per second (after debouncing), every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What the slider counts

• The slider counts completion requests after debouncing, which means waiting for a short pause in typing.
• Stage 1 has no debounce. It sends one request per keystroke, about 4 for every debounced one, and its single GPU serves about 40 a second.

What one GPU can do

• With a 1,500-token prefix (stage 2): about 37 requests a second.
• With a 2,000-token prompt (stage 3): about 28.
• Once the prefix cache supplies 70% of each prompt (stage 4): about 83.
• With 8-bit weights (stage 6): about 130.

Headroom and filtering

• GPU pools are kept near 60% busy, because waiting in a queue eats the latency budget.
• In stage 6 the trigger model skips about 30% of requests.
• About 55% of requests finish and reach the filters.

Step 7: a separate pool for big models. Product then adds chat and multi-file edits. Their prompts are 10,000 tokens or more, on a bigger model. On the completion GPUs, one such prompt would stall every suggestion in its batch. So they get their own GPUs behind the same gateway. A 300 ms product and a 2 second product never compete for one GPU.

Code completion assistant · architecture

Stage 7: A separate pool for big models

Grow itTrace a requestBreak it

Color keyServiceDatabaseCacheQueue or streamBackground workerModelNew in this stageAsync or occasional

Play

PreviousNext

Stage

1234567

Completion requests per second (after debouncing):

35,000

12k average35k peak350k (10×)

Stage 7 of 7

A separate pool for big models

Why this stage

Product adds chat and multi-file edits.

Their prompts are 10,000 tokens or more, and on the completion GPUs they would wreck the 300 ms p95.

What we add

A separate pool for a larger model, reached through the same gateway, with its own capacity and its own limits.

What it costs

Two fleets to size and run instead of one.

At 35k completion requests per second (after debouncing), every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What the slider counts

• The slider counts completion requests after debouncing, which means waiting for a short pause in typing.
• Stage 1 has no debounce. It sends one request per keystroke, about 4 for every debounced one, and its single GPU serves about 40 a second.

What one GPU can do

• With a 1,500-token prefix (stage 2): about 37 requests a second.
• With a 2,000-token prompt (stage 3): about 28.
• Once the prefix cache supplies 70% of each prompt (stage 4): about 83.
• With 8-bit weights (stage 6): about 130.

Headroom and filtering

• GPU pools are kept near 60% busy, because waiting in a queue eats the latency budget.
• In stage 6 the trigger model skips about 30% of requests.
• About 55% of requests finish and reach the filters.

Go deeperMore ways to save GPUs1 min

Lever	Saves	Catch
Scale with the daily curve	Up to half the GPU hours	Loading a model takes minutes. Keep warm spares
A smaller model	Another 2 to 3 times	Quality must hold in every top language
Prompt budgets per language	Prompt reading where context helps little	Needs a test per language
Caching whole suggestions	Almost nothing	Prompts are nearly unique. Reusing what still fits works better

A fleet sized for the peak all day costs about 3 times the average load. So scaling with the daily curve is the first lever once the design is stable.

A slow region

Interviewerminute 34

Your 95th percentile in Europe is 450 ms. Find the time.

What they are testing

Whether you measure each hop, joining the plugin’s timings with the server’s by request ID, and know which hops can change. Tuning the model first when the network is the problem is the red flag.

Go back to the budget drawing. Each hop has its own metric. A slow region almost always comes from one of three places, and the model is rarely the first suspect:

• Requests leaving the region. A routing or DNS mistake sends European developers to the US. The network hop jumps from 40 ms to 150 ms or more.
• Queueing. The pool runs too busy, so requests wait for a slot.
• Cold caches. Routing broke, or GPUs restarted, so every prompt is read from scratch.

Every request also carries a deadline. If it is still waiting in a queue when the plugin would stop waiting, the gateway drops it. A suggestion that can’t arrive in time is pure cost.

Go deeperEvery hop, and how to cut it2 min

Hop	Typical	What sets it	How to cut it
Wait for a pause	75 ms	A setting	Tune per language. Skip after a block opens
Packing	10 ms	Index lookups on the laptop	Prepare when the cursor enters a function
Network, both ways	40 ms	Distance to the region	GPUs in each region. One warm connection
Gateway and queue	10 ms	Policy, routing, a free slot	Cached policy. Pools about 60% busy
Reading the prompt	10 ms	New prompt tokens	Prompt reuse and the stable order
Writing the answer	50 ms	Tokens × time per step	Stop at the end of the line. Guess ahead
Checks and drawing	15 ms	Checks per line, drawing	Release line by line
Total	210 ms		Slow cases add queueing, cold caches, long lines

A bank asks where its code goes

Interviewerminute 37

A bank’s security team asks where their code goes. What do you tell them?

What they are testing

Whether you can answer precisely: what leaves the laptop, where it is processed, how long it lives, who can see it, and what is never shared. “It’s encrypted” on its own is the red flag.

Follow one line of the bank’s code:

One line of the bank's code, from laptop to gone

Scroll or pinch to zoom, drag to pan

• Color key
• On the laptop
• On the way
• On the GPU
• Afterwards

Requests never move to another region, even under overload. The bank gets fewer suggestions instead. Crash dumps are off on the GPU servers, so a crash can’t write code to disk.

Common mistake

Sharing the prefix cache across customers

Open-source boilerplate is the same at every company, so a shared cache looks like free speed. But a reused prompt is measurably faster. That timing can reveal whether another customer recently sent the same text. Keep the cache per customer, as the routing code does.

Go deeperFour ways to deploy, by how much isolation the customer needs1 min

Option	Isolation	Cost	Pick when
Shared regional GPUs, cache per customer	Logical. Nothing kept	Lowest. Batches mix customers	The default
Dedicated GPUs per customer	Physical	The customer pays for idle nights	Regulated customers who require it
Inside the customer’s own cloud account	Their network and keys	They run it. We ship it	Code must never cross their network edge
On the device	Nothing leaves	A small model, slow on many laptops	Air-gapped or offline work

When things break

Interviewerminute 40

The European GPU pool loses half its capacity. What do developers see?

What they are testing

Whether you degrade in steps and keep the rules: completion is optional, a late suggestion is never shown, and a customer never leaves its region. Moving a European bank to the US is the red flag.

Completion may get worse, or disappear for a while. It must never freeze the editor, show something late, or move code where it isn’t allowed.

A pool that runs short gives up quality in steps, each cheaper than the last. First a 1,024-token budget. Then single-line suggestions only. Then a 150 ms wait for a pause. Then no suggestions for some requests. Moving to another region happens only where the customer allows it.

Click any box to break it and see what developers would notice:

Code completion assistant · architecture

Break it: what fails, who notices, what users see

Color keyServiceDatabaseCacheQueue or streamBackground workerModelNew in this stageAsync or occasional

Break

IDE pluginLocal repo indexCompletion gatewayA GPU poolPost-processingPublic code indexEvent streamQuality pipelineMetrics storeLarge model pool

Failure: click any box to break it

A GPU pool goes down

What breaks

Too few GPUs for the load, or a region’s pool is down.

How we notice

Queue time, KV cache use and p95 latency per pool.

What users see

Shorter or fewer suggestions, then none.

Never slow ones: a late suggestion is dropped.

What we do

Degrade in steps: smaller context budget, single line only, longer debounce, then skip.

Spill to another region only if the tenant allows it.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Go deeperEvery failure, and what we do2 min

Part or risk	Failure	What developers see	What we do
GPU pool	Short of capacity, or down	Shorter, fewer suggestions, then none	The steps above. Move region only where allowed
Gateway	A server down	Seconds without suggestions	No state, so plugins just reconnect
Plugin	Crash or hang	No suggestions. Typing still works	Work off the main thread. The editor restarts it
Local index	Old, or still building	Weaker suggestions	Rebuild in the background
Public code check	Errors	Long suggestions held back where required	Fail closed where the policy requires the check
The model	Calls a function that doesn’t exist	Code that doesn’t compile	Names in the prompt. Check names against the index
The model	Unsafe code, like SQL built from strings	A security hole accepted	Security tests before release. The developer still reviews
Repo content	A dependency’s comment gives the model orders	A harmful suggestion	No tools, so a person must accept it. Skip vendored folders
A new model	Gets worse	Lower acceptance, or slower	A/B guardrails. Roll back by routing

No suggestion is a fine failure. A late one, a wrong one, or one served from the wrong region is not.

Wrap up

The whole design

Here is everything we built, in the order we built it. Use the buttons to step through it.

Code completion assistant · architecture

The full design, stage by stage

Color keyServiceDatabaseCacheQueue or streamBackground workerModelNew in this stageAsync or occasional

Play

PreviousNext

Stage

1234567

Completion requests per second (after debouncing):

35,000

12k average35k peak350k (10×)

Stage 7 of 7

A separate pool for big models

Why this stage

Product adds chat and multi-file edits.

Their prompts are 10,000 tokens or more, and on the completion GPUs they would wreck the 300 ms p95.

What we add

A separate pool for a larger model, reached through the same gateway, with its own capacity and its own limits.

What it costs

Two fleets to size and run instead of one.

At 35k completion requests per second (after debouncing), every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.The whole design. Step back through the stages to rebuild it from memory.

The numbers behind the busy bars

What the slider counts

• The slider counts completion requests after debouncing, which means waiting for a short pause in typing.
• Stage 1 has no debounce. It sends one request per keystroke, about 4 for every debounced one, and its single GPU serves about 40 a second.

What one GPU can do

• With a 1,500-token prefix (stage 2): about 37 requests a second.
• With a 2,000-token prompt (stage 3): about 28.
• Once the prefix cache supplies 70% of each prompt (stage 4): about 83.
• With 8-bit weights (stage 6): about 130.

Headroom and filtering

• GPU pools are kept near 60% busy, because waiting in a queue eats the latency budget.
• In stage 6 the trigger model skips about 30% of requests.
• About 55% of requests finish and reach the filters.

The decisions, and what each one costs

Decision	We chose	We gave up	We’d choose differently if
Model	A small fill-in-the-middle model on GPUs	The quality of a large model	The budget were 1 second, not 300 ms
Where it runs	Our own GPUs in each region	No fleet to run	Volume were small
How often we ask	Wait for a pause, reuse, a trigger model	A few suggestions that would have been taken	GPUs were free
Context	2,048 tokens picked on the laptop	Whole files. Packing on the server	Repos were too big for a laptop index
Prompt order	Most stable first, code above the cursor last	Ordering by score	The model server couldn’t reuse prompts
Routing	Hash of customer and file, with a busy cap	Perfectly even load	Requests shared nothing
Cancelling	All the way to the GPU, with deadlines	Simplicity in every layer	Cancels were rare
Privacy	Nothing kept, one region, cache per customer	Learning from failures. A shared cache	A customer chose to share data

How to spend 45 minutes

• 0 to 6. Speed, privacy, repo size, chat or not. Then the three tools.
• 6 to 12. The numbers: 35,000 requests a second, 2,000 prompt tokens against 24 out, about 700 GPUs with prompt reuse. The 300 ms budget.
• 12 to 26. Build it: one GPU, wait and cancel, context, regions with routing by file, then checks and measuring. A mid-level answer can stop here.
• 26 to 30. How a new model ships.
• 30 to 42. Stress it: ten times, a slow region, and the bank’s question.
• 42 to 45. Losing half a pool, then your own questions.

Never skip cancelling, the latency budget, or prompt reuse.

Follow-up questions

Interviewer asksAcceptance went up 3 points, but kept code fell. Ship it?

No. Developers take more suggestions and then delete them, which is more time reviewing and fixing. If kept characters per hour fell, the new model is worse. The kept code check exists for exactly this.

Go deeperMore follow-up questions3 min

A customer has a monorepo with 5 million files. What changes? The laptop index can’t hold it, and most of it doesn’t matter to one developer. Index the files they touch and their imports on the laptop. For the rest, build an index on our servers in the customer’s region, with their permission. Cache its answers, because it adds a network hop to the 10 ms packing budget.

Why not call a hosted model API? For a small product, do. At 2 trillion prompt tokens a day, we need control over region, per-customer caching, and stopping mid-answer. At this volume, our own GPUs usually win, and running them is the price.

How does it learn a customer’s code without training on it? Mostly through context. The names and similar code the packer finds already carry the customer’s style. For customers who opt in, train a small add-on to the model on their code, keep it in their region, and load it only for their requests.

When do you show multi-line suggestions? On an empty line after a line that opens a block, like a function signature. There, developers take whole blocks. The answer streams line by line, each line checked, so the first line still arrives in time. Elsewhere, one line only.

A developer pastes an API key into a file. Could it reach someone else’s suggestion? No. Nothing is stored or trained on, and the cache is per customer and gone in minutes. Inside the same customer it could appear as context, so the secret check drops any suggestion shaped like a key.

The developer switches files. Does prompt reuse still help? Not for the first request. That one reads about 2,000 tokens cold, about 25 ms slower, still inside the budget. Routing by file means the old file’s prompt may still be on its GPU when the developer comes back.

Further reading

• On this site: How LLM inference works (prompt reading, the prefix cache, guessing ahead), retrieval-augmented generation, vector search, evaluation and guardrails, and lab step 1, where you measure prompt reuse yourself. Related problems: a ChatGPT-style assistant and RAG over company docs.
• Efficient Training of Language Models to Fill in the Middle, Bavarian et al., 2022: how code models learn to fill a gap.
• Evaluating Large Language Models Trained on Code, Chen et al., 2021: HumanEval and the pass@k metric.
• Productivity Assessment of Neural Code Completion, Ziegler et al., 2022: which usage measures track how productive developers feel.