Design a hosted agent platform
Design a managed agents platform that runs thousands of AI agents for other companies, step by step. Durable runs with an event log, microVM sandboxes, an MCP tool gateway with per-tenant credentials, fair scheduling, token budgets and model provider fallback, with code in Java, Python, C#, Go and TypeScript.
“Design a platform that runs AI agents for other companies. Customers define an agent, give it tools, and start runs through an API. Some runs take ten seconds, some take two hours.”
A company describes its agent: a model, a system prompt and a list of tools. The platform runs it, thousands of times at once, for thousands of companies. We call each company a tenant.
The agents belong to the tenants. The platform owns everything around them: where the loop runs, what survives a crash, where model-written code runs, whose passwords a tool call uses, and who gets the next free slot when everyone is busy. So this is an infrastructure problem with a model in the middle. We start with one server running one loop in memory, and grow it only when a number forces us to.
The answer in one minute
Section titled “The answer in one minute”If you only have a minute, this is the design. The rest of the page explains each part.
- Every run is a log of steps. Each model reply and tool result is saved before the run moves on. Any worker can pick up any run after a crash, from its last step.
- A tool call that changes something gets a key from its place in the log. A resumed run sends the same key, so the tool does the refund once, not twice.
- Model-written code runs in its own tiny virtual machine, one per run, never reused, with no network by default.
- Every tool call goes through one gateway. It checks the allowed list, asks for approvals, and adds the tenant’s credentials. Passwords never enter a prompt.
- Slots and model capacity are shared fairly by tenant, so one customer’s burst never makes others wait.
Scope it
Section titled “Scope it”A few questions change the design a lot. For the rest, we assume an answer and move on.
Design a platform that runs other companies’ AI agents.
Whether you find the questions that decide the shape: who writes the loop, how long runs last, whether agents run code, and how isolated tenants must be. Talking about prompts and models before asking who the tenants are reads as a chatbot answer to an infrastructure question.
These five questions matter most:
- How many tenants and runs? About 2,000 companies, 500,000 runs a day.
- How long does a run last? Seconds to hours, 4 minutes on average. Some wait days for a person to approve a step. If every run finished in 30 seconds, it could run inside one API call, with nothing saved.
- Who writes the agent loop? We do. Tenants give us settings: model, prompt, tools, limits.
- Which tools? Tenants’ own tool servers, plus built-in code running and a browser.
- How isolated must tenants be? One shared fleet, but no tenant may ever reach another’s data. EU tenants’ data stays in the EU.
We leave out writing agents for tenants, training models, and a no-code agent builder.
Go deeperThe full requirements, and more questions3 min
Must do. Versioned agent settings, where each run is pinned to one version. Start, stream, cancel and get the result of runs. A run may wait up to 7 days for approval. Tools with the tenant’s credentials. Code and a browser in a sandbox. Limits per tenant (runs at once, tokens per minute, monthly spend) and per run. A trace, replay, usage and cost for every run.
Must be true.
- Durability. A saved step is never lost. A run survives losing any worker, machine or zone.
- Side effects. A tool call that changes something happens exactly once when the tool accepts a key, and at most once otherwise.
- Isolation. No tenant can reach another’s data, secrets or network. One sandbox per run, never reused.
- Fairness. A tenant within its plan waits under 5 seconds for 95% of runs, whatever others do.
- Speed. Creating a run takes under 200 ms. Platform work per step stays under 150 ms, not counting model and tool time. A sandbox is ready in under 300 ms.
- Availability. 99.95% for the API. Under 0.1% of runs fail because of us.
- Billing. Usage per tenant within 1% of the model providers’ invoices.
| More questions | We assume | If the answer were different |
|---|---|---|
| Which models? | Several hosted providers. Tenants may bring keys | Our own models make GPUs our problem (see the chat assistant) |
| How do tenants pay? | Per token and sandbox minute | A flat price per seat means hard caps become mandatory |
| Memory across runs? | Out of scope. Tenants pass context in | A memory store per agent, with its own privacy rules |
| Agents starting other agents? | Yes, each child is its own run | See orchestration |
How much AI it needs
Section titled “How much AI it needs”Where is the AI in this system, and what exactly does your platform own?
Whether you see that the platform itself is infrastructure, and draw a clear line between what it owns and what the tenant owns. Proposing to fine-tune a model for customers, or to judge their answers, is the red flag. The platform can’t know what a good answer is in every domain.
The AI lives in the tenants’ agents. The platform’s own decisions, like which run goes next, whether a tool call is allowed, or what a tenant has spent, are about money, security and fairness. So they are plain code. The real question is how much of the agent the platform takes over:
- Color key
- Too little
- What we run
- Picked
- Too far
We pick a managed loop. Every model call and tool call passes through our code, so we can save, meter, block and replay it. That is what makes crash recovery, fairness and isolation possible.
So the line is clear. The platform owns the loop, the run state, sandboxes, the tool gateway and budgets. The tenant owns the prompt, the model, its tool servers, approvals and what a good answer is. Tenants with unusual loops can run their code as a tool in a sandbox.
The numbers that decide it
Section titled “The numbers that decide it”How big is this? What does a run cost?
Whether you size runs in flight (not just requests per second), model calls and tokens, and notice which cost dominates. Sizing GPUs for a platform that buys model calls from providers, or stopping at runs per day, both read badly.
- Runs a day, all tenants
- 500,000
- Peak over average
- 3×
- Minutes per run, mean
- 4
- Model calls per run
- 15
- Input tokens per call
- 8000
- Output tokens per call
- 400
- Dollars per million input tokens
- 3
- Dollars per million output tokens
- 15
- Runs that need a sandbox
- 40%
- Sandbox cold start, seconds
- 2
- Event log per run
- 50 KB
- Days of hot retention
- 30
500,000 runs a day ÷ 86,400 seconds, and about 3 times that at peak.
So we starting runs is easy. A small API and a queue handle this.
17 runs a second × 240 seconds per run. (Runs in flight = arrival rate × time each one takes. This is called Little’s law.) A two-hour run holds a slot as long as 30 average ones.
So we size workers by runs held, not by requests. A worker waiting on the model holds about 200 runs, so about 21 workers.
17 runs a second × 15 model calls per run. Each call sends the conversation so far, about 8,000 tokens, and gets 400 back.
So we spread calls over several provider accounts. One account allows about 100 calls a second at these sizes.
15 calls × (8,000 input tokens at $3 per million + 400 output at $15 per million) is 15 × 3 cents, at typical prices for a mid-sized model. A sandbox for 4 minutes is well under a cent.
So we put token budgets on every tenant and every run. Servers are a rounding error next to tokens.
40% of the 4,200 runs in flight hold a sandbox for their whole life.
So we run about 55 sandbox machines, and keep about 140 ready per region so a burst never waits.
Two numbers decide this platform. It is sized by runs in flight, not requests per second. And 95% of the cost is tokens. So budgets and prompt reuse matter far more than packing machines tightly.
Go deeperThe calculator, the event log, and the warm pool2 min
- Runs started, peak
- 17/s
- Runs in flight, peak
- 4,200
- About 21 workers at 200 runs each: a worker spends its time waiting on the model.
- Model calls, peak
- 260/s
- Past one provider account (about 100 calls/s at these sizes): a model gateway spreads calls over accounts and providers.
- Tokens per second, peak
- 2,200,000/s
- Input is most of it, because every call resends the conversation so far.
- Tokens a day
- 63 billion
- Model cost per run, in cents
- 45
- Before prompt caching. Sandboxes, workers and storage add well under one cent.
- Model cost per 1,000 runs, in dollars
- 450
- Model spend a day, in dollars
- 230,000
- Every dollar belongs to a tenant: metering must match provider invoices.
- Sandboxes in use, peak
- 1,700
- About 55 hosts at 50 microVMs each, kept 60% busy.
- Warm sandboxes per cell (10× burst)
- 140
- Covers ten times the peak start rate for one cold start. Small next to the sandboxes in use.
- Event log writes, peak
- 780/s
- One Postgres primary takes this with room to spare.
- Event log, hot retention
- 750 GB
- Fits one database node.
The event log. About 3 events per model call: 780 writes a second. At about 50 KB per run (big outputs live in object storage), 30 days is 750 GB. One Postgres primary per region handles that.
The warm pool. Sandboxes start about 7 times a second. A cold start takes 2 seconds. To cover a burst ten times bigger for one cold start: 7 × 10 × 2 = 140 ready.
Change an assumption and see what flips. Ten times the runs puts the event log past one primary. Runs of an hour instead of 4 minutes multiply runs in flight and sandboxes by 15, without changing model calls a second.
Build the design
Section titled “Build the design”We start with one server. Then each step fixes the biggest problem left. The drawing grows with each step.
Step 1. One server, one loop
Section titled “Step 1. One server, one loop”Start simple. One customer, one agent. What runs it?
Whether you can state the loop and its budgets in two minutes and name what is naive. Drawing sandboxes and queues before saying what one step does reads as memorized.
An agent server runs the agent loop in memory. Ask the model. Run the tools it asks for. Add the results to the conversation. Repeat. Budgets are checked every turn, because a confused model will loop until someone stops it.
Stage 1: One server, one loop
Stage 1 of 6
One server, one loop
Why this stage
Nothing yet.
This is the smallest design that runs one tenant’s agent end to end.
What we add
An agent server that takes a task, runs the loop in memory (model call, tool call, repeat) with a step budget, and returns the answer.
What it costs
A crash or deploy loses every run in flight, one server holds about 200 runs, and tools run inside our own process.
At 17 runs started per second, Agent workers is 2125% busy. Stage 2 is the fix.
The numbers behind the busy bars
What one run looks like
- A run lasts about 4 minutes. It makes 15 model calls and about 10 tool calls.
- Each step writes about 3 events to the database.
What one machine can do
- A worker holds about 200 runs at once, because it mostly waits on the model. So one worker takes about 0.8 new runs a second.
- One Postgres primary takes about 5,000 small writes a second.
- A sandbox host runs about 50 microVMs.
Outside limits
- One model provider account allows about 100 calls a second at our prompt sizes. The stage 5 gateway pools about 10 accounts across providers.
Why sandboxes add up
- About 40% of runs keep a sandbox for their whole life. That means about 96 sandboxes open for each run started per second.
// Limits the tenant sets per agent. They are all that stands between a confused model and// an endless bill, so the loop checks them on every turn, not at the end.record Budget(int maxSteps, int maxTokens) {}
// Stage 1: one tenant, one process, everything in memory.class AgentLoop { private final Model model; private final Map<String, Tool> tools; private final Budget budget;
AgentLoop(Model model, Map<String, Tool> tools, Budget budget) { this.model = model; this.tools = tools; this.budget = budget; }
String run(String task) { // The whole state of the run is this list. If the process dies, the run is gone: // that is the flaw stage 2 fixes. List<Message> messages = new ArrayList<>(List.of(Message.user(task))); int tokens = 0; for (int step = 0; step < budget.maxSteps(); step++) { Reply reply = model.complete(messages, tools.values()); tokens += reply.tokens(); messages.add(reply.message()); if (reply.toolCalls().isEmpty()) { return reply.text(); // no tool asked for: the model has answered } for (ToolCall call : reply.toolCalls()) { messages.add(Message.tool(call.id(), call(call))); } if (tokens > budget.maxTokens()) { // Stop between steps, never inside one, so the transcript stays valid. return "Stopped: token budget used up."; } } return "Stopped: step budget used up."; }
private String call(ToolCall call) { Tool tool = tools.get(call.name()); if (tool == null) { // Models do ask for tools that don't exist. Tell the model; don't crash the run. return "Unknown tool " + call.name() + "."; } try { // Second flaw: the tool runs inside our process, with our permissions. return tool.invoke(call.args()); } catch (Exception e) { // A failed tool is information the model can act on, not a platform failure. return "Tool error: " + e.getMessage(); } }}# Limits the tenant sets per agent. They are all that stands between a confused model and# an endless bill, so the loop checks them on every turn, not at the end.@dataclassclass Budget: max_steps: int = 30 max_tokens: int = 200_000
# Stage 1: one tenant, one process, everything in memory.class AgentLoop: def __init__(self, model: Model, tools: dict[str, Tool], budget: Budget): self.model, self.tools, self.budget = model, tools, budget
def run(self, task: str) -> str: # The whole state of the run is this list. If the process dies, the run is gone: # that is the flaw stage 2 fixes. messages = [Message("user", task)] tokens = 0 for _ in range(self.budget.max_steps): reply = self.model.complete(messages, list(self.tools.values())) tokens += reply.tokens messages.append(reply.message) if not reply.tool_calls: return reply.text # no tool asked for: the model has answered for call in reply.tool_calls: messages.append(Message("tool", self._call(call), call.id)) if tokens > self.budget.max_tokens: # Stop between steps, never inside one, so the transcript stays valid. return "Stopped: token budget used up." return "Stopped: step budget used up."
def _call(self, call: ToolCall) -> str: tool = self.tools.get(call.name) if tool is None: # Models do ask for tools that don't exist. Tell the model; don't crash the run. return f"Unknown tool {call.name}." try: # Second flaw: the tool runs inside our process, with our permissions. return tool.invoke(call.args) except Exception as e: # A failed tool is information the model can act on, not a platform failure. return f"Tool error: {e}"// Limits the tenant sets per agent. They are all that stands between a confused model and// an endless bill, so the loop checks them on every turn, not at the end.record Budget(int MaxSteps = 30, int MaxTokens = 200_000);
// Stage 1: one tenant, one process, everything in memory.class AgentLoop(IModel model, Dictionary<string, ITool> tools, Budget budget){ public string Run(string task) { // The whole state of the run is this list. If the process dies, the run is gone: // that is the flaw stage 2 fixes. var messages = new List<Message> { new("user", task) }; var tokens = 0; for (var step = 0; step < budget.MaxSteps; step++) { var reply = model.Complete(messages, tools.Values); tokens += reply.Tokens; messages.Add(reply.Message); if (reply.ToolCalls.Count == 0) return reply.Text; // no tool asked for: the model has answered foreach (var call in reply.ToolCalls) messages.Add(new("tool", Call(call), call.Id)); if (tokens > budget.MaxTokens) // Stop between steps, never inside one, so the transcript stays valid. return "Stopped: token budget used up."; } return "Stopped: step budget used up."; }
string Call(ToolCall call) { if (!tools.TryGetValue(call.Name, out var tool)) // Models do ask for tools that don't exist. Tell the model; don't crash the run. return $"Unknown tool {call.Name}."; try { // Second flaw: the tool runs inside our process, with our permissions. return tool.Invoke(call.Args); } catch (Exception e) { // A failed tool is information the model can act on, not a platform failure. return $"Tool error: {e.Message}"; } }}
// The contracts the loop uses, reduced to what it needs.record Message(string Role, string Content, string? ToolCallId = null);record ToolCall(string Id, string Name, JsonElement Args);record Reply(Message Message, string Text, IReadOnlyList<ToolCall> ToolCalls, int Tokens);interface IModel { Reply Complete(IReadOnlyList<Message> messages, IEnumerable<ITool> tools); }interface ITool { string Invoke(JsonElement args); }// Budget holds the limits the tenant sets per agent. They are all that stands between a// confused model and an endless bill, so the loop checks them on every turn, not at the end.type Budget struct { MaxSteps int // 30 MaxTokens int // 200,000}
// AgentLoop is stage 1: one tenant, one process, everything in memory.type AgentLoop struct { Model Model Tools map[string]Tool Budget Budget}
func (a *AgentLoop) Run(ctx context.Context, task string) (string, error) { // The whole state of the run is this slice. If the process dies, the run is gone: // that is the flaw stage 2 fixes. messages := []Message{{Role: "user", Content: task}} tokens := 0 for step := 0; step < a.Budget.MaxSteps; step++ { reply, err := a.Model.Complete(ctx, messages, a.Tools) if err != nil { return "", err } tokens += reply.Tokens messages = append(messages, reply.Message) if len(reply.ToolCalls) == 0 { return reply.Text, nil // no tool asked for: the model has answered } for _, call := range reply.ToolCalls { out := a.call(ctx, call) messages = append(messages, Message{Role: "tool", Content: out, ToolCallID: call.ID}) } if tokens > a.Budget.MaxTokens { // Stop between steps, never inside one, so the transcript stays valid. return "Stopped: token budget used up.", nil } } return "Stopped: step budget used up.", nil}
func (a *AgentLoop) call(ctx context.Context, call ToolCall) string { tool, ok := a.Tools[call.Name] if !ok { // Models do ask for tools that don't exist. Tell the model; don't crash the run. return fmt.Sprintf("Unknown tool %s.", call.Name) } // Second flaw: the tool runs inside our process, with our permissions. out, err := tool.Invoke(ctx, call.Args) if err != nil { // A failed tool is information the model can act on, not a platform failure. return "Tool error: " + err.Error() } return out}// Limits the tenant sets per agent. They are all that stands between a confused model and// an endless bill, so the loop checks them on every turn, not at the end.interface Budget { maxSteps: number; // 30 maxTokens: number; // 200,000}
// Stage 1: one tenant, one process, everything in memory.class AgentLoop { constructor( private readonly model: Model, private readonly tools: Map<string, Tool>, private readonly budget: Budget, ) {}
async run(task: string): Promise<string> { // The whole state of the run is this array. If the process dies, the run is gone: // that is the flaw stage 2 fixes. const messages: Message[] = [{ role: 'user', content: task }]; let tokens = 0; for (let step = 0; step < this.budget.maxSteps; step++) { const reply = await this.model.complete(messages, [...this.tools.values()]); tokens += reply.tokens; messages.push(reply.message); if (reply.toolCalls.length === 0) { return reply.text; // no tool asked for: the model has answered } for (const call of reply.toolCalls) { messages.push({ role: 'tool', content: await this.call(call), toolCallId: call.id }); } if (tokens > this.budget.maxTokens) { // Stop between steps, never inside one, so the transcript stays valid. return 'Stopped: token budget used up.'; } } return 'Stopped: step budget used up.'; }
private async call(call: ToolCall): Promise<string> { const tool = this.tools.get(call.name); if (!tool) { // Models do ask for tools that don't exist. Tell the model; don't crash the run. return `Unknown tool ${call.name}.`; } try { // Second flaw: the tool runs inside our process, with our permissions. return await tool.invoke(call.args); } catch (e) { // A failed tool is information the model can act on, not a platform failure. return `Tool error: ${e instanceof Error ? e.message : String(e)}`; } }}It has three problems. A crash or deploy loses every run in flight. Tools run inside our process, with our permissions. And one server holds about 200 runs, while the peak needs 4,200.
Step 2. Runs that survive crashes
Section titled “Step 2. Runs that survive crashes”A run takes two hours and you deploy twice a day. What happens to it?
Whether you save the run’s state step by step, and explain what a resumed run does differently. “Put it on a queue and retry”, without saying what is saved, is the red flag. The retry starts from scratch and repeats every side effect.
Runs must outlive the machines that run them. So a run becomes a log of saved steps. This is called durable execution (see deployment models).
- The API saves the run and its first event, and puts the run’s ID on a queue.
- Any worker takes it, with a lease: a claim that expires after 30 seconds unless renewed.
- The worker reads the run’s log, rebuilds the conversation, and takes the next step.
- Every model reply and tool result is saved before the run moves on.
Stage 2: Durable runs
Stage 2 of 6
Durable runs
Why this stage
Runs last from seconds to hours and some wait days for approval.
We deploy daily.
At the 17 runs/s peak, about 4,200 runs are in flight: 21 servers’ worth.
What we add
A runs API, a run queue with leases, a pool of stateless workers, and an event log that records every step, so any worker can resume any run.
What it costs
Every step writes to the log, and a resumed step may repeat a tool call: idempotency keys become mandatory.
At 17 runs started per second, Model providers is 255% busy. Stage 3 is the fix.
The numbers behind the busy bars
What one run looks like
- A run lasts about 4 minutes. It makes 15 model calls and about 10 tool calls.
- Each step writes about 3 events to the database.
What one machine can do
- A worker holds about 200 runs at once, because it mostly waits on the model. So one worker takes about 0.8 new runs a second.
- One Postgres primary takes about 5,000 small writes a second.
- A sandbox host runs about 50 microVMs.
Outside limits
- One model provider account allows about 100 calls a second at our prompt sizes. The stage 5 gateway pools about 10 accounts across providers.
Why sandboxes add up
- About 40% of runs keep a sandbox for their whole life. That means about 96 sandboxes open for each run started per second.
record Event(int seq, String type, Map<String, Object> data) {}
// Another worker already wrote this slot of the log.class ConflictException extends RuntimeException {}
interface EventLog { List<Event> read(String runId);
// INSERT INTO run_events (run_id, seq, type, data). The primary key (run_id, seq) throws // ConflictException when the slot is taken, so the log doubles as the run's lock. void append(String runId, int seq, String type, Map<String, Object> data);}
// Stage 2: a run is its event log. Any worker can load any run and take the next step,// because everything the run knows was written down before it was acted on.class DurableRun { private final EventLog log; private final Model model; private final ToolCaller tools; private final String runId; private final List<Event> events;
DurableRun(EventLog log, Model model, ToolCaller tools, String runId) { this.log = log; this.model = model; this.tools = tools; this.runId = runId; this.events = new ArrayList<>(log.read(runId)); // resuming and starting are one path }
private void append(String type, Map<String, Object> data) { int seq = events.size() + 1; // the next free slot; whoever writes it first owns the step log.append(runId, seq, type, data); // throws ConflictException for a stale worker events.add(new Event(seq, type, data)); }
// Takes one step. Returns false once the run has finished. boolean advance() { if (events.get(events.size() - 1).type().equals("RunFinished")) return false; Event reply = null; for (Event e : events) if (e.type().equals("ModelReplied")) reply = e; if (reply != null) { Set<Object> done = new HashSet<>(); for (Event e : events.subList(reply.seq(), events.size())) if (e.type().equals("ToolCompleted")) done.add(e.data().get("key")); @SuppressWarnings("unchecked") List<ToolCall> calls = (List<ToolCall>) reply.data().get("calls"); for (int i = 0; i < calls.size(); i++) { // The key comes from the reply's slot in the log, so it is identical on every // retry. A tool that already acted on this key returns its first result. String key = runId + ":" + reply.seq() + ":" + i; if (!done.contains(key)) { String result = tools.call(calls.get(i), key); append("ToolCompleted", Map.of("key", key, "result", result)); return true; // one effect per step, so a crash repeats at most one call } } } // Rebuilt from the log, the prompt is exactly what the crashed worker would have sent. Reply r = model.complete(Messages.from(events)); append("ModelReplied", Map.of("text", r.text(), "calls", r.toolCalls(), "tokens", r.tokens())); // saved before any tool runs if (r.toolCalls().isEmpty()) { append("RunFinished", Map.of("answer", r.text())); return false; } return true; }
// The lease (30 s, renewed every step) says which worker may run this; the log's // primary key catches the one case the lease can't: a worker that paused past it. static void work(Lease lease, DurableRun run) { try { while (lease.renew() && run.advance()) { } } catch (ConflictException e) { // Someone else owns the run now; walking away is the correct move. } }}@dataclass(frozen=True)class Event: seq: int type: str # RunCreated, ModelReplied, ToolCompleted, RunFinished, ... data: dict
# Another worker already wrote this slot of the log.class Conflict(Exception): ...
class EventLog(Protocol): def read(self, run_id: str) -> list[Event]: ...
# INSERT INTO run_events (run_id, seq, type, data). The primary key (run_id, seq) # raises Conflict when the slot is taken, so the log doubles as the run's lock. def append(self, run_id: str, seq: int, type: str, data: dict) -> None: ...
# Stage 2: a run is its event log. Any worker can load any run and take the next step,# because everything the run knows was written down before it was acted on.class DurableRun: def __init__(self, log: EventLog, model: Model, tools: ToolCaller, run_id: str): self.log, self.model, self.tools, self.run_id = log, model, tools, run_id self.events = log.read(run_id) # resuming and starting are the same code path
def _append(self, type: str, data: dict) -> None: seq = len(self.events) + 1 # the next free slot; whoever writes it first owns the step self.log.append(self.run_id, seq, type, data) # raises Conflict for a stale worker self.events.append(Event(seq, type, data))
# Takes one step. Returns False once the run has finished. def advance(self) -> bool: if self.events[-1].type == "RunFinished": return False replies = [e for e in self.events if e.type == "ModelReplied"] if replies: reply = replies[-1] done = {e.data["key"] for e in self.events[reply.seq :] if e.type == "ToolCompleted"} for i, call in enumerate(reply.data["calls"]): # The key comes from the reply's slot in the log, so it is identical on every # retry. A tool that already acted on this key returns its first result. key = f"{self.run_id}:{reply.seq}:{i}" if key not in done: result = self.tools.call(call, key) self._append("ToolCompleted", {"key": key, "result": result}) return True # one effect per step, so a crash repeats at most one call # Rebuilt from the log, the prompt is exactly what the crashed worker would have sent. reply = self.model.complete(to_messages(self.events)) self._append("ModelReplied", {"text": reply.text, "calls": reply.tool_calls, "tokens": reply.tokens}) # saved before any tool runs if not reply.tool_calls: self._append("RunFinished", {"answer": reply.text}) return False return True
def work(lease: Lease, run: DurableRun) -> None: # The lease (30 s, renewed every step) says which worker may run this; the log's # primary key catches the one case the lease can't: a worker that paused past it. try: while lease.renew() and run.advance(): pass except Conflict: pass # someone else owns the run now; walking away is the correct moverecord Event(int Seq, string Type, JsonElement Data);
// Another worker already wrote this slot of the log.class ConflictException : Exception { }
interface IEventLog{ List<Event> Read(string runId);
// INSERT INTO run_events (run_id, seq, type, data). The primary key (run_id, seq) throws // ConflictException when the slot is taken, so the log doubles as the run's lock. void Append(string runId, int seq, string type, object data);}
// Stage 2: a run is its event log. Any worker can load any run and take the next step,// because everything the run knows was written down before it was acted on.class DurableRun(IEventLog log, IModel model, IToolCaller tools, string runId){ // Resuming and starting are the same code path. readonly List<Event> events = log.Read(runId);
void Append(string type, object data) { var seq = events.Count + 1; // the next free slot; whoever writes it first owns the step log.Append(runId, seq, type, data); // throws ConflictException for a stale worker events.Add(new Event(seq, type, JsonSerializer.SerializeToElement(data))); }
// Takes one step. Returns false once the run has finished. public bool Advance() { if (events[^1].Type == "RunFinished") return false; var reply = events.LastOrDefault(e => e.Type == "ModelReplied"); if (reply is not null) { var done = events.Skip(reply.Seq).Where(e => e.Type == "ToolCompleted") .Select(e => e.Data.GetProperty("key").GetString()).ToHashSet(); var calls = reply.Data.GetProperty("calls").Deserialize<List<ToolCall>>()!; for (var i = 0; i < calls.Count; i++) { // The key comes from the reply's slot in the log, so it is identical on every // retry. A tool that already acted on this key returns its first result. var key = $"{runId}:{reply.Seq}:{i}"; if (done.Contains(key)) continue; var result = tools.Call(calls[i], key); Append("ToolCompleted", new { key, result }); return true; // one effect per step, so a crash repeats at most one call } } // Rebuilt from the log, the prompt is exactly what the crashed worker would have sent. var r = model.Complete(Messages.From(events)); Append("ModelReplied", new { text = r.Text, calls = r.ToolCalls, tokens = r.Tokens }); if (r.ToolCalls.Count == 0) { Append("RunFinished", new { answer = r.Text }); return false; } return true; }
// The lease (30 s, renewed every step) says which worker may run this; the log's primary // key catches the one case the lease can't: a worker that paused past it. public static void Work(ILease lease, DurableRun run) { try { while (lease.Renew() && run.Advance()) { } } catch (ConflictException) { // Someone else owns the run now; walking away is the correct move. } }}
// The contracts this file uses, reduced to what it needs.record ToolCall(string Id, string Name, JsonElement Args);record Reply(string Text, IReadOnlyList<ToolCall> ToolCalls, int Tokens);record Message(string Role, string Content);interface IModel { Reply Complete(IReadOnlyList<Message> messages); }interface IToolCaller { string Call(ToolCall call, string idempotencyKey); }interface ILease { bool Renew(); }static class Messages{ // Turns the log back into the conversation the model sees. public static List<Message> From(List<Event> events) => events.Select(e => new Message(e.Type, e.Data.GetRawText())).ToList();}type Event struct { Seq int Type string // RunCreated, ModelReplied, ToolCompleted, RunFinished, ... Data map[string]any}
// ErrConflict means another worker already wrote this slot of the log.var ErrConflict = errors.New("event slot already taken")
type EventLog interface { Read(ctx context.Context, runID string) ([]Event, error) // Append runs INSERT INTO run_events (run_id, seq, type, data). The primary key // (run_id, seq) returns ErrConflict when the slot is taken, so the log doubles as the // run's lock. Append(ctx context.Context, runID string, seq int, typ string, data map[string]any) error}
// DurableRun is stage 2: a run is its event log. Any worker can load any run and take the// next step, because everything the run knows was written down before it was acted on.type DurableRun struct { log EventLog model Model tools ToolCaller runID string events []Event // loaded with log.Read: resuming and starting are the same code path}
func (r *DurableRun) append(ctx context.Context, typ string, data map[string]any) error { seq := len(r.events) + 1 // the next free slot; whoever writes it first owns the step if err := r.log.Append(ctx, r.runID, seq, typ, data); err != nil { return err // ErrConflict for a stale worker } r.events = append(r.events, Event{seq, typ, data}) return nil}
// Advance takes one step. It returns false once the run has finished.func (r *DurableRun) Advance(ctx context.Context) (bool, error) { if r.events[len(r.events)-1].Type == "RunFinished" { return false, nil } var reply *Event for i := range r.events { if r.events[i].Type == "ModelReplied" { reply = &r.events[i] } } if reply != nil { done := map[string]bool{} for _, e := range r.events[reply.Seq:] { if e.Type == "ToolCompleted" { done[e.Data["key"].(string)] = true } } for i, call := range reply.Data["calls"].([]ToolCall) { // The key comes from the reply's slot in the log, so it is identical on every // retry. A tool that already acted on this key returns its first result. key := fmt.Sprintf("%s:%d:%d", r.runID, reply.Seq, i) if done[key] { continue } result, err := r.tools.Call(ctx, call, key) if err != nil { return false, err } // One effect per step, so a crash repeats at most one call. return true, r.append(ctx, "ToolCompleted", map[string]any{"key": key, "result": result}) } } // Rebuilt from the log, the prompt is exactly what the crashed worker would have sent. m, err := r.model.Complete(ctx, ToMessages(r.events)) if err != nil { return false, err } saved := map[string]any{"text": m.Text, "calls": m.ToolCalls, "tokens": m.Tokens} if err := r.append(ctx, "ModelReplied", saved); err != nil { // saved before any tool runs return false, err } if len(m.ToolCalls) > 0 { return true, nil } return false, r.append(ctx, "RunFinished", map[string]any{"answer": m.Text})}
// Work drives one leased run. The lease (30 s, renewed every step) says which worker may// run it; the log's primary key catches the one case the lease can't: a worker that paused// past it.func Work(ctx context.Context, lease Lease, run *DurableRun) error { for lease.Renew(ctx) { more, err := run.Advance(ctx) if errors.Is(err, ErrConflict) { return nil // someone else owns the run now; walking away is the correct move } if err != nil || !more { return err } } return nil}interface RunEvent { seq: number; type: string; // RunCreated, ModelReplied, ToolCompleted, RunFinished, ... data: Record<string, unknown>;}
// Another worker already wrote this slot of the log.class ConflictError extends Error {}
interface EventLog { read(runId: string): Promise<RunEvent[]>; // INSERT INTO run_events (run_id, seq, type, data). The primary key (run_id, seq) throws // ConflictError when the slot is taken, so the log doubles as the run's lock. append(runId: string, seq: number, type: string, data: Record<string, unknown>): Promise<void>;}
// Stage 2: a run is its event log. Any worker can load any run and take the next step,// because everything the run knows was written down before it was acted on.class DurableRun { private constructor( private readonly log: EventLog, private readonly model: Model, private readonly tools: ToolCaller, private readonly runId: string, private readonly events: RunEvent[], ) {}
// Resuming and starting are the same code path: read the log. static async load(log: EventLog, model: Model, tools: ToolCaller, runId: string) { return new DurableRun(log, model, tools, runId, await log.read(runId)); }
private async append(type: string, data: Record<string, unknown>) { const seq = this.events.length + 1; // the next free slot; whoever writes it first owns it await this.log.append(this.runId, seq, type, data); // throws ConflictError if stale this.events.push({ seq, type, data }); }
// Takes one step. Resolves to false once the run has finished. async advance(): Promise<boolean> { if (this.events[this.events.length - 1].type === 'RunFinished') return false; const reply = this.events.filter((e) => e.type === 'ModelReplied').pop(); if (reply) { const done = new Set( this.events .slice(reply.seq) .filter((e) => e.type === 'ToolCompleted') .map((e) => e.data.key), ); const calls = reply.data.calls as ToolCall[]; for (const [i, call] of calls.entries()) { // The key comes from the reply's slot in the log, so it is identical on every // retry. A tool that already acted on this key returns its first result. const key = `${this.runId}:${reply.seq}:${i}`; if (done.has(key)) continue; const result = await this.tools.call(call, key); await this.append('ToolCompleted', { key, result }); return true; // one effect per step, so a crash repeats at most one call } } // Rebuilt from the log, the prompt is exactly what the crashed worker would have sent. const r = await this.model.complete(toMessages(this.events)); await this.append('ModelReplied', { text: r.text, calls: r.toolCalls, tokens: r.tokens }); if (r.toolCalls.length > 0) return true; // saved before any tool runs await this.append('RunFinished', { answer: r.text }); return false; }}
// The lease (30 s, renewed every step) says which worker may run this; the log's primary// key catches the one case the lease can't: a worker that paused past it.async function work(lease: Lease, run: DurableRun): Promise<void> { try { while ((await lease.renew()) && (await run.advance())) {} } catch (e) { if (!(e instanceof ConflictError)) throw e; // Someone else owns the run now; walking away is the correct move. }}Two details make this correct:
- Two workers can’t split a run. Each event goes into a numbered slot: 1, 2, 3, with no gaps. Saving fails if the slot is taken. A worker that froze past its lease tries to save, finds the slot taken, and walks away.
- A repeated tool call is safe. Each tool call’s key comes from its place in the log: the run, the reply number, and the call number. A resumed run sends the same key, and the tool returns its first result instead of acting twice.
- Color key
- Service
- Background worker
- Queue or stream
The cost is about 3 log writes per model call. And every tool that changes state needs to accept a key. “Stress it” covers tools that can’t.
Go deeperFollow a new run, and the tables behind the log3 min
Trace a request: Start a run
Start a run: step 1 of 4
Tenant apps to Runs API
POST /v1/runs with the tenant’s API key and an Idempotency-Key. The API checks the key, that the agent version exists, and the tenant’s queue depth limit.
- Tenant apps → Runs API5 ms
- Runs API → Run event log6 ms
- Runs API → Fair scheduler3 ms
- Response
CREATE TABLE runs ( run_id TEXT PRIMARY KEY, tenant_id TEXT NOT NULL, agent_id TEXT NOT NULL, agent_version INT NOT NULL, -- the fixed settings the run is pinned to status TEXT NOT NULL, -- queued, running, waiting, finished, failed, cancelled lease_owner TEXT, lease_expires_at TIMESTAMPTZ, -- 30 s, renewed every step last_seq INT NOT NULL DEFAULT 0, tokens_in BIGINT NOT NULL DEFAULT 0, tokens_out BIGINT NOT NULL DEFAULT 0, cost_micros BIGINT NOT NULL DEFAULT 0, idempotency_key TEXT, created_at TIMESTAMPTZ NOT NULL, UNIQUE (tenant_id, idempotency_key));
CREATE TABLE run_events ( run_id TEXT NOT NULL, seq INT NOT NULL, -- 1, 2, 3 ... with no gaps type TEXT NOT NULL, -- RunCreated, ModelReplied, ToolCompleted, ... data JSONB NOT NULL, -- up to 8 KB here. Larger outputs by reference created_at TIMESTAMPTZ NOT NULL, PRIMARY KEY (run_id, seq) -- the write that wins a slot owns the step);The primary key on (run_id, seq) is the most important line on this page. It is what stops two
workers from splitting a run. Large outputs and sandbox files go to Amazon S3.
Why not a workflow engine like Temporal? The agent loop has one fixed shape: call the model, call tools, repeat. It needs a log and leases, not a general engine, and we want to schedule by tenant at every step. The cost is that leases and recovery are ours to build and test. A team that can’t own them should use Temporal.
Step 3. Sandboxes
Section titled “Step 3. Sandboxes”The agent writes Python and wants to run it. Where does that happen?
Whether you treat model-written code as hostile and choose an isolation boundary on purpose. “In a Docker container on the worker”, with no word about the shared kernel, the network or reuse across tenants, is the red flag.
About 40% of runs execute code the model wrote, or drive a browser. That code may have been steered by a hidden instruction in a web page (a prompt injection). So treat it as untrusted code from the internet. It can’t run on a worker next to other tenants’ runs.
Each such run gets its own sandbox: a microVM, a tiny virtual machine with its own operating system kernel, such as Firecracker. It has 1 CPU, 2 GB, 60 seconds per execution, and no network unless the tenant allows specific hosts. It lives as long as the run, so files persist between steps. At the end it is destroyed, never reused.
Stage 3: Sandboxes
Stage 3 of 6
Sandboxes
Why this stage
About 40% of runs execute code the model wrote, or drive a browser.
That code is untrusted and must never run on a worker that holds other tenants’ runs.
What we add
A sandbox fleet: one microVM per run, with CPU, memory and time limits, no network by default, destroyed when the run ends.
What it costs
About 1,700 sandboxes in use at peak, and a cold start of about 2 s unless a warm pool hides it.
At 17 runs started per second, Model providers is 255% busy. Stage 4 is the fix.
The numbers behind the busy bars
What one run looks like
- A run lasts about 4 minutes. It makes 15 model calls and about 10 tool calls.
- Each step writes about 3 events to the database.
What one machine can do
- A worker holds about 200 runs at once, because it mostly waits on the model. So one worker takes about 0.8 new runs a second.
- One Postgres primary takes about 5,000 small writes a second.
- A sandbox host runs about 50 microVMs.
Outside limits
- One model provider account allows about 100 calls a second at our prompt sizes. The stage 5 gateway pools about 10 accounts across providers.
Why sandboxes add up
- About 40% of runs keep a sandbox for their whole life. That means about 96 sandboxes open for each run started per second.
// Stage 3: warm microVMs for one image in one cell. A run takes one, owns it for its whole// life, and it is destroyed afterwards. A sandbox never serves a second run, so nothing one// tenant's code leaves behind can reach another tenant.class SandboxPool { private final Vmm vmm; private final String image; private final int target; // about 140 per cell: ten times the peak start rate × 2 s private final ConcurrentLinkedDeque<Sandbox> warm = new ConcurrentLinkedDeque<>();
SandboxPool(Vmm vmm, String image, int targetWarm) { this.vmm = vmm; this.image = image; this.target = targetWarm; }
Sandbox acquire(String tenantId, String runId, List<String> egress) { Sandbox sb = warm.pollFirst(); if (sb == null) { // Pool empty during a burst: pay the cold start rather than fail the step. sb = vmm.boot(image); } // Bind only after leaving the pool. Warm sandboxes belong to nobody: no tenant label, // no files, no network. The tenant's egress allowlist is opened here, for this run. sb.configure( Map.of("tenant", tenantId, "run", runId), egress, // empty by default: no internet, no internal network 1, 2048, Duration.ofSeconds(60)); return sb; }
void release(Sandbox sb) { // Destroy, never recycle. Cleaning a used VM well enough to trust it with another // tenant costs more than restoring a fresh one from a snapshot. vmm.destroy(sb); }
// Runs every second in the background, so cold starts happen before anyone waits. void refill() { for (int missing = target - warm.size(); missing > 0; missing--) { warm.addLast(vmm.boot(image)); // restored from a memory snapshot: runtime loaded } }}# Stage 3: warm microVMs for one image in one cell. A run takes one, owns it for its whole# life, and it is destroyed afterwards. A sandbox never serves a second run, so nothing one# tenant's code leaves behind can reach another tenant.class SandboxPool: def __init__(self, vmm: Vmm, image: str, target_warm: int): self.vmm, self.image = vmm, image self.target = target_warm # about 140 per cell: ten times the peak start rate × 2 s self.warm: deque[Sandbox] = deque() self.lock = threading.Lock()
def acquire(self, tenant_id: str, run_id: str, egress: list[str]) -> Sandbox: with self.lock: sb = self.warm.popleft() if self.warm else None if sb is None: # Pool empty during a burst: pay the cold start rather than fail the step. sb = self.vmm.boot(self.image) # Bind only after leaving the pool. Warm sandboxes belong to nobody: no tenant label, # no files, no network. The tenant's egress allowlist is opened here, for this run. sb.configure( labels={"tenant": tenant_id, "run": run_id}, egress_allow=egress, # empty by default: no internet, no internal network cpu=1, memory_mb=2048, step_timeout_s=60, ) return sb
def release(self, sb: Sandbox) -> None: # Destroy, never recycle. Cleaning a used VM well enough to trust it with another # tenant costs more than restoring a fresh one from a snapshot. self.vmm.destroy(sb)
# Runs every second in the background, so cold starts happen before anyone waits. def refill(self) -> None: with self.lock: missing = self.target - len(self.warm) for _ in range(max(0, missing)): sb = self.vmm.boot(self.image) # restored from a memory snapshot: runtime loaded with self.lock: self.warm.append(sb)// Stage 3: warm microVMs for one image in one cell. A run takes one, owns it for its whole// life, and it is destroyed afterwards. A sandbox never serves a second run, so nothing one// tenant's code leaves behind can reach another tenant.class SandboxPool(IVmm vmm, string image, int targetWarm){ // targetWarm is about 140 per cell: ten times the peak start rate × a 2 s cold start. readonly ConcurrentQueue<ISandbox> warm = new();
public ISandbox Acquire(string tenantId, string runId, IReadOnlyList<string> egress) { // Pool empty during a burst: pay the cold start rather than fail the step. if (!warm.TryDequeue(out var sb)) sb = vmm.Boot(image); // Bind only after leaving the pool. Warm sandboxes belong to nobody: no tenant label, // no files, no network. The tenant's egress allowlist is opened here, for this run. sb.Configure( labels: new() { ["tenant"] = tenantId, ["run"] = runId }, egressAllow: egress, // empty by default: no internet, no internal network cpus: 1, memoryMb: 2048, stepTimeout: TimeSpan.FromSeconds(60)); return sb; }
// Destroy, never recycle. Cleaning a used VM well enough to trust it with another // tenant costs more than restoring a fresh one from a snapshot. public void Release(ISandbox sb) => vmm.Destroy(sb);
// Runs every second in the background, so cold starts happen before anyone waits. public void Refill() { for (var missing = targetWarm - warm.Count; missing > 0; missing--) warm.Enqueue(vmm.Boot(image)); // restored from a memory snapshot: runtime loaded }}
// The contracts this file uses, reduced to what it needs.interface ISandbox{ void Configure(Dictionary<string, string> labels, IReadOnlyList<string> egressAllow, int cpus, int memoryMb, TimeSpan stepTimeout);}interface IVmm{ ISandbox Boot(string image); void Destroy(ISandbox sandbox);}// SandboxPool is stage 3: warm microVMs for one image in one cell. A run takes one, owns it// for its whole life, and it is destroyed afterwards. A sandbox never serves a second run,// so nothing one tenant's code leaves behind can reach another tenant.type SandboxPool struct { vmm VMM image string target int // about 140 per cell: ten times the peak start rate × a 2 s cold start
mu sync.Mutex warm []Sandbox}
func (p *SandboxPool) Acquire(ctx context.Context, tenantID, runID string, egress []string) (Sandbox, error) { p.mu.Lock() var sb Sandbox if n := len(p.warm); n > 0 { sb, p.warm = p.warm[n-1], p.warm[:n-1] } p.mu.Unlock() if sb == nil { // Pool empty during a burst: pay the cold start rather than fail the step. var err error if sb, err = p.vmm.Boot(ctx, p.image); err != nil { return nil, err } } // Bind only after leaving the pool. Warm sandboxes belong to nobody: no tenant label, // no files, no network. The tenant's egress allowlist is opened here, for this run. err := sb.Configure(ctx, SandboxConfig{ Labels: map[string]string{"tenant": tenantID, "run": runID}, EgressAllow: egress, // empty by default: no internet, no internal network CPUs: 1, MemoryMB: 2048, StepTimeout: 60 * time.Second, }) return sb, err}
// Release destroys, never recycles. Cleaning a used VM well enough to trust it with another// tenant costs more than restoring a fresh one from a snapshot.func (p *SandboxPool) Release(ctx context.Context, sb Sandbox) error { return p.vmm.Destroy(ctx, sb)}
// Refill runs every second in the background, so cold starts happen before anyone waits.func (p *SandboxPool) Refill(ctx context.Context) { p.mu.Lock() missing := p.target - len(p.warm) p.mu.Unlock() for ; missing > 0; missing-- { sb, err := p.vmm.Boot(ctx, p.image) // restored from a memory snapshot: runtime loaded if err != nil { return // try again on the next tick; acquire falls back to a cold boot meanwhile } p.mu.Lock() p.warm = append(p.warm, sb) p.mu.Unlock() }}// Stage 3: warm microVMs for one image in one cell. A run takes one, owns it for its whole// life, and it is destroyed afterwards. A sandbox never serves a second run, so nothing one// tenant's code leaves behind can reach another tenant.class SandboxPool { private readonly warm: Sandbox[] = [];
constructor( private readonly vmm: Vmm, private readonly image: string, private readonly target: number, // about 140 per cell: 10× the peak start rate × 2 s ) {}
async acquire(tenantId: string, runId: string, egress: string[]): Promise<Sandbox> { // Pool empty during a burst: pay the cold start rather than fail the step. const sb = this.warm.shift() ?? (await this.vmm.boot(this.image)); // Bind only after leaving the pool. Warm sandboxes belong to nobody: no tenant label, // no files, no network. The tenant's egress allowlist is opened here, for this run. await sb.configure({ labels: { tenant: tenantId, run: runId }, egressAllow: egress, // empty by default: no internet, no internal network cpus: 1, memoryMb: 2048, stepTimeoutMs: 60_000, }); return sb; }
// Destroy, never recycle. Cleaning a used VM well enough to trust it with another // tenant costs more than restoring a fresh one from a snapshot. release(sb: Sandbox): Promise<void> { return this.vmm.destroy(sb); }
// Runs every second in the background, so cold starts happen before anyone waits. async refill(): Promise<void> { for (let missing = this.target - this.warm.length; missing > 0; missing--) { this.warm.push(await this.vmm.boot(this.image)); // from a memory snapshot } }}A cold microVM takes about 2 seconds to be ready. So each region keeps a pool of ready ones, started from a saved memory image. A ready sandbox belongs to nobody until a run takes it.
When a run waits a day for an approval, its sandbox’s files are saved to object storage, and the VM is destroyed. A fresh one gets the files when the run resumes.
Go deeperWhy not containers?2 min
| Option | Boundary | Start | Fits when |
|---|---|---|---|
| Container | Shares the host’s kernel | Under a second | Code you trust, from one tenant |
| gVisor-style kernel | A kernel written as a normal program | About a second | Untrusted code, if some speed can be given up |
| MicroVM (Firecracker) | Hardware virtualization, its own kernel | About 2 s with our image | Untrusted code from many tenants: our case |
| WebAssembly or V8 isolates | The language runtime | Milliseconds | Small pure functions, not “run Python with pip” |
A container escape needs one bug in the kernel, and every tenant on the machine shares that kernel. A microVM escape needs a bug in a small virtual machine monitor, a much smaller target. Firecracker was built for exactly this: AWS Lambda’s functions from many customers.
Step 4. A tool gateway
Section titled “Step 4. A tool gateway”The agent needs to call the customer’s billing system with the customer’s credentials. How?
Whether credentials stay out of the prompt and the sandbox, and whether the allowed tools are enforced in code, not by asking the model nicely. An API key in the system prompt is the red flag. So is letting the model decide when an approval is needed.
Tenants offer their systems as MCP servers. (MCP is a standard way for agents to call tools.) Every tool call from every run now goes through one service, the tool gateway. It runs five checks in order:
- Color key
- From the model
- A check our code runs
- The allowed list. The tool must be on this agent version’s list. Nothing in a prompt can change it.
- Approval. For tools the tenant marked, the worker saves the exact arguments and parks the run.
- Credentials. A short-lived token from the secrets vault, for this tenant and this server, goes into this one call. It never enters a prompt, the log or a sandbox. A model talked into leaking it has nothing to leak.
- The key. The key from the log goes with the request.
- The output. It is capped at 8 KB, and wrapped so the model can tell data from instructions.
Built-in tools (code, browser) go through the same gateway to the run’s sandbox. So one place enforces the rules for every tool.
Stage 4: Tool gateway
Stage 4 of 6
Tool gateway
Why this stage
Agents call tenants’ own systems through MCP servers, with the tenant’s credentials.
Those secrets must never enter a prompt or a sandbox, and some calls need a person to approve them.
What we add
A tool gateway that every tool call passes through: allowlist per agent version, approval rules, credentials from a per-tenant vault, idempotency keys, audit.
What it costs
A hop on every tool call (about 5 ms) and one more critical service.
Approvals turn runs into things that wait for days.
At 17 runs started per second, Model providers is 255% busy. Stage 5 is the fix.
The numbers behind the busy bars
What one run looks like
- A run lasts about 4 minutes. It makes 15 model calls and about 10 tool calls.
- Each step writes about 3 events to the database.
What one machine can do
- A worker holds about 200 runs at once, because it mostly waits on the model. So one worker takes about 0.8 new runs a second.
- One Postgres primary takes about 5,000 small writes a second.
- A sandbox host runs about 50 microVMs.
Outside limits
- One model provider account allows about 100 calls a second at our prompt sizes. The stage 5 gateway pools about 10 accounts across providers.
Why sandboxes add up
- About 40% of runs keep a sandbox for their whole life. That means about 96 sandboxes open for each run started per second.
// Part of an agent version, fixed when the tenant publishes it. Nothing in a prompt or a// tool result can change it, so an injected instruction cannot widen what the agent may do.record ToolPolicy(Set<String> allowed, Set<String> needsApproval) {}
// Stage 4: every tool call from every run passes through here, and nowhere else.class ToolGateway { private final McpRegistry servers; private final Vault vault; private final Audit audit;
ToolGateway(McpRegistry servers, Vault vault, Audit audit) { this.servers = servers; this.vault = vault; this.audit = audit; }
ToolResult call(RunContext run, String name, Map<String, Object> args, String key) { ToolPolicy policy = run.agentVersion().policy(); if (!policy.allowed().contains(name)) { // Goes back to the model as a tool error it can recover from, not as a crash. return ToolResult.error("Tool " + name + " is not enabled for this agent."); } if (policy.needsApproval().contains(name) && !run.isApproved(key)) { // The worker records ApprovalRequested and parks the run. Nothing waits in memory, // and the approver sees these exact arguments, not the model's summary of them. return ToolResult.needsApproval(name, args); } McpServer server = servers.pinned(run.tenantId(), name); // the version the tenant reviewed // A short-lived token for this tenant and this server only. It travels in the header // of this one call: never in a prompt, the event log, or a sandbox. String token = vault.token(run.tenantId(), server.id(), server.scopes()); audit.record(run.tenantId(), run.runId(), name, args, key); // MCP has no idempotency field, so the key rides in the request's _meta and an HTTP // header. Tenants' servers that change things must honor it (see deep dive 1). McpResult result = server.callTool(name, args, token, Map.of("idempotencyKey", key), Duration.ofSeconds(30)); // Tool output is untrusted text bound for the next prompt: cap it, and fence it so // the model can tell data from the tenant's instructions. String text = result.text().substring(0, Math.min(8192, result.text().length())); return new ToolResult(Fence.wrap(text), result.isError()); }}# Part of an agent version, fixed when the tenant publishes it. Nothing in a prompt or a# tool result can change it, so an injected instruction cannot widen what the agent may do.@dataclass(frozen=True)class ToolPolicy: allowed: frozenset[str] needs_approval: frozenset[str] # writes the tenant wants a person to confirm
# Stage 4: every tool call from every run passes through here, and nowhere else.class ToolGateway: def __init__(self, servers: McpRegistry, vault: Vault, audit: Audit): self.servers, self.vault, self.audit = servers, vault, audit
def call(self, run: RunContext, name: str, args: dict, key: str) -> ToolResult: policy = run.agent_version.policy if name not in policy.allowed: # Goes back to the model as a tool error it can recover from, not as a crash. return ToolResult.error(f"Tool {name} is not enabled for this agent.") if name in policy.needs_approval and not run.is_approved(key): # The worker records ApprovalRequested and parks the run. Nothing waits in memory, # and the approver sees these exact arguments, not the model's summary of them. return ToolResult.needs_approval(name, args) server = self.servers.pinned(run.tenant_id, name) # the version the tenant reviewed # A short-lived token for this tenant and this server only. It travels in the header # of this one call: never in a prompt, the event log, or a sandbox. token = self.vault.token(run.tenant_id, server.id, server.scopes) self.audit.record(run.tenant_id, run.run_id, name, args, key) # MCP has no idempotency field, so the key rides in the request's _meta and an HTTP # header. Tenants' servers that change things must honor it (see deep dive 1). result = server.call_tool(name, args, token=token, meta={"idempotencyKey": key}, timeout_s=30) # Tool output is untrusted text bound for the next prompt: cap it, and fence it so # the model can tell data from the tenant's instructions. return ToolResult(fence(result.text[:8192]), is_error=result.is_error)// Part of an agent version, fixed when the tenant publishes it. Nothing in a prompt or a tool// result can change it, so an injected instruction cannot widen what the agent may do.record ToolPolicy(IReadOnlySet<string> Allowed, IReadOnlySet<string> NeedsApproval);
// Stage 4: every tool call from every run passes through here, and nowhere else.class ToolGateway(IMcpRegistry servers, IVault vault, IAudit audit){ public ToolResult Call(RunContext run, string name, JsonElement args, string key) { var policy = run.Policy; if (!policy.Allowed.Contains(name)) // Goes back to the model as a tool error it can recover from, not as a crash. return ToolResult.Error($"Tool {name} is not enabled for this agent."); if (policy.NeedsApproval.Contains(name) && !run.IsApproved(key)) // The worker records ApprovalRequested and parks the run. Nothing waits in memory, // and the approver sees these exact arguments, not the model's summary of them. return ToolResult.ApprovalNeeded(name, args); var server = servers.Pinned(run.TenantId, name); // the version the tenant reviewed // A short-lived token for this tenant and this server only. It travels in the header // of this one call: never in a prompt, the event log, or a sandbox. var token = vault.Token(run.TenantId, server.Id, server.Scopes); audit.Record(run.TenantId, run.RunId, name, args, key); // MCP has no idempotency field, so the key rides in the request's _meta and an HTTP // header. Tenants' servers that change things must honor it (see deep dive 1). var result = server.CallTool(name, args, token, new Dictionary<string, string> { ["idempotencyKey"] = key }, TimeSpan.FromSeconds(30)); // Tool output is untrusted text bound for the next prompt: cap it, and fence it so the // model can tell data from the tenant's instructions. var text = result.Text.Length > 8192 ? result.Text[..8192] : result.Text; return new ToolResult($"<tool-output>\n{text}\n</tool-output>", result.IsError); }}
// The contracts this file uses, reduced to what it needs.record ToolResult(string Text, bool IsError, bool NeedsApproval = false){ public static ToolResult Error(string text) => new(text, true); public static ToolResult ApprovalNeeded(string name, JsonElement args) => new($"{name} {args.GetRawText()}", false, true);}record RunContext( string TenantId, string RunId, ToolPolicy Policy, IReadOnlySet<string> ApprovedKeys){ public bool IsApproved(string key) => ApprovedKeys.Contains(key);}record McpResult(string Text, bool IsError);interface IMcpServer{ string Id { get; } IReadOnlyList<string> Scopes { get; } McpResult CallTool(string name, JsonElement args, string token, Dictionary<string, string> meta, TimeSpan timeout);}interface IMcpRegistry { IMcpServer Pinned(string tenantId, string tool); }interface IVault { string Token(string tenantId, string serverId, IReadOnlyList<string> scopes); }interface IAudit{ void Record(string tenantId, string runId, string tool, JsonElement args, string key);}// ToolPolicy is part of an agent version, fixed when the tenant publishes it. Nothing in a// prompt or a tool result can change it, so an injected instruction cannot widen what the// agent may do.type ToolPolicy struct { Allowed map[string]bool NeedsApproval map[string]bool // writes the tenant wants a person to confirm}
// ToolGateway is stage 4: every tool call from every run passes through here, and nowhere else.type ToolGateway struct { servers McpRegistry vault Vault audit Audit}
func (g *ToolGateway) Call(ctx context.Context, run RunContext, name string, args json.RawMessage, key string) (ToolResult, error) { policy := run.Policy if !policy.Allowed[name] { // Goes back to the model as a tool error it can recover from, not as a crash. return ToolResult{Text: "Tool " + name + " is not enabled for this agent.", IsError: true}, nil } if policy.NeedsApproval[name] && !run.Approved[key] { // The worker records ApprovalRequested and parks the run. Nothing waits in memory, // and the approver sees these exact arguments, not the model's summary of them. return ToolResult{NeedsApproval: true, Text: name + " " + string(args)}, nil } server, err := g.servers.Pinned(ctx, run.TenantID, name) // the version the tenant reviewed if err != nil { return ToolResult{}, err } // A short-lived token for this tenant and this server only. It travels in the header of // this one call: never in a prompt, the event log, or a sandbox. token, err := g.vault.Token(ctx, run.TenantID, server.ID(), server.Scopes()) if err != nil { return ToolResult{}, err } g.audit.Record(ctx, run.TenantID, run.RunID, name, args, key) // MCP has no idempotency field, so the key rides in the request's _meta and an HTTP // header. Tenants' servers that change things must honor it (see deep dive 1). ctx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() res, err := server.CallTool(ctx, name, args, token, map[string]string{"idempotencyKey": key}) if err != nil { return ToolResult{}, err } // Tool output is untrusted text bound for the next prompt: cap it, and fence it so the // model can tell data from the tenant's instructions. text := res.Text if len(text) > 8192 { text = text[:8192] } return ToolResult{Text: "<tool-output>\n" + text + "\n</tool-output>", IsError: res.IsError}, nil}// Part of an agent version, fixed when the tenant publishes it. Nothing in a prompt or a// tool result can change it, so an injected instruction cannot widen what the agent may do.interface ToolPolicy { readonly allowed: ReadonlySet<string>; readonly needsApproval: ReadonlySet<string>; // writes the tenant wants a person to confirm}
// Stage 4: every tool call from every run passes through here, and nowhere else.class ToolGateway { constructor( private readonly servers: McpRegistry, private readonly vault: Vault, private readonly audit: Audit, ) {}
async call(run: RunContext, name: string, args: unknown, key: string): Promise<ToolResult> { const policy = run.agentVersion.policy; if (!policy.allowed.has(name)) { // Goes back to the model as a tool error it can recover from, not as a crash. return { text: `Tool ${name} is not enabled for this agent.`, isError: true }; } if (policy.needsApproval.has(name) && !run.isApproved(key)) { // The worker records ApprovalRequested and parks the run. Nothing waits in memory, // and the approver sees these exact arguments, not the model's summary of them. return { text: `${name} ${JSON.stringify(args)}`, isError: false, needsApproval: true }; } const server = await this.servers.pinned(run.tenantId, name); // the reviewed version // A short-lived token for this tenant and this server only. It travels in the header // of this one call: never in a prompt, the event log, or a sandbox. const token = await this.vault.token(run.tenantId, server.id, server.scopes); await this.audit.record(run.tenantId, run.runId, name, args, key); // MCP has no idempotency field, so the key rides in the request's _meta and an HTTP // header. Tenants' servers that change things must honor it (see deep dive 1). const res = await server.callTool(name, args, { token, meta: { idempotencyKey: key }, signal: AbortSignal.timeout(30_000), }); // Tool output is untrusted text bound for the next prompt: cap it, and fence it so the // model can tell data from the tenant's instructions. const text = `<tool-output>\n${res.text.slice(0, 8192)}\n</tool-output>`; return { text, isError: res.isError }; }}Our lab showed why the allowed list must be code. Over 18 tries with a document that hid orders to file a refund ticket, no ticket was filed, but one answer told the user it had been. What made the call impossible was the allowed list on a read-only task, not the model’s judgment.
An approval turns a run into something that waits for days, at no cost. The run is just a row in a table until the approval arrives:
- Color key
- Queue or stream
- Service
- Background worker
This is enough for a mid-level answer. Runs as saved logs on workers, sandboxes for code, and one gateway for tools with credentials kept out of prompts. The rest of the page is what makes it a senior answer: proving the platform works, sharing it fairly, and the hard failure cases.
Go deeperFollow one step, the API, and the classes4 min
Follow one step of a run with Next hop:
Trace a request: One step of a run
One step of a run: step 1 of 6
Agent workers to Model providers
The worker holds the run’s lease. It sends the system prompt, the tool list and the earlier steps, about 8,000 tokens. The first token comes after about 0.5 s; 400 tokens later the model asks to call run_python.
- Agent workers → Model providers4500 ms
- Agent workers → Run event log5 ms
- Agent workers → Tool gateway5 ms
- Tool gateway → Sandbox fleet1200 ms
- Agent workers → Run event log5 ms
- Response
The platform’s own work is about 15 ms of the 5.7 second step. The rest is the model and the tool. That is why this page spends its effort on scheduling, cost and failure, not on milliseconds.
The API. Agents are settings, and runs start asynchronously.
POST /v1/runsAuthorization: Bearer <tenant-api-key>Idempotency-Key: 4b0c-...
{ "agentId": "agt_31", "version": 7, "input": "Refund order 5512, it arrived broken", "webhook": "https://acme.example/hooks/agent" }
202 Accepted{ "runId": "run_8f2", "status": "queued", "events": "/v1/runs/run_8f2/events" }202 means the work hasn’t happened yet. A retry with the same key returns the same run. A tenant
over its limit of runs at once gets queued, not rejected. Only a very long queue returns 429.
GET /v1/runs/run_8f2/events?after=23Accept: text/event-stream
event: approval_requesteddata: {"seq":26,"approvalId":"apr_9","tool":"issue_refund","args":{"order":5512,"amount":89.0}}The events are the log itself, so after=23 resumes a dropped stream with no gap and no repeat.
Each agent version is fixed once created. Editing an agent never changes a run in progress. Every
endpoint is scoped by the API key’s tenant, so there is no way to name another tenant’s run.
- Color key
- Facade
- Entity
- Value object
- Enum
- Interface
- Implementation
DurableRun holds nothing that isn’t in its events, so any worker can load it. AgentVersion and
ToolPolicy are fixed values, so a run can’t change its own permissions. And ToolGateway is the
only path from a run to the outside world.
Prove it works
Section titled “Prove it works”Your customers’ agents give wrong answers sometimes. Is that your problem? How do you know your platform is good?
Whether you separate the platform’s quality (did we run the agent faithfully, safely and on time?) from the agent’s quality (was the answer right?), and give tenants tools for the second. Promising to judge every tenant’s answers yourself is the red flag.
There are two kinds of quality, with different owners. The tenant owns whether its agent’s answers are right. The platform owns whether it ran the agent faithfully: every step saved, none repeated, no tool outside the rules, no tenant slowed by another, every token billed correctly.
Replay is the strongest test the platform has. The log records every input the loop saw: model replies and tool results. Feed them back in place of the real model and tools, and a correct new version of our code makes exactly the same decisions. Before every release, we replay 10,000 saved runs. If even one run takes a different path, the release doesn’t ship.
We also kill workers in the middle of 1,000 test runs, with a fake tool that counts keys. Every run must finish, and no key may arrive twice. And attack documents try to make agents call tools off their allowed list. Zero such calls may get through.
For the tenant’s quality, the platform gives tools, not verdicts. Each agent can have a test set (see evaluation) that runs before a new version or a new model goes live, scored next to the current one. The tenant decides.
Go deeperAll the release metrics1 min
| Metric | How we measure it | Target | Blocks a release? |
|---|---|---|---|
| Replay differences | 10,000 saved runs replayed on the new code | 0 runs differ | Yes |
| Resume after a worker kill | Kill workers mid-step, 1,000 runs in staging | 100% finish | Yes |
| Repeated side effects | The same test, with a fake tool that counts keys | 0 | Yes |
| Tool rule escapes | Attack documents asking for tools off the list | 0 calls made | Yes |
| Sandbox escape and network | Known escape attempts and network probes | 0 succeed | Yes |
| Runs failed by us | Failed runs whose last error is ours | Under 0.1% | Canary rollback |
| Platform time per step | Trace time minus model and tool time, 99th percentile | Under 150 ms | Canary rollback |
| Billing accuracy | Our token counts against provider invoices, daily | Within 1% | Alert |
Stress it
Section titled “Stress it”The design works for one tenant at a time. Now we push it through four hard situations: one tenant’s burst, a refund that could happen twice, a tenant trying to read another’s data, and a model provider going down.
One tenant starts 3,000 runs at once
Section titled “One tenant starts 3,000 runs at once”One customer just started 3,000 runs at once, and your model provider is returning 429s. What happens to everyone else?
Whether you schedule by tenant, not by arrival, and treat provider limits as a shared budget to divide. “Autoscale the workers” is the red flag. Workers aren’t the scarce resource. Model capacity and slots are.
Two shared things need dividing: run slots and model capacity. Nothing so far stops one tenant from filling every slot. And 260 model calls a second is more than one provider account allows.
Step 5: fair scheduling and a model gateway. The run queue becomes a fair scheduler, with one queue per tenant. Each tenant has a share, set by its plan. A free slot goes to the tenant that is using the smallest part of its share. A tenant with idle share lends it to whoever has work, and gets it back as its own runs arrive. Try it with a batch job:
With one shared queue, A’s batch of 3,000 makes B and C wait about 21 minutes, even though both are inside their plans. A hard cap protects them, but holds A to 100 slots while others sit idle. Fair share gives B and C a wait of 1 to 2 seconds, and A borrows every idle slot.
// Model tokens per minute for one tenant. It refills continuously, so a quiet tenant can// burst, but never by more than one minute's worth.class TokenBucket { private final double ratePerNano, cap; private double level; private long at = System.nanoTime();
TokenBucket(double perMinute) { this.ratePerNano = perMinute / 60e9; this.cap = perMinute; this.level = perMinute; }
private void refill() { long now = System.nanoTime(); level = Math.min(cap, level + (now - at) * ratePerNano); at = now; }
boolean has(double n) { refill(); return level >= n; }
// Charged after each model call with the real count. It may go below zero, and then the // tenant's next run waits until the bucket refills: overspending costs only them. void spend(double n) { refill(); level -= n; }}
class Tenant { final String id; final double weight; // from the plan: the share of the cell this tenant is guaranteed final int maxRunning; // a hard ceiling, so even borrowing has a limit final TokenBucket tokens; final Deque<String> queue = new ArrayDeque<>(); int running;
Tenant(String id, double weight, int maxRunning, TokenBucket tokens) { this.id = id; this.weight = weight; this.maxRunning = maxRunning; this.tokens = tokens; }}
record Admitted(String tenantId, String runId) {}
// Stage 5: which tenant's run gets the next free slot. Not first come, first served: the// tenant using the smallest part of its weight goes first. Idle capacity is lent to whoever// has work, and comes back as soon as the owner's runs show up. One scheduler per cell owns// this state, so nothing here needs a lock.class FairScheduler { private final Map<String, Tenant> tenants = new HashMap<>(); private final double tokensPerRun; // about 126,000: 15 calls × 8,400 tokens
FairScheduler(List<Tenant> all, double tokensPerRun) { for (Tenant t : all) tenants.put(t.id, t); this.tokensPerRun = tokensPerRun; }
void submit(String tenantId, String runId) { tenants.get(tenantId).queue.addLast(runId); }
Optional<Admitted> nextRun() { Tenant best = null; for (Tenant t : tenants.values()) { if (t.queue.isEmpty() || t.running >= t.maxRunning) continue; if (!t.tokens.has(tokensPerRun)) continue; // out of budget: only its runs wait if (best == null || t.running / t.weight < best.running / best.weight) best = t; } if (best == null) return Optional.empty(); best.running++; return Optional.of(new Admitted(best.id, best.queue.pollFirst())); }
void finished(String tenantId) { tenants.get(tenantId).running--; }}# Model tokens per minute for one tenant. It refills continuously, so a quiet tenant can# burst, but never by more than one minute's worth.class TokenBucket: def __init__(self, per_minute: float): self.rate, self.cap = per_minute / 60, per_minute self.level, self.at = per_minute, time.monotonic()
def _refill(self) -> None: now = time.monotonic() self.level = min(self.cap, self.level + (now - self.at) * self.rate) self.at = now
def has(self, n: float) -> bool: self._refill() return self.level >= n
def spend(self, n: float) -> None: # Charged after each model call with the real count. It may go below zero, and then # the tenant's next run waits until the bucket refills: overspending costs only them. self._refill() self.level -= n
@dataclassclass Tenant: id: str weight: float # from the plan: the share of the cell this tenant is guaranteed max_running: int # a hard ceiling, so even borrowing has a limit tokens: TokenBucket queue: deque = field(default_factory=deque) running: int = 0
# Stage 5: which tenant's run gets the next free slot. Not first come, first served:# the tenant using the smallest part of its weight goes first. Idle capacity is lent to# whoever has work, and comes back as soon as the owner's runs show up. One scheduler per# cell owns this state, so nothing here needs a lock.class FairScheduler: def __init__(self, tenants: list[Tenant], tokens_per_run: float): self.tenants = {t.id: t for t in tenants} self.tokens_per_run = tokens_per_run # about 126,000: 15 calls × 8,400 tokens
def submit(self, tenant_id: str, run_id: str) -> None: self.tenants[tenant_id].queue.append(run_id)
def next_run(self) -> tuple[str, str] | None: best = None for t in self.tenants.values(): if not t.queue or t.running >= t.max_running: continue if not t.tokens.has(self.tokens_per_run): continue # out of budget: this tenant's runs wait, nobody else's do if best is None or t.running / t.weight < best.running / best.weight: best = t if best is None: return None best.running += 1 return best.id, best.queue.popleft()
def finished(self, tenant_id: str) -> None: self.tenants[tenant_id].running -= 1// Model tokens per minute for one tenant. It refills continuously, so a quiet tenant can// burst, but never by more than one minute's worth.class TokenBucket(double perMinute){ double level = perMinute; long at = Environment.TickCount64;
void Refill() { var now = Environment.TickCount64; level = Math.Min(perMinute, level + (now - at) * perMinute / 60_000); at = now; }
public bool Has(double n) { Refill(); return level >= n; }
// Charged after each model call with the real count. It may go below zero, and then the // tenant's next run waits until the bucket refills: overspending costs only them. public void Spend(double n) { Refill(); level -= n; }}
class Tenant(string id, double weight, int maxRunning, TokenBucket tokens){ public string Id => id; public double Weight => weight; // from the plan: the share of the cell it is guaranteed public int MaxRunning => maxRunning; // a hard ceiling, so even borrowing has a limit public TokenBucket Tokens => tokens; public Queue<string> Waiting { get; } = new(); public int Running { get; set; }}
// Stage 5: which tenant's run gets the next free slot. Not first come, first served: the// tenant using the smallest part of its weight goes first. Idle capacity is lent to whoever// has work, and comes back as soon as the owner's runs show up. One scheduler per cell owns// this state, so nothing here needs a lock.class FairScheduler(IEnumerable<Tenant> all, double tokensPerRun){ // tokensPerRun is about 126,000: 15 calls × 8,400 tokens. readonly Dictionary<string, Tenant> tenants = all.ToDictionary(t => t.Id);
public void Submit(string tenantId, string runId) => tenants[tenantId].Waiting.Enqueue(runId);
public (string TenantId, string RunId)? NextRun() { Tenant? best = null; foreach (var t in tenants.Values) { if (t.Waiting.Count == 0 || t.Running >= t.MaxRunning) continue; if (!t.Tokens.Has(tokensPerRun)) continue; // out of budget: only its runs wait if (best is null || t.Running / t.Weight < best.Running / best.Weight) best = t; } if (best is null) return null; best.Running++; return (best.Id, best.Waiting.Dequeue()); }
public void Finished(string tenantId) => tenants[tenantId].Running--;}// TokenBucket holds model tokens per minute for one tenant. It refills continuously, so a// quiet tenant can burst, but never by more than one minute's worth.type TokenBucket struct { perMinute, level float64 at time.Time}
func (b *TokenBucket) refill() { now := time.Now() b.level = math.Min(b.perMinute, b.level+now.Sub(b.at).Minutes()*b.perMinute) b.at = now}
func (b *TokenBucket) Has(n float64) bool { b.refill() return b.level >= n}
// Spend is charged after each model call with the real count. It may go below zero, and then// the tenant's next run waits until the bucket refills: overspending costs only them.func (b *TokenBucket) Spend(n float64) { b.refill() b.level -= n}
type Tenant struct { ID string Weight float64 // from the plan: the share of the cell this tenant is guaranteed MaxRunning int // a hard ceiling, so even borrowing has a limit Tokens *TokenBucket queue []string running int}
// FairScheduler is stage 5: which tenant's run gets the next free slot. Not first come, first// served: the tenant using the smallest part of its weight goes first. Idle capacity is lent// to whoever has work, and comes back as soon as the owner's runs show up. One scheduler per// cell owns this state, so nothing here needs a lock.type FairScheduler struct { tenants map[string]*Tenant tokensPerRun float64 // about 126,000: 15 calls × 8,400 tokens}
func (s *FairScheduler) Submit(tenantID, runID string) { t := s.tenants[tenantID] t.queue = append(t.queue, runID)}
func (s *FairScheduler) NextRun() (tenantID, runID string, ok bool) { var best *Tenant for _, t := range s.tenants { if len(t.queue) == 0 || t.running >= t.MaxRunning { continue } if !t.Tokens.Has(s.tokensPerRun) { continue // out of budget: this tenant's runs wait, nobody else's do } if best == nil || float64(t.running)/t.Weight < float64(best.running)/best.Weight { best = t } } if best == nil { return "", "", false } best.running++ runID, best.queue = best.queue[0], best.queue[1:] return best.ID, runID, true}
func (s *FairScheduler) Finished(tenantID string) { s.tenants[tenantID].running--}// Model tokens per minute for one tenant. It refills continuously, so a quiet tenant can// burst, but never by more than one minute's worth.class TokenBucket { private level: number; private at = performance.now();
constructor(private readonly perMinute: number) { this.level = perMinute; }
private refill() { const now = performance.now(); this.level = Math.min(this.perMinute, this.level + ((now - this.at) / 60_000) * this.perMinute); this.at = now; }
has(n: number): boolean { this.refill(); return this.level >= n; }
// Charged after each model call with the real count. It may go below zero, and then the // tenant's next run waits until the bucket refills: overspending costs only them. spend(n: number) { this.refill(); this.level -= n; }}
interface Tenant { id: string; weight: number; // from the plan: the share of the cell this tenant is guaranteed maxRunning: number; // a hard ceiling, so even borrowing has a limit tokens: TokenBucket; queue: string[]; running: number;}
// Stage 5: which tenant's run gets the next free slot. Not first come, first served: the// tenant using the smallest part of its weight goes first. Idle capacity is lent to whoever// has work, and comes back as soon as the owner's runs show up. One scheduler per cell owns// this state, so nothing here needs a lock.class FairScheduler { private readonly tenants: Map<string, Tenant>;
constructor( all: Tenant[], private readonly tokensPerRun: number, // about 126,000: 15 calls × 8,400 tokens ) { this.tenants = new Map(all.map((t) => [t.id, t])); }
submit(tenantId: string, runId: string) { this.tenants.get(tenantId)!.queue.push(runId); }
nextRun(): { tenantId: string; runId: string } | undefined { let best: Tenant | undefined; for (const t of this.tenants.values()) { if (t.queue.length === 0 || t.running >= t.maxRunning) continue; if (!t.tokens.has(this.tokensPerRun)) continue; // out of budget: only its runs wait if (!best || t.running / t.weight < best.running / best.weight) best = t; } if (!best) return undefined; best.running++; return { tenantId: best.id, runId: best.queue.shift()! }; }
finished(tenantId: string) { this.tenants.get(tenantId)!.running--; }}Then tokens. Each tenant has a token budget per minute and a monthly cap. A tenant that overspends waits. Nobody else does.
Then model capacity. A model gateway sits between workers and providers. It holds about ten provider accounts, tracks how much room each has left, and sends each call to one with room. When all are busy, it queues calls in the same fair order. Steps of runs already in flight go before first steps of new runs, because a half-finished run is holding a slot and a sandbox.
Stage 5: Fair scheduling and a model gateway
Stage 5 of 6
Fair scheduling and a model gateway
Why this stage
At peak the platform makes about 260 model calls/s and one provider account allows about 100.
One tenant’s batch of 3,000 runs can fill every slot.
What we add
A fair scheduler with a queue per tenant, weights from each plan, and token budgets; a model gateway that spreads calls over provider accounts and providers; a usage meter.
What it costs
Admission is no longer first come, first served, and fallback to another model changes behavior, so it is opt-in per agent.
At 17 runs started per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one run looks like
- A run lasts about 4 minutes. It makes 15 model calls and about 10 tool calls.
- Each step writes about 3 events to the database.
What one machine can do
- A worker holds about 200 runs at once, because it mostly waits on the model. So one worker takes about 0.8 new runs a second.
- One Postgres primary takes about 5,000 small writes a second.
- A sandbox host runs about 50 microVMs.
Outside limits
- One model provider account allows about 100 calls a second at our prompt sizes. The stage 5 gateway pools about 10 accounts across providers.
Why sandboxes add up
- About 40% of runs keep a sandbox for their whole life. That means about 96 sandboxes open for each run started per second.
Prompt reuse is the biggest cost lever. About 80% of each call’s input is the same start the provider saw seconds ago. Providers charge much less for that, as little as a tenth. A run drops from about 45 cents to roughly 20 to 25. That reuse lives with one provider and region, so the gateway keeps each run on the same account, and moves it only on failure.
Step 6: cells per region. At ten times the load, the event log takes about 7,700 writes a second, too many for one primary. And EU tenants need their data and model calls to stay in the EU. So we split into cells: complete copies of the stack, each sized for about 20,000 runs in flight. Each tenant has one home cell in its region, and a thin global layer routes by tenant. A bad deploy also reaches one cell first, which limits the damage. Large or regulated tenants get a cell of their own.
Stage 6: Cells per region
Stage 6 of 6
Cells per region
Why this stage
At 10× (170 runs/s), the event log takes about 7,700 writes/s, past one primary.
EU tenants require their data and model calls to stay in the EU.
What we add
The whole stack copied into cells.
A tenant lives in one home cell in its region; the event log is partitioned by run id inside each cell.
What it costs
A thin global routing layer, cross-cell moves for tenants that grow, and more fleets to deploy, one cell at a time.
At 17 runs started per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one run looks like
- A run lasts about 4 minutes. It makes 15 model calls and about 10 tool calls.
- Each step writes about 3 events to the database.
What one machine can do
- A worker holds about 200 runs at once, because it mostly waits on the model. So one worker takes about 0.8 new runs a second.
- One Postgres primary takes about 5,000 small writes a second.
- A sandbox host runs about 50 microVMs.
Outside limits
- One model provider account allows about 100 calls a second at our prompt sizes. The stage 5 gateway pools about 10 accounts across providers.
Why sandboxes add up
- About 40% of runs keep a sandbox for their whole life. That means about 96 sandboxes open for each run started per second.
A refund that could happen twice
Section titled “A refund that could happen twice”Your worker crashed right after the refund API returned success. The resumed run calls it again. Does the customer get two refunds?
Whether you know that no amount of saving closes this gap on its own, and that the fix needs the tool’s help. Claiming “exactly once” without a key on the tool side is the red flag.
There is always a moment after a tool has acted, and before its result is saved. After a crash there, retrying risks a duplicate. Not retrying risks a lost action.
The fix needs the tool’s help. The key comes from the log (run_8f2:24:0), so the crashed worker and
the new one compute the same key without talking to each other. The tool stores keys it has seen.
When the same key arrives again, it returns the first result instead of paying again. (See
idempotency.)
- Color key
- Background worker
- Database
- Service
Some tools can’t store keys. The tenant marks them as such. After a crash, the platform doesn’t retry them. It saves the outcome as “unknown”, and tells the model to check before trying again. An honest “unknown” beats a silent second refund.
A tenant tries to read another’s data
Section titled “A tenant tries to read another’s data”A malicious tenant writes an agent whose only goal is to read another tenant’s data. Walk me through what stops it.
Whether isolation holds at every layer (data, secrets, compute, network, model context), and whether each layer holds even if the one above fails. One answer (“we check the tenant ID”) is the red flag.
No single wall is enough. Each layer assumes the one above it might fail. This is called defense in depth:
| Layer | The attack | What stops it |
|---|---|---|
| API and data | Ask for another tenant’s run ID | The tenant comes from the API key, never the request. The database checks it again |
| Secrets | Get the agent to print a credential | Credentials exist only in the vault and the gateway |
| Compute | Escape the sandbox to the machine | One microVM per run, never reused |
| Network | Reach our internal services from a sandbox | All outbound traffic goes through a proxy that blocks everything not allowed |
| Model context | Get another tenant’s data through shared prompt reuse | No prompt ever contains two tenants’ data |
| Tool servers | A third-party tool server slips instructions into its description | Servers pinned to the version the tenant reviewed |
| Noisy neighbor | Use up slots, tokens or a shared tool server | Fair scheduler, token budgets, limits per tenant at the gateway |
The database’s second check catches the query someone forgot to scope. The network proxy catches code that escaped the language but not the VM. No single bug crosses tenants.
A model provider goes down mid-run
Section titled “A model provider goes down mid-run”Your main provider is down for 40 minutes. 3,000 runs are halfway through. What do you do?
Whether you know that switching models mid-run changes the agent’s behavior, and that saved runs make waiting cheap. “Fail over to another model automatically”, with no word about behavior or the tenant’s consent, is the red flag.
Try these in order:
- Retry, waiting a little longer each time.
- The same model, from another account or region. The run continues as before. Prompt reuse is lost, so the next steps cost about twice as much.
- The tenant’s chosen backup model, if it listed one.
- Otherwise, park the run. It releases its slot and resumes when the provider recovers. Parking costs only time. The run is a row in a table, and it resumes at the step it stopped on.
A circuit breaker per provider stops calls to a failing endpoint, and lets test calls through later (see resilience).
Why not switch models silently? Because the tenant tested its agent on one model. Another model may call tools in a different order, ignore an instruction, or format arguments differently. A silent switch turns our outage into the tenant’s quality problem. So backup models are part of the agent’s settings, chosen by the tenant, and shown in the run’s trace.
When things break
Section titled “When things break”Pick any box and break it. What do your customers see?
Whether every part has a story, and whether AI failures (a runaway agent, an injection, a model that changed) are on the list beside crashed servers.
Every failure should end in one of three states. The run continues somewhere else, the run waits, or the model is told what failed. A run should never just vanish, and an action should never happen twice.
Click any box to break it and see what tenants would notice:
Break it: what fails, who notices, what users see
Failure: click any box to break it
An agent worker goes down
What breaks
Its 200 runs stop mid-step.
Their leases stop being renewed.
How we notice
Missed heartbeats; leases expire after 30 s.
What users see
A pause of about 30 s in those runs, then they continue.
Nothing is lost.
What we do
Other workers take the expired leases, replay each run’s event log and continue from the last completed step.
Tool calls in flight repeat with the same idempotency key.
Three failures are special to agents:
- A runaway agent loops and retries, burning the tenant’s budget. Budgets on steps, tokens and cost per run, and detecting repeats, stop it.
- A hidden instruction in a tool’s output makes the agent try something the task shouldn’t. The allowed list, approvals for writes, and marked outputs stop it.
- A provider changes its model overnight, and agents behave differently. We pin model versions, and upgrades are opt-in, with the tenant’s tests.
Go deeperEvery failure, and what we do2 min
| Part or failure | What tenants see | What we do |
|---|---|---|
| A worker crashes | Its 200 runs pause about 30 s | Another worker reads each log and continues. Keys make repeats safe |
| Event log primary down | Every run in the cell pauses | A replica takes over in about 30 s. Workers stop, never guess |
| A model provider is down | Runs on that model slow down or park | Other accounts and regions, the tenant’s backup model, else park |
| A sandbox machine is lost | About 50 runs lose their sandbox | A new sandbox from the pool, files restored from the last save |
| A tenant’s tool server down | That tenant’s tool calls fail | A breaker per server. The model gets an error. Other tenants unaffected |
| A whole cell down | Its tenants’ runs pause | Runs resume when it recovers. Dedicated tenants can fail over |
Under overload, shed load in this order: delay new runs first, then batch runs, then interactive ones. Steps of runs in flight slow down last, because each holds a slot and a sandbox.
Wrap up
Section titled “Wrap up”The whole design
Section titled “The whole design”Here is everything we built, in the order we built it. Use the buttons to step through it.
The full design, stage by stage
Stage 6 of 6
Cells per region
Why this stage
At 10× (170 runs/s), the event log takes about 7,700 writes/s, past one primary.
EU tenants require their data and model calls to stay in the EU.
What we add
The whole stack copied into cells.
A tenant lives in one home cell in its region; the event log is partitioned by run id inside each cell.
What it costs
A thin global routing layer, cross-cell moves for tenants that grow, and more fleets to deploy, one cell at a time.
At 17 runs started per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
What one run looks like
- A run lasts about 4 minutes. It makes 15 model calls and about 10 tool calls.
- Each step writes about 3 events to the database.
What one machine can do
- A worker holds about 200 runs at once, because it mostly waits on the model. So one worker takes about 0.8 new runs a second.
- One Postgres primary takes about 5,000 small writes a second.
- A sandbox host runs about 50 microVMs.
Outside limits
- One model provider account allows about 100 calls a second at our prompt sizes. The stage 5 gateway pools about 10 accounts across providers.
Why sandboxes add up
- About 40% of runs keep a sandbox for their whole life. That means about 96 sandboxes open for each run started per second.
The decisions, and what each one costs
Section titled “The decisions, and what each one costs”| Decision | We chose | We gave up | We’d choose differently if |
|---|---|---|---|
| What we own | The loop, state, sandboxes, tools, budgets | Tenants’ freedom to bring any loop | Tenants were mostly framework authors |
| Run state | Our own event log with numbered slots | A ready-made workflow engine | The team couldn’t own leases and recovery (Temporal) |
| Side effects | Keys from the log. “Unknown” for the rest | Silent retries | Every tool were ours and safe to repeat |
| Running code | A microVM per run, never reused, a ready pool | Container speed and density | Only one trusted tenant ran code |
| Tool access | One gateway with fixed rules and a vault | A hop per call, and a critical service | Agents had no outside tools |
| Scheduling | Fair share, with borrowing | First come, first served | There were a handful of tenants with their own capacity |
| Model capacity | A gateway over accounts, sticky routing, parking | Silent switches to other models | Tenants asked for “any model, cheapest first” |
| Layout | Cells per region | One global pool | No residency rules, and under 20,000 runs in flight |
How to spend 45 minutes
Section titled “How to spend 45 minutes”- 0 to 6. Who writes the loop, how long runs last, whether agents run code. Then the line between what we own and what the tenant owns.
- 6 to 11. The numbers: 4,200 runs in flight, 260 model calls a second, 45 cents a run, 1,700 sandboxes.
- 11 to 23. Build it: the loop, the event log with resume, sandboxes, the tool gateway. A mid-level answer can stop here.
- 23 to 27. Our quality against the agent’s quality. Replay as the release test.
- 27 to 41. Stress it: fair scheduling, the double refund, isolation, the provider outage.
- 41 to 45. Failures, then your own questions.
Never skip the event log, the keys, or the tool gateway.
Follow-up questions
Section titled “Follow-up questions”Interviewer asksA run has 2,000 steps and a 300,000-token conversation. What breaks?
The prompt. Every step resends the whole conversation, so cost per step grows with the run, and eventually passes the model’s limit. The tenant can turn on summarizing, where older steps become one summary event, saved in the log like any other, so replay still works. Budgets per run stop runs that would grow forever.
Go deeperMore follow-up questions3 min
You deploy new code. What happens to runs that started on the old one? Events carry a format version, and the replay test ran the new code on saved runs before it shipped. So old logs load in new code. For a change that can’t replay old runs, the old code stays deployed until its runs finish.
A tenant cancels a run while a tool call is in flight. Cancel is an event saved to the log. The worker sees it at the next step and stops. The tool call already sent finishes, and its result is saved, because a request that may have acted can’t be pulled back. The sandbox is destroyed.
A tenant brings its own model API key. The model gateway uses that key for that tenant’s calls, so its limits are the tenant’s own. Budgets still apply. The key goes in the vault, like tool credentials.
How do you bill accurately when provider usage arrives a day later? Every model reply event stores the token counts the provider returned. So the log is the billing record, written at the moment of use. A daily job compares our sums with the invoices, and alerts on more than 1% difference.
One tenant is 30% of all traffic. Fair share still works, but that tenant shouldn’t share a cell. Give it its own cell, with its own provider capacity.
The agent’s code tries to reach the cloud’s internal address for machine credentials. It fails. All outbound traffic from a sandbox goes through a proxy that blocks everything except the tenant’s allowed hosts, and internal addresses are blocked even if a tenant tries to allow them. The attempt is logged, and repeated attempts flag the agent for review.
How does an approver know what they’re approving? The approval shows the tool and the exact arguments that will be sent, not the model’s description of them. It is tied to the call’s key, so approving one refund can’t authorize a different one.
What changes at 100 times the runs? More cells, not bigger ones: about 25 to 30. Model capacity becomes the hard limit, negotiated per region with several providers.
Further reading
Section titled “Further reading”- Agent deployment models, MCP and orchestration on this site, plus idempotency and the job scheduler for leases.
- The support agent problem designs one tenant’s agent that could run on this platform.
- Firecracker: Lightweight Virtualization for Serverless Applications, NSDI 2020: the microVM design behind AWS Lambda, and why it beats containers for code from many customers.
- Model Context Protocol specification: the messages and authorization the tool gateway speaks.
- Temporal documentation: durable execution as a product, the main alternative to our own event log.
The designbeyondcode digest, every Saturday
New deep dives, interview tips and site news in one short email. Free. Leave with one click.
We never sell or share your email. We only use it to send this newsletter.