Agent deployment models

An agent that answers in five seconds can run like any other API call. An agent that researches for ten minutes, waits a day for a manager’s approval, and then files a refund cannot.

Three questions become design decisions:

• Where does the loop run?
• What happens when its machine dies?
• Whose credentials does it act with?

They’re what interviewers mean by “how would you deploy this?”.

The problem

A support agent handles a refund request. It looks up the order, checks the policy, drafts a refund, and waits for a team lead to approve anything over 500 euros. The team lead approves the next morning. Meanwhile, the service was redeployed twice, and one of its machines crashed.

If the agent’s state lived in a web request or in a process’s memory, it’s gone. The customer asks again, and the agent starts over. And if the first attempt got as far as issuing the refund, the second may issue it again.

Three ways to run an agent

1. Inside the request

The API handler runs the loop and streams the answer back. This is simple, and right for interactive tasks that finish in seconds, like a chat answer or a quick lookup. The limits are the request’s limits:

• Load balancer and client timeouts, often 30 to 120 seconds.
• Nothing survives a crash or a deploy.
• Each running task holds a connection open.

2. Background job

The API puts the task on a queue and returns a task ID. Workers take tasks from the queue and run the loop. The client checks for status, or gets a webhook or a server-sent event when it’s done.

Now long tasks don’t hold connections, and workers scale with the queue (see message queues). But on its own, this still loses work in progress when a worker crashes. The queue delivers the task again, and it starts from the beginning.

3. Durable execution

Here the task’s state is saved after every meaningful step: the plan, each completed tool call and its result, each worker’s notes. This saved state is a checkpoint. If a worker crashes or is redeployed, another worker picks up the task, loads the last checkpoint, and continues.

Waiting for a person becomes a saved state, not a sleeping process. So a task can wait for days at no cost. Workflow engines like Temporal and AWS Step Functions provide this as infrastructure. A small system can do it with a database table and careful code, as the lab does.

Durable agent runs

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Queue or stream
• Background worker
• Database

Crashes and side effects

Checkpoints fix lost work. They don’t fix repeated side effects. A crash can happen after a tool call takes effect, but before the checkpoint records it. Then the retry can’t tell whether the call happened. Try it:

The process dies during stage

In the web request Checkpoint after each stage Checkpoints and idempotency keys

Stages run twice

Tokens spent twice

Tickets filed

The fix is the same one payment APIs use: idempotency keys. Each tool call that changes something carries a key built from the task and the step, like task-81f3:step-4:file_ticket. The tool stores the keys it has seen. A repeat returns the original result, instead of acting again. Then “retry until it succeeds” is safe. See idempotency.

Two more rules make resume work:

• Save tool results, not just progress. On resume, replay the saved results into the conversation, instead of calling the tools again. Then the resumed run sees exactly what the first one saw.
• Version the checkpoint. A checkpoint written by last week’s code may not load in this week’s code. Store a version. During deploys, finish or migrate old runs.

Waiting for people

An approval step works like this:

1. The task is saved with the status “waiting for approval”, and a notification goes out.
2. The worker is done with it for now.
3. When the approver clicks, the API queues a “resume” for that task.
4. Any worker continues from the checkpoint.

Add a timeout, like auto-reject or escalate after 48 hours, so waiting tasks don’t pile up forever. And show the approver exactly what will happen, with the real arguments. Not the model’s summary of them.

Where the agent’s actions run

Credentials. The agent acts on behalf of a user. So tool calls use that user’s delegated token, narrowly scoped and short-lived (OAuth scopes, short expiry). They don’t use a service account that can do everything. Then the agent can never do more than the user could, and audit logs show who it acted for.

Sandboxes. An agent that runs code or uses a browser does it in an isolated, short-lived environment:

• a container or micro-VM per task
• no access to internal networks by default
• an allowlist of outside hosts it may reach
• limits on CPU, memory and time
• destroyed when the task ends

Treat everything the model writes as untrusted code. It may have been steered by injected instructions.

Limits per tenant. Cap concurrent tasks, steps per task, tokens per day and dollars per month for each customer. One customer’s runaway agents shouldn’t use up everyone’s model capacity.

TradeoffA workflow engine instead of hand-written checkpoints

We gain

Retries, timers, resume after a crash, waiting for signals, and a history of every step. All solved and tested.

We give up

Another system to run and learn. Its rules (deterministic workflow code, versioning) apply to all your agent code.

Common mistake

Retrying a whole agent task on failure

By default, a queue delivers a failed task again, and it runs from the start. Every model call is paid for twice, and every side effect may happen twice. Checkpoint completed steps, replay their results on resume, and put idempotency keys on every tool that changes state.

How it fails

Failure	What happens	Fix
Request timeout on a long task	User sees an error; work is lost	Background job with a task ID
Crash mid-task	Work redone or lost	Checkpoint per step; resume from the last
Crash after a side effect	Duplicate refund, ticket or email	Idempotency keys on every write
Deploy changes the code under running tasks	Resume fails or misbehaves	Versioned checkpoints; drain or migrate
Approval never comes	Tasks wait forever, holding resources	Timeouts and escalation
Agent runs with broad credentials	An injected instruction reaches anything	Per-user scoped, short-lived tokens
Runaway tenant	Everyone else is rate limited	Per-tenant concurrency and budget caps

At scale

Take 50,000 agent tasks an hour. Each takes 2 minutes and 12 model calls on average.

• Concurrency. 50,000 × 2 minutes ÷ 60 ≈ 1,700 tasks running at any time. Workers mostly wait on model calls, so each can run many tasks at once with async I/O. The limit is the model provider’s rate limit, not worker CPU.
• Model capacity. 50,000 × 12 = 600,000 model calls an hour, about 170 a second. That’s the number to negotiate rate limits for, or to size GPUs against (see LLM inference).
• Checkpoint writes. One per step, so about 170 a second, a few kilobytes each. That’s a modest load for any database. But if tool results are large, keep them out of it, and store a reference instead.
• Observability. Every task has a trace with its steps, tokens, cost and outcome, that you can query by tenant. Stuck tasks, with no progress in N minutes, raise an alert.

In the interview

Interviewer asksThis agent can take ten minutes. How do you serve it?

Not in the request. The API checks the task, stores it, queues it, and returns a task ID. The client follows progress over server-sent events or polling. Workers run the loop and checkpoint each step, so after a crash or a deploy, the task resumes from the last step. Tools that change state take an idempotency key, so a resumed step can’t act twice.

Interviewer asksThe agent needs a manager's approval before issuing a refund.

The task saves its state as “waiting for approval”, and the worker moves on. The manager sees the exact action and arguments, and approves. That queues a resume. Any worker loads the checkpoint and continues, and the refund call carries an idempotency key. If no one answers in 48 hours, it escalates or cancels.

Mid-level covers

Moves long tasks to a queue with workers and a task ID. Retries failed tasks.

Senior adds

Chooses request, background or durable execution by task length and risk. Checkpoints steps and replays saved tool results. Uses idempotency keys for side effects. Models approvals as saved states with timeouts. Runs code in sandboxes. Uses per-user scoped credentials, and caps each tenant.

Try it on your laptop

Step 4: Orchestrate and resume →

Kill your orchestrator halfway through a task with one flag, resume it from its checkpoint, and compare the time and tokens with starting over.

Recap

In the request for tasks that finish in seconds. Background jobs for long ones, with a task ID. Durable execution when tasks are long, wait for people, or must survive crashes.

Checkpoints save each step and its tool results, and resume replays them. Idempotency keys make retried side effects safe.

Approvals are saved states with timeouts, not sleeping processes.

Containment: per-user scoped credentials, sandboxes for code, and per-tenant caps.

Further reading

• Idempotency and message queues for the two building blocks this page leans on.
• Temporal documentation: durable execution: how a workflow engine provides checkpoints, timers and signals.