Agent orchestration

One agent in one conversation works for small tasks. Larger tasks break it in predictable ways:

• The conversation fills with results that matter to only one part of the task.
• The model loses track of the rest.
• Every step reads everything again.
• Nothing runs in parallel.

Orchestration is how you split work across several model calls or agents. It also decides who makes the split: your code, or a model.

The problem

A user asks: “Compare how our incident, deploy and on-call policies handle a SEV1 at night, and tell me who gets paged.” Answering needs research in three areas, then one answer that combines them.

A single agent searches for incidents and reads the results. Then it searches deploys and reads those, and so on. By the last search, its context holds every earlier result, and most of it is irrelevant to the current step. It’s slow, because every step waits for the one before. And it’s expensive, because every call reads the whole history again.

The patterns

These go from most predictable to most flexible. In the first four, code decides the flow. In the last two, a model decides it.

1. Prompt chain

Fixed steps. Each step is a model call whose output feeds the next. For example: extract the facts, then draft, then check the draft against the facts. Code can check the output between steps and stop early. Use it when the steps are known in advance.

2. Routing

A first call classifies the request: billing question, bug report, refund. Then code sends it to the prompt, tools or model suited to that kind. Cheap requests go to a small model, and hard ones to a large one.

3. Parallel calls

Independent parts run at the same time, and code combines the results. The parts can be different subtasks, like checking an answer for tone, facts and policy all at once. Or they can be the same task run several times, for a vote.

4. Evaluator and optimizer

One call produces something. Another grades it against criteria. The first revises until the grade passes, or a round limit is hit. This is useful where quality is easy to check, like code that must pass tests.

5. Orchestrator and workers

A model plans the subtasks at run time. It hands each one to a worker: an agent with its own fresh context, and only the tools it needs. Then it writes the final answer from the workers’ short reports. This is the pattern for tasks whose parts aren’t known in advance.

Orchestrator and workers

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Background worker

Compare it with one agent doing everything, then with the workers in parallel:

One agentOrchestrator, workers in turnOrchestrator, workers in parallel

Task: "Compare how incidents, deploys and on-call handle a SEV1 at night, and tell me who gets paged."

Wall time

Tokens read

Largest context

Model calls

The win isn’t the number of agents. Each worker’s context holds only its own topic, so calls stay small and focused, and independent work overlaps. The orchestrator never sees raw search results, only the notes. That’s why it can combine many workers without filling its context.

That win needs a task big enough for one agent’s context to grow large. On small tasks, the extra calls cost more than they save. In the lab, on three handbook questions, the orchestrator used 3 to 4 times the tokens of one agent. Its answers were no better overall. Running its workers in parallel saved little, because they all shared one model on a CPU.

6. Handoffs

Agents pass the conversation to each other. A triage agent hands a billing question to a billing agent, which can hand it on to a refunds agent. Each agent has its own instructions and tools. This suits customer-facing flows that move between departments. The cost is that no single place sees the whole flow, so tracing and loop limits matter more.

Choosing

TradeoffOrchestrator and workers instead of one agent

We gain

Smaller, focused contexts. Parallel work. Each worker gets only its own tools. And the parts can be tested on their own.

We give up

More model calls, with a plan step and a write step on top. More moving parts. And information can be lost between what a worker found and its summary.

Rules of thumb:

• Start with the simplest pattern that handles the task. Measure before adding agents. A chain or a router solves most production problems.
• Use orchestrator and workers when the parts are independent and each needs real research. Don’t use it when the parts depend tightly on each other, because workers can’t see each other’s findings.
• Use a smaller, cheaper model for workers, and a stronger one to plan and write. Planning and combining are where reasoning quality shows.
• Have workers return structured notes (JSON with findings and sources), not prose. Then the orchestrator can combine them reliably.

Common mistake

Multi-agent as the default architecture

“A researcher agent, a writer agent and a critic agent” sounds like good design. It often performs worse than one well-prompted call. Every handoff loses context. Every agent adds latency and cost. And failures are harder to trace. Add agents to solve a measured problem: a context that is too large, work that could run in parallel, or tools that must be kept apart. Not to copy a human team.

How it fails

Failure	What happens	Fix
Bad plan	Workers research the wrong things; the answer misses the point	Check the plan in code (count, schema), allow a re-plan, evaluate plans
Lossy notes	A worker found the key fact; its summary dropped it	Structured notes with required fields and sources
Duplicate work	Two workers research the same thing	Plan with clearly separate subtasks; dedupe
One worker fails	The whole task fails, or the answer silently has a hole	Retry the worker; mark missing parts in the answer
Cost multiplies	Ten workers each make ten calls	Budgets per worker and per task; cap parallelism
Rate limits	Parallel workers hit the provider’s limits together	A shared limiter; queue workers

At scale

• Parallelism has a cap. The cap is the model provider’s rate limits and your GPU capacity, not how many workers the plan asks for. Run workers through a bounded pool or queue.
• Checkpoint between stages. A task with a plan, five workers and a write step can take minutes. Save the plan, and each worker’s notes as they finish. Then a crash resumes from the last finished worker, instead of starting over. See deployment.
• Trace the tree. Each worker’s trace is a child of the task’s trace. So one view shows the whole run, its cost, and where the time went.
• Evaluate each level. Plans, worker notes and final answers each get their own eval set. Then a bad final answer can be traced to the stage that caused it.

In the interview

Interviewer asksWould you use multiple agents for this?

Only if I can name the problem they solve. If the task’s parts are independent, and each needs its own research, an orchestrator with workers keeps each context small and runs the parts in parallel. If the steps are fixed, a prompt chain is simpler and cheaper. I’d start with the simplest pattern, and measure quality, latency and cost. Then I’d add agents where the measurements point.

Interviewer asksA worker fails halfway through a ten-minute task. What happens?

The plan and each finished worker’s notes are checkpointed. So only the failed worker runs again, with a retry limit. If it still fails, the final answer says which part is missing, instead of hiding it. Tools are idempotent, so a retried worker doesn’t repeat side effects.

Mid-level covers

Describes chaining, routing and orchestrator-workers, and when to use parallel calls.

Senior adds

Starts from the simplest pattern, and justifies each added agent with a measured problem. Keeps worker contexts small with structured notes. Uses different model sizes per role. Caps parallelism against rate limits. Checkpoints stages, and evaluates and traces each level.

Try it on your laptop

Step 4: Orchestrate and resume →

Build an orchestrator with workers over your MCP tools, compare it with one agent on the same questions, then kill it mid-run and resume from its checkpoint.

Recap

Code decides the flow: a chain (fixed steps), routing (classify, then send), parallel calls (independent parts, or votes), and an evaluator loop (produce, grade, revise).

A model decides the flow: orchestrator and workers (plan, hand off with fresh contexts, combine notes), and handoffs (agents pass the conversation on).

Why split work: smaller contexts, parallel work, and tools kept apart. Not to imitate a team.

Costs: more calls, summaries that lose detail, and more to trace. Budget, checkpoint and evaluate each level.

Further reading

• Building effective agents, Anthropic: the workflow and agent patterns in this page, with examples.
• How we built our multi-agent research system, Anthropic: orchestrator and workers in production, including token costs.
• Agents and tool use for the single loop each worker runs.