Step 4: Orchestrate and resume

Turn one agent into an orchestrator with worker agents on a laptop, compare quality, tokens and time against a single agent, then kill a run halfway and resume it from a checkpoint.

Senior40 min read

Read first: Agent orchestration Agent deployment models

Some questions have several parts, like “summarize how we handle a SEV1, from the first page to the postmortem”. In this step, the step 3 agent becomes a worker. An orchestrator splits the task into research questions. It hands each one to a worker with a fresh, short context. Then it writes the answer from their notes. Finally, you kill it halfway through and bring it back.

You’ll need: step 3 working, then cd ai-assistant/step-4, pip install -r requirements.txt and python -m assistant.index ../docs.

What you’re adding

orchestrate.py: plan, work, write. The plan is one model call in JSON mode that returns 2 to 4 research questions. Each worker is the step 3 agent with only search_docs allowed and a budget of 4 steps. The writer gets the task and the workers’ notes, never the raw search results.
checkpoint.py: saves the run’s state as one JSON file after every stage. The state is the task, the plan, each finished worker’s notes, and the tokens so far. It writes with an atomic rename, so a crash mid-write can’t leave half a file.

def orchestrate(task, llm, run=None, crash_after=None, parallel=1):
    run = run or checkpoint.new_run(task)
    if run["plan"] is None:                     # resumed runs skip planning
        run["plan"], tokens = plan(task, llm)
        checkpoint.save(run)

    # Only the workers that haven't finished. On a fresh run that's all of them.
    todo = [i for i in range(len(run["plan"])) if str(i) not in run["results"]]
    with ThreadPoolExecutor(parallel) as pool:
        for i, result in zip(todo, pool.map(lambda i: work(run["plan"][i]), todo)):
            run["results"][str(i)] = result
            checkpoint.save(run)                # saved before anything else can go wrong
            if crash_after is not None and len(run["results"]) >= crash_after:
                os._exit(1)                     # a hard kill: no cleanup, like a lost machine
    ...                                          # then the writer, then a final save

python -m assistant.orchestrate "Summarize how we handle a SEV1, from the first page to the postmortem"
python -m assistant.orchestrate --parallel 4 "..."

Measure it

Measure

One agent against an orchestrator

python -m assistant.experiment ../docs

It asks three questions that each need several documents. Each is answered three ways:

One agent, allowed 8 steps.
The orchestrator, with workers one at a time.
The orchestrator, with 4 workers at once.

“Facts” counts how many of five key facts the answer contains. The facts are numbers and rules from the handbook.

We ran it twice, on two machines of the same type (2 CPU cores, qwen2.5:3b). Each cell shows run 1 / run 2. The second machine was slower, so compare times within a run, not across runs.

Task	Setup	Facts	Tokens	Time
SEV1, page to postmortem	One agent	3 / 3	1,593 / 1,623	40 s / 103 s
	Orchestrator, 1 at a time	3 / 3	6,692 / 6,663	129 s / 356 s
	Orchestrator, 4 at a time	3 / 4	5,338 / 6,678	107 s / 370 s
What stops a bad change	One agent	5 / 2	1,766 / 1,579	45 s / 85 s
	Orchestrator, 1 at a time	1 / 3	5,524 / 5,808	109 s / 330 s
	Orchestrator, 4 at a time	1 / 1	5,425 / 2,954	108 s / 171 s
Compare retention	One agent	3 / 5	1,471 / 1,528	29 s / 89 s
	Orchestrator, 1 at a time	5 / 5	5,954 / 5,907	101 s / 281 s
	Orchestrator, 4 at a time	5 / 5	5,903 / 6,012	101 s / 282 s

Facts are out of 5.

Read this table slowly, because it is not the result most people expect.

Quality: no clear winner. Add up the facts. In run 1, one agent got 11 of 15 and the orchestrator got 9. In run 2, one agent got 10 and the orchestrator got 11 (one at a time) or 10 (four at a time). The same setup on the same task often scored differently between runs. One agent got 5 of 5 on “bad change” in run 1, and 2 of 5 in run 2. The difference between setups is smaller than the difference between runs. With a small model, you need many runs before you can say one design is better. A third run agreed: one agent got 9 of 15, the orchestrator 10 and 9.

Cost: a clear loser. The orchestrator used 3 to 4 times the tokens in almost every case. It also took about 3 times as long. That result was the same in both runs. There are more model calls: one to plan, two or more per worker, and one to write. And each worker searches on its own, so the model reads similar handbook chunks several times.

When splitting helps. The retention question asks about four separate things: app logs, audit logs, metrics and backups. Each worker took one and found its number, so the orchestrator got all 5 facts in both runs. One agent got 3, then 5. When the parts are truly separate, splitting makes the result steadier. The “bad change” question is different. Deploy waves, rollbacks, flags and reviews are linked, and the facts sit close together in two files. Split four ways, the writer only sees the workers’ notes, never the documents. When a note leaves out a number, the writer can’t put it back.

Running 4 workers at once barely helped. In 5 of 6 comparisons it saved little or nothing, and once it was slower. The one big saving (330 s down to 171 s) came with half the tokens. That run’s workers simply searched less, and it also got fewer facts. All four workers share one model on 2 CPU cores, and the CPU was already busy. Parallel workers save time when the model server has spare capacity, such as a GPU or a hosted API. On a laptop CPU they mostly wait in line.

So the lesson is the one from the orchestration page. Split a task when its parts are truly independent, and measure more than once. Don’t split it because it sounds more advanced.

Break it

Kill it halfway

python -m assistant.orchestrate --crash-after 2 "Summarize how we handle a SEV1, from the first page to the postmortem"

The process exits the moment the second worker’s notes are saved, the way a machine dies: no cleanup, no finally. Without the checkpoint, everything it did is gone and the only option is to run the whole task again.

Fix it

Resume from the checkpoint

python -m assistant.orchestrate --resume <run id>

The run id is the first line the crashed run printed. Resuming loads the plan and the finished workers’ notes, runs only the workers that hadn’t finished, and writes the answer.

The experiment does exactly this with the SEV1 task, and measures both halves:

Part	Workers done	Time	Tokens
Before the crash	2 of 4	177 s	3,229
Resume: 2 workers, then write	4 of 4	176 s	3,365
Total		353 s	6,594

A clean run of the same task used 6,663 tokens and about 6 minutes on the same machine. So the crash cost almost nothing. The resumed run picked up the two saved workers’ notes, and did only the rest. Without the checkpoint, you would pay the first 3,229 tokens and 3 minutes again. On a real run with a hosted model, that is money. The longer and more expensive a run is, the more a checkpoint after every stage is worth.

Checkpoint

You now have

An orchestrator that plans, delegates to workers with small contexts, and writes from their notes
Measured quality, tokens and time for one agent against the orchestrator
Runs that survive a hard crash and resume from the last finished stage

The finished code is in ai-assistant/step-4. This step puts into practice: Agent orchestration, Agent deployment models.

Next, in step 5, the assistant gets what a production service needs. That’s a cache, a rate limit, traces, and an eval gate that refuses a worse model.