Agents and tool use

An agent is a model in a loop:

1. It reads the task, and decides on an action.
2. Your code carries the action out.
3. The model reads the result, and decides again.
4. This repeats until it can answer.

The “action” is a call to a tool you’ve described to it: search, look up an order, run a query, file a ticket. That small loop lets a model do work, instead of just writing text. It is also what makes agents slower, more expensive and harder to trust than a single model call.

The problem

A RAG assistant retrieves once, then answers. That fails on a request like “the payments service keeps timing out; find who owns it and file them a ticket”.

This request needs two different systems, an owner directory and a ticket tracker. It needs them in order, and the second call depends on the result of the first. You could hard-code that sequence, but it would work for this request and no other. An agent decides the sequence at run time.

How it works

Step 1: Describe the tools

Each tool is a name, a description, and a JSON Schema for its arguments:

{
  "name": "lookup_owner",
  "description": "Find which team owns a service.",
  "inputSchema": {
    "type": "object",
    "properties": { "service": { "type": "string" } },
    "required": ["service"]
  }
}

The descriptions go into the prompt. They’re the main thing the model uses to decide what to call. So writing them is prompt engineering. Say what the tool is for and when to use it, not just what it does.

In the lab, a 3B model skipped the handbook search on policy questions, and answered from nothing. One sentence naming which questions need the search fixed most of the misses. But one question still never searched.

Step 2: Run the loop

The model doesn’t call anything itself. It returns a structured request, like “call lookup_owner with service: payments”. Your code decides whether to run it.

messages = [system_prompt, user_task]
for step in range(MAX_STEPS):                       # a budget, never `while True`
    reply = model.chat(messages, tools=tool_specs)
    messages.append(reply)
    if not reply.tool_calls:                        # no tool requested: this is the answer
        return reply.text
    for call in reply.tool_calls:
        output = run_tool(call)                     # your code, your checks (step 4)
        messages.append(tool_result(call, output))  # the model reads it on the next turn
return "Stopped: step budget reached."

That’s the whole pattern. Every agent framework is this loop, plus conveniences.

The agent loop: the model proposes, your code decides and acts

Scroll or pinch to zoom, drag to pan

• Color key
• Service

Step 3: Understand what it costs

Each model call reads the whole conversation again: the system prompt, the tool descriptions, the task, and every call and result so far. So a task with n steps reads roughly n times the average context. And the context grows each step.

For example, ten steps with a tool result of 1,000 tokens each read over 50,000 tokens in total, for one task. Latency adds up the same way. Each step is a full model call, typically 0.5 to 3 seconds, plus the tool’s own time.

Step through the scenarios below. Watch the token counter, and what happens when a tool keeps failing.

Two tools, one approvalTool keeps failing, no guardsTool keeps failing, with guardsInjected instruction in a documentNext stepRun to the endReset

Model calls0

Tokens read so far0

Tools run0

Step 4: Guard the loop

A model choosing its own next step will sometimes choose badly. The guards are plain code around the loop:

• A step budget. A maximum number of model calls per task, so a confused model can’t loop all night. Five to fifteen covers most tasks. Needing more usually means the task should be split.
• Repeat detection. Say the same tool with the same arguments already failed. It won’t succeed the third time. Don’t run it. Tell the model it’s blocked, and that it should say so.
• An allowlist per task. Offer only the tools this kind of task needs. A read-only question never sees a tool that writes.
• Approval for actions with side effects. Reading is safe to automate. Sending, paying, deleting and filing wait for a person, or at least for a policy check.
• The user’s permissions. Tools run with the permissions of the person who asked, never with the service’s own credentials. So the agent can’t do anything the user couldn’t.
• Tool output marked as data. Wrap results so the model can tell them apart from instructions. Say in the system prompt to ignore instructions inside results. This lowers the rate at which injected instructions work. When it fails, the allowlist and approvals still hold.

Here is what happened in the lab’s real runs. The 3B model gave up gracefully on a failing tool. In 18 tries, with or without guards, it never made the injected tool call. But some answers still repeated the injected instruction to the user. And one falsely claimed the ticket was filed. That’s the point of guards: the outcome doesn’t depend on how the model behaves on a given day.

When not to use an agent

Agents are the most flexible option, and the least predictable. Reach for one last:

The task	Use
Same steps every time	Plain code calling the model where it needs language (a workflow)
Answer from documents	RAG: one retrieval, one call
Choose one of a few paths	A model call that classifies, then code that routes
Steps depend on what earlier steps find, and vary by request	An agent

TradeoffAn agent instead of a fixed workflow

We gain

Handles requests you didn’t plan for, and uses tools in any order.

We give up

Several model calls per task, so it’s slower and costlier. It behaves differently from run to run, so it’s harder to test. And when it goes wrong, it can break more.

Many production “agents” are mostly workflows, with one agentic step where the path really does vary. See orchestration for the patterns in between.

Common mistake

Giving the agent every tool

Fifty tool descriptions cost thousands of prompt tokens on every step. And models pick the wrong tool more often as the list grows. Group tools by task, and offer each task only what it needs. That is cheaper, more accurate and safer, all at once.

How it fails

Failure	What it looks like	Guard
Loops	The same failing call, again and again	Step budget, repeat detection
Wrong tool or wrong arguments	Plausible calls that don’t do what was meant	Clear descriptions, strict schemas, validate arguments
Doesn’t use the tool	Answers from memory	Say which questions need which tool
Acts on an injected instruction	Calls a tool no one asked for	Allowlist, approval, user permissions
Gives up silently	Says it’s done when a step failed	Check the result in code, not just the final message
Costs explode	One task reads 100,000 tokens	Step and token budgets per task, trimmed tool output

At scale

• Trim tool output. A tool that returns a 20,000-token page puts 20,000 tokens into every later step. Return the top results, summaries or IDs, and let the model ask for more.
• Budget per task, alert per tenant. Track steps, tokens and dollars per task. A runaway loop is a cost incident, so treat it like one.
• Trace every step. Record the task, each model call with its prompt size, and each tool call with its arguments, result and duration. Without traces, “the agent did something weird” can’t be debugged.
• Make tools idempotent. Agents retry. A ticket tool that takes an idempotency key files one ticket, however many times it’s called (see idempotency).
• Long tasks need a runtime. A task that takes minutes across dozens of steps can’t live in one web request. See deployment.

In the interview

Interviewer asksWhen would you use an agent rather than a fixed pipeline?

When the steps depend on what earlier steps find, and vary from request to request. For example, investigating an incident, or a support case that may need a refund, a reship or an escalation. If the steps are the same every time, a fixed workflow is cheaper, faster and easier to test. Then I’d use the model only where language is needed.

Interviewer asksHow do you stop an agent from doing something harmful?

In code, not in the prompt. Each task gets an allowlist of tools. Anything that changes state needs approval or a policy check. Tools run with the requesting user’s permissions. A step and token budget stops loops, and repeated failing calls are blocked. Tool output is marked as data to reduce injection. But I assume some injections will work, and make sure a fooled agent can’t do much.

Mid-level covers

Explains tool descriptions and the loop. Adds a step limit, and approval for writes.

Senior adds

Chooses agent or workflow per task. Accounts for cost growing with context at each step. Designs guards in code: allowlists, permissions, repeat detection, budgets. Treats injection as certain. Trims tool output, makes tools idempotent, and traces every step.

Try it on your laptop

Step 3: Add tools with MCP →

Give your assistant three tools over MCP, then watch a small model skip a tool it needed, and measure how much one sentence of instructions fixes.

Recap

The loop. The model proposes a tool call. Your code checks it and runs it. The result goes back. Repeat until the model answers or the budget runs out.

Cost. Every step reads the whole conversation again. Steps multiply latency and tokens.

Guards, in code. A step budget, repeat detection, a per-task allowlist, approval for side effects, the user’s permissions, and tool output marked as data.

Use one last. A workflow if the steps are fixed. RAG if it’s one lookup. An agent when the path depends on what it finds.

Further reading

• Building effective agents, Anthropic: workflows versus agents, and the common patterns.
• ReAct: Synergizing Reasoning and Acting in Language Models, Yao et al. 2022: the reason-act-observe loop.
• MCP for how tools are exposed to agents, and orchestration for agents that coordinate other agents.