AI Agents and Tool Calling: Why Orchestration Is Harder Than the Model

The ability of language models to call external functions turns them from answering machines into components that can act. But experience building these systems shows a consistent pattern: failures rarely come from a model that isn't smart enough. They come from the planning and error handling around it.

What actually happens when a model calls a tool

The model doesn't execute anything. It produces structured text stating an intent to call a particular function with particular arguments. The code around it reads that intent, runs the real function, and returns the result into the conversation.

The distinction matters because it puts responsibility in the right place. Argument validation, permission limits, and failure handling all sit on the application side. A model producing invalid arguments is an event to anticipate, not an anomaly.

The failures that show up most often

Three patterns repeat. First, the model calls the same tool over and over because it doesn't read the first result as an answer — usually because the tool's output is too long or its format is unclear. Second, the model gives up too quickly when a call returns an error, then invents an answer as if the call had succeeded.

Third, and most troublesome, the model calls a series of tools that are correct individually but together don't solve the problem. It loses its way midway through a long task because the conversation history grows too long and the original instructions sink out of view.

Design patterns that help

Tool descriptions are the interface that matters most. Clear function names, few parameters, and a short note on when a tool should not be used have far more impact than scaling up the model.

Tool output should be concise and consistent. Returning an entire JSON object with fifty fields forces the model to filter out irrelevant information; returning the three fields it needs produces far more accurate decisions.

Errors should come back as actionable messages. "Failed" gives the model no way out; "the date must be in YYYY-MM-DD format" makes a fix possible on the next attempt.

Autonomy limits you need to set

An agent system that runs without an iteration limit is a system that will eventually spin forever. Set a maximum number of steps, and treat hitting that limit as a failure to report, not a condition to ignore.

For actions that are hard to undo — sending email, moving funds, deleting data — human confirmation isn't a step backwards but correct design. The cost of one confirmation is far smaller than the cost of one wrong action that can't be taken back.

Measure before you scale up

The biggest temptation when an agent system misbehaves is to switch to a bigger model. In many cases the same improvement comes from clarifying tool descriptions and tidying their output — at no extra cost per call.

Build a set of test cases that represent real tasks, run every change against them, and record the success rate. Without that measurement, every prompt tweak is just a guess that feels like an improvement.