AI agent orchestration in production: what actually breaks

Routing, not prompting

AI agent orchestration is the layer that decides which agent works on what, in what order, with which context, and under whose supervision. The agents themselves don't decide it.

Orchestration sits above the agents, and it turns a group of capable individuals into a team.

I build one of these systems, so I see orchestration from the inside. Aldena is a platform where you hire a team of AI agents into a project.

A project is one isolated workspace where each conversation gets its own sandbox. You wire the agents into a hierarchy and hand real work to the top of it.

My product does orchestration all day. This post is the engineering view: the control structure, how work gets split, and what keeps agents consistent across runs.

It also covers where the human sits, what fails in production, and what it costs.

The one public write-up I keep going back to is Anthropic's 2025 post on building their multi-agent research system, because it is written from the operating side too.

Their agents do open-ended research, and mine build software. The hard parts are the same: splitting the work, coordination overhead, durable state, and failure recovery.

And the token bill grows faster than you expect.

Workflow engine or orchestrator

AI agent orchestration is easy to confuse with workflow orchestration, and the difference decides which kind of system you need.

A workflow engine runs a graph you drew before the run started: step A, then B, retry on failure, branch on a condition.

It is predictable, cheap, and easy to debug. If you can draw the whole path in advance, use one and skip agents entirely.

Agentic orchestration is for work where the path is only discovered during the run. Nobody can draw a flowchart for "build this feature" in advance.

The plan depends on what the spec says and what the tests catch. So routing moves into the run, where an agent reads the work and picks the next step.

The mistake is treating those two as opposites. The systems that hold up in production are strict about structure and free about routing.

In Aldena the org chart, the map of who reports to whom, fixes who may hand work to whom. Tool permissions fix what any agent may touch.

Inside those limits the agents choose the sequence at runtime.

If you are still deciding whether multiple agents are worth it, I wrote multi-agent systems for that question. This post assumes you are past it.

Hierarchy beats swarms

A flat swarm puts every agent in one group chat to organize itself, and it is the first architecture most people reach for.

It sounds elegant, and it turns into noise. Every message costs every listener context, and agents reply to updates that were not addressed to them.

Two agents grab the same task because nothing made either one the owner.

Anthropic saw versions of this even inside their orchestrator-worker design. Their early agents talked too much and duplicated work when a brief was vague.

In one failure mode they spawned 50 subagents for a query one agent could have answered. They fixed it with stricter structure, not smarter agents.

Their production architecture is a lead agent that splits a query up, spawns focused subagents, and combines what comes back.

On the internal research benchmark Anthropic reported, it beat a single-agent baseline by 90.2 percent. That test ran Claude Opus 4 leading Sonnet 4 subagents, against single-agent Opus 4.

That is a hierarchy one level deep. Aldena turns the same shape into a full org chart, because software delivery has more distinct roles than research does.

The rules are strict on purpose. The chart has to be a tree, and Aldena rejects any connection that would make an agent its own ancestor.

An agent can reach its direct reports and its own manager, and nothing sideways. Work enters at the top in plain language and travels down as scoped hand-offs.

It comes back up as a status line, with the detail behind it. The tree can nest far deeper than you will ever need.

The intake agent over the delivery agent over specialists covers almost everything. You wire it by dragging a line on a canvas, documented on the org chart page.

A hierarchy gives you three things a swarm cannot:

• Attribution. Each piece of work traveled down one line from one manager. When the output is wrong, you know who split the task and who did it.
• Bounded fan-out. An agent can only create work for its own direct reports. That makes runaway delegation, one agent handing work to more and more agents, structurally impossible.
• One place for results. Results come back to the manager that asked for them. Conflicting answers meet there instead of spreading through a group chat.

Decomposition decides the outcome

Anthropic's lead agent had to learn to give every subagent four things: an objective, an output format, guidance on which tools to use, and clear task boundaries.

Vague briefs produced duplicated and orphaned work. The team also wrote effort scaling rules straight into the prompts.

Simple fact-finding gets one agent with 3 to 10 tool calls. Complex research gets ten or more subagents with divided responsibilities.

Even the tool descriptions mattered. In the numbers they published, rewriting those descriptions cut task completion time for later agents by 40 percent.

The bottleneck was the quality of the delegation, never the intelligence of the agents.

I reached the same conclusion from a different direction. Build the split into the agents themselves, so nobody works it out again on every request.

Aldena has a roster of prebuilt agents, and each one has clear rules about what it owns and what it refuses to do:

• The requirements agent asks its clarifying questions before it writes any of the spec, then writes acceptance criteria.
• The architecture agent turns requirements into components, a data model, and contracts.
• The backlog planning agent slices an approved spec into work items.
• The delivery agent delegates the build and drives the review loop, and never writes code itself.
• The code review agent reviews the diff and never edits what it reviews.

Refusal matters most. A hand-off is only a contract when both sides know what the receiver will not do. The full roster is on the agents page.

A manager routes work by capability. Every agent is named for its job, so the manager finds the right report by name, and restaffing a project rewrites nobody's instructions.

Briefs alone do not stop runaway fan-out, so the caps are structural. An agent can only hand out a few pieces of work in one turn.

A chain of hand-offs can only run so deep before it stops. Both caps live in the runtime and refuse the hand-off outright.

Code sharing is looser. Every agent in a project works on its single checkout, so the delivery agent's instructions say to run one report at a time.

That instruction, not the runtime, keeps two agents off the same working copy.

If the right agent is missing, the manager does the work itself when reasonable, or reports the gap upward instead of waiting on a teammate you never hired.

Those rules are my answer to the 50-subagent failure mode. Make runaway fan-out impossible instead of discouraging it.

Memory keeps teams consistent

Aldena keeps two durable memory layers, one private per agent and one shared per project, because a context window is scratch space that disappears when a run ends.

A team that only has context windows re-learns the project every morning. Anthropic hit this inside a single research session.

Their lead agent writes its plan to memory before the context window truncates it, because even a 200K token window runs out mid-task.

A delivery team that runs for weeks has the harder version. Consistency has to survive many separate runs, across several agents.

Each agent has private memory that only it can read and write. Each project has shared memory that every agent in the project reads.

Both layers hold a limited number of entries, and each entry has to be short. Writing past a limit fails until something is pruned.

The limits are deliberate. Memory should read as a list of current facts rather than a transcript. A memory with no limit recreates the context problem one layer up.

The split decides what goes where. Shared project memory holds what helps everyone: the repository, the default branch, the stack, the conventions, and decisions that last.

Private memory holds working state that would be noise to another agent. The delivery agent keeps its delivery state there, one line per feature.

On the next turn it resumes a half-finished pipeline instead of restarting it.

The code review agent records which failures were already there before the change, so the team stops re-diagnosing the same broken test.

You can also inspect memory. A Memory tab in project settings lists every entry with its author and timestamp.

Asking "why did it think that" becomes one lookup instead of a long search through the history. Tell an agent a fact is stale and the entry gets pruned.

Memory is scoped to the project. The same agent hired into two projects keeps two memories, so one client's conventions stay out of another client's work.

Details are on the agent memory page.

Gate the project boundary

Human attention is the scarcest resource in an AI agent team, so Aldena places its approval gate at the trust boundary, which is the edge of the project.

There are two ways to waste that attention. Ask a person to approve everything and they learn to approve without reading.

Ask them to approve nothing and they find out what happened afterwards, in the incident review.

Hand-offs between agents inside the project flow freely. Actions that leave the project pass a point a person controls.

In Aldena that point is a policy on every tool: allow, ask, or deny, set per project. Reading files and running the test suite belong on allow.

Writes, shell commands, and anything that touches your connected accounts belong on ask. When a tool is on ask, the run pauses and shows you the exact call first.

Aldena saves the pause as state instead of holding a connection open. Answer an hour or a day later and the run resumes from the same step.

Answering always allow writes the policy back, so you are never asked the same decision twice.

Asking a person a question is also a tool, and the only one nobody can switch off. An agent facing ambiguity can always stop and ask instead of guessing.

Two gates come set up for you. For engineering work the delivery agent branches, commits, pushes, and opens a pull request against your default branch, then stops.

Its instructions say the open request is your review gate, so it never merges its own work. The diff, test results and reasoning are in front of you.

The backlog planning agent presents the whole proposed backlog and waits for an explicit go-ahead before it creates a single ticket in your tracker.

Even a routine an agent set for itself is listed in the project's Routines tab, where you can cancel it. The whole layer is documented on human in the loop.

So agents run at full speed inside the project. Everything that crosses into a connected account passes a checkpoint you own.

That holds for a push to GitHub, a ticket in Jira, and every other connector and MCP server the project has.

What breaks in production

Anthropic's post is at its best on production failures, and its list of what breaks long-running agents matches the one I keep for Aldena.

Long-running agents cannot restart from scratch on every error, so Anthropic built durable execution that resumes from the point of failure.

Deploying an update while agents are mid-run breaks them, so they move traffic across gradually with rainbow deployments.

Debugging is hard because the same input takes a different path every run. So they trace decision patterns instead of expecting the same result twice.

The same pressures shaped my orchestration architecture:

• A pause is saved state. An approval wait or a clarifying question suspends the run, and the answer resumes it a minute or a day later.
• Depth is capped because errors compound. A misread requirement at the first level becomes confident nonsense several levels down. The depth cap limits how far a mistake travels.
• Shared state gets serialized. Two developers editing one working copy is the classic multi-agent corruption bug. So the delivery agent works one report at a time.
• Parallelism moves up. I trade some parallelism inside a project for edits that never mix. Projects share no filesystem, credentials, or memory, so they run side by side.
• Damage is a design decision. A project's agents reach nothing outside the project, so a bad run stops at its boundary. You only ask what happened in this project.
• You watch it live. The canvas lights each manager-to-report line as work moves. Each agent's card says whether it is thinking, running a command, or waiting on you.

A hand-off that goes to the wrong place shows up while it happens, instead of in the logs afterwards.

The hierarchy adds waiting time where results meet. A manager waits for its reports before deciding the next step, which is the bottleneck Anthropic describes too.

I accept that trade. The alternative is agents carrying on with partial results nobody combined, and that is how a team delivers contradictions.

Token economics

Anthropic's published data puts agents at about 4 times the tokens of a chat interaction, and multi-agent systems at about 15 times.

On BrowseComp, a benchmark for open-ended web research, token usage alone explained about 80 percent of the performance difference. Tool calls and model choice took that to 95 percent.

Tokens genuinely buy capability. But a multi-agent system has to point at work valuable enough to justify spending roughly fifteen times as much.

So cost is an orchestration problem, and Aldena's cost levers sit at the same level as the routing. Model choice is set per agent rather than per account.

Run the code review agent on a heavy reasoning model and the project setup agent on something cheap and fast.

Swapping a model later leaves the agent's instructions, memory, and place in the chart untouched.

Each agent also has a reasoning effort setting, from low up to max, so you pay for deep thinking only where it helps.

Each model's list rate is on the models page, so you can price a swap before you make it.

In a project that runs code, each conversation gets its own sandbox on your team's own sandbox provider account, which the provider bills directly.

The plan on the pricing page is the only thing Aldena bills, so the bill never surprises you.

The other cost decision is team size, and people get that one wrong first. Anthropic's effort scaling rule matches the number of agents to the task's complexity.

The same rule applies to staffing. The requirements agent, one build agent, and the code review agent are a normal starting team.

Hire another agent when a hand-off you keep doing yourself should belong to someone, never just because the roster has one you have not tried.

Running agents the task did not need is how you pay the multi-agent token bill for work one agent could have done.

One manager, three reports

Aldena's orchestration is something you can watch end to end in a single afternoon, and I think every orchestration claim should meet that bar.

Open one project. Hire the requirements agent, a build agent, and the code review agent under a manager.

Wire the three lines, hand the manager one deliverable you actually need, and watch the lines light up.

The spec comes back to you for answers, the build travels down, and the review comes back up.

At the end, open the pull request the manager created. That is where you see the gate. The work arrived built and tested, and the merge is still yours.

How to build an AI agent team is the step-by-step staffing version of this post. There are other ways to run agent teams.

This one held up under real work. The tree, the caps, the memory limits, and the gate at the boundary each exist because the version without them broke first.