AI agent orchestration in production: hierarchy, shared memory, approval gates, and what breaks

AI agent orchestration is a routing problem, not a prompting problem

AI agent orchestration is the layer that decides which agent works on what. It also decides in what order, with which context, and under whose supervision. The agents themselves do not do that work. Orchestration sits above them, and it turns a group of capable individuals into a team.

I build one of these systems, so I look at orchestration from the inside. Aldena is a platform where you hire a team of AI agents into a room, which is one isolated workspace with its own server. You wire the agents into a hierarchy and hand real work to the top of it. Orchestration is what my product does all day. This post is the engineering view. It covers the control structure, how work gets split up, and what keeps agents consistent across runs. It also covers where the human sits, what fails in production, and what it costs.

The one public write-up I keep going back to is Anthropic's 2025 post on building their multi-agent research system. It is written from the operating side too. Their agents do open-ended research. My agents ship software. The hard parts are the same in both cases: splitting the work up, coordination overhead, durable state, and failure recovery. The token bill also grows faster than you expect.

A workflow engine is not an orchestrator

People often confuse this with workflow orchestration, and the difference decides which kind of system you need. A workflow engine runs a graph you drew before the run started: step A, then B, retry on failure, branch on a condition. It is predictable, cheap, and easy to debug. If you can draw the whole path in advance, use one and skip agents entirely.

Agentic orchestration is for work where the path is only discovered during the run. Nobody can draw a flowchart for "build this feature" in advance. The plan depends on what the spec turns out to say, and on what the tests turn out to catch. So the routing decisions move into the run itself. An agent reads the current state of the work and decides what happens next.

The mistake is treating those two as opposites. The systems that hold up in production are strict about structure and free about routing. In Aldena the org chart, meaning the map of who reports to whom, fixes who may hand work to whom. Tool permissions fix what any agent may touch. Inside those limits the agents choose the actual sequence at runtime. Structure is fixed, routing is live. If you are still deciding whether multiple agents are worth it for your problem, I wrote multi-agent systems for that question. This post assumes you are past it.

Why a hierarchy beats a flat swarm

The first architecture most people reach for is the flat swarm: put every agent in one group chat and let them organize themselves. It sounds elegant, and it turns into noise. Every message costs every listener context. Agents reply to updates that were not addressed to them. Two agents grab the same task because nothing made either one the owner. Anthropic saw versions of this even inside their orchestrator-worker design. Their early agents talked too much. They duplicated work when a brief was vague. In one failure mode they spawned 50 subagents for a query one agent could have answered. They fixed it with stricter structure, not with smarter agents.

Their production architecture is a lead agent that splits a query up, spawns focused subagents, and merges what comes back. On the internal research benchmark Anthropic reported in that post, it beat a single-agent baseline by 90.2 percent. That test ran Claude Opus 4 leading Sonnet 4 subagents, against single-agent Opus 4. That is a hierarchy one level deep. Aldena takes the same shape and turns it into a full org chart, because software delivery has more distinct roles than research does.

The rules are strict on purpose. The chart has to be a tree, so a connection that would make an agent its own ancestor is rejected. An agent can reach its direct reports and its own manager, and nothing sideways. Work enters at the top in plain language and travels down as scoped hand-offs. It comes back up as a status line, with the detail behind it. The tree can nest far deeper than you will ever need, though a project manager over an engineering manager over specialists covers almost everything. You wire it by dragging a line on a canvas, documented on the org chart page.

A hierarchy gives you three things a swarm cannot:

• Attribution. Every piece of work traveled down a specific line from a specific manager. When the output is wrong, you know who split the task up and who did it.
• Bounded fan-out. An agent can only create work for its own direct reports. Runaway delegation is where one agent keeps creating work for more and more agents. The structure makes that impossible, so nobody has to rely on a prompt instruction holding.
• One place where results merge. Results come back to the manager that asked for them. Conflicting answers meet in one place instead of spreading through a group chat.

Decomposition is where orchestration is won or lost

Anthropic's sharpest lesson is about the hand-off itself. Their lead agent had to learn to give every subagent four things: an objective, an output format, guidance on which tools to use, and clear task boundaries. Vague briefs produced duplicated and orphaned work. They also wrote effort scaling rules straight into the prompts. Simple fact-finding gets one agent with 3 to 10 tool calls. Complex research gets ten or more subagents with divided responsibilities. Even the tool descriptions mattered: in the numbers they published, rewriting those descriptions cut task completion time for later agents by 40 percent. The bottleneck was the quality of the delegation. The intelligence of the agents was never the limit.

I reached the same conclusion from a different direction. Build the split into the roles themselves, so nobody works it out again on every request. Aldena ships a roster of prebuilt roles, and each one has clear rules about what it owns and what it refuses to do. The business analyst asks its clarifying questions before it writes any of the spec, then writes acceptance criteria. The solution architect turns requirements into components, a data model, and contracts. The scrum master slices an approved spec into work items. The engineering manager delegates the build and drives the review loop, and never writes code itself. The staff engineer reviews the diff and never edits what it reviews. Refusal is the part that matters. A hand-off is only a contract when both sides know what the receiver will not do. The full roster is on the agents page.

A manager routes work by capability rather than by name. It looks up the right report by role or by trait and hands the work over, so restaffing a room does not rewrite anyone's instructions. Briefs alone do not stop runaway fan-out, so the caps are structural. An agent can only hand out a few pieces of work in one turn. A chain of hand-offs can only run so deep before it stops. Both caps live in the runtime and refuse the hand-off outright. Sharing the code is held together more loosely. Every agent in a room works on that room's single checkout, so the delivery manager is instructed to run one report at a time. That instruction is what keeps two agents off the same working copy at once. If a role is missing, the manager does the work itself when that is reasonable. Otherwise it reports the gap upward, instead of waiting on a teammate you never hired. Those rules are my answer to the 50-subagent failure mode. Make runaway fan-out impossible instead of discouraging it.

Shared memory keeps a team consistent

A context window is scratch space. It disappears when the run ends. A team that only has context windows re-learns the project every morning. Anthropic ran into this inside a single research session. Their lead agent writes its plan to memory before the context window truncates it, because even a 200K token window runs out mid-task. A delivery team that runs for weeks has the harder version of that problem. Consistency has to survive many separate runs, across several agents.

Aldena keeps two durable layers. Each agent has private memory that only it can read and write. Each room has shared memory that every agent in the room reads. Both layers hold a limited number of entries, and each entry has to be short. Writing past a limit fails until something is pruned. The limits are deliberate. Memory should read as a list of current facts rather than a transcript. A memory with no limit just recreates the context problem one layer up.

The split decides what goes where. Shared room memory holds what helps everyone: the repository, the default branch, the stack, the conventions, and decisions that last. Private memory holds working state that would be noise to another agent. The engineering manager keeps its delivery state in private memory, one line per feature. On the next turn it resumes a half-finished pipeline instead of restarting it. Reviewers record which failures were already there before the change, so the team stops re-diagnosing the same broken test.

You can also inspect memory. A Memory tab in room settings lists every entry with its author and timestamp. Asking "why did it think that" becomes one lookup instead of a long search through the history. Tell an agent that a fact is stale and the entry gets pruned. All of it is scoped to the room. The same role hired into two rooms keeps two separate memories, which is what keeps one client's conventions out of another client's work. Details on the agent memory page.

Put the approval gate at the boundary, not inside every hand-off

Human attention is the scarcest resource in the whole system, and there are two ways to waste it. Ask a person to approve everything and they learn to approve without reading. Ask them to approve nothing and they only find out what happened afterwards, in the incident review. The placement that works is the trust boundary, which here is the edge of the room. Hand-offs between agents inside the room flow freely. Actions that leave the room pass a point a person controls.

In Aldena that is a policy on every tool: allow, ask, or deny, set per room. Reading files and running the test suite belong on allow. Writes, shell commands, and anything that touches your connected accounts belong on ask. When a tool is on ask, the run pauses and shows you the exact call before it happens. The pause is saved as state rather than held open on a connection. So you can answer an hour or a day later, and the run resumes from the same step. Answering always allow writes the policy back, so a decision you already made is not put to you twice. Asking a person a question is itself a tool, and it is the one tool that can never be switched off. An agent that finds something ambiguous can always stop and ask instead of guessing.

Two gates come set up for you. For engineering work the delivery manager branches, commits, pushes, and opens a pull request against your default branch, then stops. Its instructions say the open request is your review gate, so it does not merge its own work. Merging is also a write tool under the room policy, which means it pauses for you unless you set it to allow. The diff, the test results, and the reasoning are in front of you before anything is merged. The scrum master presents the whole proposed backlog and waits for an explicit go-ahead before it creates a single ticket in your tracker. Even scheduled work that an agent set up for itself is listed in room settings, where you can cancel it. The whole layer is documented on human in the loop. So agents run at full speed inside the room. Everything that crosses into a connected account passes a checkpoint you own. That holds for a push to GitHub, a ticket in Jira, and every other connector the room has.

What breaks in production

Anthropic's post is at its best on this topic, and their list matches mine. Long-running agents cannot be restarted from scratch on every error, so they built durable execution that resumes from the point of failure. Deploying an update while agents are mid-run breaks them, so they move traffic across gradually with rainbow deployments. Debugging is hard because the same input takes a different path every run. So they trace decision patterns instead of expecting the same result twice. The same pressures shaped my orchestration architecture:

• A pause is saved state. An approval wait, a clarifying question, or an empty credit balance suspends the run. When the balance runs out, work already running pauses and new requests stop instead of adding to the bill. Adding credit resumes the paused work from where it stopped. Without that, a team that runs for weeks becomes a cost you cannot control.
• Depth is capped because errors compound. A misread requirement at the first level becomes confident nonsense several levels down. The cap on hand-off depth is there to limit how far a mistake travels. It is not a technical ceiling.
• Shared state that agents can change gets serialized, meaning one agent at a time. Two developers editing one working copy is the classic multi-agent corruption bug. That is why delivery managers work one report at a time. I trade some parallelism inside a room for never merging two agents' half-finished edits. The parallelism happens one level up. Rooms run at the same time without interfering, because two rooms never share a filesystem, credentials, or memory.
• The damage a bad run can do is a design decision. A room's agents can reach nothing outside the room, so a bad run stops at the room boundary. When something goes wrong, you only have to ask what happened in this room. You never have to ask what else the agents could reach.
• You watch it live. The room canvas draws every manager-to-report line and lights it up while work moves along it. Each agent's card says whether it is thinking, running a command, or waiting on you. You see a hand-off go to the wrong place while it happens, instead of reconstructing it from logs afterwards.

The hierarchy adds waiting time at the point where results merge. A manager waits for its reports before it can decide the next step. That is the same bottleneck Anthropic describes in their lead agent. I accept that trade. The alternative is agents continuing with partial results that nobody merged, and that is how teams ship contradictions.

Token economics decide whether any of this is worth running

The most useful numbers in Anthropic's post are the economic ones. In the data they published, agents burn about 4 times the tokens of a chat interaction, and multi-agent systems burn about 15 times. On BrowseComp, a benchmark for open-ended web research, token usage on its own explained about 80 percent of the difference in performance. Adding tool calls and model choice took that to 95 percent. Tokens genuinely buy capability. But a multi-agent system has to be pointed at work valuable enough to justify spending roughly fifteen times as much.

So cost is an orchestration problem, and that is why the spend controls in Aldena sit at the same level as the routing. Model choice is set per agent rather than per account. Run the reviewer on a heavy reasoning model and the room assistant on something cheap and fast. Change your mind next week without touching the role, its memory, or its place in the chart. Each agent also carries a reasoning effort setting, from low up to max, so you pay for deep thinking only where it helps. Usage comes out of one prepaid team balance at the per-model rates published on the models page. The room's server is metered by the hour on the same balance, priced by size, with the current rates on the pricing page. The whole system stops when the budget runs out, so the bill never surprises you.

The other cost decision is team size, and that is the one people get wrong first. Anthropic's effort scaling rule says to match the number of agents to the complexity of the task. The same rule applies to staffing. An analyst, one developer, and a reviewer is a normal starting team. Hire another role when a hand-off you keep doing yourself should belong to someone. Do not hire one just because there is another role on the roster you have not tried yet. Running agents the task did not need is how you pay the multi-agent token bill for work one agent could have done.

Start with one manager and three reports

You can watch everything in this post happen in a single afternoon. That is the standard I think any orchestration claim should meet. Open one room. Hire an analyst, a developer, and a reviewer under a manager. Wire the three lines, then hand the manager one deliverable you actually need. Then watch the lines light up. The spec comes back to you for answers, the build travels down, and the review comes back up. Open the pull request it created at the end. That is where you see the gate. The work arrived finished, and the merge is still yours.

If you are comparing platforms in this category, AI workforce platform covers the management layer around the orchestration. How to build an AI agent team is the step-by-step staffing version of this post. There are other ways to run agent teams. This is the one that held up under real work. Every piece of it exists because the version without it broke first. The tree, the caps, the memory limits, and the gate at the boundary all came from something going wrong.