Multi-agent platforms: what to demand before you buy one

Multi-agent platforms: what to demand before you buy one

Multi-agent platforms carry the orchestration and the permission system that a framework like LangGraph leaves for you to build. What to test, and where guardrails go.

Aaron Delasy
Aaron Delasyfounderupdated

You are buying the parts a demo never shows

Every multi-agent demo looks the same. Five agents chat with each other, a plan scrolls past, and something clever appears at the end. None of that tells you whether the system will hold up inside your company. The model already does the clever part, and the model is not what a platform charges you for.

You are buying everything around the model: how work routes between agents, who holds the credentials, and what needs sign-off before it runs. You are also paying for what the team remembers next week. You are paying for whether finished work reaches the tools your company already uses. None of that looks good in a demo. All of it decides whether the platform was the right purchase.

A multi-agent platform runs several specialized AI agents that hand work to each other. Above them sits an orchestration layer, the part that decides who does what. One agent takes the request and breaks it into pieces. It passes each piece to the teammate best suited to it, then puts the results back together. If the idea is new to you, start with what is a multi-agent system and come back here. This post is the buyer's side of the topic.

A request entering an orchestration layer where one agent breaks it into three pieces, passes each to a specialized agent, and puts the results back together

I run aldena. It is a platform in exactly this category, so I am not a neutral judge. I will flag every place where I use my own product as the example. The tests hold whether or not you ever try it.

A platform carries what a framework hands you

Confusing these two purchases is the fastest way to waste three months.

A multi-agent framework is a code library. LangGraph, CrewAI, AutoGen and the OpenAI Agents SDK (software development kit) all sit at that layer. They give you the building blocks: agent definitions, message passing, and graph or crew abstractions. They are genuinely good at that part.

They also leave everything else to you. You write the orchestration logic, host the runtime, and wire up state storage. You build the permission system, connect every tool your company uses, and build an interface a non-engineer can drive. Then you pay people to maintain all of it. That trade is right when agents are your product and the supporting code is your advantage over competitors. I compare the major options in ai agent frameworks.

A multi-agent orchestration platform ships those layers already running: hosting, orchestration, credential handling, permissions, memory, integrations, and an interface someone outside engineering can drive. Credal's guide to multi-agent platforms makes the enterprise case for this well. Its point is that the next use case should not need a new build or another software subscription. I would state it more strictly. A platform is worth its price only when the second, fifth and twentieth workflow cost configuration instead of engineering.

My version of that test is a headcount estimate. Count the engineers who will still be maintaining agent infrastructure six months after the pilot. On a framework that number is one or two, and it never falls to zero. That is an acceptable answer when you budgeted for those salaries. If your business case promised that the agents pay for themselves, a framework breaks that promise. Those salaries stay on the books.

Table comparing a multi-agent framework, where you write orchestration, host the runtime and build permissions, with a platform that ships those layers running

Orchestration is the first thing I test

Orchestration is the layer that turns one request into coordinated work. It splits the request, assigns the pieces, passes work between agents, and puts the answers back together. It also has to keep the context intact while all of that happens. Every vendor claims it. Few of them let you watch it happen, and watching is what a buyer needs. Four tests.

Can you see the routing? During a live run, one screen should tell you which agent holds the work right now. The same screen should tell you who handed it to them. Reading the logs afterwards does not count. If delegation is invisible, every failure turns into a slow investigation.

Can you change it without code? Team structure should be configuration. Moving a review step or adding a specialist should not need a code change and a deploy. If it does, orchestration stays with your engineers forever and the platform never gets past the pilot.

Are there hard caps? Runaway delegation is the classic multi-agent failure. Agent A asks B, B asks C, C asks A, and the loop bills you until someone notices. Ask the vendor whether hard limits on hand-offs exist at all. Ask whether the platform enforces them, or whether the model is trusted to stop on its own. Ask where those limits are written down. "The model is smart about that" is not an answer, because a limit the model can decide to ignore is not a limit. In aldena the caps are explicit. One turn sends at most 3 hand-offs, replies included. A chain of hand-offs stops at 200 hops. A chain holds at most 10 chats of one agent, which is the cap that stops a loop like A to B to C to A. The runtime enforces all three, so no prompt raises them. Your platform's limits can differ from mine. What matters is that they exist, that you can find them written down, and that the model cannot change them.

Agent A asks B, B asks C and C asks A in a runaway loop, beside the three caps the aldena runtime enforces: at most 3 hand-offs in one turn, a chain that stops at 200 hops, and at most 10 chats of one agent in a chain, which stops that loop

Does context survive the hand-off? When a manager agent passes work down, the agent below needs the real constraints, not a one-line summary of a two-page request. Ask to see what one hand-off actually contains.

I wrote up how I built aldena's orchestration layer in ai agent orchestration. That post covers the failures that forced those caps.

Agents should never hold your keys

Tool calls are where a multi-agent system touches the real world. That makes them the place your security review should spend its time. The pattern to demand has four parts. Credentials live in the platform and never inside an agent's context. Every tool carries explicit limits. Every call is checked against policy before it runs, and every call is logged with its arguments.

An agent holding an API key in its context next to an agent that asks the platform to act, with credentials, tool limits, policy checks and call logs kept in the platform

The buyer questions that follow from it:

  • Where do credentials live? If the answer involves pasting an API key into a prompt, or an agent "knowing" a password, do not buy it. An agent that holds a credential in its context can leak it. An agent that asks the platform to perform the action never sees the credential.
  • What is the scope of a connector? A connector is the link between the platform and one outside tool. Connecting your issue tracker for one project should not expose it to every other workspace on the account.
  • Is permission set per tool or per integration? "Jira: on" is not a policy. Reading issues and deleting issues deserve different treatment, and that means the platform has to gate each tool on its own.
  • What would a prompt injection reach? A prompt injection is a hostile instruction hidden inside content an agent reads. Agents read email, web pages and tickets, so any of those can carry one. A smarter model does not fix this. The fix is scoping. Whatever an agent is tricked into attempting stays inside one project's boundary, and it still has to pass that project's approval policies.

Delivery decides whether the work was real

A multi-agent platform that stops at chat lets the agents "finish" and hands you a transcript to copy by hand into GitHub or Jira, where the work actually counts. That is a chatbot with extra steps, and it limits the value of everything the agents did before it.

Real delivery means the work lands in your systems of record, the systems your company treats as the official version. That means a pull request in GitHub or Bitbucket, tickets in Jira or Linear, or a message in Slack. Delivery into those systems is also what makes governance workable. A pull request can be reviewed, commented on, and undone. A block of code pasted into a chat window cannot.

Chat output copied by hand into GitHub or Jira next to work landing as a pull request, tickets or a Slack message that can be reviewed, commented on and undone

Guardrails are policy, not prompts

There are two kinds of guardrails and only one of them counts. An instructed guardrail is a line in a system prompt, such as "never delete production data". It fails quietly the first time the model reads something unexpected. Nobody is told. An enforced guardrail is built into the runtime, so no amount of model confusion changes it. Four enforcements to demand.

Approval gates in front of side effects. An approval gate is a stop the run cannot get past without a human answer. Before an agent does anything with real consequences, the run pauses and a person sees the exact action. It continues only after that person approves. The pause has to be durable, so an approval given an hour later resumes the run rather than restarting it.

Role boundaries the runtime enforces. A reviewer that cannot edit code, a manager that cannot merge, a scheduler that cannot execute. If those boundaries are persona text and not real differences in tool access, they will not hold.

Isolation. One project's agents, credentials and memory are sealed off from another's. A mistake stops at that boundary instead of reaching everything the account can see.

Readable state. You can read what an agent remembers, see what it scheduled for itself, and audit what it called. You can then correct any of it. If you cannot inspect the state, you are trusting the vendor rather than governing the system.

Output validation means running a second model as a judge over the first one's work. It appears in most enterprise agent guides, including Credal's, and it does help with quality. But a judge only reads the output after the work is done. The guardrails that keep you out of an incident sit in front of the action.

A system prompt guardrail next to runtime-enforced guardrails, with the approval gate in front of the action and output validation after the work is done

How the approval gate works in aldena

Aldena is my own product, so treat this section as the worked example of those four enforcements. Agents in aldena work in projects. A project is an isolated workspace with its own connected integrations and credentials and its own memory, and each conversation in it gets its own sandbox. Agents cannot reach anything outside their project, which is the isolation piece.

Inside a project, every tool an agent can reach runs under one of three policies: allow, ask, or deny. The Permissions tab groups tools by where they come from. There are built-ins for files, commands, web access and memory, plus whatever each connected integration adds. You can set a whole group at once and then override single tools. That is how a rule like "reads run free, writes ask first" becomes a setup that takes a couple of minutes.

Aldena project permissions tab showing allow, ask, and deny policies per tool

When a tool is set to ask, the run pauses and shows the exact call the agent wants to make. You answer allow once, always allow, or deny, and the run resumes from the step it paused on. Always allow writes the policy back, so the same decision is never put to you twice. The pause is durable, so answering the next morning is fine. Asking a human a question is itself a tool, and it is the one tool that can never be switched off. An agent that is unsure can always stop and ask instead of guessing.

Approval gate pausing the delivery agent's run to show the exact memory write before it executes

Two boundaries come set up for you, and I would keep both. The delivery agent is written to open the pull request and stop, never to merge its own request. That one is an instruction to the agent, so by my own test above it is the weaker kind. The enforced part sits underneath it. Merging is a write tool under the allow, ask, or deny policy, so it stops for you unless you set it to allow. Engineering work ends with a pull request that holds the diff, the test results, and the reasoning. The backlog planning agent presents its whole proposed backlog and waits for your approval before it creates a single ticket in your tracker. Every routine the agents set for themselves is listed in project settings with an off switch. Every agent's memory is readable and correctable from the same place. The full write-ups are at human in the loop and isolated projects.

The pilot checklist I would actually run

Run one real workflow, with a real deliverable someone is waiting for, and score the platform on these:

  1. Watch the full run. Could you tell which agent held the work at every moment?
  2. Restructure the team in the middle of the pilot. Did routing follow the change without a deploy?
  3. Set a destructive tool to require approval, then trigger it. Were you shown the exact call, and did the run resume cleanly after you answered?
  4. Deny a tool outright. Confirm the agent cannot reach it at all, rather than being asked nicely not to.
  5. Find the output. Is it a pull request, a ticket, or a document in the system of record, or is it only a transcript?
  6. Ask where credentials are stored, and what one compromised project could reach.
  7. Pull the audit trail for the run. Every tool call should be attributable to one agent.
  8. Price the workflow before you run it. You should be able to work out what a run costs from a published price list, not from a "contact sales" quote.
  9. Count the engineers this still needs after the pilot ends. That number answers the platform versus framework question with data instead of a slide.

A platform that passes all nine is rare. One that fails the approval-gate test or the delivery test is not a platform yet, whatever the sales deck claims.

ready when you are

spin up your first project.

one project per client, product, or piece of work, staffed with intake, requirements, build and code review agents.