llama vs gpt vs claude: pick a model per role, not a winner

the single-winner question is the wrong shape

Ask which of Llama, GPT, or Claude is best, and most pages that rank for that question will pick one winner. That answer works for a chatbot. One conversation, one model, pick the strongest. It stops working the moment the model runs agents. An agent team does several different jobs at once. A lead role splits the request into tasks. Specialists work through those narrow tasks in volume. A reviewer judges what comes back. Those three jobs want different models, for the same reason they would want different people.

The comparison I keep sending people to is Vellum's Llama 3.1 70B vs GPT-4o vs Claude 3.5 Sonnet evaluation. I do not send it for the numbers. By 2026 every model on that page has been replaced, some of them twice. I send it for the method, which held up. Price the models. Measure throughput, meaning tokens per second, and latency. Look at the public benchmarks, then put the leaderboard aside. Run your own evals next, meaning test sets you score the models on yourself. Score them on accuracy, precision, recall, and F1, which combines precision and recall into one number. Vellum ended with a verdict per use case instead of one best model overall. My version goes one step further, down to a model per agent role.

what vellum measured, and the detail the title hides

The setup matters more here than the results do. The title names GPT-4o and Claude 3.5 Sonnet. The models Vellum actually ran its custom tasks against were GPT-4o mini, Claude 3.5 Haiku, and Gemini 1.5 Flash. That choice was the right one. A 70B open model, meaning 70 billion parameters, was never a rival to the closed flagships. Its real rivals were the cheap, fast closed models doing the same kind of work. Models compete against the others in their own job tier, and the brand counts for less than the tier. That is the first lesson about roles, and it arrives before any eval runs.

The checklist itself is the part I reuse. Cost per million tokens. Throughput in tokens per second. Latency to the first token. The benchmark differences each provider reports. Then three custom tasks, which were math riddles, support ticket classification, and reasoning. Each task scored with accuracy, plus precision, recall, and F1 wherever plain accuracy would hide the way a model fails.

cost: list price times an agent's appetite

Vellum's price table at the time ran like this. GPT-4o mini at $0.15 per million input tokens and $0.60 per million output. Claude 3.5 Haiku at $0.25 and $1.25. Gemini 1.5 Flash at $0.35 and $1.05. Llama 3.1 70B priced by whichever host serves it. Those figures are all history now, but the point attached to the table held up better than the prices did. The closed providers keep cutting their rates, so the cost advantage of an open model shrinks every time a cheaper mini tier ships.

No price table shows the multiplier. A chat message costs one request and one reply. An agent doing delegated work reads much more than that. On every step it re-reads its instructions, its memory, the files it opened, and every tool result. Agent work therefore spends most of its tokens on input. The input column matters far more than chat use would suggest. Cache read pricing matters for the same reason.

A model that is a little cheaper and a little more wrong does not save you money. Each retry is a second full pass over the same context. Running Llama on your own hardware changes which costs you pay. It does not remove them. I listed those costs in what it takes to self-host an always-on agent.

speed and latency compound down a hand-off chain

Speed is where the open model won outright. In Vellum's tests at the time, Llama 3.1 70B ran at around 250 tokens per second, depending on the host. Gemini 1.5 Flash managed 166, Claude 3.5 Haiku 128, and GPT-4o mini 103. Latency to the first token was 0.52 seconds for Haiku, 0.56 for GPT-4o mini, and 1.05 for Flash. Their reading of that result still holds in 2026. Serving speed stays an open-model advantage, because the hosts that serve open weights compete on it.

Agent work changes what those numbers mean. In a chat, latency is how long the user waits for a reply. In a hierarchy, the work moves from agent to agent. A chain of hand-offs in aldena can only run so deep before it stops. Every step in that chain is a prompt, a batch of tool calls, and a report sent back to the agent above. Latency per step multiplies by the depth of the chain.

Accuracy per step compounds worse. Say a model gets each step right 95 percent of the time, and the work passes through ten of those steps. The chain then comes out right only about 60 percent of the time. That is 0.95 multiplied by itself ten times. So for an agent, streaming speed is the wrong number to watch. What matters is how often a step has to be done twice. One retry costs more time than fast tokens save. I wrote about the routing side of this in ai agent orchestration.

the custom evals beat the leaderboard, three times

Three tasks, and every number below is Vellum's, from that same run.

Math riddles, 14 problems. GPT-4o mini 86 percent, Gemini 1.5 Flash 71, Llama 3.1 70B 64, Claude 3.5 Haiku 29. Classification, 100 customer support tickets. The accuracy scores sat close together, GPT-4o mini on top at 72 percent and the rest near 68. GPT-4o mini led on precision at 89 percent, while Haiku had the best F1 at 75. Reasoning, 14 questions. GPT-4o mini 63 percent, Llama and Flash at 44, Haiku at 38. The Llama 70B before it had scored 56 on that same reasoning set. So the newer model got worse at reasoning while it gained 14 points on math.

Every model in those tables is retired now. Three lessons from them still apply.

First, leaderboard rank did not predict task rank. The same Haiku that came last on the riddles had the best F1 on classification. A model is not good or bad on its own. It is good or bad at one task.

Second, the metric is a choice. Say your agent sorts tickets into categories, and a wrong category costs you a customer's trust. Precision is then the score that matters. Say it screens for incidents instead, where the expensive mistake is a miss. Recall is the score that matters. Plain accuracy made every model in that classification table look better than it was.

Third, upgrades can go backwards. A newer, bigger, better-marketed release lost 12 points on Vellum's own reasoning set. Public benchmarks will not catch that for your workload. An eval you own will catch it, as long as you re-run it on every model change.

One caution about the format. With 14 questions, one flipped answer moves a score by 7 points. Treat Vellum's sets, and any set that small, as a hint about direction. A set that small cannot prove anything. Build your own set bigger.

the open vs closed question in 2026

Since that post, the open side moved from Llama 3.1 through 3.3 to the Llama 4 generation. The GPT-4o family was replaced by GPT-5. Claude is on the Opus and Sonnet lines I covered in claude opus 4.5 benchmarks and claude sonnet 5 benchmarks. So the comparison people type now is Llama 4 vs GPT-5 vs Claude Opus 4.5. Every specific number above is history. The differences between the three families are still in the same places they were two years ago.

Open weights still buy you control. You pick the host, or you become the host. Hosts compete on price and on speed. Nobody can retire the weights while you still depend on them, and your data can stay wherever your policy requires.

Closed flagships are still better at the long jobs. I run this exercise again with every new generation. The strongest closed models are still the ones I trust to plan work, break it into pieces, and review it across many steps. That gap is closing, but more slowly than the marketing on either side suggests.

The tier Vellum documented by accident is still where open models compete hardest. High volume, narrow, well-specified steps. For that work, a cheap fast model with a clear brief is the sensible choice.

That is the verdict, and it matches the one Vellum reached: the application decides the model. There is no single best one. There are roles to fill, and each role can hold a different model.

how per-role model choice works in aldena

I built aldena so that changing a model is a setting rather than a migration. You pick the model per agent, and the choice is not tied to the whole account. A room is one chat with its own agents and its own server. Every agent in that room carries its own model and its own reasoning effort, from low up to max. So the reviewer that checks every diff can run a heavy reasoning model at high effort. The room assistant that files facts runs something cheap and fast.

The agent roster ships a set of prebuilt roles that follow that split. The org chart, which sets who reports to whom, is what makes the split safe. Work is handed down through the hierarchy, and the strong model reviews what the cheap model produced before anything comes back to you.

Swapping a model is a dropdown. An agent keeps its memory, its skills, and its place in the hierarchy when the model changes. Running Llama in a specialist role for a week therefore costs you a week and nothing more. Usage meters against one prepaid team balance at each provider's list rate. Every supported model is priced on the model pricing page, so you can look up the cost of a swap before you make it.

The approval gate, which is the rule that some actions need your yes first, keeps the experiment cheap. Risky tools are set to ask. Engineering work arrives as an open pull request. The delivery role never merges its own request. Merging is a write tool too, so it defaults to asking you first. The worst day a cheap model can give you is a denied prompt or a closed pull request, not a production incident.

run the same method on your own tickets

What lasts from Vellum's post is the procedure, and you can run it in an afternoon. Pull 50 to 100 real examples per role out of your own history. Real tickets, real bug reports, real classification calls. Write down what a correct answer looks like before any model runs.

Score accuracy, and add precision, recall, and F1 anywhere the classes are uneven. Price the winner at your real token mix, which for agents means counting input tokens heavily. Then assign a model per role instead of one model for the whole company.

Re-run that eval set every time a provider releases a new model. When a new model gets worse on your own examples, your own set is the only thing that will tell you.

The winning model changes every few months. The roles, and the method you use to fill them, are what you keep.