CrewAI Evals Beta: Built for Claude Code, Codex and Antigravity First

Something worth evaluating drops about every two weeks now.

Every week I see a new open-weight model lands on Hugging Face or a frontier lab ships a cheaper tier I ask myself, should I migrate my flow and crews to use it? will it be better? will it be faster? will it be cheaper? I just get genuinely curious as an engineer. But every team out there is asking the same, they wonder how much more efficient they can get on their managed agents and agentic systems.

This year a new model went from release to powering 100+ CrewAI crews in a month roughly every five days.

When evaluating use cases and model there are 4 vectors people care about: quality, speed, cost and agent design.

When we looked at 2 billion agentic workflow executions, we found that most production workloads still ran on smaller models once teams measured what each step actually required to be the most efficient. You do not need a frontier model for every role, but for that assessment you need to know the deviation from a trusted baseline across all 4 vectors when considering adjust an agent or swaping a model.

Historically, answering that question meant pausing your real work for days to stand up an sidecar harness, wire up fixture datasets, write custom graders, and stare at a web dashboard. Because the setup costs that much, a lot teams skip it and that didn't feel right, understanding and assessing your use case should not cost you speed at this day and age.

When looking at the data, across our users, we notice that switching models is a big component of long term production impact, more than half of the long running CrewAI projects in the wild switch models along the way, and a third go through three or more.

Even us internally, since starting this this, started to realize we actually could move faster and get amazing wins with few adjustments on our own use cases, but for that we had to have an opinionated take on how an extremelly easy way to evaluate agentic use cases with CrewAI, while also optimizing for coding loops.

Dashboards are built for human eyes, but your coding harness is the one writing the fix

We as an industry, we can't afford having model comparisons and evals stuck in browser tabs, the way we write agentic systems changed. All of the builders I talk to live inside coding harnesses like Claude Code, Antigravity and Codex.

We can tell it on out data already, where 70% or our docs access is coming from AI Agents, and 20% of all CrewAI projects now are being created by coding agents.

Historically most have thought of a human visual dashboard as the finish line of an agentic run. Even as tools start adding CLI and MCP access alongside their UIs, the default workflow still expects a human to open a browser, squint at a trace tree, figure out which step failed, and then act as a copy-paste router back into the terminal so the coding agent can fix it.

That is backwards though, we believe the CrewAI way for agent evaluations should be built for coding agent harnesses first and human dashboards second.

crewai eval turns your traces run into a gate your coding agent can read

Over the past few weeks we shipped an experimental open-source CLI called crewai eval. We are shipping it as an experimental builder tool designed to put evaluation directly inside your terminal loop. Since then we started to see adoption across many different coding harnesses:

We solved the cold-start problem by leaning on the execution substrate you already use. Because CrewAI has conventions around the structure of your deterministic Flows and autonomous Crews, crewai eval can evaluate your last traced run without forcing you to build a fixture library from scratch first, but creating one for yourself and allowing you and your agents to extend it, easily turning your existing process traces become the initial test suite.

When you or your coding harness run crewai eval, it pulls the execution trace so you can measure a change immediately against a real workflow, and it outputs a structured markdown brief formatted specifically for an coding harness to parse, keeping context clean and pointing straight to the evidence.

It reports the exact role and Flow step where a stopped or failing model broke execution, Claude Code, Antigravity, Codex, or your CI pipeline gets an unmistakable binary signal to keep iterating or revert.

Nowadays using a more direct way to stack rank models for a specific use case is even more relevant, given that we have noticed that the coding harnesses itself have quite a few tendencies when picking models themselves without any grounding.

Projects started in Claude Code run Anthropic models 4× as often as hand-built ones, and projects started in Antigravity run Google models 2.5× as often. So your default models arrives with the harness, one more reason to measure before you accept it.

Model bake-offs and agent self-improvement are two different loops

When you give a coding harness a machine-legible eval command, you can run two very different loops. Mixing them up is how teams break their systems, so we keep them distinct.

Model-selection loop.

The first is the model-selection loop. This is the weekly question: a new open-weight or frontier model drops, and you want to know if it belongs in your architecture. You check correctness on the tasks you care about first, and then compare efficiency across latency, tool calls, and cost.

In a multi-agent system, you rarely swap the model for the entire application at once. With crewai eval, you use model and role overlays to test combinations without rewriting your codebase. You can keep a heavy frontier model on your synthesizer or planner agent while overlaying a fast, cheap open-weight model onto a researcher or formatter role, benchmarking both against your baseline across quality, cost, speed, and tokens. That is what model-agnosticism actually looks like in practice: zero lock-in, and hard proof of which model earns its spot at each step.

Self-improvement loop

The second is the self-improvement loop. Here, the models stay fixed, and you let Claude Code or Codex improve the Crew or Flow itself.

In back-to-back conversations this year, first in Agent Harnesses Are Dead and then on the Main Branch podcast, I talked about what comes after basic harnesses: entangled agents that watch runs, learn over memories, and open pull requests against their own code for human engineers to review. Internally, we already run agents that stay in watch mode and open PRs to improve their own skills and flows. crewai eval is the measurement gate that makes that loop repeatable for any developer. You point Claude Code at your repo, let it run crewai eval, let it read the markdown brief to see which role missed the rubric or where a flow step choked, and let it adjust the prompt, tool definition, or flow routing until the exit gate clears.

Why a one-command loop needs guardrails, not blind self-modification

If you tell a seasoned systems engineer that a coding agent is going to rewrite a multi-agent workflow in a closed loop until a test passes, their first reaction should be skepticism. Left unguarded, an automated loop will happily overfit a tiny eval set, game a soft rubric, or rewrite your Flow topology into spaghetti just to turn a red exit code green.

Prithvi Rajasekaran at Anthropic documented this clearly: separating the agent doing the work from the agent judging it is a massive lever, but self-evaluation defaults to being overly lenient, and long optimization loops can plateau or bloat implementation complexity. And as Anthropic notes in their broader guide to agent evals, when you evaluate an agent, you are always evaluating the harness and the model working together.

A score on a single run is not an autopilot license. It is a routing mechanism for human attention.

I saw the cleanest proof of this with a 150,000-employee customer we worked with. Before putting a discount-approval agent into production, they ran a backwards test against 30 days of historical human decisions. The agent matched the human decision 94% of the time. The reason that deployment succeeded is what they did next: they did not celebrate a 94% score and turn on 100% automation. They used the evaluation to isolate the specific 6% of cases where the agent diverged, routed that 6% to humans, and cut overall process time down to about 3% of what it used to take.

That is how you should run crewai eval with a coding harness. The non-zero exit gate and role-level failure attribution stop the coding agent from shipping broken iterations, while the brief gives the human engineer the exact evidence needed before merging the PR. The harness does the legwork of testing overlays and proposing fixes but you still own the architecture.

Multi-agent evals still have rough edges we are hardening in the open

We are releasing crewai eval as an experimental feature because multi-agent evaluation still has hard, unsolved edges in the ecosystem.

First, run-to-run variance is real. Evaluating your last traced run gets you moving in seconds, but a single execution does not capture non-determinism across tool calls or subtle prompt coupling. Second, grading agents also require checking state changes alongside outputs and trajectories. Right now, balancing strict trajectory checks against outcome checks still takes judgment.

We built the CLI contract so we can harden those primitives in the open with the builders who push Crews and Flows the hardest. If a coding agent is going to write the fix, the evaluation has to speak its language first.