How do you evaluate an AI agent before you trust it in production?

Why public benchmarks cannot answer this

A leaderboard tells you that a model is broadly capable. It cannot tell you whether your agent retrieves the right document, calls the right tool in the right order, or refuses the request it should have refused. Those outcomes depend on your data, your prompts, your tool definitions, and your definition of correct, none of which appear in a public benchmark.

This is why every serious evaluation workflow starts from your own cases. The open-source framework published by OpenAI makes the point structurally: alongside a public registry of benchmarks, it supports building private evals on your own data without exposing that data publicly. The private half is the half that decides whether you can trust the system.

Step 1: write criteria you can count

The most common reason an evaluation effort stalls is that nobody wrote down what good means in countable terms. Anthropic's evaluation guidance sets the bar directly: criteria should be specific, measurable, achievable and relevant, and it demonstrates the difference with a safety example. The weak version is the goal safe outputs. The strong version is fewer than 0.1 percent of outputs across 10,000 trials flagged for toxicity by a content filter.

The second version is usable because it names a threshold, a sample size, and the instrument doing the measuring. Write your criteria in that shape before you collect a single test case, because the criteria determine what you need to collect.

• Name the metric, the threshold, and the measuring instrument in the same sentence.
• Set the target from prior runs or a documented baseline, not from optimism.
• Write separate criteria for correctness, for refusal behaviour, and for cost or latency if they matter.
• If a criterion cannot be counted, either find a proxy that can or drop it.

Step 2: build a small golden set from real failures

You do not need a large dataset to get a usable signal. A set of 20 to 50 cases with known-correct outcomes will catch most regressions, provided the cases are drawn from reality rather than invented at a whiteboard. The best sources are support tickets, logged sessions where the agent went wrong, and the edge cases your team already argues about.

Weight the set toward failure. A suite where the agent passes everything on the first run is measuring nothing, because it has no room to detect a regression. Include the ambiguous inputs, the adversarial ones, and the cases where the correct behaviour is to refuse or to ask a clarifying question. Store the set in version control next to the code so a case can never be quietly edited to make a run pass.

• Draw cases from real logs and real complaints, not from imagination.
• Record the expected outcome for each case at the time you add it.
• Include refusals and clarification cases, not only successful completions.
• Version the set alongside the code and treat edits to expectations as reviewable changes.

Step 3: pick the cheapest grader that works

Grading methods form a ladder, and the guidance from Anthropic spans it explicitly, from exact match checks through to model-based grading. Climb only as far as you must. Exact match, schema validation, a regular expression, or an assertion that a specific tool was called are all deterministic, fast, and free of judgment error. Use them wherever the answer is checkable.

Model-based grading, commonly called LLM-as-a-judge, is the tool for open-ended output where no deterministic check exists. It works, but it has documented failure modes. The MT-Bench and Chatbot Arena paper names position bias, verbosity bias and self-enhancement bias alongside limited reasoning ability, while also reporting that strong judges reached over 80 percent agreement with human preferences, comparable to agreement between humans themselves.

Read both halves of that result honestly. A model judge is good enough to be useful and biased enough to need controls. Randomize the order of compared answers, keep the rubric explicit and short, and spot-check a sample of judgments by hand rather than assuming the judge is calibrated.

• Deterministic checks first: exact match, schema, regex, tool-call assertions.
• Model-based grading only for genuinely open-ended output.
• Randomize presentation order to blunt position bias.
• Hand-check a sample of judge decisions regularly; never treat the judge as ground truth.

Step 4: run it as a gate, not as a ritual

An evaluation you run by hand before a launch is a ceremony. An evaluation that runs automatically on every change is a control. Open-source harnesses are built for this: promptfoo documents use as a CLI, as a library, or inside a CI pipeline, and describes its aim as test-driven development for LLM work rather than trial and error. DeepEval occupies the same category from a different maintainer, which is a useful signal that the pattern is not vendor-specific.

Wire the suite to the events that actually change behaviour. That means prompt edits, model version changes, tool or schema changes, retrieval index rebuilds, and dependency upgrades. Model providers ship updates on their own schedule, so a suite that only runs when you change your code will miss the day the model changes underneath it.

What this does not tell you

An eval suite measures the cases it contains. It cannot certify behaviour on inputs you never imagined, and a passing suite is not evidence of safety in a high-stakes domain. Treat it as a regression detector and a comparison instrument, not as a guarantee.

Scores also drift for uninteresting reasons. Sampling temperature, a changed system prompt, and a silently updated model version all move numbers without anyone editing the agent. Record the model version, the prompt version and the settings with every run, because a score without that context cannot be compared to last week's.

RepoRadar has not benchmarked the frameworks named here against each other. They are cited as documented examples of the category, not as a ranking or a recommendation, and no pricing or performance claim is made about any of them.

A useful next action

Open your logs, find the ten most recent cases where the agent produced the wrong result, and write down what the right result would have been. That file is your first eval set, and it took an afternoon rather than a quarter.

Run it against your current setup to get a baseline number, then run it again after your next prompt or model change. The first time it catches a regression you would otherwise have shipped, the practice will justify itself. If you also want to read public benchmarks more critically, RepoRadar's answer on how accurate AI benchmarks are covers what those numbers can and cannot support.