How to Build a Production AI Agent Eval Suite in 2026 (CI/CD Gating Playbook)
Published Sources checked 4 min read · By the Eval Suites editors
What a production agent eval suite actually contains
An agent eval suite has three parts: tasks, environments, and scorers. The task is the instruction plus any starting state. The environment is what the agent can touch, like a repo, a database, a fake airline API. The scorer decides pass or fail, and it should read the final state of the environment whenever it can, not the agent's own description of what it did.
That last point matters more than people expect. Agents narrate success they didn't achieve. τ-bench grades by comparing the database state at the end of a conversation against an annotated goal state, which is why its results are hard to game with confident prose.[1]
- Tasks: 30 to 200 real cases pulled from production logs, support tickets, and bugs you've already shipped once.
- Environments: containers or mocks the agent can break without hurting anything. Inspect AI supports Docker and other sandboxes per sample.[2]
- Scorers: state checks and schema checks first, model-graded rubrics only where nothing deterministic exists.
How many times should each agent task run?
Run each task more than once. A single pass tells you the agent can succeed, not that it will. The τ-bench authors introduced pass^k, the chance an agent succeeds on all k independent tries of the same task, and reported that strong function-calling agents dropped below 25% on pass^8 in the retail domain.[1] That gap between "sometimes works" and "works every time" is exactly what customers feel.
In practice, three trials per task is a sensible floor for nightly runs, and five to eight for release candidates if your budget allows it. Write down the cost per full run before you commit to a number. Agent suites with real tool calls get expensive fast.
Which checks should block a merge?
Block merges only on checks that are fast, cheap, and stable. Everything else should report, not block. A gate that flakes once a week teaches the team to click "re-run" without reading it.
| Tier | When it runs | What it checks | Blocks merge? |
|---|---|---|---|
| Smoke | Every pull request | Tool-call JSON validity, schema shape, refusal on 10 to 20 known-bad prompts | Yes |
| Regression | Every pull request touching prompts, tools, or model config | 30 to 100 recorded tasks, 1 trial, deterministic state checks | Yes, with a tolerance band |
| Reliability | Nightly | Full suite, 3+ trials per task, pass^k and cost per task | No, posts a report |
| Release | Before a model or prompt version ships | Full suite, 5 to 8 trials, red-team set, human review of diffs | Yes, by a named owner |
Promptfoo ships a GitHub Action that runs an eval on pull requests and comments the before-and-after results, which covers the smoke and regression tiers for most teams.[3] DeepEval documents a pytest-style flow for the same job if your stack is Python-first.[4]
# .github/workflows/evals.yml (regression tier)
name: agent-evals
on: pull_request
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 22 }
- run: npx promptfoo@latest eval -c evals/regression.yaml -o results.json
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}How do you keep score changes from being noise?
Report a confidence interval next to every score. On a 50-task suite, one task is two percentage points, so small moves are usually noise. Anthropic's "Adding Error Bars to Evals" paper recommends standard errors, clustered when questions share a source, and paired comparisons when you test two models on the same questions.[5]
- Compare the new version and the old version on the same tasks in the same run, so per-task difficulty cancels out.
- Set the regression tolerance from the observed spread across reruns, not from a number that sounded strict.
- Pin model versions, temperature, and tool mocks. Log all three with each result.
Should you test the user side of the conversation too?
Yes, if your agent works with people who also take actions. τ²-bench models this as a dual-control environment, where both the agent and a simulated user can change shared state, as in telecom troubleshooting where the user has to toggle settings on their own phone.[6] The authors report a significant performance drop when agents move from solo work to guiding a user.[6]
You don't need their whole framework to apply the idea. Add a handful of tasks where success requires the agent to give correct instructions and then verify the user followed them.
Where to start this week
- Pull 30 real failures from logs or tickets and write each as a task with an end-state check.
- Wire those 30 into a PR gate with a single trial and a tolerance band.
- Schedule a nightly job that runs them three times and records pass^3 and cost.
- Add public suites for coverage you can't build yourself, such as tool calling or prompt injection, from the agents category in this registry.
If you already use Inspect AI, the inspect_evals repository has many public benchmarks packaged as tasks you can run with one command.[7]
Frequently asked questions
Suites mentioned in this guide
Related guides
Sources
- 1. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains · arXiv:2406.12045 (Sierra) · Paper · Sources checked
- 2. Inspect AI: Sandboxing · UK AI Security Institute · Official docs · Sources checked
- 3. Promptfoo: GitHub Action integration · Promptfoo · Official docs · Sources checked
- 4. DeepEval: Unit testing in CI/CD · Confident AI · Official docs · Sources checked
- 5. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations · arXiv:2411.00640 (Anthropic) · Paper · Sources checked
- 6. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment · arXiv:2506.07982 (Sierra) · Paper · Sources checked
- 7. inspect_evals repository · UK AI Security Institute (GitHub) · Repository · Sources checked