Skip to content
Eval Suites50

How to Build a Production AI Agent Eval Suite in 2026 (CI/CD Gating Playbook)

Published Sources checked 4 min read · By the Eval Suites editors

At-a-glance answer (TL;DR)

A production agent eval suite is a versioned set of tasks, each with a sandboxed environment and a check you trust, run several times per change so you can see reliability, not just one lucky pass. Gate merges on cheap deterministic checks for every pull request, run the slow multi-trial agent suite nightly or before release, and report results with error bars so a two-point swing on 50 tasks doesn't block anyone.

What a production agent eval suite actually contains

An agent eval suite has three parts: tasks, environments, and scorers. The task is the instruction plus any starting state. The environment is what the agent can touch, like a repo, a database, a fake airline API. The scorer decides pass or fail, and it should read the final state of the environment whenever it can, not the agent's own description of what it did.

That last point matters more than people expect. Agents narrate success they didn't achieve. τ-bench grades by comparing the database state at the end of a conversation against an annotated goal state, which is why its results are hard to game with confident prose.[1]

  • Tasks: 30 to 200 real cases pulled from production logs, support tickets, and bugs you've already shipped once.
  • Environments: containers or mocks the agent can break without hurting anything. Inspect AI supports Docker and other sandboxes per sample.[2]
  • Scorers: state checks and schema checks first, model-graded rubrics only where nothing deterministic exists.

How many times should each agent task run?

Run each task more than once. A single pass tells you the agent can succeed, not that it will. The τ-bench authors introduced pass^k, the chance an agent succeeds on all k independent tries of the same task, and reported that strong function-calling agents dropped below 25% on pass^8 in the retail domain.[1] That gap between "sometimes works" and "works every time" is exactly what customers feel.

In practice, three trials per task is a sensible floor for nightly runs, and five to eight for release candidates if your budget allows it. Write down the cost per full run before you commit to a number. Agent suites with real tool calls get expensive fast.

Which checks should block a merge?

Block merges only on checks that are fast, cheap, and stable. Everything else should report, not block. A gate that flakes once a week teaches the team to click "re-run" without reading it.

Suggested gating tiers for an agent eval suite
TierWhen it runsWhat it checksBlocks merge?
SmokeEvery pull requestTool-call JSON validity, schema shape, refusal on 10 to 20 known-bad promptsYes
RegressionEvery pull request touching prompts, tools, or model config30 to 100 recorded tasks, 1 trial, deterministic state checksYes, with a tolerance band
ReliabilityNightlyFull suite, 3+ trials per task, pass^k and cost per taskNo, posts a report
ReleaseBefore a model or prompt version shipsFull suite, 5 to 8 trials, red-team set, human review of diffsYes, by a named owner

Promptfoo ships a GitHub Action that runs an eval on pull requests and comments the before-and-after results, which covers the smoke and regression tiers for most teams.[3] DeepEval documents a pytest-style flow for the same job if your stack is Python-first.[4]

Minimal regression gate
# .github/workflows/evals.yml (regression tier)
name: agent-evals
on: pull_request
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npx promptfoo@latest eval -c evals/regression.yaml -o results.json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

How do you keep score changes from being noise?

Report a confidence interval next to every score. On a 50-task suite, one task is two percentage points, so small moves are usually noise. Anthropic's "Adding Error Bars to Evals" paper recommends standard errors, clustered when questions share a source, and paired comparisons when you test two models on the same questions.[5]

  • Compare the new version and the old version on the same tasks in the same run, so per-task difficulty cancels out.
  • Set the regression tolerance from the observed spread across reruns, not from a number that sounded strict.
  • Pin model versions, temperature, and tool mocks. Log all three with each result.

Should you test the user side of the conversation too?

Yes, if your agent works with people who also take actions. τ²-bench models this as a dual-control environment, where both the agent and a simulated user can change shared state, as in telecom troubleshooting where the user has to toggle settings on their own phone.[6] The authors report a significant performance drop when agents move from solo work to guiding a user.[6]

You don't need their whole framework to apply the idea. Add a handful of tasks where success requires the agent to give correct instructions and then verify the user followed them.

Where to start this week

  1. Pull 30 real failures from logs or tickets and write each as a task with an end-state check.
  2. Wire those 30 into a PR gate with a single trial and a tolerance band.
  3. Schedule a nightly job that runs them three times and records pass^3 and cost.
  4. Add public suites for coverage you can't build yourself, such as tool calling or prompt injection, from the agents category in this registry.

If you already use Inspect AI, the inspect_evals repository has many public benchmarks packaged as tasks you can run with one command.[7]

Frequently asked questions

Suites mentioned in this guide

Related guides

Sources

  1. 1. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains · arXiv:2406.12045 (Sierra) · Paper · Sources checked
  2. 2. Inspect AI: Sandboxing · UK AI Security Institute · Official docs · Sources checked
  3. 3. Promptfoo: GitHub Action integration · Promptfoo · Official docs · Sources checked
  4. 4. DeepEval: Unit testing in CI/CD · Confident AI · Official docs · Sources checked
  5. 5. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations · arXiv:2411.00640 (Anthropic) · Paper · Sources checked
  6. 6. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment · arXiv:2506.07982 (Sierra) · Paper · Sources checked
  7. 7. inspect_evals repository · UK AI Security Institute (GitHub) · Repository · Sources checked