Skip to content
Eval Suites50

AI agent benchmarks: SWE-bench Verified, τ-bench and GAIA

Sources checked 2 min read · By the Eval Suites editors

In brief

SWE-bench Verified tests issue-solving in software repositories, τ-bench tests tool use and domain-policy adherence in conversations, and GAIA tests general-assistant questions needing reasoning and tools. Their tasks and scorers differ, so a score from one is not a substitute for the others.[1][2][3]

SWE-bench Verified
500 human-validated samples [1]
GAIA original benchmark
466 questions [3]
Application release guarantee
None; add your own workflow tests

What benchmarks measure multi-turn tool use and agent planning?

τ-bench directly studies tool-agent-user interaction; GAIA studies general-assistant reasoning and tool use; SWE-bench Verified studies repository issue-solving. Planning is exercised through task completion, not measured as one interchangeable “planning score.”[2][3][1]

Benchmark tasks and what a successful run establishes
BenchmarkTaskEvidence of successLimit
SWE-bench VerifiedResolve a real repository issueApplicable tests pass after the patchCoding subset, not broad agent safety
τ-benchFollow policy while conversing and using toolsFinal database state matches annotated goalSimulated domains and users
τ²-benchCoordinate agent and user actions in shared stateEnvironment and task evaluationDual-control reliability, not a security certification
GAIAAnswer general-assistant questions requiring toolsAnswer correctness for the taskNot your application’s permission policy

The table follows the original papers and the Verified release description. Keep benchmark version, subset, model, tool access and scoring details alongside results.[1][2][4][3]

How does SWE-bench Verified differ from the original SWE-bench?

SWE-bench Verified is a 500-sample subset screened by human annotators for issues such as underspecified descriptions and problematic tests. OpenAI released it with the SWE-bench authors; it is not a new collection of 500 synthetic tasks.[1]

Verified makes the evaluated subset more clearly scoped, but it does not imply that every possible correct patch will pass, that no task has leaked into training, or that the agent can handle an arbitrary production repository. Read failures and reproduce the environment before blaming the model.

Container setup, test execution and repeated model calls add operational cost. There is no universal run price: measure your model usage, infrastructure and timeout behavior on the selected tasks rather than borrowing a leaderboard’s cost assumption.

What is tau-bench and how does it test agent policy violations?

τ-bench gives an agent domain-specific tools and policy guidelines while an LLM simulates the user; it compares the final database state with an annotated goal. Its pass^k metric studies consistency across repeated trials, not just one successful attempt.[2]

A policy-related task can fail because the agent changes the wrong state or cannot complete the permitted goal. Inspect the task and trace before labeling every failed case a security violation; goal-state accuracy and prohibited-action detection are related but different checks.[2]

τ²-bench adds a dual-control domain where user and agent both act in the shared environment. That models coordination, such as guiding a user to change a device setting; it should not be described as a published prompt-injection attack suite.[4]

What does GAIA add, and how should I budget benchmark runs?

GAIA adds general-assistant questions involving reasoning, multimodal inputs, browsing and tools. The original paper describes 466 questions and retains some answers for leaderboard evaluation.[3]

  1. Choose the benchmark aligned to the capability you need, not whichever leaderboard score is highest.
  2. Run a controlled subset to verify environment setup and record token use, elapsed time and failed requests.
  3. Pin the benchmark subset and execution settings. Report excluded cases and timeout policy.
  4. Add application-specific permissions, data and end-state checks before using any public score in a release decision.

Use the individual directory entries for official repository or dataset links. Public benchmarks remain independently maintained resources; Eval Suites does not bundle them into paid packs.

Related suites

Related reading

Sources

  1. 1. Introducing SWE-bench Verified · OpenAI and SWE-bench · Official docs · Sources checked
  2. 2. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains · arXiv:2406.12045 (Sierra) · Paper · Sources checked
  3. 3. GAIA: a benchmark for General AI Assistants · arXiv:2311.12983 · Paper · Sources checked
  4. 4. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment · arXiv:2506.07982 (Sierra) · Paper · Sources checked