AI agent benchmarks: SWE-bench Verified, τ-bench and GAIA
Sources checked 2 min read · By the Eval Suites editors
In brief
SWE-bench Verified tests issue-solving in software repositories, τ-bench tests tool use and domain-policy adherence in conversations, and GAIA tests general-assistant questions needing reasoning and tools. Their tasks and scorers differ, so a score from one is not a substitute for the others.[1][2][3]
- SWE-bench Verified
- 500 human-validated samples [1]
- GAIA original benchmark
- 466 questions [3]
- Application release guarantee
- None; add your own workflow tests
What benchmarks measure multi-turn tool use and agent planning?
τ-bench directly studies tool-agent-user interaction; GAIA studies general-assistant reasoning and tool use; SWE-bench Verified studies repository issue-solving. Planning is exercised through task completion, not measured as one interchangeable “planning score.”[2][3][1]
| Benchmark | Task | Evidence of success | Limit |
|---|---|---|---|
| SWE-bench Verified | Resolve a real repository issue | Applicable tests pass after the patch | Coding subset, not broad agent safety |
| τ-bench | Follow policy while conversing and using tools | Final database state matches annotated goal | Simulated domains and users |
| τ²-bench | Coordinate agent and user actions in shared state | Environment and task evaluation | Dual-control reliability, not a security certification |
| GAIA | Answer general-assistant questions requiring tools | Answer correctness for the task | Not your application’s permission policy |
The table follows the original papers and the Verified release description. Keep benchmark version, subset, model, tool access and scoring details alongside results.[1][2][4][3]
How does SWE-bench Verified differ from the original SWE-bench?
SWE-bench Verified is a 500-sample subset screened by human annotators for issues such as underspecified descriptions and problematic tests. OpenAI released it with the SWE-bench authors; it is not a new collection of 500 synthetic tasks.[1]
Verified makes the evaluated subset more clearly scoped, but it does not imply that every possible correct patch will pass, that no task has leaked into training, or that the agent can handle an arbitrary production repository. Read failures and reproduce the environment before blaming the model.
Container setup, test execution and repeated model calls add operational cost. There is no universal run price: measure your model usage, infrastructure and timeout behavior on the selected tasks rather than borrowing a leaderboard’s cost assumption.
What is tau-bench and how does it test agent policy violations?
τ-bench gives an agent domain-specific tools and policy guidelines while an LLM simulates the user; it compares the final database state with an annotated goal. Its pass^k metric studies consistency across repeated trials, not just one successful attempt.[2]
A policy-related task can fail because the agent changes the wrong state or cannot complete the permitted goal. Inspect the task and trace before labeling every failed case a security violation; goal-state accuracy and prohibited-action detection are related but different checks.[2]
τ²-bench adds a dual-control domain where user and agent both act in the shared environment. That models coordination, such as guiding a user to change a device setting; it should not be described as a published prompt-injection attack suite.[4]
What does GAIA add, and how should I budget benchmark runs?
GAIA adds general-assistant questions involving reasoning, multimodal inputs, browsing and tools. The original paper describes 466 questions and retains some answers for leaderboard evaluation.[3]
- Choose the benchmark aligned to the capability you need, not whichever leaderboard score is highest.
- Run a controlled subset to verify environment setup and record token use, elapsed time and failed requests.
- Pin the benchmark subset and execution settings. Report excluded cases and timeout policy.
- Add application-specific permissions, data and end-state checks before using any public score in a release decision.
Use the individual directory entries for official repository or dataset links. Public benchmarks remain independently maintained resources; Eval Suites does not bundle them into paid packs.
Related suites
Related reading
Sources
- 1. Introducing SWE-bench Verified · OpenAI and SWE-bench · Official docs · Sources checked
- 2. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains · arXiv:2406.12045 (Sierra) · Paper · Sources checked
- 3. GAIA: a benchmark for General AI Assistants · arXiv:2311.12983 · Paper · Sources checked
- 4. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment · arXiv:2506.07982 (Sierra) · Paper · Sources checked