Skip to content
Eval Suites50

Agents & Tool Use eval suites

Agent and tool-use eval suites test whether a model finishes multi-step tasks in real or simulated environments, graded on the end state rather than the agent's own account of what it did.

8 of 8 suites

Fixes real GitHub issues in Python repos; a patch passes only if the repo's hidden tests pass. Verified is the 500-task human-validated subset.

Size
500 tasks
Runs with
Docker / Inspect AI
License
MIT
Category
Agents

Multi-turn agents follow domain policy while talking to a simulated user. τ²-bench adds a telecom domain where agent and user both act on shared state (dual-control).

Size
165 tasks (τ) · 278 (τ²)
Runs with
Python
License
MIT
Category
Agents

GAIA

production

General-assistant questions that need web browsing, file reading, and multi-step tool use, with short exact answers.

Size
466 questions
Runs with
Inspect AI
License
Not stated
Category
Agents

OSWorld

frontier

Open-ended computer-use tasks on real desktop apps in Ubuntu, checked by execution-based scripts.

Size
369 tasks
Runs with
Sandbox VM
License
Apache-2.0
Category
Agents

Hard tasks in real terminal environments, from compiling code to configuring servers, run through the Harbor framework.

Size
89 tasks
Runs with
Harbor
License
Apache-2.0
Category
Agents

Scores simple, parallel, nested, and multi-turn function calls against expected calls and executable checks.

Size
Varies by version
Runs with
Python
License
Apache-2.0
Category
Agents

WebArena

frontier

Browser agents complete tasks on self-hosted e-commerce, forum, GitLab, and CMS sites, graded on final page or DB state.

Size
812 tasks
Runs with
Self-hosted sandbox
License
Apache-2.0
Category
Agents

WorkArena

production

Knowledge-work tasks inside ServiceNow: forms, lists, dashboards, and catalogs. WorkArena++ adds compositional tasks.

Size
33 task types (682 in ++)
Runs with
BrowserGym
License
Not stated
Category
Agents

Size and license come from each suite's primary source. Not stated means the source doesn't declare a license; check the repo before commercial use.