SWE-bench Verified
productionFixes real GitHub issues in Python repos; a patch passes only if the repo's hidden tests pass. Verified is the 500-task human-validated subset.
- Size
- 500 tasks
- Runs with
- Docker / Inspect AI
- License
- MIT
- Category
- Agents
Agent and tool-use eval suites test whether a model finishes multi-step tasks in real or simulated environments, graded on the end state rather than the agent's own account of what it did.
8 of 8 suites
Fixes real GitHub issues in Python repos; a patch passes only if the repo's hidden tests pass. Verified is the 500-task human-validated subset.
Multi-turn agents follow domain policy while talking to a simulated user. τ²-bench adds a telecom domain where agent and user both act on shared state (dual-control).
General-assistant questions that need web browsing, file reading, and multi-step tool use, with short exact answers.
Open-ended computer-use tasks on real desktop apps in Ubuntu, checked by execution-based scripts.
Hard tasks in real terminal environments, from compiling code to configuring servers, run through the Harbor framework.
Scores simple, parallel, nested, and multi-turn function calls against expected calls and executable checks.
Browser agents complete tasks on self-hosted e-commerce, forum, GitLab, and CMS sites, graded on final page or DB state.
Knowledge-work tasks inside ServiceNow: forms, lists, dashboards, and catalogs. WorkArena++ adds compositional tasks.
Size and license come from each suite's primary source. Not stated means the source doesn't declare a license; check the repo before commercial use.