Skip to content
Eval Suites50

SWE-bench Verified

Fixes real GitHub issues in Python repos; a patch passes only if the repo's hidden tests pass. Verified is the 500-task human-validated subset.

SWE-bench Verified at a glance

Maintainer
SWE-bench team; Verified released with OpenAI
Size
500 tasks[1]
Category
Agents & Tool Use
Runs with
Docker / Inspect AI
License
MIT
Tier
production

Failure modes SWE-bench Verified catches

Broken patches
Test regressions
Wrong file edits

Starter command

A starting point, not a guarantee. Commands change between releases, so confirm against the official README.

SWE-bench Verified starter
pip install inspect-ai inspect-evals
inspect eval inspect_evals/swe_bench --model openai/gpt-4.1 --limit 20

Sources

  1. 1. SWE-bench Verified official page or repository · SWE-bench team; Verified released with OpenAI · Repository · Sources checked
  2. 2. SWE-bench Verified paper · arXiv · Paper · Sources checked