SWE-bench Verified
Fixes real GitHub issues in Python repos; a patch passes only if the repo's hidden tests pass. Verified is the 500-task human-validated subset.
Failure modes SWE-bench Verified catches
Broken patches
Test regressions
Wrong file edits
Starter command
A starting point, not a guarantee. Commands change between releases, so confirm against the official README.
pip install inspect-ai inspect-evals
inspect eval inspect_evals/swe_bench --model openai/gpt-4.1 --limit 20Sources
- 1. SWE-bench Verified official page or repository · SWE-bench team; Verified released with OpenAI · Repository · Sources checked
- 2. SWE-bench Verified paper · arXiv · Paper · Sources checked