Skip to content
Eval Suites50

SWE-Lancer

Real Upwork freelance software tasks valued at $1 million total, including manager-style proposal selection, graded by end-to-end tests.

SWE-Lancer at a glance

Maintainer
OpenAI
Size
1,488 tasks[1]
Category
Coding & Systems
Runs with
Docker
License
MIT
Tier
frontier

Failure modes SWE-Lancer catches

Incomplete features
UI regressions
Bad tradeoff choices

Starter command

A starting point, not a guarantee. Commands change between releases, so confirm against the official README.

SWE-Lancer starter
git clone https://github.com/openai/SWELancer-Benchmark
# follow the repo README for setup and run commands

Sources

  1. 1. SWE-Lancer official page or repository · OpenAI · Repository · Sources checked
  2. 2. SWE-Lancer paper · arXiv · Paper · Sources checked