Skip to content
Eval Suites50

Coding & Systems eval suites

Coding eval suites run generated code or patches against tests, so a pass means the code works, not that it looks right.

6 of 6 suites

An open RL-trained coding agent, not a benchmark. Its release reports results on SWE-bench Verified with test-based verifiers; listed here for its reproducible evaluation setup.

Size
Model, not a dataset
Runs with
R2E-Gym
License
Not stated
Category
Coding

LiveCodeBench

production

Competitive programming problems collected over time with release dates, so you can score only problems published after a model's cutoff.

Size
Grows by release
Runs with
Python
License
MIT
Category
Coding

KernelBench

frontier

Asks models to write GPU kernels for PyTorch workloads and checks both correctness and speedup over the reference.

Size
250 workloads
Runs with
Python / CUDA
License
MIT
Category
Coding

HumanEval's 164 problems with roughly 80x more tests, catching code that passed the original thin test suites.

Size
164 problems
Runs with
EvalPlus
License
Apache-2.0
Category
Coding

Aider Polyglot

production

Code-editing benchmark from hard Exercism exercises in C++, Go, Java, JavaScript, Python, and Rust; tests whether edits apply cleanly.

Size
225 exercises
Runs with
Aider CLI
License
Apache-2.0
Category
Coding

SWE-Lancer

frontier

Real Upwork freelance software tasks valued at $1 million total, including manager-style proposal selection, graded by end-to-end tests.

Size
1,488 tasks
Runs with
Docker
License
MIT
Category
Coding

Size and license come from each suite's primary source. Not stated means the source doesn't declare a license; check the repo before commercial use.