Skip to content
Eval Suites50

How Eval Suites verifies listings

Sources checked

What we check on every Eval Suites listing

Every listing's maintainer, size, and license come from the suite's own repository, dataset card, or paper. If those sources don't state a license, we say "Not stated" instead of guessing.

  • Size uses the unit the source uses: tasks, questions, prompts, or task types. We don't add units together.
  • License is the SPDX id from the repo. Noncommercial licenses, like CC-BY-NC, aren't counted as open source.
  • Starter commands are starting points. Commands change between releases, so each one links back to the official README.

What we corrected during the first check

Some commonly repeated descriptions didn't match their sources, so the listings say what the source says. GAIA has 466 questions, not 450. DeepSWE is a trained coding agent, not a benchmark. Awesome Agent Evals is a reading list. The paper behind "OmniBias / OmniJudge" names its benchmark D3-Omni. AbstentionBench uses a noncommercial license.

How sponsorship and paid listings are kept separate

Paid placements are labeled Sponsored or Featured. They change where a listing appears, never its facts. A $79 Fast-Track listing gets a faster review, not an easier one. See sponsor options and submission tiers.

How to report an error

If a listing is wrong, resubmit it through the submission form with a link to the source that shows the correct fact.