Community collection of benchmarks packaged for Inspect AI: safety, cyber, coding, reasoning, and knowledge. Maintained by Generality Labs; founded with the UK AI Security Institute.
- Size
- Collection (many evals)
- Runs with
- Inspect AI
- License
- MIT
- Category
- Security
Professional capture-the-flag challenges from four competitions, with optional subtasks for partial credit.
- Size
- 40 CTF tasks
- Runs with
- Inspect AI
- License
- Apache-2.0
- Category
- Security
Agents try to exploit real critical-severity web application CVEs in sandboxed containers.
- Size
- 40 CVEs
- Runs with
- Docker
- License
- Apache-2.0
- Category
- Security
Meta's PurpleLlama suite for insecure code generation, prompt injection, and offensive-cyber risk.
- Size
- Multiple sub-benchmarks
- Runs with
- Python
- License
- MIT
- Category
- Security
Standardized red-teaming framework comparing automated jailbreak attacks and refusal robustness.
- Size
- 510 behaviors
- Runs with
- Python
- License
- MIT
- Category
- Security
Extends τ²-bench with adversarial dual-control security scenarios, where the user or shared environment tries to manipulate the agent.
- Size
- 8 security domains
- Runs with
- Python
- License
- MIT
- Category
- Security
Prompt injection attacks arrive through tool outputs while agents do realistic tasks; scores utility and attack success together.
- Size
- 97 tasks · 629 security cases
- Runs with
- Python
- License
- MIT
- Category
- Security
Question sets that test reliance on social stereotypes across nine protected categories, in ambiguous and disambiguated contexts.
- Size
- 58,492 examples
- Runs with
- Inspect AI
- License
- CC-BY-4.0
- Category
- Security
Size and license come from each suite's primary source. Not stated means the source doesn't declare a license; check the repo before commercial use.