Skip to content
Eval Suites50

Security & Red-Teaming eval suites

Security and red-teaming suites measure prompt injection resistance, jailbreak robustness, cyber capability, and bias, with utility tracked alongside attack success.

8 of 8 suites

Inspect Evals

production

Community collection of benchmarks packaged for Inspect AI: safety, cyber, coding, reasoning, and knowledge. Maintained by Generality Labs; founded with the UK AI Security Institute.

Size
Collection (many evals)
Runs with
Inspect AI
License
MIT
Category
Security

Cybench

frontier

Professional capture-the-flag challenges from four competitions, with optional subtasks for partial credit.

Size
40 CTF tasks
Runs with
Inspect AI
License
Apache-2.0
Category
Security

CVE-Bench

frontier

Agents try to exploit real critical-severity web application CVEs in sandboxed containers.

Size
40 CVEs
Runs with
Docker
License
Apache-2.0
Category
Security

CyberSecEval 3

production

Meta's PurpleLlama suite for insecure code generation, prompt injection, and offensive-cyber risk.

Size
Multiple sub-benchmarks
Runs with
Python
License
MIT
Category
Security

HarmBench

production

Standardized red-teaming framework comparing automated jailbreak attacks and refusal robustness.

Size
510 behaviors
Runs with
Python
License
MIT
Category
Security

DUMA-Bench

frontier

Extends τ²-bench with adversarial dual-control security scenarios, where the user or shared environment tries to manipulate the agent.

Size
8 security domains
Runs with
Python
License
MIT
Category
Security

AgentDojo

production

Prompt injection attacks arrive through tool outputs while agents do realistic tasks; scores utility and attack success together.

Size
97 tasks · 629 security cases
Runs with
Python
License
MIT
Category
Security

Question sets that test reliance on social stereotypes across nine protected categories, in ambiguous and disambiguated contexts.

Size
58,492 examples
Runs with
Inspect AI
License
CC-BY-4.0
Category
Security

Size and license come from each suite's primary source. Not stated means the source doesn't declare a license; check the repo before commercial use.