Skip to content
Eval Suites50

Harnesses & Runners eval suites

Harnesses and runners are the tools you run evals with. They don't ship a fixed dataset; you bring tests or load public suites.

5 of 5 suites

Promptfoo

starter

Declarative YAML test suites with deterministic and model-graded assertions, red-teaming, and a GitHub Action for PR gates.

Size
Tool (bring your tests)
Runs with
Promptfoo CLI
License
MIT
Category
Harnesses

DeepEval

starter

Pytest-style LLM testing with 50+ metrics for RAG, agents, and custom G-Eval rubrics.

Size
Tool (50+ metrics)
Runs with
DeepEval / Pytest
License
Apache-2.0
Category
Harnesses

Open-source scorer library (factuality, closed QA, JSON diff, embeddings) usable alone or with Braintrust's commercial platform.

Size
Library
Runs with
TypeScript / Python
License
MIT
Category
Harnesses

orq-arena

starter

Round-robin, head-to-head model comparisons on your own prompts with LLM judges and Elo rankings.

Size
Your prompts
Runs with
CLI / Python
License
MIT
Category
Harnesses

A curated reading list of agent eval resources and notes, not a runnable suite. Useful for finding patterns and papers.

Size
443+ curated links
Runs with
GitHub / Markdown
License
CC0-1.0
Category
Harnesses

Size and license come from each suite's primary source. Not stated means the source doesn't declare a license; check the repo before commercial use.