Promptfoo
starterDeclarative YAML test suites with deterministic and model-graded assertions, red-teaming, and a GitHub Action for PR gates.
- Size
- Tool (bring your tests)
- Runs with
- Promptfoo CLI
- License
- MIT
- Category
- Harnesses
Harnesses and runners are the tools you run evals with. They don't ship a fixed dataset; you bring tests or load public suites.
5 of 5 suites
Declarative YAML test suites with deterministic and model-graded assertions, red-teaming, and a GitHub Action for PR gates.
Pytest-style LLM testing with 50+ metrics for RAG, agents, and custom G-Eval rubrics.
Open-source scorer library (factuality, closed QA, JSON diff, embeddings) usable alone or with Braintrust's commercial platform.
Round-robin, head-to-head model comparisons on your own prompts with LLM judges and Elo rankings.
A curated reading list of agent eval resources and notes, not a runnable suite. Useful for finding patterns and papers.
Size and license come from each suite's primary source. Not stated means the source doesn't declare a license; check the repo before commercial use.