Skip to content
Eval Suites50

RAG & Factuality eval suites

RAG and factuality suites check whether answers are supported by evidence, whether retrieval found the right context, and whether a model declines when it doesn't know.

7 of 7 suites

Ragas

starter

Metrics library for RAG pipelines: faithfulness, response relevancy, context precision, and context recall, many reference-free.

Size
Library (bring your data)
Runs with
Python / Ragas
License
Apache-2.0
Category
RAG

DeepSearchQA

frontier

Hand-crafted prompts for deep-research agents that need many search steps to assemble complete answer sets.

Size
900 prompts
Runs with
Python
License
Apache-2.0
Category
RAG

FRAMES

production

Multi-hop questions that combine several Wikipedia articles, testing retrieval, reasoning, and factuality together.

Size
824 questions
Runs with
Python
License
Apache-2.0
Category
RAG

SimpleQA

starter

Short fact-seeking questions graded correct, incorrect, or not attempted, so declining beats a confident wrong answer.

Size
4,326 questions
Runs with
Inspect AI
License
MIT
Category
RAG

TruthfulQA

starter

Questions where common human misconceptions are the tempting answer; measures whether models repeat falsehoods.

Size
817 questions
Runs with
Inspect AI
License
Apache-2.0
Category
RAG

HaluEval

starter

Generated and human-annotated hallucination samples across QA, dialogue, and summarization for detection testing.

Size
35,000 samples
Runs with
Python
License
MIT
Category
RAG

AbstentionBench

production

Collects 20 datasets of unanswerable, underspecified, false-premise, and stale questions to test whether models abstain. Noncommercial license.

Size
20 datasets
Runs with
Python
License
CC-BY-NC-4.0
Category
RAG

Size and license come from each suite's primary source. Not stated means the source doesn't declare a license; check the repo before commercial use.