Skip to content
Eval Suites50

The Complete Checklist for RAG Evaluation Metrics (Faithfulness, Recall & Hallucination Traps)

Published Sources checked 3 min read · By the Eval Suites editors

At-a-glance answer (TL;DR)

Evaluate a RAG system in two halves. Retrieval: did the right passages come back (context recall and context precision)? Generation: is every claim in the answer supported by those passages (faithfulness), and does it address the question (response relevancy)? Then add hallucination traps, meaning questions your corpus can't answer, and check that the system says so instead of inventing an answer.

Why RAG evaluation splits into retrieval and generation

Split the score, or you won't know what to fix. A wrong answer can come from bad retrieval, a model that ignored good context, or both. The Ragas paper frames RAG evaluation around exactly these dimensions: whether retrieved context is relevant and focused, and whether the answer is faithful to it.[1]

The RAG metric checklist

Core RAG metrics, what they catch, and their blind spots
MetricMeasuresNeeds a reference answer?Blind spot
Context recallShare of the facts needed for the reference answer that appear in retrieved contextYesSays nothing about the generated answer
Context precisionWhether relevant chunks rank above irrelevant onesUsuallyA precise retriever can still miss key facts
FaithfulnessShare of answer claims supported by retrieved contextNoA faithful answer can still be off-topic or incomplete
Response relevancyWhether the answer addresses the question askedNoRewards on-topic answers even when they're wrong
Abstention rate on trapsHow often the system declines unanswerable questionsNo, but traps need labelsToo high means the system refuses real questions

Definitions follow the Ragas documentation, which lists these and more, including noise sensitivity and factual correctness.[2] Ragas computes faithfulness by breaking the answer into individual claims and checking each one against the retrieved context.[3]

How to build hallucination traps that actually catch something

Write questions that look answerable but aren't, then check that the system says it doesn't know. Faithfulness alone won't catch this. A model can paraphrase an irrelevant chunk faithfully and still mislead someone.

  • Near-miss entities: ask about a product version, policy year, or customer tier that doesn't exist in the corpus.
  • False premises: "Why did we raise the refund window to 90 days?" when it's still 30.
  • Split facts: questions where half the answer is in the docs and half isn't, so a confident full answer means invention.
  • Stale facts: put an old and a new version of a document in the index and check which one the answer follows.

OpenAI's SimpleQA work is a useful model here: 4,326 short, fact-seeking questions graded as correct, incorrect, or not attempted, which rewards a model for declining instead of guessing.[4] Copy that three-way grading for your own traps.

Testing multi-hop questions

If users ask questions that need two or more documents, test that directly. Google's FRAMES benchmark has 824 multi-hop questions that require combining several Wikipedia articles, and its authors found that retrieval quality made a large difference to accuracy.[5] Build 20 to 50 multi-hop questions from your own corpus and track them as their own slice.

A minimal RAG eval setup

Ragas evaluation (API names vary by release; check the docs)
from ragas import evaluate
from ragas.metrics import faithfulness, context_recall, context_precision, answer_relevancy

result = evaluate(dataset, metrics=[faithfulness, context_recall, context_precision, answer_relevancy])
print(result)

Metric names and imports have changed across Ragas releases, so pin the version and check the current docs before copying.[2] These metrics use an LLM judge, which means they inherit judge bias. Spot-check 30 verdicts by hand before you trust a threshold.

Frequently asked questions

Suites mentioned in this guide

Related guides

Sources

  1. 1. Ragas: Automated Evaluation of Retrieval Augmented Generation · arXiv:2309.15217 · Paper · Sources checked
  2. 2. Ragas: Available metrics · Ragas · Official docs · Sources checked
  3. 3. Ragas: Faithfulness · Ragas · Official docs · Sources checked
  4. 4. Measuring short-form factuality in large language models · arXiv:2411.04368 (OpenAI) · Paper · Sources checked
  5. 5. Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation · arXiv:2409.12941 (Google) · Paper · Sources checked