The Complete Checklist for RAG Evaluation Metrics (Faithfulness, Recall & Hallucination Traps)
Published Sources checked 3 min read · By the Eval Suites editors
Why RAG evaluation splits into retrieval and generation
Split the score, or you won't know what to fix. A wrong answer can come from bad retrieval, a model that ignored good context, or both. The Ragas paper frames RAG evaluation around exactly these dimensions: whether retrieved context is relevant and focused, and whether the answer is faithful to it.[1]
The RAG metric checklist
| Metric | Measures | Needs a reference answer? | Blind spot |
|---|---|---|---|
| Context recall | Share of the facts needed for the reference answer that appear in retrieved context | Yes | Says nothing about the generated answer |
| Context precision | Whether relevant chunks rank above irrelevant ones | Usually | A precise retriever can still miss key facts |
| Faithfulness | Share of answer claims supported by retrieved context | No | A faithful answer can still be off-topic or incomplete |
| Response relevancy | Whether the answer addresses the question asked | No | Rewards on-topic answers even when they're wrong |
| Abstention rate on traps | How often the system declines unanswerable questions | No, but traps need labels | Too high means the system refuses real questions |
Definitions follow the Ragas documentation, which lists these and more, including noise sensitivity and factual correctness.[2] Ragas computes faithfulness by breaking the answer into individual claims and checking each one against the retrieved context.[3]
How to build hallucination traps that actually catch something
Write questions that look answerable but aren't, then check that the system says it doesn't know. Faithfulness alone won't catch this. A model can paraphrase an irrelevant chunk faithfully and still mislead someone.
- Near-miss entities: ask about a product version, policy year, or customer tier that doesn't exist in the corpus.
- False premises: "Why did we raise the refund window to 90 days?" when it's still 30.
- Split facts: questions where half the answer is in the docs and half isn't, so a confident full answer means invention.
- Stale facts: put an old and a new version of a document in the index and check which one the answer follows.
OpenAI's SimpleQA work is a useful model here: 4,326 short, fact-seeking questions graded as correct, incorrect, or not attempted, which rewards a model for declining instead of guessing.[4] Copy that three-way grading for your own traps.
Testing multi-hop questions
If users ask questions that need two or more documents, test that directly. Google's FRAMES benchmark has 824 multi-hop questions that require combining several Wikipedia articles, and its authors found that retrieval quality made a large difference to accuracy.[5] Build 20 to 50 multi-hop questions from your own corpus and track them as their own slice.
A minimal RAG eval setup
from ragas import evaluate
from ragas.metrics import faithfulness, context_recall, context_precision, answer_relevancy
result = evaluate(dataset, metrics=[faithfulness, context_recall, context_precision, answer_relevancy])
print(result)Metric names and imports have changed across Ragas releases, so pin the version and check the current docs before copying.[2] These metrics use an LLM judge, which means they inherit judge bias. Spot-check 30 verdicts by hand before you trust a threshold.
Frequently asked questions
Suites mentioned in this guide
Related guides
Sources
- 1. Ragas: Automated Evaluation of Retrieval Augmented Generation · arXiv:2309.15217 · Paper · Sources checked
- 2. Ragas: Available metrics · Ragas · Official docs · Sources checked
- 3. Ragas: Faithfulness · Ragas · Official docs · Sources checked
- 4. Measuring short-form factuality in large language models · arXiv:2411.04368 (OpenAI) · Paper · Sources checked
- 5. Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation · arXiv:2409.12941 (Google) · Paper · Sources checked