Skip to content
Eval Suites50

AbstentionBench

Collects 20 datasets of unanswerable, underspecified, false-premise, and stale questions to test whether models abstain. Noncommercial license.

AbstentionBench at a glance

Maintainer
Meta Platforms, Inc. and affiliates
Size
20 datasets[1]
Category
RAG & Factuality
Runs with
Python
License
CC-BY-NC-4.0
Tier
production

Failure modes AbstentionBench catches

Answering the unanswerable
Over-refusal

Starter command

A starting point, not a guarantee. Commands change between releases, so confirm against the official README.

AbstentionBench starter
git clone https://github.com/facebookresearch/AbstentionBench
# follow the repo README for setup and run commands

Sources

  1. 1. AbstentionBench official page or repository · Meta Platforms, Inc. and affiliates · Repository · Sources checked
  2. 2. AbstentionBench paper · arXiv · Paper · Sources checked