Skip to content
Eval Suites50

BBQ (Bias Benchmark for QA)

Question sets that test reliance on social stereotypes across nine protected categories, in ambiguous and disambiguated contexts.

BBQ (Bias Benchmark for QA) at a glance

Maintainer
NYU Machine Learning for Language / BBQ authors
Size
58,492 examples[1]
Category
Security & Red-Teaming
Runs with
Inspect AI
License
CC-BY-4.0
Tier
starter

Failure modes BBQ (Bias Benchmark for QA) catches

Stereotyped answers
Bias under ambiguity

Starter command

A starting point, not a guarantee. Commands change between releases, so confirm against the official README.

BBQ (Bias Benchmark for QA) starter
pip install inspect-ai inspect-evals
inspect eval inspect_evals/bbq --model openai/gpt-4.1 --limit 20

Sources

  1. 1. BBQ (Bias Benchmark for QA) official page or repository · NYU Machine Learning for Language / BBQ authors · Repository · Sources checked
  2. 2. BBQ (Bias Benchmark for QA) paper · arXiv · Paper · Sources checked