BBQ (Bias Benchmark for QA)
Question sets that test reliance on social stereotypes across nine protected categories, in ambiguous and disambiguated contexts.
Failure modes BBQ (Bias Benchmark for QA) catches
Stereotyped answers
Bias under ambiguity
Starter command
A starting point, not a guarantee. Commands change between releases, so confirm against the official README.
pip install inspect-ai inspect-evals
inspect eval inspect_evals/bbq --model openai/gpt-4.1 --limit 20Sources
- 1. BBQ (Bias Benchmark for QA) official page or repository · NYU Machine Learning for Language / BBQ authors · Repository · Sources checked
- 2. BBQ (Bias Benchmark for QA) paper · arXiv · Paper · Sources checked