SimpleQA
Short fact-seeking questions graded correct, incorrect, or not attempted, so declining beats a confident wrong answer.
Failure modes SimpleQA catches
Confident hallucination
Poor calibration
Starter command
A starting point, not a guarantee. Commands change between releases, so confirm against the official README.
pip install inspect-ai inspect-evals
inspect eval inspect_evals/simpleqa --model openai/gpt-4.1 --limit 20Sources
- 1. SimpleQA official page or repository · OpenAI · Repository · Sources checked
- 2. SimpleQA paper · arXiv · Paper · Sources checked