Skip to content
Eval Suites50

SimpleQA

Short fact-seeking questions graded correct, incorrect, or not attempted, so declining beats a confident wrong answer.

SimpleQA at a glance

Maintainer
OpenAI
Size
4,326 questions[1]
Category
RAG & Factuality
Runs with
Inspect AI
License
MIT
Tier
starter

Failure modes SimpleQA catches

Confident hallucination
Poor calibration

Starter command

A starting point, not a guarantee. Commands change between releases, so confirm against the official README.

SimpleQA starter
pip install inspect-ai inspect-evals
inspect eval inspect_evals/simpleqa --model openai/gpt-4.1 --limit 20

Sources

  1. 1. SimpleQA official page or repository · OpenAI · Repository · Sources checked
  2. 2. SimpleQA paper · arXiv · Paper · Sources checked