MMLU-Pro
Harder MMLU with 10 answer options and more reasoning-heavy questions across 14 subjects.
Failure modes MMLU-Pro catches
Guessing
Shallow knowledge
Starter command
A starting point, not a guarantee. Commands change between releases, so confirm against the official README.
pip install inspect-ai inspect-evals
inspect eval inspect_evals/mmlu_pro --model openai/gpt-4.1 --limit 20Sources
- 1. MMLU-Pro official page or repository · TIGER-Lab · Repository · Sources checked
- 2. MMLU-Pro paper · arXiv · Paper · Sources checked