Skip to content
Eval Suites50

MMLU-Pro

Harder MMLU with 10 answer options and more reasoning-heavy questions across 14 subjects.

MMLU-Pro at a glance

Maintainer
TIGER-Lab
Size
12,032 questions[1]
Category
Reasoning & Frontier
Runs with
Inspect AI
License
Not stated by the source. Check before commercial use.
Tier
starter

Failure modes MMLU-Pro catches

Guessing
Shallow knowledge

Starter command

A starting point, not a guarantee. Commands change between releases, so confirm against the official README.

MMLU-Pro starter
pip install inspect-ai inspect-evals
inspect eval inspect_evals/mmlu_pro --model openai/gpt-4.1 --limit 20

Sources

  1. 1. MMLU-Pro official page or repository · TIGER-Lab · Repository · Sources checked
  2. 2. MMLU-Pro paper · arXiv · Paper · Sources checked