MMMU
productionCollege-level questions over charts, diagrams, maps, and scientific images across 30 subjects.
- Size
- 11,500 questions
- Runs with
- Inspect AI
- License
- Not stated
- Category
- Multimodal
Voice, multimodal, and instruction-following suites cover images, audio, latency, and strict format constraints that text-only benchmarks miss.
4 of 4 suites
College-level questions over charts, diagrams, maps, and scientific images across 30 subjects.
Prompts with verifiable instructions such as word limits, required keywords, and formats, checked by code instead of a judge.
Pipecat's eval tooling runs text or audio scenarios against voice agents with semantic checks, latency budgets, and LLM judging. You define the scenarios.
Diagnostics for multimodal LLM-as-judge bias across image, video, and speech. The benchmark in the paper is named D3-Omni, not OmniBias.
Size and license come from each suite's primary source. Not stated means the source doesn't declare a license; check the repo before commercial use.