Skip to content
Eval Suites50

Voice & Multimodal eval suites

Voice, multimodal, and instruction-following suites cover images, audio, latency, and strict format constraints that text-only benchmarks miss.

4 of 4 suites

MMMU

production

College-level questions over charts, diagrams, maps, and scientific images across 30 subjects.

Size
11,500 questions
Runs with
Inspect AI
License
Not stated
Category
Multimodal

IFEval

starter

Prompts with verifiable instructions such as word limits, required keywords, and formats, checked by code instead of a judge.

Size
541 prompts
Runs with
Inspect AI
License
Apache-2.0
Category
Multimodal

Pipecat Evals

production

Pipecat's eval tooling runs text or audio scenarios against voice agents with semantic checks, latency budgets, and LLM judging. You define the scenarios.

Size
Your scenarios
Runs with
Pipecat
License
BSD-2-Clause
Category
Multimodal

Diagnostics for multimodal LLM-as-judge bias across image, video, and speech. The benchmark in the paper is named D3-Omni, not OmniBias.

Size
10,671 samples
Runs with
Python
License
Not stated
Category
Multimodal

Size and license come from each suite's primary source. Not stated means the source doesn't declare a license; check the repo before commercial use.