Skip to content
Eval Suites50

How to build a golden dataset for your LLM application

Sources checked 3 min read · By the Eval Suites editors

In brief

A golden dataset is a reviewed reference set of inputs and expected behavior for your application. Start with real tasks, define what success means, and keep a held-out set separate from prompt tuning. Synthetic cases can fill coverage gaps, but they need review before they become reference labels.[1]

Purpose
Application-specific regression and evaluation reference
Sample-size guarantee
None; depends on uncertainty and risk
Required review
Verify labels, permissions and coverage before use

How do I create a golden test dataset for evaluating my LLM application?

Create the golden dataset around an observable outcome, not a collection of plausible prompts. OpenAI’s evaluation guidance recommends task-specific objectives and examples spanning typical, edge and adversarial cases, with human judgment used to calibrate automated scoring.[1]

  1. Choose one workflow, such as answering a policy question or issuing a permitted refund. Write its success and failure conditions.
  2. Collect consented, minimized production examples and known bugs. Remove secrets and personal data before sharing cases.
  3. Assign each case a stable ID and a slice: ordinary, difficult, ambiguous, unanswerable or adversarial. These are proposed planning labels, not universal benchmark categories.
  4. Write the expected tool calls, required evidence, acceptable answer and prohibited side effects. An open-ended answer can have a rubric instead of one exact sentence.
  5. Have a reviewer resolve ambiguous labels, then version the dataset and reserve unseen cases for evaluation.
Proposed reference-case record, using fictional data
{
  "id": "refund-duplicate-001",
  "slice": "edge",
  "input": "Refund the same order twice",
  "starting_state": {"order_id": "fixture-1042", "refunded": false},
  "expected_tool_calls": [{"name": "refund", "arguments": {"order_id": "fixture-1042"}}],
  "expected_context_ids": ["refund-policy-v1"],
  "reference_output": "The order was refunded once.",
  "forbidden_effects": ["second refund"],
  "label_status": "reviewed"
}

The record above is an editorial template, not a format required by every tool. Your adapter must translate it into the runner’s schema. Keep the starting state because the same request can require different behavior after a previous refund.

How many test cases do I need in a golden dataset for reliable eval?

There’s no universal minimum that makes an LLM evaluation reliable. A pilot of 50–200 curated cases can help you debug labels and coverage; it does not guarantee statistical significance. Experiment planning depends on the uncertainty and model difference you need to measure.[2]

For an illustrative independent binary pass/fail sample, the approximate 95% margin is 1.96 × sqrt(p × (1 − p) / n). At p = 0.5, 50 cases give about ±13.9 percentage points and 200 give about ±6.9. These are calculated normal approximations, not guarantees; correlated cases, rare failures and small slices need different analysis.[2]

Illustrative sample planning, independent binary outcomes at p = 0.5
CasesApproximate 95% marginInterpretation
50±13.9 percentage pointsUseful pilot; small score changes remain uncertain
200±6.9 percentage pointsMore precision; not rare-event safety assurance

Repeated runs of the same case measure run-to-run reliability; they don’t create new independent task coverage. Report important slices separately and use paired comparisons when both versions answer the same cases.[2]

Should golden test cases be synthetic or human-annotated?

Use both, but separate case generation from label approval. Production examples show actual workloads; synthetic examples can target gaps. OpenAI explicitly lists both as possible data sources and recommends expert labeling and calibration.[1]

Don’t let one model write the question, answer and verdict without independent review. That workflow can reproduce the generator’s blind spots instead of exposing them. Treat this as a design risk: check answers against authoritative domain material and record who resolved disagreements.

  • Reject duplicate paraphrases that inflate the apparent dataset size without adding a new decision.
  • Include missing evidence and conflicting evidence, not just easy positive examples.
  • Review a sample from every slice, then expand review when disagreement appears. No fixed review percentage is guaranteed to be enough.
  • Retain a changelog of case edits. Keep a tuning set separate from the held-out evaluation set.

How do I keep a golden dataset useful after launch?

Add verified failures as regressions and retire stale cases deliberately. Continuous evaluation and growing the test set as new failures appear are part of OpenAI’s recommended workflow.[1]

When policy changes, version the reference and mark which deployments it applies to. Don’t silently change yesterday’s labels and compare today’s score as though the test stayed the same. Record dataset version, model identifier, prompts, scorer configuration, cost and elapsed time with each run.

Related suites

Related reading

Sources

  1. 1. Evaluation best practices · OpenAI · Official docs · Sources checked
  2. 2. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations · arXiv:2411.00640 (Anthropic) · Paper · Sources checked