Skip to content
Eval Suites50

G-Eval explained: rubrics, probabilities and judge limits

Sources checked 2 min read · By the Eval Suites editors

In brief

G-Eval uses a language model to derive evaluation steps from criteria and assess a generated response with a structured scoring form. Liu and colleagues studied GPT-4-based evaluation in 2023; a modern library implementation is not proof that another judge reproduces the paper’s results or removes bias.[1][2]

Original paper
Liu et al., 2023 [1]
Evaluation type
LLM-as-a-judge, not deterministic verification
Model availability
Check your provider; no current availability assumed

How does G-Eval work for LLM-as-a-judge evaluation?

G-Eval turns task-specific criteria into evaluation steps, then asks a judge to score the candidate response. The original paper describes chain-of-thought and a form-filling approach for natural-language generation evaluation.[1]

Use a rubric for qualities that don’t have a reliable exact check, such as whether a summary omits a consequential qualification. Use code for a valid JSON object, permitted tool call or final database value. A judge can sound confident about an incorrect verdict.

  1. State one criterion and provide the input fields needed to judge it.
  2. Generate evaluation steps or supply reviewed fixed steps. DeepEval accepts criteria or evaluation_steps, not both.
  3. Score a held-out human-labeled set and examine disagreements before choosing a release threshold.
  4. Version the rubric, steps, judge model and scoring implementation with every result.

DeepEval’s G-Eval documentation explains the criteria-versus-steps contract and notes that fixed steps skip the step-generation stage.[2]

How does G-Eval generate evaluation criteria and probabilities?

The evaluator doesn’t discover your product requirements for you: you provide the criterion, and it can generate steps for applying it. Probability-weighted scoring uses the model’s score-token probabilities, where available, to form a weighted score rather than treating one sampled numeral as the whole result.[2]

An illustrative weighted mean is Σ(score × probability) / Σ(probability) over allowed score values. For scores 1 and 2 with probabilities 0.25 and 0.75, it is 1.75. This simplified calculation is not a claim that every provider exposes the required token probabilities or that every implementation normalizes identically.

G-Eval does not eliminate position bias or central-tendency bias. The original paper flags preference for LLM-generated text, and the MT-Bench judge study documents position and verbosity biases in LLM judging. Calibrate, test order swaps where relevant and preserve uncertainty.[1][3]

How do I implement G-Eval with GPT-4o or Claude 3.5 Sonnet in Python?

Use a supported judge model or a documented custom adapter, and first confirm that your provider still serves the requested model. GPT-4o and Claude 3.5 Sonnet are not the original paper’s evaluated model names; this guide does not assume they remain available.[1][4]

Documented DeepEval pattern; requires a configured, available judge
import os
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

# Choose a model supported by your installed release and provider.
judge_model = os.environ["EVAL_JUDGE_MODEL"]
metric = GEval(
    name="Evidence consistency",
    criteria="Does the actual output preserve the facts in the expected output?",
    evaluation_params=[
        LLMTestCaseParams.ACTUAL_OUTPUT,
        LLMTestCaseParams.EXPECTED_OUTPUT,
    ],
    model=judge_model,
)
case = LLMTestCase(
    input="Summarize the fixture policy",
    actual_output="Refunds need review.",
    expected_output="Refunds require a review before approval.",
)
metric.measure(case)
print(metric.score, metric.reason)

For Claude or another provider not accepted by the string model option, use the documented DeepEvalBaseLLM custom-model interface for your installed release. Verify synchronous and asynchronous generation, structured-output handling and any log-probability limitations; don’t present a fallback sampled score as identical to the original method.[4][2]

When should a G-Eval score block a release?

Let a G-Eval score block a release only after you’ve checked its agreement with reviewed labels for that task. OpenAI recommends calibrating automated scoring with human judgment, and statistical evaluation guidance explains why small score changes need uncertainty analysis.[5][6]

Log judge calls and cost separately from application calls. Keep ambiguous failures in a review lane until you understand them; changing the threshold to silence a flaky judge hides the problem. The deterministic-assertions guide covers checks that can move out of the judge entirely.

Related suites

Related reading

Sources

  1. 1. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment · Yang Liu et al., arXiv:2303.16634 · Paper · Sources checked
  2. 2. G-Eval metric · Confident AI · Official docs · Sources checked
  3. 3. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · arXiv:2306.05685 · Paper · Sources checked
  4. 4. Metrics and custom evaluation models · Confident AI · Official docs · Sources checked
  5. 5. Evaluation best practices · OpenAI · Official docs · Sources checked
  6. 6. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations · arXiv:2411.00640 (Anthropic) · Paper · Sources checked