An LLM feature can improve on a benchmark and get worse for users. The benchmark may miss the real task distribution, the grader may reward style over correctness, or a prompt change may fix common inputs while breaking a high-risk edge case.

Evaluation is therefore not one score. It is a layered system for detecting different kinds of failure before and after release.

The release question should come first: what evidence would make us ship, hold, or roll back this change? Without that decision, an evaluation suite becomes a dashboard of numbers that nobody owns.

Define the unit of success

Start with the product decision the model supports. A customer-support answer may need:

  • correct resolution;
  • grounding in allowed sources;
  • no unsupported policy claim;
  • correct escalation when evidence is insufficient;
  • acceptable tone and latency.

“Answer quality” hides these independent requirements. Each deserves a metric or rubric with a clear failure example.

Build a dataset from task slices

Random production samples overrepresent common easy requests. Construct slices:

  • frequent successful tasks;
  • rare but costly failures;
  • adversarial or ambiguous inputs;
  • empty and conflicting evidence;
  • permission boundaries;
  • multilingual or formatting cases when relevant.

Keep the raw example, expected behavior, source evidence, slice labels, and dataset version. Do not force a single reference answer when several answers could be correct; store criteria instead.

Use deterministic checks first

Before asking another model to grade, validate everything software can determine:

  • schema and type validity;
  • required citations;
  • permitted tool calls;
  • authorization and policy decisions;
  • numerical calculations;
  • exact identifiers and links;
  • latency and token budget.

These checks are cheap, repeatable, and explainable. A model judge should not decide whether JSON parses.

Use model graders with calibration

Model grading is useful for semantic properties such as completeness or groundedness, but it introduces its own bias.

A grader prompt should contain the task, rubric, evidence, candidate answer, and a constrained result schema. Evaluate the grader against a human-labeled calibration set. Measure agreement by slice, not just in aggregate.

Common failure modes include position bias, preference for longer answers, sensitivity to phrasing, and judging factuality without access to the required evidence.

Calibration needs its own dataset. Have qualified reviewers label a blinded set, retain disagreements, and compare the automated grader against the adjudicated labels by slice. A single agreement percentage hides whether the grader systematically misses unsafe refusals or favors verbose answers.

For pairwise grading, run both orders:

judge(candidate_A, candidate_B)
judge(candidate_B, candidate_A)

If the preference changes when order changes, mark the case unstable rather than forcing a winner. The MT-Bench/Chatbot Arena work documents position and verbosity bias; its reported human agreement is an experimental result for its setup, not a warranty for every judge prompt.

Separate retrieval from generation

For RAG, an end-to-end failure does not reveal whether the system retrieved the wrong evidence or generated poorly from good evidence.

Track at least:

  • retrieval recall on answer-bearing sources;
  • ranking quality;
  • access-filter correctness;
  • context utilization;
  • claim-level support;
  • answer completeness;
  • appropriate refusal when evidence is weak.

This decomposition turns “the answer got worse” into a component a team can change.

Make the evaluation executable

Every candidate model, prompt, retriever, chunking change, or tool definition should produce a versioned evaluation run.

run = {
  dataset_version,
  application_version,
  model_and_parameters,
  prompt_version,
  retrieval_configuration,
  per_example_results,
  slice_aggregates,
  cost_and_latency
}

A release gate should express risk tolerance. For example: no regression on permission checks, no more than a defined decline on high-risk slices, and bounded cost and p95 latency.

Compare candidates as paired observations

Run baseline and candidate on the same cases. For binary pass/fail criteria, record four counts: both pass, both fail, baseline-only pass, and candidate-only pass. The last two show the direction of change more clearly than two rounded aggregate percentages.

LLM output is stochastic. Repeat cases when sampling is enabled, retain the sampling parameters, and report uncertainty. A one-point movement on 40 examples is not evidence of improvement simply because a spreadsheet formats it green.

Keep the run manifest

An evaluation is reproducible only when the artifact records:

  • dataset version and case IDs;
  • model snapshot and sampling parameters;
  • system/developer prompt version;
  • tool definitions and environment version;
  • retriever, index, and corpus versions;
  • grader model, rubric, and code version;
  • raw output, component trace, latency, tokens, and cost;
  • human adjudication and reviewer guidance.

This separates a model regression from a changed retriever, unavailable tool, or grader drift.

Evaluate the system, not only the final paragraph

For an agentic or RAG workflow, grade component boundaries:

  1. Was the correct retrieval/tool path selected?
  2. Were tool arguments valid and authorized?
  3. Did the tool return sufficient evidence?
  4. Did the model use that evidence without inventing claims?
  5. Did the product choose the correct completion, refusal, or escalation state?

End-to-end quality can improve while a dangerous component regression is masked by easy cases. Conversely, retrieval recall can improve without improving the final answer if the generator ignores the evidence.

Close the online loop

Offline data freezes yesterday’s distribution. Production supplies new queries, failure clusters, user corrections, escalations, abandonment, and operational incidents.

Sample these signals with privacy controls, review them, and promote representative cases into the offline dataset. Keep an untouched holdout set so repeated prompt tuning does not overfit the entire evaluation suite.

Treat evaluation as infrastructure

The durable system has dataset ownership, versioning, reproducible runs, reviewer guidance, release thresholds, and a process for adding newly discovered failures.

The objective is not to prove that a model is intelligent. It is to make a specific product behavior measurable enough that the team can improve it without relying on demos and intuition.

References