How to evaluate LLM features in production: test sets, judges, metrics and budgets

You evaluate an LLM feature the way you test any other software, with one difference: the outputs are open-ended and not repeatable, so you need a written definition of a good answer, a test set drawn from real usage, and a grader for each quality you care about. Run that suite offline on every change, watch what users actually do once it is live, and treat model upgrades, safety, cost and latency as release gates rather than afterthoughts.

This write-up gathers what the published research and the model providers' own documentation say about each of those steps, and what we recommend from an engineering point of view. It is about evaluating any LLM feature: drafting, summarising, classifying, extracting, chatting or calling tools. If your feature answers questions from your own documents, retrieval needs its own metrics, covered in our piece on what decides RAG answer quality. If you are still deciding what to build, start with how to add AI to your product.

Why LLM features need a different kind of testing

Ordinary unit tests assume that the same input gives the same output. Language models break that assumption in three ways, and each one changes how you test.

• Outputs vary between runs. Even at temperature 0, hosted inference is often not repeatable. Thinking Machines sampled the same prompt 1,000 times at temperature 0 and got 80 distinct completions, tracing the cause to batch sizes that change with server load rather than to randomness in the model. A single passing run proves little.
• The model under you changes. Chen, Zaharia and Zou compared the March and June 2023 releases of GPT-4 and found its accuracy on identifying prime numbers fell from 84% to 51%, alongside other shifts in behaviour and formatting. A feature that worked last quarter can degrade without a line of your code changing if you do not control the model version.
• There is rarely one right answer. A summary or a drafted reply can be good in many ways and bad in many more. You cannot assert equality; you have to define qualities and measure them.

There is a fourth, human problem. Shankar and colleagues describe criteria drift: people need criteria to grade outputs, but grading outputs is how they discover their criteria. In their study, some standards only appeared once participants had seen real outputs. In practice this means you cannot write a perfect rubric in a workshop before anything exists. You write a first version, grade real outputs, and revise it, and that loop is part of the work rather than a sign it is going wrong.

Start with a written definition of good

Before any test set, agree what success means for this feature, in writing, with the person who owns the business outcome. Anthropic's guidance on defining success criteria asks for criteria that are specific, measurable, achievable and relevant, and its worked example combines several dimensions at once: a quality score, a share of non-toxic outputs, the severity of errors and a response-time target. That multidimensional shape is the important part. A feature that is accurate but too slow, or cheap but occasionally unsafe, has not met its bar.

Two rules make this list useful. First, mark which criteria are hard gates (safety, format, severe errors) and which are scores to improve (tone, completeness). Hard gates are never averaged away by a good mean. Second, put the definition next to the code, versioned, so that a change to what good means is as visible in review as a change to the prompt.

Build the offline test set from real usage

The offline test set, sometimes called a golden set, is the single most valuable artefact in an LLM project. OpenAI's evaluation best practices recommend mixing sources (production data, historical data, domain-specific and human-curated examples, synthetic cases) and covering typical, edge and adversarial inputs. The same guide lists biased datasets that do not reflect production traffic among the common anti-patterns.

Where good cases come from

• Production logs, sampled across the whole range of usage rather than only the interesting examples, anonymised where personal data is involved. Before launch, use the closest equivalent: support tickets, emails, search queries, form submissions.
• Domain experts, who know the awkward cases: the policy exception, the ambiguous request, the customer who writes in two languages.
• Failures: every output a user flags, every escalation and every incident becomes a test case with the correct expected behaviour attached.
• Synthetic cases, generated to fill gaps the real data does not cover, such as rare categories or very long inputs. Use them to widen coverage, not to replace real traffic, and review them by hand: generated cases tend to be cleaner and more predictable than real ones.
• Adversarial cases: prompt injection attempts, requests for other users' data, attempts to extract the system prompt, inputs designed to make the feature run up cost.

Slice the results, not just the average

Tag every case with the attributes that matter for your product: language, customer segment, input length, document type, intent, difficulty, whether it should be refused. Then report results per slice. An overall score that rises while one slice collapses is common, particularly after a model change, and the average will hide it. The slices also tell you where to add cases: if one language has only a handful of examples, its score is too noisy to act on.

Keep it honest

• Keep a held-out portion that nobody tunes prompts against, and check it less often. If few-shot examples in the prompt are copied from the test set, the test set stops measuring anything.
• Version the test set with the code, and record which version produced which score, so results stay comparable over time.
• Retire stale cases when the business changes, rather than leaving expected answers that are now wrong.
• Store expected behaviour, not only expected text: "mentions the 30-day return window and offers a label" is more robust than one reference reply.

Choose a grader for each criterion

Each criterion needs a way of being scored, and the right grader depends on the criterion, not on fashion. OpenAI's guide sets out the trade-off plainly: metric-based checks are fast and suit regression testing but miss nuance, human evaluation is the highest quality but slow and expensive, and LLM-as-judge sits in between. Use the cheapest grader that measures the criterion faithfully.

A good suite layers them. Code checks run on every case and catch contract failures immediately. A calibrated LLM judge scores the open-ended qualities. People review a sample, every disagreement between judge and human, and every hard-gate failure.

LLM-as-judge: useful, biased, and in need of calibration

Using one model to grade another is now the default way to score open-ended output at scale, and the evidence supports it with caveats. In the paper that popularised the approach, Zheng and colleagues found that strong judges such as GPT-4 reached over 80% agreement with human preferences, the same level at which humans agreed with each other. The same paper documented position bias, verbosity bias, self-enhancement bias and limited reasoning ability, for example when grading maths.

Position bias can be large. Wang and colleagues showed that simply changing the order in which two answers were presented could flip a judge's verdict: with ChatGPT as the evaluator, Vicuna-13B beat ChatGPT on 66 of 80 queries in one ordering. Their remedies are practical: have the judge produce evidence before its rating, and aggregate results across both orderings. Self-preference is subtler. Panickssery and colleagues found that GPT-4 and Llama 2 can recognise their own outputs and that this self-recognition is linked to scoring them higher, even where human annotators rate the outputs as equal.

An LLM judge is a measuring instrument. Like any instrument, it is only trustworthy once it has been calibrated against a known standard, and it needs recalibrating whenever you change it.

OpenAI's evaluation guide gives consistent advice: prefer pairwise comparison or pass/fail over fine-grained scales, control for response length, ask the judge to reason before it scores, and scale the judge up only once it consistently agrees with human annotations. Putting the research and the documentation together, this is the procedure we recommend:

1. Label a calibration set by hand. A few hundred cases graded by domain experts against the rubric, covering every slice and including deliberate failures.
2. Write one judge per criterion. A judge asked to rate faithfulness, tone and completeness at once does all three badly. Narrow questions with binary answers are easier to calibrate.
3. Use binary or pairwise verdicts. "Does the reply contain any claim not supported by the context: yes or no?" is more stable than a score from 1 to 10.
4. Ask for reasoning first, verdict last, in a structured field your code can parse.
5. Swap positions in every pairwise comparison and count a win only if it holds both ways.
6. Use a different model family as the judge from the one being tested where you can, to reduce self-preference.
7. Measure agreement with the human labels, separately for passes and failures. A judge that agrees on easy passes but misses half the real failures is not fit for a release gate.
8. Read the disagreements. They usually reveal an ambiguous rubric rather than a bad judge, and fixing the rubric helps human reviewers too.
9. Recalibrate whenever the judge model, the rubric or the feature changes significantly.

Reading results: noise, sample size and error bars

An evaluation run is an experiment with a sample, and its result has an error bar whether or not you report one. Miller's paper Adding Error Bars to Evals makes the case that the evaluation literature has ignored standard experimental statistics, and sets out formulas for analysing results, comparing two models and planning how many questions an evaluation needs. Three habits follow from it for product teams.

• Compare versions on the same cases. A paired comparison (old and new prompt on identical inputs) detects real differences with far fewer cases than comparing two unrelated runs.
• Know your noise floor. Run the current version twice and see how much the score moves on its own. Any change smaller than that is not a result. Because outputs vary between runs, sampling each case more than once and averaging is often worth the cost for important decisions.
• Report uncertainty with the score, especially per slice. A slice with few cases can swing wildly; the fix is more cases, not more meetings about the number.

Online metrics: what users actually do

Offline evaluation tells you whether a change is safe to ship. Only production tells you whether the feature is working. The difficulty is that the most obvious signal, explicit feedback, is the weakest: few users press thumbs up or down, and those who do skew towards strong reactions. Behavioural signals are more plentiful and usually more honest.

For changes with real uncertainty, such as a new model or a redesigned prompt, release gradually. Shadow mode runs the new version on live inputs without showing its output, so you can grade it on real traffic at no risk. A canary or A/B test shows it to a small share of users and compares behavioural metrics against the current version, with a predefined stopping rule and a rollback switch. Log enough to reconstruct any interaction (input, context, model version, prompt version, output, user reaction), with retention and access controls agreed with whoever owns data protection.

Finally, read. A small weekly sample of real interactions, reviewed by someone who knows the domain, finds problems no metric was designed to catch. Every confirmed failure goes into the offline set, so the same mistake cannot ship twice.

Regression testing on model upgrades

Model changes are the largest regression risk in most LLM features, and they are not optional: providers retire models on a published schedule. OpenAI's deprecation policy commits to at least six months' notice for generally available models and at least three months for specialised variants, warns that preview models can be retired on much shorter notice, and advises against preview models for business-critical workloads. Anthropic gives at least 60 days' notice before retiring publicly released models and recommends testing replacements well before the retirement date. Google's Vertex AI documentation says a scheduled retirement gives at least 45 days to migrate.

Notice periods of weeks to months mean a model migration is a planned piece of work with a deadline, and the evaluation suite is what makes it routine rather than frightening. Two practices make the difference. Pin a dated model version in production rather than an alias that can move underneath you, so that changes happen when you choose. And track deprecation dates for every model you use, including the judge models in your evaluation suite. Parameters change too: Anthropic's deprecation page lists request parameters that newer API versions reject, which can break a feature as surely as a retired model.

1. Run the full offline suite on the candidate model with the current prompts, and compare per slice and per criterion against the pinned version.
2. Check the contract first. Structured output, tool calling and refusal behaviour are where new models most often differ in ways that break code.
3. Run a pairwise comparison between old and new outputs on the same cases, with positions swapped, and have people review the cases where the verdict flips.
4. Adjust prompts deliberately. A new model often needs prompt changes; make them against the training portion of the test set and confirm on the held-out portion.
5. Measure latency and cost on realistic inputs, since both can move substantially between model generations.
6. Recalibrate any LLM judge that is itself changing model.
7. Shadow, then canary, watching the online metrics against the current version.
8. Keep the rollback to the previous pinned version available until the retirement date.

Safety and abuse evaluations

Quality evaluations ask whether the feature helps honest users. Safety evaluations ask what a dishonest user, or a malicious document, can make it do. The OWASP Top 10 for LLM Applications (2025) is the most useful checklist for building that adversarial set. Its entries map directly onto test cases:

• Prompt injection (LLM01): direct attempts in user input and indirect ones hidden in documents, emails or web pages the feature reads.
• Sensitive information disclosure (LLM02) and system prompt leakage (LLM07): requests for other users' data, for credentials, or for the instructions themselves.
• Improper output handling (LLM05): outputs that would be dangerous if passed unchecked into HTML, SQL, shell commands or downstream APIs.
• Excessive agency (LLM06): for features that call tools, attempts to make the model take actions beyond the user's permissions or intent.
• Misinformation (LLM09): confident false statements on topics where errors cause harm, which overlaps with faithfulness grading.
• Unbounded consumption (LLM10): inputs crafted to trigger very long outputs, loops or repeated tool calls, running up cost and degrading service.

Treat these as hard gates with pass/fail verdicts per case, and never average them into a quality score. Most of the real defences are architectural rather than prompt-based: least-privilege tools, permission checks in code, output encoding, rate limits and spending caps. The evaluation's job is to prove those defences hold, and to keep proving it after every change. Run a focused red-team exercise before launch and after major changes, and add every successful attack to the suite.

Cost and latency budgets

Cost and latency are product requirements, not operational details to tidy up later. Set budgets per request and, more usefully, per successful task, because a cheap call that fails and is retried three times is not cheap. Providers price input tokens, output tokens and, increasingly, cached tokens differently, so the shape of your prompts and responses matters as much as the model you choose. Long prompts, verbose outputs, retries and agent loops are the usual reasons a feature costs far more in production than in the prototype.

For latency, measure two things: time to first token, which decides whether the interface feels responsive, and total time, which decides when the user can act on the result. Report both at the 95th percentile. OpenAI's latency optimisation guide gives a useful rule of thumb: generating output is almost always the slowest step, so cutting 50% of output tokens may cut roughly 50% of latency, while cutting 50% of the prompt may improve latency by only 1 to 5%. It calls streaming the single most effective way to reduce the time users spend waiting, and lists not using an LLM at all among its principles.

Prompt caching is the other large lever when many requests share a long prefix, such as a system prompt, tool definitions or a reference document. In Anthropic's implementation, cache reads are priced at 0.1 times the base input price for most models and 5-minute cache writes at 1.25 times, the cache lasts five minutes by default and is refreshed each time it is used, and a hit requires the cached segment to be identical. The practical consequence is architectural: put stable content first and variable content last, and include cache hit rate in your monitoring, because a small prompt change can silently drop it to zero.

• Record tokens, latency and cost for every case in every evaluation run, so a prompt change that improves quality but doubles output length is visible before it ships.
• Set hard limits on output length, tool-call depth and retries, which is also the main defence against unbounded consumption.
• Route by difficulty where it pays: a smaller, faster model for routine cases and a larger one for the hard slice, with the router itself evaluated.
• Alert on cost per task and cache hit rate, not just on total monthly spend, which moves too slowly to catch a regression.

Putting evaluation into the delivery pipeline

Evaluation only works if it runs without anyone remembering to run it. The OpenAI guide lists vibe-based evaluation, and waiting until after shipping to add evaluations, among the anti-patterns; the cure is to wire the suite into the same pipeline as your other tests, a theme we cover more broadly under quality assurance.

1. On every pull request that touches prompts, model settings, retrieval or output handling: code checks and a fast subset of judged cases, with hard gates blocking the merge.
2. Nightly or before each release: the full suite, every slice, latency and cost, compared against the last release.
3. On every model change: the migration procedure above, including judge recalibration.
4. Continuously in production: online metrics, safety flags and budgets on a dashboard someone owns, with alerts.
5. Weekly: a human read of sampled interactions, with new failures added to the test set.

Keep the test set, rubrics, judge prompts and results in your own repository in a portable format such as JSON or CSV, whatever tooling you use to run them. Evaluation platforms come and go: OpenAI's own guide notes that its hosted Evals platform is being deprecated, with a planned shutdown on 30 November 2026. The test data is the asset that outlives every tool and every model, and it should belong to you. Operating this loop over time is what we mean by LLMOps.

How Authect approaches this

On generative AI projects, we build the evaluation set with the client in the first weeks, from their real data, and treat it as the specification of the feature. Every project is led by a senior engineer, and the evaluation suite runs in the pipeline from the first prompt, alongside cost and latency budgets agreed up front. Clients own the code, the test set, the rubrics and the infrastructure from day one, so they can change models, vendors or teams without losing the definition of a good answer. You talk to the founders who build it, never a salesperson. If you have an LLM feature that works in a demo but not reliably for users, tell us about it.