Evals: from demo to decision

A demo shows that something can work. An evaluation shows how often it does, on which cases, and how it fails. For a customer deciding whether to put a system in front of their staff, only the second is evidence. Evals are also how you stay sane: without them, every change to a prompt is an argument.

Why demos aren't evidence#

Demos are chosen. Production is not. A system that answers ten hand-picked questions beautifully can still be wrong on one in five real ones, and nobody finds out until a customer complains. Worse, without a fixed set of test cases, a prompt change that improves tone can quietly reduce accuracy, and you won't notice.

Different stakeholders also mean different things by "good". The sponsor wants to know about risk and business impact, engineers want to know why it failed, users care whether it fits their work, and security wants evidence of control. A well-built eval answers all four from the same data.

Start by reading failures#

Before choosing metrics, read outputs. Run the system on 50 to 100 real inputs and read every answer. Write a short note on each one that is wrong, then group the notes. Practitioners such as Hamel Husain argue that this unglamorous step is where most of the value lies, and it matches experience. You will find failure categories nobody predicted. Those categories tell you what to measure.

Build the golden set with the customer#

A golden set is a collection of real questions or cases, each with what a good answer must contain, which sources it should cite and what it must not do. Build it from the customer's own work, with their experts, because they are the ones who will judge whether to launch.

• Source it from real traffic: top ticket types, common questions, past escalations, cases that went wrong.
• Cover the shapes, not just the topics: simple lookups, answers that span several documents, ambiguous questions, questions with no answer in the sources, unsafe requests, and requests the user isn't permitted to make.
• Size it to the stage: a few dozen cases in discovery, low hundreds for a serious pilot, more before full production, and keep growing it.
• Hold some back: keep a private slice that nobody tunes prompts against, so you can tell real improvement from overfitting.
• Version everything: the set, the documents, the prompt, the model and the index. A failing case you can't reproduce is useless.

Asking customer experts to write expected answers has a side benefit. They often disagree with each other. When two experts give different answers to the same case, the problem isn't the model. Their policy is ambiguous, and that is worth knowing before launch.

Five properties that come apart#

These come apart in practice. An answer can be correct but ungrounded (the model knew it from training), grounded but incorrect (the source is out of date), or correct and grounded but unhelpful. Measure them separately, or a single blended score will hide exactly the failure you care about.

Layer your scorers#

Use cheap, deterministic checks for everything they can cover, an LLM judge for qualities that need reading, and people for the cases where being wrong is expensive. Before trusting an LLM judge, score a sample by hand and check that the judge agrees with your experts. Research such as Shankar and colleagues' work on validating validators shows that judges need calibrating, and that the criteria themselves often shift once people see real outputs.

Every scorer has a blind spot. Use several, and know which one each catches.

Find where it failed#

When an answer is wrong, the model is often the last link in a chain that broke earlier. Before touching a prompt, find which link broke.

For the retrieval layer, measure it directly. Did the right passage appear in the top results at all (recall)? How much of what was retrieved was relevant (precision)? How high did the first relevant result rank? Were relevant documents removed by a permission filter they should have passed? Each points to a different fix: chunking, search settings, the reranker, or access control.

Turn evals into gates#

1. 1

   Agree targets per segment, before you build

   For example: 90% correct on routine questions, 95% of material claims cited, a correct 'I don't know' on 90% of no-answer cases, median latency under 4 seconds.
2. 2

   Separate hard invariants from targets

   Some things must hold 100% of the time: no cross-customer data, no unapproved actions, no answers from excluded sources. A single failure blocks launch.
3. 3

   Run the suite on every change

   Prompt, model, index, chunking, tool definitions. Compare against the last version by segment, not just in aggregate.
4. 4

   Report two views

   For the sponsor: launch readiness against the targets and the open risks. For engineers: failing cases with traces.
5. 5

   Close the loop

   Every complaint, escalation and incident becomes a test case. The set grows with the deployment.