Evaluation & quality
Prove it works before launch, and know the moment it stops working after.
Roadmap: AI foundations & first builds, Production GenAI & integration
Why it matters
"It works in the demo" is the most dangerous sentence in AI deployment. Evals are how you turn a demo into something a customer can trust and you can safely change. They're also where many candidates come apart in final rounds: everyone can build a RAG demo, far fewer can show it's good.
What it's made of
- Golden datasets built from real user requests and real failures
- Separate metrics for retrieval, answer quality, safety and citations
- Offline evals vs online signals (edits, overrides, escalations)
- LLM-as-judge, calibrated against human ratings
- Regression suites run on every prompt, model or index change
- Release gates and stop conditions
- Measuring trust: edit rate and unedited-use rate
Start with 30 real examples
You don't need thousands of examples to start. Collect 30 real requests from the target users, including messy ones, out-of-scope ones and a few adversarial ones. Write the expected outcome for each. Run your system against them before every change. This one habit catches most regressions.
Measure each layer separately
One score hides where things break. Track at least:
- Retrieval: was the right source found (recall at k)?
- Grounding: is every claim supported by the retrieved text?
- Citations: do they point to the right passage?
- Task success: did the user get what they needed?
- Safety: correct refusals, no permission leaks.
Trust is its own metric
A system can be accurate and still unused. Track how often people use the output without editing it, how often they override it, and how often they escalate. If accuracy is high but the edit rate is too, the problem is trust or fit, not the model.
Gates, not vibes
Before launch, agree on written gates with the customer: for example citation accuracy above 95%, zero permission leaks in the test suite, p95 latency under 4 seconds. Agree on stop conditions too: what result pauses the rollout. Decisions get much easier when the bar was set before the results came in.
Mid-level looks like
Defines one offline metric and one online metric, and builds a small test set.
Senior looks like
Designs an evidence plan that could prove the recommendation wrong, ties release gates to risk, and links quality metrics to adoption and business outcomes.
Common pitfalls
- "We tested it manually and it looked good."
- Reporting accuracy with no baseline to compare it to.
- Changing the prompt and the model at the same time, so you can't tell which helped.
- Using only test cases the demo team invented instead of real requests.
Practice drills
Drill 1
Build a 30-item eval set for an HR policy Q&A bot, including out-of-scope and adversarial questions.
A strong answer: Real-looking questions, expected answers with source passages, clear refusal cases, and a scoring rule for each.
Drill 2
Your LLM-as-judge disagrees with human ratings on 30% of cases. Decide which to trust and what to change.
A strong answer: Sampling disagreements, finding systematic judge errors, tightening the rubric, and re-measuring agreement.
Drill 3
Write release gates and a rollback trigger for a 50-user pilot of a support-reply assistant.
A strong answer: Quality, safety, latency and adoption thresholds, plus a named owner for the go/no-go call.
Read next
AI Engineering
Chip Huyen
The most complete practical guide to building products on foundation models: evaluation, prompting, RAG, agents, fine-tuning, latency and cost. Close to a syllabus for the technical side of an AI-focused FDE role.
Designing Machine Learning Systems
Chip Huyen
Focuses on ML in production: data pipelines, feature engineering, deployment, monitoring and distribution shift. Still relevant for the classic ML pieces of customer deployments and for thinking about model drift.
Hands-On Large Language Models
Jay Alammar, Maarten Grootendorst
A visual, code-first walkthrough of embeddings, semantic search, RAG, classification and fine-tuning. Good for building intuition about what is happening under the API calls you wire into customer systems.
Prompt Engineering for LLMs
John Berryman, Albert Ziegler
Written by engineers who built GitHub Copilot, it treats prompts as engineered artifacts: context assembly, few-shot design, tool use and workflow design. Directly useful when tuning an LLM feature against a customer's real data.