Evaluation, safety & observability
Build an LLM evaluation harness
Golden sets, LLM-as-judge, regression suites, scoring in CI before shipping prompt/model changes.
~12 focused hours·intermediate
Tools: LangSmith, Ragas, DeepEval, OpenAI Evals, pytest, LLM-as-judge
Market relevance — share of job ads asking for this
What employers mean
You should be able to…
- Build a golden dataset of representative inputs and expected outputs/criteria for your LLM feature
- Implement an LLM-as-judge grader with a clear rubric, and validate the judge against human ratings
- Compute RAG-specific metrics (faithfulness, answer relevancy, context precision/recall) with a library like Ragas
- Run a regression suite automatically on every prompt or model change, before deploying
- Wire eval scoring into CI so a prompt change that regresses quality blocks the merge
- Compare two prompts or two models side-by-side on the same golden set and report which wins and why
- Track eval scores over time to catch silent quality drift after a model or vendor update
- Design evals that catch both correctness and safety/tone regressions, not just accuracy
Needs first: Design and version prompts systematically
Learn — free, link-checked
The few resources that matter
Read · intermediate · 30 min · docs.smith.langchain.com
LangSmith Evaluation
Explains golden datasets, LLM-as-judge evaluators and online vs offline evals, the vocabulary used in eval-harness interviews. — LangChain
Read · intermediate · 30 min · platform.openai.com
Evals
Reference docs for OpenAI's hosted evals API (graders, datasets, run comparisons) if you don't want to self-host a harness. — OpenAI
Read · intermediate · 35 min · docs.confident-ai.com
DeepEval - The LLM Evaluation Framework
Pytest-style LLM eval framework with built-in agent/trajectory metrics, plugs straight into a CI regression suite. — Confident AI
Read · intermediate · 40 min · cookbook.openai.com
Getting Started with OpenAI Evals
Runnable notebook building a graded eval from scratch, copy this pattern straight into a CI regression suite. — OpenAI
Read · intermediate · 40 min · docs.ragas.io
Ragas
Open-source metrics (faithfulness, answer relevancy, context precision) purpose-built for scoring RAG and agent outputs. — Ragas
Course · intermediate · 90 min · deeplearning.ai
Evaluating and Debugging Generative AI
Shows how to version prompts/configs and trace experiments so eval regressions are debuggable, not just scored. — DeepLearning.AI (with Weights & Biases)
Practice
Eval harness + CI gate for a Hindi/English support-ticket summarizer
Build a summarizer for bilingual (Hindi + English) customer support tickets, then build an eval harness with a 30-example golden set, an LLM-as-judge rubric for faithfulness and tone, and Ragas-style faithfulness scoring. Wire it into a CI script that fails if the mean score drops below a threshold, and demonstrate it catching a deliberately-regressed prompt.
Done when
- A golden set of 30+ bilingual ticket/summary pairs with human-reviewed reference criteria
- An LLM-as-judge grader with a written rubric, validated against at least 10 human-labeled examples (report agreement rate)
- Ragas or equivalent faithfulness/relevancy metrics computed and logged per run
- A CI script that fails the build when mean score drops below a set threshold, demonstrated on a regressed prompt
Prove it
Evidence a recruiter can check
- Public GitHub repo with the golden set, judge rubric, and eval scores over 2+ prompt versions
- A report showing judge-vs-human agreement rate on a sample
- CI logs/screenshot showing a run failing due to a regression, then passing after a fix
Interview
Questions you'll get asked
- How do you build a golden dataset when you don't have a lot of labeled examples yet?
- What are the pitfalls of using an LLM as a judge, and how do you validate the judge itself?
- Walk me through the Ragas faithfulness metric — what does it actually measure and how?
- How would you set up a CI gate that blocks a prompt change if eval scores drop?
- How do you evaluate a chatbot's tone/safety, not just factual correctness?
- Describe how you'd detect silent quality drift after switching from GPT-4o to a cheaper model.
- What's the difference between offline evals and online (production) evals, and when do you need both?