RoleGPSPath to your AI role
Retrieval & knowledge systems

Evaluate and improve retrieval quality

Build eval sets, measure recall/faithfulness (RAGAS-style), tune chunking, reranking, hybrid search.

~12 focused hours·intermediate

Tools: RAGAS, LangSmith evaluation, Custom eval sets (golden Q&A), Faithfulness/relevance metrics, A/B testing of chunking strategies

Market relevance — share of job ads asking for this
Prerequisite capability — not asked for directly, but needed for others.
What employers mean

You should be able to…

  1. Build a labeled eval set (questions + ground-truth answers/sources) for a RAG system
  2. Measure retrieval recall/precision and generation faithfulness (RAGAS-style metrics)
  3. Tune chunk size, overlap, and top-k based on eval scores, not guesswork
  4. Compare reranking on/off and hybrid vs vector-only retrieval with measured metrics
  5. Detect and reduce hallucination rate using faithfulness scoring
  6. Set up regression testing so RAG quality doesn't silently degrade after a change

Needs first: Build a grounded RAG application with citations

Learn — free, link-checked

The few resources that matter

Read · intermediate · 40 min · docs.smith.langchain.com

Evaluation

Shows how to build labeled datasets and run automated regression evals so RAG quality doesn't silently degrade. — LangChain
Read · intermediate · 40 min · docs.ragas.io

Ragas

Open-source metrics (faithfulness, answer relevancy, context precision) purpose-built for scoring RAG and agent outputs. — Ragas
Read · advanced · 40 min · huggingface.co

Advanced RAG on Hugging Face Documentation using LangChain

Covers reranking and query optimization plus a worked evaluation section, bridging pipeline-building and eval in one place. — Hugging Face
Course · advanced · 90 min · deeplearning.ai

Building and Evaluating Advanced RAG Applications

Pairs advanced retrieval techniques (sentence-window, auto-merging) directly with the eval metrics used to justify them. — DeepLearning.AI (Jerry Liu, Andrew Ng)
Practice

Eval harness for a RAG assistant: chunking-strategy shootout

Build a RAGAS-based (or equivalent) evaluation harness for a RAG pipeline, reusing the HR/labour-law assistant or any prior RAG project, with a hand-labeled set of 20-30 question/answer/source triples. Run the same corpus through 3 different chunking strategies and report faithfulness, answer relevance, and retrieval recall for each so you can justify which one wins.

Done when
  • A labeled eval set of at least 20 question/answer/source triples exists and is version-controlled
  • RAGAS or equivalent metrics (faithfulness, answer relevance, context recall) are computed automatically, not eyeballed
  • At least 3 chunking/retrieval configurations are compared with a results table showing which scores best on which metric
  • The eval harness can be re-run after a pipeline change to catch regressions
Prove it

Evidence a recruiter can check

  • Public GitHub repo with the eval set, harness code, and a results table comparing configurations
  • A regression example: a deliberately-worsened config shows lower scores, proving the harness actually detects degradation
  • README explaining what each metric measures and why it was chosen
  • A short note on the winning configuration and the tradeoffs (cost vs. quality) it represents
Interview

Questions you'll get asked

  1. How do you build an evaluation set for a RAG system when you don't have pre-existing labeled data?
  2. What's the difference between faithfulness and answer relevance as RAG evaluation metrics?
  3. How would you use RAGAS or a similar framework to compare two chunking strategies objectively?
  4. How do you catch a regression in RAG quality before it ships, not after users complain?
  5. How would you measure whether a reranker is actually improving your pipeline's end results?