Annotation, quality & human feedback
Write labeling guidelines and evaluation rubrics
Turn a vague quality goal into a rubric with examples and edge cases that others can apply.
~8 focused hours·intermediate
Tools: rubric templates, golden/reference examples, scoring rubrics (Likert/pass-fail)
Market relevance — share of job ads asking for this
What employers mean
You should be able to…
- Turn a vague quality bar ('good customer support response') into scored, checkable criteria
- Provide 2-3 worked examples per score level so raters apply the rubric consistently
- Cover edge cases explicitly instead of leaving them to individual judgment
- Pilot a rubric on a small sample and revise it before rolling out to a full annotation team
- Balance rubric strictness with practical rater throughput
- Version and document rubric changes so historical ratings stay interpretable
- Write rubrics usable by both human raters and automated LLM-judge pipelines
Needs first: Evaluate and rank model responses (RLHF / preference data)
Learn — free, link-checked
The few resources that matter
Read · beginner · 30 min · huggingface.co
Evaluate
Official docs for standard NLP quality metrics — vocabulary you need to write rationales that cite concrete numbers, not vibes. — Hugging Face
Read · beginner · 60 min · guidelines.raterhub.com
Search Quality Rater Guidelines
The actual rubric Google trains its own quality raters on — the closest thing to a real take-home for the 'AI Quality Evaluator' roles flooding Indian job boards. — Google
Read · intermediate · 30 min · docs.claude.com
Develop test cases
Anthropic's own guide to writing test cases and grading criteria for evaluating model outputs at scale. — Anthropic
Build from · intermediate · 90 min · github.com
OpenAI Evals
Open-source framework with real eval examples showing how pairwise and rubric-based model grading is actually implemented in code. — OpenAI
Course · intermediate · 90 min · deeplearning.ai
Automated Testing for LLMOps
Free short course on building automated eval pipelines and rubrics for LLM applications, from prompt to production. — DeepLearning.AI
Build from · intermediate · 120 min · github.com
Anthropic courses
Anthropic's own hands-on notebooks for tool use, structured extraction, and grading model output quality. — Anthropic
Practice
Rubric + golden set for grading AI customer-support replies
Write a 4-criterion rubric (accuracy, helpfulness, tone, policy compliance) with a 1-5 scale and 2 worked examples per score level for grading AI-generated customer support replies (include a Hindi-language example). Build a 20-item golden set with your own scores and rationales, then have the rubric scored by an LLM-as-judge and compare.
Done when
- Rubric has 4+ criteria, each with a 1-5 scale and at least 2 worked examples per level
- A 20-item golden set is scored manually with rationales referencing specific rubric criteria
- An LLM-as-judge is run against the same rubric on the golden set and results are compared to manual scores
- A short revision log documents at least one rubric change made after piloting, and why
Prove it
Evidence a recruiter can check
- Public repo with the rubric doc, golden set, manual scores, and LLM-judge comparison
- Agreement/correlation number between manual and LLM-judge scores on the golden set
- Revision log showing the rubric was piloted and iterated, not written once and shipped
Interview
Questions you'll get asked
- Turn 'the response should be helpful' into a concrete, scoreable rubric with 3+ criteria.
- How do you pilot-test a new rubric before rolling it out to 20 annotators?
- What do you include in a rubric so two different raters reach the same score on a borderline case?
- How would you version a rubric so old ratings remain interpretable after you update it?
- How is a rubric for a human rater different from one designed for an LLM-as-judge pipeline?
- Give an example of a rubric criterion that sounded clear but caused disagreement in practice — how did you fix it?
- How do you balance a rubric being thorough vs. annotators being able to apply it quickly?