Annotation, quality & human feedback
Audit label quality and compute agreement
Sampling, inter-annotator agreement, error taxonomies, feedback to annotators.
~8 focused hours·intermediate
Tools: sampling plans, Cohen's kappa/Krippendorff's alpha, Excel/Python for QA reports, error taxonomies
Market relevance — share of job ads asking for this
What employers mean
You should be able to…
- Design a sampling plan to audit label quality without re-reviewing every item
- Compute inter-annotator agreement (Cohen's kappa or similar) and interpret what the number means
- Build an error taxonomy that groups mistakes into actionable categories, not just 'wrong'
- Give specific, example-backed feedback to annotators that improves future accuracy
- Track quality metrics over time and flag a drifting annotator or a broken guideline early
- Distinguish a genuine annotator error from a guideline ambiguity that needs fixing upstream
- Report QA results to stakeholders in a way that supports a go/no-go decision on a dataset
Needs first: Label data accurately against guidelines
Learn — free, link-checked
The few resources that matter
Read · intermediate · 15 min · scikit-learn.org
cohen_kappa_score
Official reference for computing Cohen's kappa, the inter-annotator agreement metric interviewers expect you to name and calculate. — scikit-learn
Read · intermediate · 20 min · nltk.org
nltk.metrics.agreement
Reference implementation of kappa, alpha, and S agreement metrics for more than two annotators, beyond the simple pairwise case. — NLTK
Read · intermediate · 25 min · labelstud.io
Set up task agreement
Shows how to configure and read inter-annotator agreement directly inside the tool you'll be evaluated in. — HumanSignal / Label Studio
Practice
QA audit and agreement report on a labeled dataset
Take a dataset labeled by 2+ annotators (simulate by having 2 people, or yourself on 2 separate passes, label 150 items), compute Cohen's kappa, build an error taxonomy for the disagreements, and write a QA report with specific feedback and a recommendation on dataset readiness.
Done when
- Cohen's kappa (or equivalent) is computed correctly and interpreted in plain language in the report
- Disagreements are categorized into at least 4 named error-taxonomy buckets, not left as an undifferentiated list
- Report includes at least 3 specific, example-backed feedback points an annotator could act on
- Report ends with a clear go/no-go recommendation on whether the dataset is ready to ship
Prove it
Evidence a recruiter can check
- Public repo with the two label sets, the kappa calculation code, and the QA report
- The error taxonomy with example items in each bucket
- A before/after: guideline clarification proposed based on the audit findings
Interview
Questions you'll get asked
- How do you calculate Cohen's kappa and what does a kappa of 0.4 actually tell you?
- Design a sampling plan to audit 2% of a 100,000-item labeled dataset — how do you choose the sample?
- How do you tell whether a labeling error is the annotator's fault or the guideline's fault?
- Walk me through an error taxonomy you'd build for a text-classification annotation project.
- How would you give feedback to an annotator whose accuracy dropped this week?
- What's the difference between accuracy against gold labels and inter-annotator agreement, and when do you need each?
- How do you decide a dataset is 'good enough' to ship to the ML team?