Data engineering & analytics
Design and read A/B tests
Hypotheses, sample size, significance, guardrail metrics, reporting results honestly.
~10 focused hours·intermediate
Tools: SQL, Python (scipy/statsmodels), sample-size calculators, experimentation platforms (Optimizely/GrowthBook)
Market relevance — share of job ads asking for this
What employers mean
You should be able to…
- Write a testable hypothesis with a clear success metric before launching an experiment
- Calculate required sample size and expected run duration given baseline rate and MDE
- Choose and report the right significance test (and correct for multiple comparisons when needed)
- Define guardrail metrics so a 'winning' variant that harms retention/revenue is still caught
- Detect and explain novelty effects, weekday/weekend seasonality, and sample ratio mismatch
- Report a null or negative result honestly instead of p-hacking toward significance
- Translate a statistical result into a clear ship/no-ship recommendation for a PM
Needs first: Query and model data with SQL
Learn — free, link-checked
The few resources that matter
Read · beginner · 20 min · optimizely.com
A/B Testing
Practitioner explainer on sample size, guardrail metrics, and common pitfalls from a leading experimentation platform. — Optimizely
Course · beginner · 90 min · khanacademy.org
Significance tests (hypothesis testing)
Builds the p-value and significance intuition you need to defend an A/B result under questioning. — Khan Academy
Course · beginner · 300 min · khanacademy.org
Statistics and probability
Free, self-paced stats foundation so the claims in your story (and your A/B test readouts) survive scrutiny. — Khan Academy
Course · intermediate · 300 min · udacity.com
A/B Testing
Google's free course on designing experiments, choosing metrics, and reading statistical significance without fooling yourself. — Google / Udacity
Practice
A/B test readout for a checkout-flow change
Simulate two variants of a checkout funnel (control vs. a simplified UPI-first flow) with realistic conversion, revenue, and a guardrail metric (refund rate). Run the experiment analysis in Python/SQL: sample size justification, significance test, guardrail check, and a written ship/no-ship recommendation.
Done when
- Pre-registers a hypothesis and required sample size calculation before 'running' the simulated test
- Uses an appropriate significance test and reports confidence interval, not just a p-value
- Checks at least one guardrail metric and explicitly discusses trade-offs if it moved the wrong way
- Ends with a one-paragraph honest recommendation, including what you'd do if the result were inconclusive
Prove it
Evidence a recruiter can check
- Public GitHub repo/notebook with the simulation, analysis, and written readout
- A one-page readout memo formatted the way you'd actually send it to a PM
- Completed Udacity A/B Testing course certificate or equivalent, linked
Interview
Questions you'll get asked
- How do you calculate the sample size needed for an A/B test given a baseline conversion rate and MDE?
- What's a guardrail metric and why do you need one even if the primary metric wins?
- How would you detect sample ratio mismatch, and why does it matter?
- A test hits significance on day 2 — do you call it? Why or why not?
- Explain p-value and statistical significance to a non-technical PM in one minute.
- How do you handle novelty effects when a new feature initially spikes engagement?
- Walk me through how you'd design an experiment to test a new UPI checkout flow.
- What would make you recommend NOT shipping a variant that won on the primary metric?