Data engineering & analytics
Build scheduled data pipelines
Airflow/dbt/Spark basics: extract, transform, load into a warehouse with tests and lineage.
~25 focused hours·intermediate
Tools: Airflow, dbt, Spark (basics), Great Expectations, a cloud warehouse (BigQuery/Snowflake/Redshift)
Market relevance — share of job ads asking for this
What employers mean
You should be able to…
- Write an Airflow DAG with proper task dependencies, retries, and scheduling
- Build dbt models with tests (not null, unique, referential integrity) and documented lineage
- Design idempotent ETL so re-running a job doesn't duplicate data
- Handle incremental loads instead of always doing a full table refresh
- Add data-quality checks that fail the pipeline loudly instead of silently loading bad data
- Debug a failed pipeline run from logs and backfill the missing partition/date
- Document data lineage so downstream users know where a column's value actually came from
Needs first: Query and model data with SQL, Write production-quality Python for AI work
Learn — free, link-checked
The few resources that matter
Read · beginner · 45 min · astronomer.io
Get started with Airflow
Practical guide to running Airflow locally with the Astro CLI and debugging your first real DAG. — Astronomer
Read · intermediate · 30 min · docs.greatexpectations.io
Quickstart
Adds automated data-quality tests to a pipeline — the 'tests and lineage' half of the job description most tutorials skip. — Great Expectations
Read · intermediate · 30 min · postgresql.org
The SQL Language — Window Functions
Official reference for RANK, LAG, and running totals — the SQL feature most Indian AI-analyst interviews probe for. — PostgreSQL Global Development Group
Read · intermediate · 60 min · airflow.apache.org
Fundamentals tutorial
Official walkthrough of writing your first DAG with tasks, dependencies, and a schedule. — Apache Airflow
Course · intermediate · 120 min · courses.getdbt.com
dbt Fundamentals
Free, certificate-backed course on modeling, testing, and documenting transformations the way modern data teams work. — dbt Labs
Practice
Daily UPI settlement pipeline with tests
Build an Airflow DAG that pulls a simulated daily UPI transaction file (drop new CSVs into a folder to mimic a daily feed), loads it incrementally into a local Postgres warehouse, transforms it with dbt models (staging -> marts), and runs dbt tests for nulls, uniqueness, and referential integrity. Schedule it to run daily and handle a manual backfill for a missed day.
Done when
- DAG has explicit task dependencies and retry configuration, not one monolithic Python task
- Loads are incremental — re-running for the same date does not duplicate rows
- At least 4 dbt tests are defined and the pipeline fails visibly when a test fails on bad input data
- README explains how to trigger a backfill for a missed date and includes a lineage diagram (even hand-drawn)
Prove it
Evidence a recruiter can check
- Public GitHub repo with the DAG, dbt project, and a lineage diagram (dbt docs generate output works)
- Screenshot/recording of the Airflow UI showing a successful run and a deliberately-failed test run
- README documenting the incremental-load and backfill design
Interview
Questions you'll get asked
- Walk me through the DAG you'd write to load daily sales data from an API into a warehouse.
- How do you make an ETL job idempotent so re-running it twice doesn't double-count?
- What's the difference between a full refresh and an incremental load, and when do you use each?
- How would you add a test that fails the pipeline if 5% of rows suddenly have null customer_id?
- A DAG failed at 3am on one task — walk me through how you'd debug and backfill it.
- What's dbt for, and how is it different from writing raw SQL scripts in a cron job?
- How do you track data lineage so someone can trace a dashboard number back to its source table?