Apache Airflow
Data · Automate
Schedule dependent data tasks and practise retries, backfills and failure recovery.
All capabilities · Data engineering & analytics
Build scheduled data pipelines
Airflow/dbt/Spark basics: extract, transform, load into a warehouse with tests and lineage.
Explore 3 tools for this projectMarket relevance
Which roles ask for this — and how often
Share of job postings in India, per role, that name this capability.
What employers mean
You should be able to…
- Write an Airflow DAG with proper task dependencies, retries, and scheduling
- Build dbt models with tests (not null, unique, referential integrity) and documented lineage
- Design idempotent ETL so re-running a job doesn't duplicate data
- Handle incremental loads instead of always doing a full table refresh
- Add data-quality checks that fail the pipeline loudly instead of silently loading bad data
- Debug a failed pipeline run from logs and backfill the missing partition/date
- Document data lineage so downstream users know where a column's value actually came from
Needs first: Query and model data with SQL, Write production-quality Python for AI work
Learn — free, link-checked
The few resources that matter
Read · beginner · 45 min · astronomer.io
Get started with Airflow
Practical guide to running Airflow locally with the Astro CLI and debugging your first real DAG. — Astronomer
Read · intermediate · 30 min · docs.greatexpectations.io
Quickstart
Adds automated data-quality tests to a pipeline — the 'tests and lineage' half of the job description most tutorials skip. — Great Expectations
Read · intermediate · 30 min · postgresql.org
The SQL Language — Window Functions
Official reference for RANK, LAG, and running totals — the SQL feature most Indian AI-analyst interviews probe for. — PostgreSQL Global Development Group
Read · intermediate · 60 min · airflow.apache.org
Fundamentals tutorial
Official walkthrough of writing your first DAG with tasks, dependencies, and a schedule. — Apache Airflow
Course · intermediate · 120 min · courses.getdbt.com
dbt Fundamentals
Free, certificate-backed course on modeling, testing, and documenting transformations the way modern data teams work. — dbt Labs
Tools for practice
Choose a tool for the job
Start with one tool for each part of your project. You don’t need to learn them all.
3 tools to explore
dbt
Data · Test
Turn SQL transformations into documented, tested data models.
BigQuery
Data
Query datasets in a cloud warehouse and inspect query cost and performance.
Practice
Daily incremental warehouse pipeline with Airflow, dbt and data tests
Use the NYC TLC trip-record parquet files as a stand-in daily feed: split one month into day-sized partitions and land them one at a time so the pipeline sees a real arriving feed. Build an Airflow DAG that loads each day incrementally into local Postgres, transforms it with dbt staging and mart models, and runs dbt tests for nulls, uniqueness and referential integrity. Then break it on purpose — feed a day with corrupted rows and confirm the run fails loudly instead of quietly writing bad marts. Finish by backfilling a day you deliberately skipped.
Start from
Milestones
- Stand up Postgres and Airflow, land one day's partition end to end · ~4h
- Build the DAG with real task dependencies, retries and a date parameter that makes re-runs idempotent · ~6h
- Write the dbt staging and mart models and generate the lineage docs · ~5h
- Add the four data tests and prove the pipeline fails on a deliberately corrupted day · ~3.5h
- Skip a day, backfill it, and document the backfill procedure · ~2.5h
Done when
- DAG has explicit task dependencies and retry configuration, not one monolithic Python task
- Loads are incremental — re-running for the same date does not duplicate rows
- At least 4 dbt tests are defined and the pipeline fails visibly when a test fails on bad input data
- README explains how to trigger a backfill for a missed date and includes a lineage diagram (even hand-drawn)
Prove it
Evidence a recruiter can check
- Airflow run history showing the same date executed twice with identical row counts — the idempotency claim, evidenced
- The deliberately-failed run: the corrupted input, the dbt test that caught it, and the marts that were left untouched
- dbt docs lineage graph from staging through to marts, generated not drawn
- The backfill: a gap in the run history, the command that filled it, and the row counts before and after
Signal it
Built a daily Airflow + dbt pipeline with incremental idempotent loads and four data tests — demonstrated it catching a corrupted day's feed before it reached the marts, and backfilling a missed date cleanly.
Interview
Questions you'll get asked
- Walk me through the DAG you'd write to load daily sales data from an API into a warehouse.
- How do you make an ETL job idempotent so re-running it twice doesn't double-count?
- What's the difference between a full refresh and an incremental load, and when do you use each?
- How would you add a test that fails the pipeline if 5% of rows suddenly have null customer_id?
- A DAG failed at 3am on one task — walk me through how you'd debug and backfill it.
- What's dbt for, and how is it different from writing raw SQL scripts in a cron job?
- How do you track data lineage so someone can trace a dashboard number back to its source table?