Self-taught data engineering — an open-course route

v0.1

Toolbelt

weeks 1–2 · ~20 h

Shell, Git and containers. Data engineering runs on other people's machines more than any other discipline here, so the tooling has to be automatic before anything else.

CourseTimeWhy it's here

The Missing Semester of Your CS EducationMIT ~12 h Shell, scripting, version control, debugging. Lectures 1–6; the data-wrangling lecture is directly on topic.

Learn Git Branchinginteractive ~4 h Pipelines live in Git and get reviewed like software. Branching fluency is assumed on every team.

Data Engineering Zoomcamp — setupDataTalks.Club ~4 h Work through the Docker and environment setup module. Free, open, and the reference course for this field.

Build

chore: initialize pipeline-log

A public repository called pipeline-log with a README and a working Docker Compose file that brings up Postgres locally. One commit per study session from here on.

Done when

You can run a containerised Postgres and connect to it from a script
You can clone, branch, commit and open a pull request without looking anything up
You can write a shell script that moves and transforms a file without opening an editor twice

v0.2

SQL, properly

weeks 3–12 · ~110 h

Not “SQL for analysts” — SQL as the language you will spend a career in. Window functions, CTEs, and knowing why a query is slow before you run it.

CourseTimeWhy it's here

CS50 SQLHarvard ~50 h A full course on relational design, querying, normalisation, indexes and transactions. Structured, rigorous, free.

Select Star SQLfree book ~20 h An interactive book that makes you write the query before showing you one. Best for building actual recall.

Use The Index, LukeMarkus Winand ~25 h How indexes really work and why your query plan looks like that. The single highest-leverage thing an engineer can learn about databases.

Mode SQL tutorialMode ~15 h Good drilling ground for window functions and analytic patterns on real datasets.

SQLBoltinteractive ref Quick refreshers when a specific clause has gone rusty.

Build

feat: a query you are not ashamed of

Load a real public dataset into Postgres and answer ten genuine questions about it — each with a query using CTEs and window functions, each with its EXPLAIN ANALYZE output and a note on what the index did.

Done when

You can write window functions without reaching for a reference
You can read an execution plan and say why a query is slow
You added an index that changed a plan, and you can explain the before and after
You understand what a transaction isolation level actually guarantees

v0.3

Python for data

weeks 13–20 · ~90 h

The glue language of the field. Enough software engineering to write pipelines other people can maintain, and enough dataframe fluency to move data without a cluster.

CourseTimeWhy it's here

Data Engineering Zoomcamp — ingestionDataTalks.Club ~40 h Ingestion, APIs, batch loading and the Python patterns the field actually uses. Follow the repository week by week.

DuckDB documentationDuckDB ~25 h An analytical database that runs in-process. Most “big data” problems below a terabyte are DuckDB problems, and knowing that saves employers a fortune.

Polars user guidePolars ~25 h Modern dataframes with lazy evaluation and a query optimiser — the concepts transfer straight to Spark.

Build

feat: ingest, transform, load

A Python pipeline that pulls from a public API, handles pagination and rate limits, validates the payload, and lands it in Postgres idempotently — rerunnable without duplicating a single row.

Done when

Your pipeline can be run twice with no duplicate rows
It fails loudly and usefully when the source API misbehaves
It is packaged, testable, and someone else could run it from the README
You can explain when a dataframe library beats SQL, and when it does not

v0.4

Databases from the inside

weeks 21–30 · ~110 h

Stop treating the database as a black box. The engineers who are trusted with the hard problems are the ones who know what is happening underneath.

CourseTimeWhy it's here

CMU 15-445 — Database SystemsCarnegie Mellon · Andy Pavlo ~70 h Storage, buffer pools, B+ trees, join algorithms, concurrency control, recovery. Lectures, notes and projects all public. The best database course in existence, free.

PostgreSQL documentationPostgreSQL ~20 h Read the chapters on indexes, the planner and MVCC properly. Unusually well-written for a manual.

CMU 15-721 — Advanced Database SystemsCarnegie Mellon ref Columnar storage, vectorised execution, modern OLAP internals. Take it after 15-445 if the field grabs you.

Build

feat: prove you understand the engine

Complete at least two of the 15-445 projects, and write up one investigation of your own: take a slow query on a large table and drive it to fast, documenting every hypothesis you tested and discarded.

Done when

You can explain how a B+ tree index answers a range query
You know what MVCC costs you and why vacuum exists
You completed two CMU projects and they pass the provided tests
You can choose between a hash join and a merge join and defend the choice

v0.5

Pipelines and orchestration

weeks 31–40 · ~110 h

The step that turns scripts into infrastructure: scheduling, dependencies, retries, backfills, and knowing what happened when something failed at 3 a.m.

CourseTimeWhy it's here

Apache Airflow documentationApache ~40 h The industry default. DAGs, operators, scheduling semantics, backfills, and the concepts every orchestrator shares.

Data Engineering Zoomcamp — orchestrationDataTalks.Club ~40 h The workflow-orchestration and data-lake modules, built end to end on a real dataset.

Dagster documentationDagster ~20 h The asset-oriented alternative. Worth knowing because it reframes pipelines as data products rather than tasks.

Start Data Engineeringblog ~10 h Practical write-ups on pipeline design and testing from someone doing the job.

Build

feat: it runs without you

Orchestrate your v0.3 pipeline: scheduled, dependency-aware, retrying on transient failure, alerting on real failure, and able to backfill a month of history without hand-holding.

Done when

A backfill of arbitrary date range runs correctly and idempotently
A transient failure retries; a real failure alerts you with something actionable
You can explain the difference between execution date and run date without hesitating
Someone else could add a new task to your DAG without reading all of it

v0.6

Warehouse and modelling

weeks 41–48 · ~90 h

Where data engineering meets the business. Modelling is the part that decides whether anyone can actually answer a question with your warehouse.

CourseTimeWhy it's here

dbt documentationdbt Labs ~35 h Models, refs, tests, incremental strategies, snapshots. dbt is the field's transformation standard and its docs teach the discipline, not just the tool.

dbt Learn coursesdbt Labs · free ~30 h The official free courses. Do dbt Fundamentals, then the modelling and testing tracks.

Data Engineering Zoomcamp — warehouseDataTalks.Club ~25 h The data-warehouse and analytics-engineering modules, including partitioning and clustering.

Build

feat: a warehouse someone can use

Model your ingested data properly — staging, intermediate and mart layers, dimensional models with surrogate keys, tests on every source, and documentation generated from the project.

Done when

Every source has freshness and uniqueness tests that actually fail when broken
You can explain a star schema and why it beats querying raw tables
Your incremental models rebuild correctly after a late-arriving record
The generated documentation is good enough for an analyst to self-serve

v0.7

Streaming and scale

weeks 49–58 · ~110 h

Batch covers most of the work. Streaming covers the rest — and distributed processing is where the interview questions get hard.

CourseTimeWhy it's here

Apache Kafka documentationApache ~40 h Topics, partitions, consumer groups, offsets, delivery semantics. The design section is worth reading twice.

Apache Spark documentationApache ~45 h The programming guide and the SQL/DataFrame guide. Understand shuffles and partitioning or Spark will punish you in production.

Data Engineering Zoomcamp — streaming and batchDataTalks.Club ~25 h The Kafka and Spark modules, applied to the same dataset you already know.

Build

feat: the same answer, two ways

Build a streaming path alongside your batch path — events into Kafka, processed with consumer semantics you chose deliberately — and reconcile the two. Explaining where they disagree is the real exercise.

Done when

You can explain at-least-once, at-most-once and exactly-once, and which you actually have
You caused and then fixed a data skew problem in Spark
Your streaming and batch results reconcile, and you can explain any gap
You can size a Kafka topic's partitions with a reason rather than a guess

v0.9

Reliability and data quality

weeks 41–66 · alongside · ~50 h

Runs in parallel from v0.6. Nobody trusts a pipeline that has silently been wrong for a month, and quietly wrong data is the field's characteristic failure.

CourseTimeWhy it's here

Great ExpectationsGX ~20 h Declarative data validation — expectations, suites, and failing a run before bad data reaches a dashboard.

dbt documentation — testsdbt Labs ~15 h Generic and singular tests, plus freshness. The cheapest data-quality win available.

Start Data Engineeringblog ~15 h Testing strategies, idempotency and pipeline design patterns, written from practice.

Build

chore: fail loudly, not silently

Add validation gates to every pipeline you own: schema checks, volume anomaly detection, freshness SLAs, and an alert that names the specific expectation that broke.

Done when

A malformed upstream record stops the pipeline instead of poisoning the warehouse
You have freshness SLAs and something tells you when they are missed
You can prove yesterday's numbers were correct, from artefacts rather than memory

v1.0

Capstone

weeks 59–66 · ~100 h

One end-to-end platform, public and documented. Data engineering portfolios are judged on whether the thing runs and whether you can explain your trade-offs.

CourseTimeWhy it's here

Data Engineering Zoomcamp — capstoneDataTalks.Club — The course ends in a project with published evaluation criteria. Use them as your bar even if you never submit.

roadmap.sh — data engineerreference ref A last gap-check against the field's expected surface area before you call yourself job-ready.

dbt documentationdbt Labs ref Reference while you build. You are not learning new tools now.

Build

release: v1.0

A complete platform on one real dataset: ingestion, orchestration, a modelled warehouse, quality gates, and a dashboard someone would actually use — with a written architecture decision record explaining every choice.

Done when

The whole thing runs from a cold start with one documented command
You wrote an architecture decision record covering at least four real trade-offs
A stranger could add a new data source by following your documentation
You can state the cost of running it per month, with reasoning

From spreadsheet to production pipeline.

The road, to scale

Ground rules that make this work

Toolbelt

SQL, properly

Python for data

Databases from the inside

Pipelines and orchestration

Warehouse and modelling

Streaming and scale

Reliability and data quality

Capstone

Reference shelf