Data Engineer

Career Track

Move and model data at scale.

You want to build the pipelines every other team depends on.

· BY THE END YOU CAN ·

01Model data so a question takes one cheap query, not five
02Run batch and streaming pipelines that fail loudly instead of silently
03Guarantee freshness, quality and lineage as volume and cost grow

Start this track→See all 110 skills

Compare all four tracks

6Phases

110Skills covered

6Portfolio projects

8Core tools

BeginnerStarts from

Your Data Engineer career begins

Every dashboard, every model, and every AI feature is downstream of a pipeline somebody built. When that pipeline is late, wrong, or quietly dropping rows, everything above it is wrong too — and nobody notices until a decision has already been made on bad numbers.

Data engineering is the discipline of making that not happen. It is SQL-first and unglamorous in the best way: modelling data so it answers questions cheaply, moving it reliably, testing it so failures are loud, and keeping the whole thing affordable as volume grows.

The track runs from SQL and Python fundamentals through dimensional modelling and warehousing, into transformation with dbt, orchestration with Airflow, distributed processing with Spark, streaming with Kafka, and finally the lakehouse, cloud, and governance layer that production platforms are built on.

Not dashboards and analysis — you build the systems those run on.

The Data Engineer job

· WHAT YOU WOULD DO ·

Design data models that make analysis fast and cheap
Build and orchestrate pipelines that run unattended
Guarantee data quality, freshness, and lineage
Keep storage and compute costs under control as volume grows

· WHAT POSTINGS ASK FOR ·

Advanced SQLPythonDimensional modellingdbtAirflowApache SparkKafkaCloud warehousesLakehouse table formatsData quality testingTerraform

· WHO HIRES FOR IT ·

Any company where decisions depend on reported numbers
Platform and infrastructure teams serving analysts and scientists
Companies migrating onto a modern warehouse or lakehouse
Regulated industries with strict lineage and retention requirements

· WHERE IT LEADS ·

1Junior Data Engineer — Builds and maintains pipelines others designed.
2Data / Analytics Engineer — Owns models and pipelines for a domain.
3Senior Data Engineer — Designs the platform and sets the standards.
4Data Platform / Staff Engineer — Owns architecture, governance, and cost.

· THE CURRICULUM ·

Every skill in the track, in the order you learn it.

6 phases, 110 named skills. Not chapter headings — the individual things you will be able to do by the end.

01
PHASE 1 OF 622 SKILLS
SQL and Python foundations
The two languages that carry most of the job.
- SELECT and filtering
- Inner and outer joins
- Self joins
- Aggregations and GROUP BY
- HAVING
- Subqueries
- Common table expressions
- Window functions
- Ranking and running totals
- lag and lead
- Set operations
- Pivoting
- Query plans and EXPLAIN
- Index strategy
- pandas for data
- APIs and pagination
- CSV, JSON and Parquet
- Avro and ORC
- Columnar storage
- Compression codecs
- Shell and cron
- Git for data projects
02
PHASE 2 OF 617 SKILLS
Data modelling and warehousing
Designing data so the questions are cheap to answer.
- OLTP vs OLAP
- Normalisation
- Deliberate denormalisation
- Dimensional modelling
- Star schemas
- Snowflake schemas
- Fact and dimension tables
- Choosing the grain
- Surrogate keys
- Slowly changing dimensions
- Bridge tables
- Data vault basics
- Partitioning strategy
- Clustering keys
- Cloud warehouse architecture
- Warehouse cost models
- Semantic layers
03
PHASE 3 OF 617 SKILLS
Pipelines and transformation
Getting data in, and turning it into something trustworthy.
- ETL vs ELT
- Batch and micro-batch ingestion
- Managed connectors
- Change data capture
- Incremental loads
- Watermarks
- Idempotency
- Safe re-runs
- Backfills
- dbt models and refs
- Materialisations
- dbt tests
- Snapshots
- Macros and Jinja
- Lineage graphs
- Generated documentation
- SQL modularity
04
PHASE 4 OF 617 SKILLS
Orchestration and reliability
Making pipelines that fail loudly instead of silently.
- Airflow DAGs
- Operators and hooks
- Sensors
- Scheduling and catchup
- Retries and timeouts
- SLAs
- Backfilling at scale
- Dynamic DAG generation
- Dagster and Prefect
- Data contracts
- Schema evolution
- Quality expectations
- Freshness and volume checks
- Anomaly detection
- Alerting people trust
- Pipeline observability
- Incident response
05
PHASE 5 OF 620 SKILLS
Distributed processing and streaming
When the data stops fitting on one machine, or stops waiting.
- Spark architecture
- RDDs vs DataFrames
- The Catalyst optimiser
- Spark SQL
- Partitioning
- Shuffles
- Data skew and salting
- Broadcast joins
- Caching and persistence
- PySpark tuning
- Structured streaming
- Kafka topics and partitions
- Offsets and consumer groups
- Delivery semantics
- Exactly-once processing
- Schema registry
- Kafka Connect
- Tumbling and sliding windows
- Session windows
- Flink stream processing
06
PHASE 6 OF 617 SKILLS
Lakehouse, cloud and governance
The platform layer, and being trusted with other people's data.
- Object storage
- Lake vs warehouse vs lakehouse
- Apache Iceberg
- Delta Lake
- ACID on object storage
- Time travel
- Snapshot isolation
- Table schema evolution
- Compaction and file sizing
- Terraform
- IAM and least privilege
- Cost monitoring
- Lifecycle policies
- PII classification
- Masking and tokenisation
- Retention and deletion
- Cataloguing and ownership

· WHAT YOU WILL BUILD ·

6 projects. Every one of them a running system.

Not exercises with a known answer. Each project is something you can deploy, send someone a link to, and defend in an interview.

01Starter

An analytics layer over a raw dataset

Take a raw, awkward dataset and model it into clean tables that answer real questions in one query instead of five — with the query plans to show why they are fast.

Advanced SQLData modellingQuery tuningDocumentation

02Starter

An incremental ingestion job with tests

A Python job that pulls from a paginated API, loads only what changed, survives being run twice, and fails loudly when the source schema shifts.

IngestionIncremental loadsIdempotencySchema checks

03Core

A batch ELT pipeline: orchestration, transformation, warehouse

Scheduled ingestion into a warehouse, transformed through a tested and documented dbt project, orchestrated with retries, SLAs, and a backfill you can trust.

AirflowdbtDimensional modellingTestingBackfills

04Core

A streaming pipeline with near-real-time aggregates

Events through Kafka into windowed aggregates that stay correct when messages arrive late, out of order, or twice.

KafkaWindowingWatermarksDelivery semantics

05Core

A lakehouse with schema evolution and time travel

An open table format over object storage: ACID writes, a column added without a rewrite, querying yesterday’s state, and compaction that keeps files sane.

Iceberg or DeltaObject storageSchema evolutionCompaction

06Capstone

A production data platform with contracts and quality gates

Batch and streaming together, governed by data contracts, guarded by quality checks that stop bad data downstream, with lineage, alerting, and a cost dashboard.

Data contractsQuality gatesLineageIAMCost monitoring

· THE DATA ENGINEER STACK ·

SQLPythonApache SparkAirflowdbtApache KafkaSnowflake / BigQueryApache Iceberg

· PLUS THE CORE, IN EVERY TRACK ·

Data Structures & AlgorithmsSystem DesignOperating SystemsComputer NetworksGit & GitHubDockerLinuxSQL

· ROLES THIS TRACK TARGETS ·

Data Engineer · Analytics Engineer · Data Platform Engineer

· HOW THE TRACK RUNS ·

From first lesson to signed offer.

Step 1

Learn

Work through the curriculum in order, from the fundamentals to the applied work the role does every day. Written lessons, not video you cannot search.

Step 2

Build

Ship the projects. Each one is a real system with real constraints, and each one ends up somewhere you can show it.

Step 3

Practice

Solve problems in the DSA arena, take AI mock interviews, and get feedback on the answers you actually gave.

Step 4

Get hired

Build an ATS-ready resume, see how well you match each opening, close the gaps it finds, and apply without leaving the platform.

· FREQUENTLY ASKED ·

Questions people ask before starting.

Do I need to be good at mathematics?

No — this is the least mathematical of the four tracks. It rewards precision and systems thinking rather than statistics. If you like making things correct and fast, it suits you.

Is data engineering the same as data analysis?

No. Analysts and scientists consume data to answer questions. Data engineers build and operate the systems that deliver that data correctly and on time. This track is the second one.

How much SQL is there really?

A lot, and deliberately so. SQL and Python carry most of the work, and the track goes well past basic queries into window functions, query plans, and modelling decisions that determine cost at scale.

Do I need cloud accounts that cost money?

The track is built around free tiers and local equivalents, and teaches cost control as a skill in its own right — which is part of the job anyway.

Does this track lead into AI work?

It underpins it. Every model and AI feature depends on data arriving correctly, and data platform experience transfers directly if you later move toward the ML Engineer track.

· NOT QUITE YOU? ·

The other three tracks.

Full Stack AI Engineer

AI Engineer

Build products on top of foundation models.

You want to build with large language models, not train them.

Not model training or research — that is the ML Engineer track.

You will build

A prompt workbench with structured outputs
Semantic search over a document collection
A production RAG assistant with citations

PythonLLM APIsRAGVector DatabasesAgents & Tool CallingEvals

Roles: AI Engineer · GenAI Engineer · LLM Application Engineer

Explore this track→

ML Engineer

Train, serve, and monitor models in production.

You want to own models end to end, from raw dataset to live endpoint.

Not calling somebody else’s API — here you build and own the model.

You will build

An end-to-end tabular model with an honest evaluation
An image classifier fine-tuned from a pretrained backbone
A reproducible training pipeline with tracked experiments

PythonPyTorchscikit-learnMLflowFeature StoresModel Serving

Roles: Machine Learning Engineer · MLOps Engineer · Applied ML Engineer

Explore this track→

· ONE PLATFORM. NO LIMITS ON WHAT YOU CAN BUILD. ·

Everything you need to grow as an engineer — software engineering, AI, real-world projects, technical challenges, hands-on practice, AI-powered guidance, interview preparation, and career development.

Don’t just learn engineering. Experience it.

Learn. Build. Solve. Practice. Prepare. Get hired.

Start Learning→