How to use this reference
- Work through topics top to bottom — Spark assumes you understand shuffle and partitioning; stream processing assumes batch DAG concepts.
- We focus on concepts and production patterns, not tool certification — Spark API trivia is less important than execution model and tuning.
- We deliberately do not repeat SQL query writing, B-tree/MVCC storage, TCP/DNS, or generic SLO mechanics — those live in sibling tracks. AI-specific ingestion (RAG, embeddings) lives in AI Systems.
- Optional tasks are pipeline design drills — partition plans, DAG sketches, backfill strategies. No cluster setup required.
- Topic 9 — Deep Cuts (Platform) is optional. Skip for general backend or analytics-engineering loops; do it for data platform, streaming, or CDC-heavy roles. Adds ≈10h on top of the core path.
The Reference
- 1
The production lens — what data engineers own, how pipelines differ from apps, and batch vs streaming mental models.
- 2
From MapReduce to DAG engines — how cluster compute works before you tune Spark.
- 3
The default batch engine — architecture, Spark SQL, and production tuning at Databricks/Google scale.
- 4
Event time, windows, and watermarks — plus Spark Structured Streaming and Flink for production.
- 5
Airflow and beyond — DAG design, dependencies, backfills, and operating scheduled pipelines.
- 6
Medallion pipelines, dimensional modeling for gold marts, and dbt as the transform layer.
- 7
Contracts, lineage, catalogs, and schema evolution — trust at scale.
- 8
Reliability semantics, failure recovery, cost, and observability for data platforms.
- 9
Optional hard-core material for data platform and streaming-heavy roles — Kafka internals, CDC, Beam, Flink state, data mesh, and warehouse engines.