What data engineers own
Data engineers build reliable paths for data — from source systems to analytics, ML features, and operational reverse-ETL. You are not:
| Role | Focus |
|---|---|
| Data scientist | Models, experiments, notebooks |
| Backend engineer | Request/response APIs, user-facing SLAs |
| ML engineer | Training, serving (see AI Systems for AI pipelines) |
| Data engineer | Ingestion, transformation, orchestration, quality, cost |
Senior signal: "I own the contract between producers and consumers — schema, freshness, and correctness."
The pipeline lifecycle
Sources → Ingest → Raw storage → Transform → Curated tables → Consumers
│ │ │ │ │
events Kafka/ data lake Spark/dbt BI, ML, APIs
APIs CDC (bronze) (silver/gold)
Each arrow is a job with schedules, retries, monitoring, and an owner.
Pipelines are distributed systems
A nightly Spark job is a distributed system with different failure modes than a microservice:
- Partial writes — job succeeds on 990/1000 partitions
- Duplicate runs — Airflow retry after a false failure
- Late data — events arrive after the window closed
- Schema drift — upstream adds a column, downstream SQL breaks
Link to Distributed Systems for delivery semantics and idempotency — we apply them to data here.
What sibling tracks cover
| Track | You use it for |
|---|---|
| SQL | Transform logic, interview query patterns |
| Databases | Storage formats (Parquet, Iceberg), OLTP vs OLAP, lake layout |
| Distributed Systems | At-least-once, exactly-once, partitioning theory |
| Observability | Metrics, logs, alerting on pipeline SLAs |
| Concurrency | Parallel execution within one machine (Spark executors) |
Anchor book
Fundamentals of Data Engineering (Reis & Housley) — the definitive modern DE textbook. DDIA Ch. 10–11 for batch/stream theory.
Interview framing
"Walk me through how clickstream data reaches the exec dashboard":
- Source (web SDK → Kafka)
- Ingest (stream or micro-batch to bronze)
- Transform (sessionization, dedup, aggregation)
- Serve (warehouse mart, BI tool)
- SLAs (freshness, completeness checks)
Mention ownership and data contracts, not just "we use Spark."
Further Reading
Hands-On Tasks (Optional)
Pipeline design drills and whiteboard exercises — DAG sketches, partition plans, backfill strategies. Assumes Databases and SQL fundamentals are in place.
- Map a company's data lifecycle20m
Pick a product (e-commerce, streaming service). Draw sources → ingestion → raw/bronze → transforms → warehouse/marts → consumers (BI, ML, reverse ETL). Label batch vs streaming paths and who owns each box.