Why data quality is a pipeline concern
One bad deploy can poison every dashboard and ML model downstream. Shift-left validation — catch issues at silver before gold.
Testing layers
| Layer | Tool | Checks |
|---|---|---|
| Unit | dbt tests | unique, not_null, accepted_values |
| Integration | Great Expectations | distributions, row counts, custom SQL |
| Anomaly | Monte Carlo, Soda | Statistical drift vs history |
| Manual | Spot checks | New source onboarding |
Data contracts
A contract between producer and consumer:
- Schema (fields, types, nullability)
- Grain — one row per what?
- Freshness SLA
- Semantics documentation
- Breaking-change process
Chad Sanderson's framing: treat data like APIs — versioned, tested, owned.
Lineage
Answer: "If I change this column, what breaks?"
OpenLineage emits events from Airflow, Spark, dbt → catalog (DataHub, Marquez).
Senior signal: block publish on failed checkpoint and emit lineage event on success.
Failure policies
| Policy | Behavior |
|---|---|
| Warn | Alert, continue (low-risk marts) |
| Block | Fail DAG, no gold update (financial data) |
| Quarantine | Bad rows to dead-letter table |
Link to Observability
Metrics: rows_processed, null_rate, freshness_lag_minutes — pipeline SLIs alongside task success.
Interview answer template
"Silver orders has a Great Expectations checkpoint: row count within 20% of 7-day median, order_id unique, amount >= 0. Fail blocks gold dbt run. OpenLineage links Spark job → Delta table → dbt model for blast-radius analysis."
Further Reading
Hands-On Tasks (Optional)
Pipeline design drills and whiteboard exercises — DAG sketches, partition plans, backfill strategies. Assumes Databases and SQL fundamentals are in place.
- Draft a data contract20m
For a `orders` silver table: schema (types, nullability), grain, freshness SLA, owner, 5 expectations (row count bounds, key uniqueness, amount >= 0), and breaking-change policy.