Schema evolution
Sources change without asking. Pipelines must handle:
| Change | Safe? | Action |
|---|---|---|
| Add optional column | Usually | Merge schema in bronze |
| Widen type (int → long) | Careful | Coerce in silver |
| Rename column | Breaking | Contract version + migration |
| Change semantics | Breaking | New table version |
Format-specific rules
Avro/Protobuf: reader schema compatibility (backward/forward/full) Parquet: column add at file level; rewrites for rename Iceberg/Delta: metadata-level schema evolution, time travel to old schema
Link to Databases (lake table formats) for storage-level detail.
Metadata catalogs
| Catalog | Ecosystem |
|---|---|
| Unity Catalog | Databricks |
| AWS Glue | AWS lake |
| DataHub | OSS, LinkedIn |
| Apache Atlas | Hadoop legacy |
| BigQuery datacatalog | GCP |
Functions:
- Discovery — search tables, owners
- Lineage — upstream/downstream graph
- ACLs — column/table grants (PII masking)
- Audit — who queried what
PII and governance
- Tag columns as PII in catalog
- Row/column-level security in warehouse
- Separate environments (dev cannot read prod PII)
Regulatory: GDPR delete propagates through medallion layers — right to erasure is a pipeline problem.
Interview answer template
"Schema changes go through contract review. Iceberg allows add-column safely. Renames require v2 table + view alias on v1 for one quarter. DataHub shows 14 downstream dbt models before we merge."
Further Reading
Hands-On Tasks (Optional)
Pipeline design drills and whiteboard exercises — DAG sketches, partition plans, backfill strategies. Assumes Databases and SQL fundamentals are in place.
- Plan a breaking schema change15m
Upstream renames `user_id` → `customer_id`. Describe: contract version bump, dual-write period, downstream dbt refs, catalog notification, rollback plan.