Medallion & Incremental Processing

ELT vs ETL

ETL (legacy)	ELT (modern)
Transform before load	Load raw first
Limited by staging capacity	Transform in warehouse/lake compute
Brittle schema upfront	Schema-on-read in bronze

Medallion implementation

Bronze:

• Append-only ingest (Kafka → Delta, S3 copy, CDC stream)
• Preserve source schema + ingest metadata (_ingest_ts, _source_file)

Silver:

• Dedup (window on event ID or MERGE)
• Type coercion, null handling
• Conformed entities (customer, product)

Gold:

• Business metrics, star-schema marts
• Often owned by dbt models

Incremental strategies

Strategy	When
Append	Immutable events (logs)
Merge/upsert	CDC, dimension updates
Overwrite partition	Daily snapshot per dt
Full refresh	Small tables, dev, disaster recovery

Delta MERGE INTO ... WHEN MATCHED ... WHEN NOT MATCHED is the workhorse for silver CDC.

Slowly changing dimensions (SCD)

Type	Behavior
SCD1	Overwrite (no history)
SCD2	Version rows with valid_from/valid_to
SCD3	Previous value column only

Implement SCD2 with merge logic or dbt snapshots. Dimensional modeling theory (star schema, grain, surrogate keys) is covered in Dimensional & Analytical Modeling — this subtopic focuses on pipeline implementation.

AI boundary

Feature store ingestion may read from gold — but embedding pipelines and vector indexes are AI Systems, not repeated here.

Interview answer template

"Bronze append from Debezium CDC. Silver merge on customer_id with SCD2 flags. Gold daily revenue is dbt incremental on order_date partition. Late events merge into silver; gold re-runs for affected dates only."