What CDC is
Change Data Capture streams row-level changes (insert, update, delete) from OLTP databases to downstream systems.
| Method | How | Downside |
|---|---|---|
| Query-based | WHERE updated_at > ? | Misses deletes, load on DB |
| Trigger-based | DB triggers | Ops burden |
| Log-based | Read transaction log (WAL, binlog) | Best for scale; Debezium |
Debezium architecture
Postgres WAL / MySQL binlog
↓
Debezium connector → Kafka topics (per table)
↓
Spark/Flink consumer → Delta silver MERGE
Features:
- Initial snapshot — consistent baseline + switch to stream
- Schema history topic — Avro/JSON schema changes
- Delete events — tombstone or
op=deleteenvelope
Event envelope
Debezium wraps changes:
{ "op": "u", "before": {...}, "after": {...}, "source": { "ts_ms": ... } }Silver merge uses after payload + op for deletes.
Handoff to medallion
Bronze: raw Debezium JSON append
Silver: MERGE on primary key, SCD2 if needed
Gold: dbt models unaware of CDC mechanics
Link to Topic 6 — incremental merge patterns.
vs batch ETL tools
Airbyte/Fivetran: managed connectors, good for onboarding. Debezium: self-operated, Kafka-native, lower latency.
AI boundary
CDC to feed feature stores is general DE. Embedding-specific ingestion stays in AI Systems.
Interview answer template
"Log-based CDC via Debezium on Postgres logical replication. Schema changes captured in history topic; consumers register Avro. Silver Delta merge on id with delete handling. Snapshot + stream avoids missing historical rows."
Further Reading
Hands-On Tasks (Optional)
Pipeline design drills and whiteboard exercises — DAG sketches, partition plans, backfill strategies. Assumes Databases and SQL fundamentals are in place.
- Design CDC from Postgres to lake20m
OLTP Postgres → Kafka → Delta silver. Cover: Debezium snapshot mode, schema changes, delete events, and merge key for silver.