Delivery semantics (pipeline view)
| Guarantee | Meaning | Typical mechanism |
|---|---|---|
| At-most-once | May lose data | Fire-and-forget |
| At-least-once | May duplicate | Retry + idempotent sink |
| Exactly-once | No dup, no loss | Transactions + idempotent + checkpoint |
True end-to-end exactly-once is hard — requires cooperation across source, processor, sink.
Link to Distributed Systems for theory — Kafka transactions, two-phase commit, idempotency keys.
Pipeline failure modes
| Failure | Symptom | Mitigation |
|---|---|---|
| Task retry | Duplicate partitions | Idempotent overwrite/merge |
| Partial cluster loss | Incomplete write | Write to temp path, atomic rename |
| Upstream schema break | Job error | Contract tests, quarantine |
| Skew/straggler | SLA miss | AQE, repartition |
| Consumer read too early | Wrong dashboard | Partition publish flag / _SUCCESS marker |
Dead-letter queues
Bad records → DLQ table/topic for inspection instead of failing entire batch:
parse JSON → valid → silver
→ invalid → dlq.raw_payload + error_reason
Checkpoint recovery (streaming)
Flink/Spark Streaming store offsets + state to durable storage. On failure, restart from last checkpoint.
Savepoints (Flink) — manual checkpoint for code upgrades with state compatibility.
_SUCCESS and publish patterns
Hive-style _SUCCESS empty file signals partition complete. Consumers filter WHERE _partition_ready.
Delta/Iceberg: transaction commit is atomic — readers see consistent snapshot.
Interview answer template
"Effective exactly-once: Kafka transactional producer + Flink checkpoint + Delta idempotent merge on event_id. Task retries can't duplicate because merge is keyed. Partial writes use replaceWhere on single dt only after job succeeds."
Further Reading
Hands-On Tasks (Optional)
Pipeline design drills and whiteboard exercises — DAG sketches, partition plans, backfill strategies. Assumes Databases and SQL fundamentals are in place.
- Recover from a partial write15m
Spark job wrote 800/1000 partitions before cluster loss. Describe detection, idempotent retry strategy, and how consumers avoid reading incomplete data.