Kafka as the durable log
Kafka is not just a message queue — it's a distributed commit log:
- Messages appended to partitions (ordered, immutable)
- Retention by time/size — replay history
- Compaction — keep latest value per key (changelog topics)
Jay Kreps' Log essay: unify batch and stream by reprocessing the log.
Core concepts
| Concept | Role |
|---|---|
| Topic | Named stream |
| Partition | Ordered shard; parallelism unit |
| Offset | Position in partition |
| Consumer group | Cooperative consumers; one reader per partition |
| Replication | Leader + ISR followers |
Ordering guarantee: per-partition only. Key choice determines co-location.
Partitioning for pipelines
key = user_id → per-user ordering
key = null (round-robin) → max throughput, no ordering
key = country → geographic batching
More partitions → more parallelism, but more overhead and consumer rebalancing cost.
Log compaction
For changelog topics (CDC, KV stores):
- Tombstones delete keys
- Consumers rebuild state by reading compacted log
Used in Kappa — reprocess entire history from Kafka.
Link to Distributed Systems
Kafka replication, leader election, and min.in.sync.replicas — messaging topic in Distributed Systems (Topic 8). Here: pipeline design on top of the log.
Interview answer template
"Clickstream to Kafka with 50 partitions keyed by session_id for session ordering. Retention 7 days for replay. Compacted changelog for user_profile CDC. Spark Structured Streaming reads with maxOffsetsPerTrigger for rate control."
Further Reading
Hands-On Tasks (Optional)
Pipeline design drills and whiteboard exercises — DAG sketches, partition plans, backfill strategies. Assumes Databases and SQL fundamentals are in place.
- Partition a high-throughput topic15m
100k events/sec clickstream, key = user_id, need parallel consumers and per-user ordering. Pick partition count, justify key choice, and explain consumer group scaling limits.