The log abstraction
A commit log is an append-only, ordered sequence of records. Kafka's core insight (Jay Kreps): the log is a distributed system primitive — source of truth, replay buffer, and integration backbone.
Replication roadmap (Topic 3) + consensus (Topic 6) underpin how Kafka replicas stay consistent.
Kafka building blocks
| Concept | Role |
|---|---|
| Topic | Named log category |
| Partition | Ordered sub-log; unit of parallelism |
| Offset | Position in partition |
| Consumer group | Cooperating consumers; each partition → one consumer in group |
| Broker | Server storing partitions |
| Controller | Cluster metadata, leader election |
Ordering guarantee
Total order within one partition only. Cross-partition order is undefined. Partition key choice (user_id, order_id) colocates related events.
Consumer groups
N consumers in group, P partitions:
- If N ≤ P — each consumer gets ≥1 partition
- If N > P — some consumers idle
- Rebalance on join/leave — brief pause; know it causes duplicate processing window
Replication in Kafka
Each partition has leader broker and followers. Producers ack when leader (and optionally min ISR followers) persist. Connects directly to leader-follower and ISR concepts from Topic 3.
Not a Kafka admin course
Skip broker tuning, ZooKeeper vs KRaft migration details unless asked. Focus on log + partition + group semantics for interviews.
Further Reading
- DDIA — Ch. 11: Stream Processing (§11.1 event streams, §11.2 messaging systems, §11.4 Kafka-style log)Book40m
- Jay Kreps — The Log: What every software engineer should know about real-time data's unifying abstractionArticle25m
- Apache Kafka Documentation — Introduction (topics, partitions, consumer groups)Reference20m
Hands-On Tasks (Optional)
Low-setup exercises — browser visualizers, paper drills, or optional Docker. No autograding; the goal is interview fluency.
- Optional: single-broker Kafka with Docker20m
If Docker is available: `docker run -d --name kafka -p 9092:9092 apache/kafka:latest` (or Confluent single-image). Produce one message, consume with console consumer, note partition and offset in output. Skip if no Docker — the reading is sufficient.