Delivery Semantics & Exactly-Once

Three semantics (misunderstood constantly)

Semantic	Meaning	Failure behavior
At-most-once	May lose messages, never duplicate	Fire-and-forget, commit offset before process
At-least-once	No loss, may duplicate	Process then commit offset; retry on failure
Exactly-once	Neither loss nor duplicate as observed	Requires idempotence + transactions or dedup

Exactly-once is almost always exactly-once effect — achieved by idempotent writes + dedup, not magic network guarantees.

End-to-end vs broker-only

Kafka "exactly-once" covers producer → broker → consumer within Kafka's transactional model. Your database write is a separate system — true end-to-end needs outbox, idempotent consumer, or transactional sink.

Idempotent consumer pattern

Store processed message IDs (or business idempotency key). On duplicate delivery, skip side effect.

if seen(message_id): return OK else: apply effect; record message_id

Fowler's Idempotent Receiver — same idea.

Dedup store design (infra depth)

Production idempotency is more than an if check:

Piece	Choice
Key	Business id (payment_id), not Kafka offset alone
Store	DB unique constraint, Redis SET with TTL, or dedicated idempotency table
TTL	Match at-least-once retry window (24–72h typical)
Atomicity	Insert key + apply effect in one transaction — duplicate insert fails cleanly

Offset-only dedup breaks if the consumer commits the offset but crashes before the side effect — prefer outbox or transactional consume-process-write.

Kafka transactions (awareness)

Transactional producer + read-process-write with offsets in one transaction — prevents duplicate within Kafka pipeline. Pair with DB idempotency for full workflow.

Choosing semantics

Use case	Typical choice
Metrics, logs	At-most-once OK
Billing, inventory	At-least-once + idempotency
Audit ledger	At-least-once + strong idempotency keys

Link to sagas

Saga steps are often at-least-once events — compensations must handle duplicates (Topic 7).

Interview answer template

"We use at-least-once from Kafka with idempotent consumers keyed by payment_id. Duplicates are harmless because the ledger table has a unique constraint on that key."