Flink State, Checkpoints & Exactly-Once

Flink state

Stateful operators keep keyed state across events:

• Aggregations (counts, sums)
• Session windows
• Join buffers

State lives in state backends:

Backend	Storage
HashMap	JVM heap (dev, small state)
RocksDB	Local disk + async to checkpoint (production)

Checkpointing

Chandy-Lamport snapshot algorithm via checkpoint barriers:

1. Job manager injects barrier into streams
2. Operators snapshot state when barrier arrives
3. All state + offsets persisted to durable storage (S3/HDFS)

On failure: restart from last completed checkpoint.

Savepoints vs checkpoints

	Checkpoint	Savepoint
Trigger	Automatic periodic	Manual
Use	Fault recovery	Upgrades, migrations, A/B
Retention	Configured count	Until deleted

Upgrade flow: stop job → savepoint → deploy new code → resume from savepoint.

Exactly-once

Flink + Kafka source (offset in checkpoint) + two-phase commit sink:

1. Begin transaction
2. Write output
3. Commit on checkpoint complete

Sink must participate (Kafka, JDBC with XA, some Delta connectors).

Link to Topic 8 and Distributed Systems — end-to-end guarantees need idempotent sinks even with checkpoints.

Backpressure

Flink propagates pressure upstream — slow sink → slower source read. Tune parallelism and buffer-timeout.

Interview answer template

"RocksDB state backend, 5-minute checkpoints to S3. Kafka offsets in checkpoint for exactly-once read. Before deploy, trigger savepoint, cancel job, resume with allowNonRestoredState only if we dropped an operator. Monitor checkpoint duration — exceeding interval means backpressure."