Spark Structured Streaming & Flink

Spark Structured Streaming

Model: treat a stream as an unbounded table — append micro-batches on a trigger interval.

Output modes:

Mode	Behavior
Append	Only new rows (no aggregations)
Complete	Full result table each trigger (small aggregates)
Update	Changed rows only (aggregates with state)

Triggers: processingTime (fixed interval), once (batch-like), continuous (experimental low-latency).

State store: RocksDB on executor for aggregations — checkpoint to durable storage for recovery.

Good for: teams already on Spark, micro-batch latency (seconds–minutes), lakehouse sinks (Delta).

Apache Flink

True streaming — record-by-record with checkpoint barriers.

Strengths:

• Low latency (sub-second)
• Sophisticated event-time and state
• Exactly-once end-to-end with Kafka + checkpointed sinks

Complexity: separate ops model, state backend tuning, savepoint upgrades.

Decision matrix

Factor	Spark Structured Streaming	Flink
Latency target	Seconds+	Sub-second to seconds
Team skills	Already use Spark	Dedicated stream team
State complexity	Moderate	Heavy (CEP, large state)
Ecosystem	Delta, Hive, batch unity	Kafka-native, Kappa

Google internal: FlumeJava / Dataflow (Beam) — same Dataflow model as Flink conceptually.

Exactly-once (preview)

Requires:

• Idempotent or transactional sink
• Checkpointed source offsets
• Two-phase commit sinks (Kafka, Delta with careful config)

Deep dive on Flink checkpoints in Topic 9 (Deep Cuts). Delivery semantics theory in Distributed Systems.

AI boundary

Real-time feature pipelines for ML may use these engines — but embedding/RAG ingestion specifics stay in AI Systems.

Interview answer template

"Sub-minute latency with complex sessionization — Flink. Five-minute dashboard refresh on the same lake stack as batch — Structured Streaming to Delta with 1-minute trigger. Same medallion sink, different engine by SLA."