Batch, Streaming & Architecture Patterns

Batch vs streaming

Dimension	Batch	Streaming
Latency	Minutes to hours	Seconds to sub-second
Cost	Cheaper per TB (amortized cluster)	Higher infra + ops complexity
Correctness	Easier to reason about fixed input set	Watermarks, late data, state
Examples	Daily aggregates, ML training sets	Fraud alerts, live dashboards

Most FAANG-scale platforms use both — not either/or.

Architecture patterns

Lambda architecture (historical)

• Speed layer — stream processing for recent data
• Batch layer — full recompute for correctness
• Serving layer — merge views

Problem: two codepaths, high maintenance. Largely superseded but still appears in legacy systems.

Kappa architecture

• One stream as source of truth; reprocess history by replaying the log
• Requires durable log (Kafka) and idempotent sinks
• Jay Kreps' argument: batch is a special case of stream (bounded input)

Medallion (lakehouse standard)

Layer	Contents	Quality
Bronze	Raw ingest, append-only	As-is from source
Silver	Cleaned, deduped, conformed schema	Validated
Gold	Business aggregates, marts	Consumer-ready

Link to Databases (Object Storage & Data Lakes) for Parquet/Iceberg/Delta storage — here we focus on pipeline flow between layers.

Micro-batch (structured streaming)

Spark Structured Streaming default: process in small batch intervals (e.g., 1 min). Bridges batch ergonomics with near-real-time latency. Not true streaming but often sufficient.

When to pick what

Requirement	Pattern
T+1 reporting	Batch (nightly Spark/dbt)
< 1 min freshness	Micro-batch or stream
< 5 s + complex joins	Flink or Spark streaming with state
Reprocess history after bug	Replay from bronze/Kafka

AI pipeline boundary

AI Systems covers RAG ingestion, embedding pipelines, and vector indexes for LLM products. This track covers general batch/stream ETL — the same bronze→silver patterns feed ML feature stores but we don't duplicate embedding-specific design here.

Interview answer template

"We use medallion on S3/Delta: bronze append from Kafka, silver dedup and schema enforcement, gold dbt marts. Fraud uses a separate Flink job — sub-30s SLA can't wait for nightly batch."