Why Beam exists
Different engines (MapReduce, Spark, Flink, Dataflow) had incompatible APIs. Apache Beam provides a portable programming model — write once, run on multiple runners.
Google Cloud Dataflow is the managed Beam runner; the theory is the Dataflow Model (same as Streaming Systems book).
Core abstractions
| Abstraction | Meaning |
|---|---|
| PCollection | Distributed dataset (bounded or unbounded) |
| PTransform | Operation on PCollection |
| Pipeline | DAG of transforms |
| Runner | Executes on Spark, Flink, Dataflow, etc. |
Windowing and watermarks are first-class in the API — not bolted on.
Runners
| Runner | Notes |
|---|---|
| Dataflow | GCP managed, autoscale, streaming-first |
| Flink | Self-hosted streaming |
| Spark | Batch-oriented runner |
| Direct | Local dev |
Same pipeline code for batch backfill (bounded source) and streaming (unbounded) — strong Kappa story.
When Beam vs native Spark/Flink
| Choose Beam | Choose native |
|---|---|
| Multi-cloud portability | Team deep on Spark only |
| GCP Dataflow standard | Databricks-native Spark |
| Unified batch+stream one codebase | Simple batch job |
Link to Topic 4
Event time and windows — Beam implements the model; Flink and Spark implement similar concepts natively.
Interview answer template
"We'd prototype on Direct runner, prod on Dataflow for autoscaling streaming. Bounded ReadFromText for backfill, unbounded ReadFromKafka for live — same ParDo transforms. Switch runner to Flink if we leave GCP."
Further Reading
Hands-On Tasks (Optional)
Pipeline design drills and whiteboard exercises — DAG sketches, partition plans, backfill strategies. Assumes Databases and SQL fundamentals are in place.
- Pick a Beam runner10m
Same pipeline for batch backfill and streaming ingest. Compare running on Dataflow vs Flink runner — ops, cost, and state backend implications.