Processing time vs event time
| Time type | Definition | Problem |
|---|---|---|
| Processing time | When the pipeline processes the event | Non-deterministic on retry/replay |
| Event time | When the event actually happened (timestamp field) | Requires handling out-of-order delivery |
Production aggregations almost always use event time for correctness.
Windows
| Window | Use case |
|---|---|
| Tumbling | Fixed, non-overlapping (5-min error rate) |
| Sliding | Overlapping (1-hour count updated every minute) |
| Session | Gap-based (user activity until 30 min idle) |
Watermarks
A watermark declares how late event-time data can arrive:
Watermark = max_event_time_seen - allowed_lateness
When watermark passes window end, the window closes and late data is dropped or sent to a side output.
Trade-off:
- Tight watermark — low latency, drops late data
- Loose watermark — correct but slow results
Stream joins (hard mode)
| Join | Challenge |
|---|---|
| Stream-stream | Buffer both sides until watermark |
| Stream-table | Broadcast/static dimension table |
| Stream-table (CDC) | Dimension changes over time — needs versioned state |
Link to Distributed Systems (Kafka semantics) — log ordering and consumer offsets underpin replay.
Interview answer template
"Clickstream events can arrive 15 minutes late from mobile offline buffers. I'd use event-time tumbling windows with a 20-minute watermark and a side output for very-late events to a correction job."
Further Reading
Hands-On Tasks (Optional)
Pipeline design drills and whiteboard exercises — DAG sketches, partition plans, backfill strategies. Assumes Databases and SQL fundamentals are in place.
- Pick a window type15m
For: (1) 5-minute error rate, (2) user session length, (3) rolling 1-hour unique users — choose tumbling, sliding, or session window and define event-time watermark policy.