Shuffle, Skew & Partitioning

Shuffle: the hidden tax

A shuffle redistributes data so all rows with the same key land on the same partition — required for groupBy, join, distinct.

Cost:

• Serialize to disk/network
• Spill if memory exhausted
• Straggler tasks if partitions uneven

Rule: minimize wide transformations; push filters before joins.

Data skew

When one key has orders of magnitude more rows than others:

• One task runs hours; 199 executors idle
• GC pressure on the hot partition
• Job fails or misses SLA

Mitigations

Technique	When
Broadcast join	Small table fits in memory (< ~100MB–1GB)
Salting	Add random suffix to hot key, join, then aggregate
AQE (Adaptive Query Execution)	Spark 3+ splits skewed partitions at runtime
Two-phase aggregation	Pre-aggregate before final shuffle

Partitioning strategy

File partitions (S3/HDFS paths):

s3://lake/events/dt=2026-07-10/hour=14/part-00001.parquet

Pick partition columns by query patterns:

• Good: dt, country if queries filter on them
• Bad: high-cardinality user_id as partition — millions of tiny files

Spark repartition vs coalesce:

• repartition(n) — full shuffle, increases/decreases partitions
• coalesce(n) — narrow, only decreases (no full shuffle)

File size targets

Aim for 128MB–1GB Parquet files. Too many small files → metadata overhead ("small file problem"). Compaction jobs merge bronze partitions.

Link to Databases (Object Storage) for file format internals — here we care about pipeline write patterns.

Interview answer template

"Join is skewed on customer_id — 0.1% of customers are whales. I'd try broadcast join if the dimension is small; else salt hot keys with rand(0,9), join on (customer_id, salt), then aggregate away salt. Enable AQE as a safety net."