Spark Architecture & Job Execution

Spark cluster anatomy

Driver (JVM) ├── SparkContext / SparkSession ├── DAG Scheduler → Task Scheduler └── talks to Cluster Manager (YARN, K8s, Mesos) Executors (JVMs on worker nodes) ├── Run tasks ├── Cache RDD/DataFrame partitions in memory/disk └── Return results to driver

Driver OOM — collected too much data (collect(), giant broadcast). Executor OOM — partition too large, skew, or cache too much.

Job → Stage → Task

1. Action triggers a job
2. Shuffle boundaries split stages
3. Each stage has tasks (one per partition)

Spark UI is mandatory for senior debugging:

• Duration per stage
• Shuffle read/write bytes
• Spill (memory → disk)
• GC time

Caching and persistence

.cache() / .persist() after an expensive computation reused downstream. But:

• Costs memory
• Stale if upstream changes
• Not a substitute for writing intermediate tables in lakehouse pipelines

Production pattern: materialize silver tables instead of chaining 15 lazy transforms.

Broadcast variables

Ship small lookup table to all executors — avoids shuffle join. Spark auto-broadcasts below spark.sql.autoBroadcastJoinThreshold (default 10MB).

Link to Concurrency

Each executor runs multi-threaded tasks on separate CPU cores. Shuffle uses disk I/O and network concurrently — thread pool tuning is cluster-manager territory.

Interview answer template

"Job has 200 tasks, 199 finish in 2 min, one runs 45 min — classic skew. I'd check the Spark UI stage detail, identify the hot key, then salt or enable AQE skew join optimization."