Cost Optimization & Pipeline Observability

Cost drivers

Driver	Lever
Compute	Right-size cluster, autoscale, spot/preemptible
Shuffle	Fewer wide transforms, broadcast joins
Storage	Lifecycle policies (S3 IA/Glacier), compaction
Warehouse	Incremental models, avoid full scans
Orchestration	Job clusters vs always-on

Nightly batch: spin up → process → tear down beats 24/7 interactive cluster.

Right-sizing Spark

• Dynamic allocation — add executors under load
• Spot instances — for fault-tolerant batch (with retry)
• Instance types — memory-heavy for shuffle, compute-heavy for CPU-bound UDFs (avoid UDFs)
• Partition tuning — too many small tasks = scheduler overhead

Storage lifecycle

Bronze: 90-day hot → archive Silver: 2-year standard Gold: warehouse retention per compliance

Small file compaction reduces list/get costs on object storage.

Pipeline observability

Don't rely only on Airflow task green/red.

Metric	Type	Alert
pipeline_freshness_lag_minutes	Gauge	> SLA
rows_written	Counter	Anomaly vs baseline
job_duration_seconds	Histogram	p95 regression
data_quality_failures	Counter	> 0 on block rules
cost_usd_estimate	Gauge	Budget threshold

Link to Observability for RED/USE, burn-rate alerting — apply to data platform on-call.

On-call for data

Data platform on-call differs from product on-call:

• Backfill decisions under time pressure
• Upstream producer coordination
• Stakeholder comms when freshness misses

Runbooks: "silver late → check sensor → check EMR → escalate to source team."

Interview answer template

"We emit freshness and row-count metrics to Datadog per mart. SLO: gold ready by 6 AM 99.5% of days. Clusters are job-scoped with spot executors; shuffle-heavy jobs get memory-optimized instances. Bronze moves to IA after 90 days."