SLAs, Backfills & Pipeline Idempotency

Data SLAs

SLA type	Example
Freshness	Gold revenue_daily ready by 6 AM UTC
Completeness	Row count within 5% of source
Correctness	Zero null in customer_id

Airflow SLA misses alert when a task instance finishes too late — different from data quality SLAs but both matter.

Backfills

When: bug fix, new column, historical reprocessing.

Steps:

1. Freeze downstream consumers or version tables
2. Delete or overwrite affected partitions (Delta replaceWhere, Iceberg overwrite)
3. Run backfill — airflow dags backfill or parallel partition jobs
4. Validate — row counts, checksums, dbt tests
5. Unblock consumers

catchup=True on a new DAG replays every missed execution_date — can overwhelm cluster. Use intentionally.

Idempotent writes

Every task must survive retry:

Pattern	Idempotent?
INSERT append	No — duplicates on retry
INSERT OVERWRITE partition	Yes — if partition is full replace
Merge (upsert) on key	Yes — with deterministic keys
Delete + insert partition	Yes — atomic in Delta/Iceberg

Link to Distributed Systems — at-least-once task execution is the default; idempotent sinks give effective exactly-once.

Late-arriving data

Options:

1. Re-run last N days nightly (simple, expensive)
2. Incremental merge job for late events
3. Streaming correction side channel

Orchestrator schedules the strategy — engine executes it.

Interview answer template

"Tasks are partition-overwrite idempotent. On retry, rewriting dt=2026-07-09 is safe. For a 30-day backfill I'd disable the downstream DAG, run parallel EMR steps per week, validate with Great Expectations, then re-enable."