Airflow: DAG Design & Operations

Airflow's job

Airflow orchestrates — it decides when and in what order tasks run. It does not process petabytes itself. Heavy lifting stays in Spark, Flink, dbt, or warehouse SQL.

Airflow DAG ├── Sensor: wait for S3 partition dt={{ ds }} ├── Operator: trigger EMR/Databricks job ├── Operator: run dbt test └── Operator: publish freshness metric

Core concepts

Concept	Meaning
DAG	Directed acyclic graph of tasks
Operator	Template for work (Bash, Python, SparkSubmit)
Sensor	Waits for external condition (file lands, partition exists)
Task instance	One run of one task for one execution_date
Executor	Where tasks run (Celery, K8s, local)

DAG design principles

1. Idempotent tasks — retry-safe (see subtopic 2)
2. Atomic partitions — one task writes one dt partition
3. Avoid heavy work in PythonOperator — trigger remote jobs
4. Explicit dependencies — no hidden ordering
5. catchup=False for new DAGs unless you mean backfill
6. Pools/queues — limit concurrent Spark submits

XCom: use sparingly

XCom passes small metadata between tasks. Never pass large DataFrames — use storage paths as handoff.

Alternatives (names to know)

Tool	Angle
Dagster	Software-defined assets, type-safe
Prefect	Dynamic flows, modern API
Luigi	Legacy Spotify batch
Step Functions	AWS-native orchestration

Airflow remains the FAANG interview default — know Dagster's asset graph as the modern contrast.

Link to Observability

Pipeline alerts: task duration anomaly, SLA miss on execution_date, data freshness metric — not just "task failed."

Interview answer template

"Airflow triggers a Databricks job for silver, then dbt Cloud for gold. Sensors gate on upstream landing. PythonOperator only passes the partition path via XCom — all compute is remote."