Airflow

Apache Airflow tutorial — DAGs to production.

The practical guide to Airflow — what a DAG actually is, which operators you'll use every day, and how to ship a pipeline you're not embarrassed by. No 400-page docs detour.
Start the orchestration course See dbt + Airflow together

The mental model in 30 seconds

An Airflow DAG is a Python file that returns a graph. Nodes are tasks (things to run — a SQL query, a Python function, a dbt build, a Spark submit). Edges are dependencies (task B waits for task A). The scheduler decides when a DAG run starts; the executor decides where each task runs. That's it.

Your first modern DAG

from datetime import datetime
from airflow.decorators import dag, task

@dag(
    dag_id="daily_revenue",
    schedule="0 6 * * *",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    tags=["finance"],
)
def daily_revenue():
    @task
    def extract() -> list[dict]:
        return fetch_orders_from_api()

    @task
    def transform(rows: list[dict]) -> list[dict]:
        return [r for r in rows if r["status"] == "paid"]

    @task
    def load(rows: list[dict]) -> None:
        warehouse_insert("revenue_daily", rows)

    load(transform(extract()))

daily_revenue()

Operators you'll actually use

  • @task / PythonOperator — anything you can write in Python.
  • SQLExecuteQueryOperator — run SQL on Postgres/Snowflake/BigQuery/Redshift.
  • BashOperator — the escape hatch. Use sparingly.
  • DbtCloudRunJobOperator / BashOperator wrapping dbt — trigger dbt from Airflow.
  • KubernetesPodOperator — run any container as a task. The pattern for isolated, heavy workloads.
  • Sensors (with deferrable=True) — wait for a file, an S3 key, an external DAG. Deferrable frees up workers.

The features seniors use every day

  • Dynamic task mapping. Fan out over a list at runtime — one task per partition, per tenant, per file.
  • Datasets / Assets. Trigger DAG B when DAG A publishes a dataset. Real cross-DAG dependencies.
  • Deferrable operators. Sensors that release their worker slot while waiting. Cuts cluster cost dramatically.
  • Pools & priority_weight. Prevent one DAG from monopolizing warehouse concurrency.
  • SLAs and callbacks. on_failure_callback that pages you, not silent failures.

Production checklist

  • Deploy from Git — never edit DAGs in the UI.
  • Pin the constraints file when installing providers.
  • Run on Kubernetes or a managed service (MWAA, Cloud Composer, Astronomer). Skip the VM.
  • Use connections + Airflow Secrets Backend — no credentials in DAG code.
  • Enable remote logging to S3/GCS from day one.
  • Alert on task duration regressions, not just failures.

Where DataForge takes you next

Our orchestration course drills DAG design patterns, dynamic mapping, and real production incidents. Pair it with the dbt tutorial (Airflow + dbt is the workhorse combo) and projects for portfolio work.

FAQ

What is Apache Airflow?
Airflow is the most widely used open-source workflow orchestrator. You define pipelines as Python DAGs (directed acyclic graphs) — each node is a task, each edge is a dependency — and Airflow schedules them, retries failures, tracks state and exposes a UI. It's the plumbing that runs data pipelines at almost every serious company.
Airflow vs Dagster vs Prefect vs Mage — which should I learn?
Learn Airflow first. It has by far the biggest job market, the deepest ecosystem of providers (AWS, GCP, Snowflake, dbt, Databricks, Kubernetes), and the most existing production DAGs you'll inherit. Dagster and Prefect are excellent newer designs and worth knowing after Airflow.
How do I run Airflow locally to learn?
Fastest path: astro dev init && astro dev start (Astronomer CLI, wraps Docker). Alternative: docker-compose from the official Airflow image. Avoid pip-installing Airflow into a random venv — the constraints file matters and breaking it wastes hours.
What is a DAG in Airflow?
A DAG is a Python file that declares tasks and their dependencies. Tasks run in topological order — a downstream task starts only after its upstream tasks succeed. DAGs have a schedule (cron or timedelta), retries, SLAs, and a UI view showing every run.
Is Airflow still relevant in 2026?
Yes. Airflow 3 is production-hardened, has task-level assets, dynamic task mapping, deferrable operators, and a modern UI. The 'Airflow is dead' takes are mostly from people selling alternatives. It's still the default for enterprise data orchestration.

Ready to start?

7 days free. Then less than a coffee per month.

Start the orchestration course