Airbnb

Airbnb's product is home-sharing. Their influence on data engineering is disproportionate — they open-sourced two of the tools that shape most modern data teams:

• Apache Airflow (originally Airbnb-internal, now the default orchestrator)
• Apache Superset (their internal BI tool, now the leading open-source BI)

Plus Dataportal (data catalog concept, spawned Amundsen and inspired DataHub) and Knowledge Repo (long-form analytical writeups).

If you ever wondered why "just use Airflow" is the default answer to orchestration questions on Stack Overflow — this is why. Understanding Airbnb's platform is understanding the design constraints those tools were built to solve.

Airbnb built Airflow in 2014 because cron + shell scripts couldn't express the DAGs their data team was writing: "recompute pricing features after the events pipeline finishes, unless the upstream trust table failed, in which case send an alert."

The core Airflow idea that stuck:

• DAGs as code in Python — versioned, reviewed, tested like any other software
• Operators encapsulate work (BashOperator, PythonOperator, SparkSubmitOperator, DBT operator...)
• Task instances are the unit of retry, alerting, and observability

A minimal but production-shaped DAG:

from airflow import DAG
from airflow.decorators import task
from datetime import datetime

with DAG(
    "listing_features_daily",
    start_date=datetime(2026, 1, 1),
    schedule="0 4 * * *",  # 4am UTC daily
    catchup=False,
    default_args={"retries": 3, "retry_delay": timedelta(minutes=10)},
) as dag:

    @task
    def wait_for_bookings():
        # sensor on upstream bronze table
        ...

    @task
    def compute_features(**context):
        # Spark job that writes to silver.listing_features
        ...

    @task
    def publish_metric():
        # Minerva metric registration
        ...

    wait_for_bookings() >> compute_features() >> publish_metric()

10,000+ DAGs like this run every day at Airbnb. It's less "orchestrator" and more "operating system for the data platform".

Airbnb's most underrated internal tool is Minerva — their metrics platform. The problem it solves: five different dashboards showing "bookings" with five different numbers.

Minerva enforces a single source of truth for metrics:

• Every metric is defined once, in YAML, with its dimensions, aggregation, and grain
• The platform generates the underlying tables in the lakehouse
• Superset, Druid, notebooks, and services all query Minerva-certified tables
• If you want a new metric, you PR it; owners approve, the platform materializes

This is the pattern that dbt Semantic Layer, Cube.dev, and Looker LookML all point at. Airbnb built it internally in 2019 because BI without a semantic layer devolves into chaos at their scale.

Superset exists because Airbnb wanted analysts, PMs, and even execs to answer their own questions without a ticket to the data team. It's a lightweight BI tool: connect to Presto/Druid/Snowflake, write SQL, save a chart, build a dashboard.

Two design choices matter for scale:

• Query caching + row-level access control — you can put Superset in front of a production warehouse without dying from repeat queries.
• SQL Lab — a first-class SQL notebook. Analysts don't go to a separate tool to explore before building a chart.

Superset now powers dashboards at Airbnb, Netflix, Twitter, Dropbox, Wikipedia, and thousands of smaller teams. It's what "self-service BI" looks like when it's built by data engineers, not sold by them.

Airbnb's medallion architecture (bronze → silver → gold) is now the reference pattern in the industry:

• Bronze: raw events landed as-is from Kafka. Schema-on-read.
• Silver: cleaned, deduplicated, joined with dimension tables. Schema enforced.
• Gold: business-level aggregates. This is what dashboards and models consume.

Same pattern Databricks now markets under the "medallion architecture" name. Airbnb was doing it in 2018 because their old flat Hive layout couldn't survive schema drift and duplicate events. The names are new; the pattern is older.

• Metrics live in a semantic layer, not a dashboard. If the same "revenue" is defined in seven Superset charts, you don't have seven charts — you have seven definitions of revenue. dbt semantic layer, Cube, or a homegrown Minerva-lite — pick one, but centralize.
• Airflow first, then question Airflow. For most teams, Airflow is the right default. Only when you feel real pain (dynamic DAGs, sub-second scheduling, or per-task cost sensitivity) do Dagster/Prefect/Temporal become obvious choices.
• The catalog is not optional. Airbnb built Dataportal because 1,500 users can't ask "which table has the real bookings data?" 1,500 times. Ship a catalog on day one, even if it's a spreadsheet.
• Medallion is a floor, not a ceiling. Bronze/silver/gold is what "just enough" data quality looks like. Skip it and every downstream table has to re-implement dedup + join.