Netflix

Netflix's north-star metric is engagement. Every artwork on the home screen, every "because you watched" row, every autoplay preview is chosen by a model trained on hundreds of billions of events per day. The data platform's job is to make that possible without the team writing 100 bespoke pipelines.

Two constraints define everything:

• Latency: personalization must reflect what you just watched — measured in seconds, not hours.
• Scale: hundreds of petabytes of viewing data, feature stores that back thousands of models, all running on commodity S3 storage rather than a proprietary warehouse.

Every client event — a play, pause, thumbs-up, artwork impression — flows into Keystone, Netflix's managed event pipeline built on top of Kafka. Keystone is not "just Kafka". It's a self-service layer where product teams register a topic, define a schema, and get:

• Automatic routing to downstream sinks (S3/Iceberg, Elasticsearch, Druid, custom Flink jobs)
• Schema evolution with backward-compatibility checks
• A managed multi-cluster Kafka fleet the producer never sees

The team publishes ~1 trillion events per day through Keystone. Producers only know a topic name; Keystone owns the durability, replay, and fan-out. This is the pattern most companies eventually rediscover: the value isn't Kafka, it's the platform on top of it.

Netflix authored Apache Iceberg because Hive tables on S3 didn't survive concurrent writes, schema changes, or partition evolution at their scale. Iceberg gave them:

• ACID commits on object storage — batch jobs and streaming jobs can write the same table without corrupting it
• Time travel — every commit is a snapshot; SELECT ... AS OF TIMESTAMP just works
• Hidden partitioning — partition scheme can change without rewriting queries
• Manifest-based planning — pruning files without listing S3

A typical Netflix Iceberg table read looks like this:

-- Read the state of the events table as it was 3 hours ago
SELECT event_type, count(*)
FROM prodhive.datalake.viewing_events
FOR TIMESTAMP AS OF current_timestamp - INTERVAL '3' HOUR
WHERE event_date = current_date - INTERVAL '1' DAY
GROUP BY event_type;

Time travel is not a demo feature — it's how they debug pipelines, backfill safely, and reproduce ML training sets.

Netflix runs a polyglot compute layer on top of the Iceberg lakehouse:

• Apache Spark — heavy batch ETL, feature generation, ML training data prep. Runs on their internal container platform, autoscaled.
• Apache Flink — thousands of stateful streaming jobs. Real-time features, near-real-time aggregations feeding personalization.
• Presto / Trino — the interactive SQL engine every analyst and engineer uses to explore the lake. Backs internal dashboards and the ad-hoc query tier.
• Metaflow — the ML workflow framework Netflix open-sourced; sits above Spark and their orchestration layer to give data scientists a Python-native experience.

Same tables, three engines. That's the point of the lakehouse — the storage is the API.

The path from a click to a personalized home row:

1. You tap play. The client emits an event; Keystone lands it in Kafka. 2. A Flink job updates your short-term interaction state within seconds (materialized to a low-latency store). 3. A batch Spark job (nightly or every few hours) recomputes long-horizon features and writes them to a feature store backed by Iceberg + Cassandra + EVCache. 4. When you open the app, the recommendation service queries the feature store, runs the ranking model, and returns the personalized home page.

The trick is that online (the model serving) and offline (training) read the *same* features from the *same* logical store. No skew between what the model saw at training time and what it sees at serving time. That is one of the hardest problems in ML systems, and the lakehouse is why they can solve it once instead of per-model.

You don't have Netflix's traffic. You probably shouldn't build Keystone. But the ideas transfer:

• Own your table format, not your warehouse. Iceberg on S3 with Trino gives you 80% of what Snowflake sells, at 20% of the cost, and you keep your data portable.
• Time travel is a debugging tool, not a demo. Turn it on the day you build the table, not the day you have your first incident.
• The event pipeline is a product. If your producers know a broker exists, you've under-invested. Give them a topic name and a schema registry, and hide the rest.
• Same features, online and offline. Design your feature store so training data and serving data are literally the same rows, at different points in time. Iceberg's snapshots make this cheap.