Uber

Uber isn't a "big data" problem — it's a big real-time problem. When you tap "request", the platform has to:

• Find nearby drivers (geo query on live driver state)
• Compute a fare (surge model, based on last-N-minute demand/supply)
• Route the trip (ETA from live traffic)
• Score for fraud (in <100ms)
• Update dozens of downstream systems (accounting, driver earnings, marketplace metrics)

All of this in about a second. And then the same event has to land in a lakehouse for tomorrow's model retraining. That dual requirement — sub-second AND analytical — is what shaped every choice in the platform.

Every meaningful event at Uber flows through Kafka. Uber runs one of the largest Kafka deployments in the world — trillions of messages a day across thousands of topics. Two design choices matter:

• Topics are keyed by entity, not by service. trip_events keyed by trip_uuid means every consumer sees ordered updates per trip.
• Producers write once; Kafka Connect / uReplicator fans out to regions and downstream sinks. Producers don't know or care where the data ends up.

Uber also open-sourced uReplicator (later replaced by MirrorMaker 2 improvements) specifically to solve cross-datacenter replication at their scale — Kafka's own tooling wasn't enough.

Traditional warehouses answer analytical queries in seconds-to-minutes. Uber needed milliseconds — to power surge pricing dashboards, fraud detection, ETA calibration. They co-created Apache Pinot to fill that gap.

Pinot is a columnar OLAP store that ingests directly from Kafka and answers slice-and-dice queries in <100ms at 100k+ QPS. A concrete example:

-- "How many completed trips per city in the last 5 minutes?"
-- Backs the surge pricing model. Runs every few seconds.
SELECT city_id, COUNT(*) AS trips
FROM trip_events
WHERE event_type = 'TRIP_COMPLETED'
  AND event_time >= ago('PT5M')
GROUP BY city_id;

Same SQL you'd write against Postgres. But Pinot pre-materializes the segments, keeps hot data in memory, and returns in 20-50ms. That's the difference between "surge pricing that reacts to reality" and "surge pricing that reacts to what happened 20 minutes ago".

Uber authored Apache Hudi in 2016, three years before "lakehouse" was a marketing term. The problem they solved: how do you do upserts on HDFS/S3? A trip can update dozens of times (accepted → started → completed → adjusted → refunded), and rewriting the whole partition every time was untenable.

Hudi introduced two table types that are now standard vocabulary:

• Copy-on-Write (CoW): rewrites files on update. Simpler reads, higher write cost. Good for analytics-heavy tables.
• Merge-on-Read (MoR): writes deltas; readers merge on-the-fly or via a compaction job. Lower write latency, higher read cost. Good for high-velocity tables like trip state.

Every modern lakehouse (Iceberg, Delta) has some version of this now. Hudi shipped it first because Uber needed it.

Michelangelo is Uber's internal ML platform. The pitch to internal teams: give us your features and your training data pointer, we handle everything else — training, deployment, monitoring, retraining. Same rails power fraud detection, ETA prediction, driver scoring, matching.

The key architectural idea is the feature store as first-class citizen. Features are defined once (in Hive/Hudi tables) and served both offline (for training) and online (via a low-latency store like Cassandra). This is the same pattern Netflix uses — and every serious ML org converges on it.

• Real-time OLAP is its own tier. If your dashboards need sub-second freshness, don't try to make Snowflake do it. Look at Pinot, Druid, or ClickHouse. They exist for this exact reason.
• Upserts on the lake are non-negotiable. The moment your data has any correction, adjustment, or slowly-changing dimension, you need a table format with upsert semantics. Hudi, Iceberg, or Delta — pick one, but pick one.
• Kafka topics are entities, not services. A topic per service creates a graph of point-to-point pipelines. A topic per entity (user, order, session, trip) creates a stable interface every consumer can share.
• The ML platform IS the ML strategy. If every team builds their own training loop, you have five ML platforms and no learnings. Michelangelo (or something like it) is the multiplier.