Spotify

Spotify's product is fundamentally a recommendation problem. Discover Weekly, Daily Mix, Blend, Radio, and even the ordering of your Home screen are all model outputs. That model is only as good as the events feeding it — every play, skip at 3 seconds, thumbs up, and search query has to land in the warehouse, cleanly, on time, every time.

At 500 billion events a day, "cleanly" is not a small ask. Spotify's entire data platform is designed around making that event pipeline boring so the interesting work — models, product, growth — has a solid floor.

Every event Spotify ships is defined in a schema registry. If a client wants to emit TrackPlayed, it registers the schema (fields, types, semantic meaning) as code. The event delivery system:

• Rejects events that don't match a known schema
• Supports backward-compatible schema evolution (new optional fields fine, breaking changes require a new event name)
• Routes events to Pub/Sub with the schema attached
• Automatically materializes a BigQuery table per event type, mirroring the schema

This is the same pattern Uber and Netflix rediscovered under different names (Keystone, Uber's Schema Service). The insight: the event schema is the interface between product engineers and data engineers. Ship it, own it, evolve it deliberately.

Spotify open-sourced Scio — a Scala SDK for Apache Beam / Google Dataflow. Beam's Java API is verbose. Scio makes Beam feel like Spark or Scalding: a fluent, typed, functional API.

A canonical Scio pipeline: enrich raw play events with track metadata, aggregate by user, write to BigQuery.

import com.spotify.scio.bigquery._
import com.spotify.scio.{ContextAndArgs, ScioContext}

object PlayEnrichment {
  def main(cmdlineArgs: Array[String]): Unit = {
    val (sc, args) = ContextAndArgs(cmdlineArgs)

    // Read schema-checked events from BigQuery (or Pub/Sub for streaming)
    val plays = sc.typedBigQuery[PlayEvent](args("input"))

    // Broadcast the track dimension for a side-input join
    val tracks = sc.typedBigQuery[Track]("tracks_curated").asMapSideInput

    plays
      .withSideInputs(tracks)
      .flatMap { (p, ctx) =>
        ctx(tracks).get(p.trackId).map { t =>
          EnrichedPlay(p.userId, p.trackId, t.artistId, t.genre, p.ms)
        }
      }
      .toSCollection
      .saveAsTypedBigQueryTable(args("output"))

    sc.run()
  }
}

Same code runs as batch (over yesterday's BigQuery table) or streaming (over the Pub/Sub topic). One codebase, two execution modes. That's the whole point of Beam.

Unlike Netflix (S3 + Iceberg + Trino) or Airbnb (S3 + Spark + Presto), Spotify made the opinionated bet on BigQuery as the warehouse. It works because:

• Separation of storage and compute is native: they don't run a cluster, they submit queries.
• Streaming inserts land Pub/Sub events with seconds of latency.
• BigQuery + Dataflow are the same product family — no impedance mismatch.

The tradeoff is vendor lock-in. Spotify accepts it because the operational cost of running Presto/Spark/HDFS at their scale would need a large infra team. BigQuery lets 30 data engineers serve 3,000 internal users. That math wins.

Do not copy this without doing the math. BigQuery at your scale might be an order of magnitude more expensive than an S3 + Trino lakehouse. Or it might be cheaper. Run the numbers.

Spotify open-sourced Backstage — a developer portal that catalogs every service, pipeline, dataset, and dashboard in the company. It's not a data-specific tool, but it solved a very data problem:

• "Which pipeline produces the user_daily_active table?"
• "Who owns the TrackPlayed event schema?"
• "What downstream dashboards break if I change this field?"

At 500 events × 3,000 users × 1,000 pipelines, discovery isn't a UX problem — it's an existential one. Backstage indexes everything and makes the graph of ownership traversable. Same problem Airbnb solved with Dataportal, LinkedIn with DataHub. Different tools, same lesson: your data platform needs a catalog, or it will collapse under its own weight.

• Schema is the contract. If your event pipeline accepts anything, it accepts everything — including tomorrow's broken client. Ship a schema registry (Avro, Protobuf, JSON Schema, doesn't matter which) before your third pipeline.
• One codebase, two modes. Beam / Scio / Spark Structured Streaming all promise the same thing: batch and streaming from the same code. Where you can, take that promise — you'll avoid one entire class of production bugs.
• BigQuery vs lakehouse is a business decision, not a technical one. BigQuery is amazing until it isn't. Lakehouse is portable but has an ops floor. Model the 3-year cost curve; don't argue on Twitter.
• The catalog is the platform. You don't have a data platform until someone else can find the dataset. Backstage, DataHub, Amundsen, or a well-maintained wiki — pick one and staff it.