PySpark

PySpark tutorial — distributed Python that scales.

The practical PySpark guide: the DataFrame API you'll actually write, the joins and windows that show up in interviews, and the tuning knobs that separate a 20-minute job from a 20-second one.
Start the Spark track See the Kafka tutorial

The mental model

Spark distributes a DataFrame across many partitions on many nodes. Transformations (select, filter, join) are lazy — Spark records them in a plan. An action (count, write, show, collect) triggers execution: Spark's Catalyst optimizer rewrites the plan, then executes it in parallel across the cluster.

Your first real DataFrame pipeline

from pyspark.sql import SparkSession, functions as F, Window

spark = SparkSession.builder.appName("orders").getOrCreate()

orders = spark.read.parquet("s3://data/orders/")
users  = spark.read.parquet("s3://data/users/")

paid = (
    orders
      .filter(F.col("status") == "paid")
      .join(F.broadcast(users), "user_id", "left")
      .withColumn(
          "order_rank",
          F.row_number().over(
              Window.partitionBy("user_id").orderBy(F.col("order_ts").desc())
          ),
      )
)

paid.write.mode("overwrite").partitionBy("event_date").parquet("s3://data/paid_orders/")

The APIs you'll use every day

  • DataFrame API. select, filter, groupBy, agg, join, withColumn. Your default.
  • Spark SQL. spark.sql("SELECT ..."). Identical performance to DataFrames — Catalyst optimizes both. Use whichever is more readable.
  • Window functions. Same semantics as SQL windows: partition, order, frame. Essential for dedup, sessionization, running aggregates.
  • UDFs — avoid when possible. Python UDFs break Catalyst optimizations and serialize row by row. Prefer built-in functions, then pandas UDFs (arrow-based, much faster), then plain UDFs as a last resort.

Joins — the #1 source of pain

  • Broadcast joins. Small dimension (default <10 MB) is copied to every executor. Cheap. Force with F.broadcast(df).
  • Shuffle sort-merge joins. Both sides big. Spark shuffles by join key, sorts, then merges. This is where jobs die.
  • Skew. One key with 10 million rows and everyone else with a few — one task holds the whole job. Enable AQE (spark.sql.adaptive.enabled=true) and skew join handling (spark.sql.adaptive.skewJoin.enabled=true).
  • Bucketed tables. If you always join on user_id, bucket both tables by user_id. Shuffles disappear.

Partitioning: files, memory, output

  • File partitions (partitionBy on write) — the folder layout on disk. Great for query pruning; disaster if cardinality is too high (10k tiny files per day = pain).
  • In-memory partitions — how the DataFrame is split across executor cores. Target ~100–200 MB per partition. Repartition or coalesce as needed.
  • Rule of thumb. partitions ≈ total data / 128 MB. Too few = no parallelism. Too many = scheduler overhead swallows the job.

Structured Streaming and lakehouse writes

Structured Streaming turns the same DataFrame API into a continuous query — read from Kafka, write to Iceberg or Delta with exactly-once semantics. In 2026 this is the standard pattern for streaming into a lakehouse: Kafka → PySpark Structured Streaming → Iceberg.

The learning path

DataForge's Spark and Spark Tuning courses cover DataFrames, joins, window functions, streaming, and the AQE + skew tuning that shows up in every senior interview. Pair with SQL for data engineering (Spark SQL is the same skill) and Python for data engineering.

FAQ

What is PySpark?
PySpark is the Python API for Apache Spark — a distributed compute engine that processes datasets too big for one machine by splitting them across a cluster. You write Python that looks a lot like pandas; Spark runs it in parallel across dozens or thousands of cores.
PySpark vs pandas — when do I use which?
Pandas if the data fits in memory on one machine (loosely: under ~10 GB). PySpark when it doesn't, when you need multi-node parallelism, or when the same code has to run against a lakehouse table. In 2026, Polars is a strong third option for single-node big data — often faster than PySpark up to a limit.
Do I need to know Scala to use Spark?
No. PySpark covers 95% of real-world Spark work — DataFrames, SQL, MLlib, Structured Streaming. You'll only reach for Scala for a custom source, a UDAF with tight performance requirements, or if you inherit a Scala codebase.
How do I run PySpark locally to learn?
pip install pyspark, then run a SparkSession locally. Even better: use Databricks Community Edition (free), or run pyspark in a Docker container. DuckDB is a fine 'PySpark-lite' for practicing DataFrame patterns without cluster overhead.
Is PySpark still worth learning in 2026?
Yes. Spark powers Databricks, Amazon EMR, Google Dataproc, and huge in-house platforms at almost every large tech company. Streaming (Structured Streaming), lakehouse writes (Iceberg + Delta), and ML pipelines all still run on Spark. It's the standard for genuinely large-scale processing.

Ready to start?

7 days free. Then less than a coffee per month.

Start the Spark track