PySpark interview questions with real answers

df = spark.read.parquet("s3://events/")   # nothing runs
df = df.filter(col("country") == "BR")     # nothing runs
df = df.groupBy("user_id").count()         # nothing runs
df.write.parquet("s3://out/")              # ACTION -> the whole plan executes

The payoff: Catalyst sees the entire plan before executing, so it can push the filter down into the Parquet scan and prune columns and partitions. The trap: errors surface at the action, far from the line that caused them.

Narrow (select, filter, withColumn): each output partition depends on one input partition — no network. Wide (groupBy, join, distinct, repartition): rows must be redistributed by key, which triggers a shuffle and creates a new stage. Stage boundaries in the Spark UI are exactly the shuffle points.

Diagnose: in the Spark UI stage view, look at task duration — median 2s, max 400s means skew.

# Option A — let AQE handle it (Spark 3+)
spark.conf.set("spark.sql.adaptive.enabled", True)
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", True)

# Option B — broadcast the small side
from pyspark.sql.functions import broadcast
orders.join(broadcast(countries), "country_code")

# Option C — salt the hot key
from pyspark.sql.functions import col, lit, rand, floor, concat_ws
SALT = 16
big   = big.withColumn("k", concat_ws("_", col("user_id"), floor(rand()*SALT)))
small = (small
    .withColumn("s", explode(array([lit(i) for i in range(SALT)])))
    .withColumn("k", concat_ws("_", col("user_id"), col("s"))))
joined = big.join(small, "k").drop("k", "s")

When the "small" side isn't small. The driver collects it and ships it to every executor, so a 2 GB broadcast with 200 executors is a driver OOM waiting to happen. Default threshold is spark.sql.autoBroadcastJoinThreshold = 10 MB. Raise it deliberately, and remember that broadcast size is the in-memory size, not the compressed Parquet size.

df.repartition(200, "customer_id")  # full shuffle, even distribution, can increase
df.coalesce(10)                     # no shuffle, merges partitions, can only decrease

Use coalesce right before writing to avoid thousands of tiny files. Use repartition when partitions are unbalanced or you need to co-locate a join key. coalesce(1) on a big DataFrame funnels everything into one task — a classic accidental bottleneck.

# Slow: row-by-row serialization JVM <-> Python, opaque to Catalyst
@udf("double")
def to_usd(cents): return cents / 100.0

# Fast: native expression, optimizable, no serialization
df.withColumn("usd", col("amount_cents") / 100.0)

# Middle ground when you truly need Python: Arrow-based pandas UDF
@pandas_udf("double")
def to_usd(s: pd.Series) -> pd.Series: return s / 100.0

The classic: latest record per key (deduplication).

from pyspark.sql import Window
from pyspark.sql.functions import row_number, col

w = Window.partitionBy("order_id").orderBy(col("ingested_at").desc())

latest = (df
    .withColumn("rn", row_number().over(w))
    .filter(col("rn") == 1)
    .drop("rn"))

Follow-up they will ask: row_number vs rank vs dense_rank, and why a window without partitionBy forces all data into a single partition.

• collect() on a large DataFrame — driver OOM. Use take(n) or write to storage.
• Too-large broadcast — lower the threshold or disable auto-broadcast for that query.
• Skewed partition — one executor holds a giant key group; salt or enable AQE.
• Over-caching — cache only reused DataFrames and unpersist() after.
• Too few partitions for the data volume — aim for ~128 MB per partition.

"Tell me about a Spark job you made faster." Structure the answer as: symptom → what the Spark UI showed → hypothesis → change → measured result. "The daily job took 4h; the UI showed one stage at 3h20 with max task 60x the median; the join key was 40% nulls; I filtered nulls before the join and enabled AQE skew handling; runtime dropped to 35 minutes." That single sentence pattern passes most interviews.

These questions are only answerable convincingly if you've broken and fixed real jobs. The DataForge PySpark course and daily Bug Hunt put you in front of broken Spark code every day, and the full interview guide covers SQL, modeling and system design.