Kafka tutorial for data engineers.

Kafka is not a queue. It's a durable log. Messages stay for a configured retention (hours, days, or forever) and consumers track their own position (offset). Multiple consumers can read the same topic independently, at their own speed, and replay from any point.

Once you internalize "log, not queue", every design decision starts making sense — retention, compaction, replay, exactly-once.

• Topic: a named stream (e.g. orders.created).
• Partition: the unit of parallelism. Ordering is guaranteed within a partition, not across a topic. Key your messages by whatever you need ordered together (user_id, order_id).
• Replication factor: typically 3 in prod. Kafka replicates partitions across brokers for durability.
• Leader / followers: writes go to the leader; followers replicate. On leader failure, a follower takes over.

Pick partition count carefully — you can add partitions later, but existing key→partition mappings change, which usually breaks ordering guarantees.

docker run -d --name=redpanda -p 9092:9092 \
  redpandadata/redpanda:latest \
  redpanda start --smp 1 --overprovisioned \
    --kafka-addr PLAINTEXT://0.0.0.0:9092 \
    --advertise-kafka-addr PLAINTEXT://localhost:9092

# produce
docker exec -it redpanda rpk topic create orders -p 3
echo '{"id":1,"amount":42}' | docker exec -i redpanda rpk topic produce orders

# consume
docker exec -it redpanda rpk topic consume orders

Same Kafka API. No ZooKeeper. Zero config. Good enough for 90% of learning and prototyping.

• acks=all. Wait for all in-sync replicas. Non-negotiable for data pipelines.
• enable.idempotence=true. Producer retries won't create duplicates.
• linger.ms + batch.size. Trade a few ms of latency for 10x throughput. Almost always worth it.
• Compression: zstd or lz4. Free throughput.

Consumers in the same group split partitions between them. If a topic has 6 partitions and you scale the consumer group to 6 pods, each pod owns 1 partition. Scale to 12 — 6 sit idle. Partition count is your parallelism ceiling.

Offsets live in the __consumer_offsets topic. Commit them:

• After processing, not on receive — otherwise you lose data on crash.
• Manually for at-least-once with control. Auto-commit is convenient and dangerous.

Rebalances happen when a consumer joins/leaves. Use cooperative rebalancing (CooperativeStickyAssignor) to avoid stop-the-world pauses.

"Exactly-once" in Kafka means read → process → write is atomic within Kafka — using idempotent producers + transactions + read-committed isolation. The moment your sink is a warehouse or an API, you're back to at-least-once unless the sink itself is idempotent (upsert on a key, transactional commit).

Practical answer: idempotent producer + idempotent sink (MERGE on unique key, Iceberg transaction) = effectively exactly-once. Say this in the interview.

You rarely write Kafka producers/consumers for standard integrations. Use Kafka Connect:

• Debezium — CDC from Postgres, MySQL, MongoDB, SQL Server into Kafka. The default way to stream a database.
• S3 / Iceberg / JDBC sinks — land Kafka topics into object storage, a lakehouse, or a warehouse without writing code.
• Schema Registry — Avro/Protobuf/JSON Schema with compatibility rules. Non-optional at scale.

Kafka → Spark Structured Streaming or Flink → Iceberg → dbt. That's the reference architecture for real-time analytics in 2026.

• Iceberg gives you ACID commits and time travel — safe to write from streaming and read from batch simultaneously.
• Watermarks handle late-arriving data. Set them explicitly; don't accept the default.
• Checkpoint state to durable storage (S3/GCS). Losing checkpoints = replaying from Kafka retention.

• Consumer lag is your #1 SLO. Monitor it per group per partition.
• Retention vs compaction. Time-based retention drops old messages. Log compaction keeps the latest per key — perfect for CDC / stateful topics.
• Repartitioning is painful. Design partition count with 2–5 year headroom.
• Poison pills — one bad message can stall a consumer. Wrap deserialization in a dead-letter queue.

The DataForge Kafka track covers everything above with runnable examples on Redpanda — producers, consumer groups, Kafka Connect, Debezium CDC, and streaming into Iceberg. Pair it with the interview questions and the portfolio projects to build streaming systems you can defend in a system-design round.