Portfolio

Data engineering portfolio projects that get you hired.

Ten projects that use the same stack production teams do — Spark, dbt, Airflow, Kafka, Iceberg, Snowflake. Free datasets, free tiers, deep enough to survive an interview.
Learn the stack Interview prep

1. NYC Taxi lakehouse (batch, beginner)

Stack: Python · DuckDB or Spark · dbt · Parquet/Iceberg.

Ingest the NYC TLC trip dataset (billions of rows across 15 years), land it as Parquet partitioned by year/month, build dbt models for revenue, tip rates, and airport traffic. Ship a small Streamlit or Metabase dashboard.

Why it works: shows partitioning, incremental models, and cost-aware design on a dataset every data person recognizes.

2. GitHub Archive trending-repos pipeline (batch, intermediate)

Stack: Airflow · Spark · dbt · Snowflake (or DuckDB) · Postgres for the API layer.

Pull hourly GitHub events from gharchive.org, compute a rolling 24h trending score per repo (stars + forks + issues weighted by decay), expose via a small FastAPI endpoint.

Why it works: orchestration, backfills, deduplication, and a downstream product people actually want to use.

3. Real-time clickstream with Kafka + Iceberg (streaming, intermediate)

Stack: Kafka (Redpanda or Confluent free) · Spark Structured Streaming or Flink · Iceberg · Trino/Athena.

Generate synthetic clickstream events (or replay Criteo/Outbrain datasets), stream into Kafka, land into Iceberg with exactly-once semantics, query with Trino. Bonus: implement late-arriving event handling and time-travel queries.

Why it works: streaming + lakehouse is the #1 topic in system-design interviews right now.

4. CDC pipeline from Postgres to a warehouse (streaming, intermediate)

Stack: Postgres · Debezium · Kafka · Snowflake or BigQuery · dbt.

Set up a source Postgres with a fake e-commerce schema, capture changes with Debezium, land them in the warehouse, and materialize SCD Type 2 tables in dbt. Prove idempotency by replaying the stream.

Why it works: CDC + slowly-changing dimensions comes up in almost every senior interview.

5. dbt analytics engineering repo (batch, beginner-intermediate)

Stack: dbt Core · Snowflake / BigQuery / DuckDB · GitHub Actions for CI.

Take a public dataset (Shopify, Stripe sample data, or Jaffle Shop) and build sources → staging → marts. Add tests, exposures, docs, and a GitHub Action that runs dbt build on every PR.

Why it works: most companies hiring right now want dbt fluency. This is the fastest way to prove it.

6. Data quality framework with Great Expectations or Soda (batch, intermediate)

Stack: Airflow · Great Expectations or Soda Core · Slack alerts.

Wrap one of the pipelines above with automated data quality checks — schema drift, null rates, freshness SLAs, distribution shifts. Fail the DAG and post to Slack when checks break.

Why it works: everyone claims they "care about data quality." Almost no one has receipts.

7. Infrastructure-as-code data platform (advanced)

Stack: Terraform · AWS (S3 + Glue + Athena) or GCP (GCS + BigQuery) · GitHub Actions.

Provision a full lakehouse from scratch with Terraform: buckets, IAM, catalog, warehouse, secrets. Version everything. Tear it down and rebuild it in one command.

Why it works: platform-engineering skills separate mid from senior. This proves you can own infra, not just SQL.

8. GDELT global events dashboard (batch, intermediate)

Stack: Spark (or BigQuery) · dbt · a BI tool (Metabase / Superset / Preset).

Ingest the GDELT 2.0 Event Database (15-min updates, terabytes of global news events), model themes/actors/locations, and build a dashboard for tracking sentiment or event volume around a topic.

Why it works: scale + real-world messy data + a visual output people can share.

9. Reddit or Bluesky sentiment pipeline (streaming, intermediate)

Stack: Kafka · Python/Spark · a small LLM or transformers model · Postgres · a simple web frontend.

Stream posts from Reddit's or Bluesky's firehose, run sentiment/topic classification, aggregate by subreddit/topic, expose live counts through a frontend.

Why it works: streaming + ML + product surface. Rare combo in a portfolio.

10. Cost-monitoring pipeline for your own cloud (advanced)

Stack: AWS Cost & Usage Report (or GCP Billing export) · dbt · Snowflake/BigQuery · Grafana or Metabase.

Ingest your own billing data, model it into service/env/team dimensions, alert when daily cost spikes past a threshold. Bonus: cross-reference with dbt model run times to find your most expensive transformations.

Why it works: proves you think about cost — the #1 concern of every data platform team hiring in 2026.

How to package a project so recruiters actually read it

  • README first. Diagram (excalidraw is fine), stack, dataset, how to run, screenshots, trade-offs discussed.
  • One-command spin-up. docker compose up or a Terraform apply. If it takes 40 steps, no one runs it.
  • Show the output. A screenshot of the dashboard or a sample query result in the README beats any wall of text.
  • Discuss trade-offs. "I chose X over Y because…" is what separates a portfolio from a tutorial.
  • Link it everywhere. LinkedIn, resume, email signature. If it's not linked, it doesn't exist.

Where DataForge fits

Every stack above is covered by the DataForge roadmap: Docker, SQL, dbt, Spark, Kafka, Iceberg, Airflow, Snowflake, Terraform. Walk the courses to learn the tools, then use these projects to prove you can wield them. Once you've shipped two or three, you're ready for the interview questions.

FAQ

How many portfolio projects do I need to get hired?
Two or three that you can explain deeply beat ten shallow ones. Recruiters skim; hiring managers dig in. Pick projects that force you through the full lifecycle — ingestion, transformation, orchestration, quality checks — not just a notebook.
Do I need a job to build a data engineering portfolio?
No. Every project below uses free/open datasets (NYC Taxi, GitHub Archive, GDELT, Reddit, Wikipedia pageviews) and runs on free tiers (Docker locally, Snowflake trial, Databricks Community, AWS/GCP free tier). The stack is the same one production teams use.
Should I host projects on GitHub?
Yes — with a real README. Diagram the pipeline, list the stack, explain trade-offs (why dbt over stored procs, why Iceberg over Parquet), show sample outputs. A repo with 200 lines of code and a great README beats 5,000 lines with none.
How do I make projects stand out to recruiters?
Solve a problem, don't just move data. 'I built a pipeline that surfaces trending GitHub repos daily' is memorable. 'I ingested some data into Snowflake' is not. Include a screenshot of the final dashboard or a metric ('processes 12M events/day at ~$3/month').
Can I use LLMs to help build these?
Yes — but you have to understand every line. Interviewers WILL ask 'why did you partition by event_date instead of event_hour?' If you can't answer, the project hurts you more than it helps.

Ready to start?

7 days free. Then less than a coffee per month.

Learn the stack