Ingest deliberately
Use spark.read, define a schema, and inspect a sample. CSV is convenient; Parquet/Delta are better long-term formats.
From first spark to production pipelines
Learn PySpark.
Actually understand it.
A visual, practical path through distributed data processing—designed to teach both what the code does and why a cluster behaves that way.
The big idea
Raw data
files, tables, events
files, tables, events
→
PySpark plan
your instructions
your instructions
Cluster workers
do work in parallel
do work in parallel
→
Useful result
table, model, report
table, model, report
One instruction plan, many workers executing pieces of it.
01 · First program
Start locally, then scale unchanged
A PySpark program can run on your laptop in local mode. The same DataFrame code can later run on a cluster; only the connection and deployment settings change.
Setup in five practical steps
1. Install a compatible Java runtime (Spark runs on the JVM). 2. Create a virtual environment so project packages remain isolated. 3. Install PySpark. 4. Write a script. 5. Run it with spark-submit, which is Spark's job launcher.
For learning, master("local[*]") means “use all local CPU cores.” On a managed platform, do not hard-code a master: the platform supplies it.
Start with a small local CSV. The goal is to learn the data flow first—not to manufacture a cluster before you need one.
# PowerShell: create and activate an isolated project
py -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install pyspark
# save the program below as first_job.py
spark-submit first_job.py
# first_job.py
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[*]").appName("first-job").getOrCreate()
df = spark.read.option("header", True).csv("data/orders.csv")
df.groupBy("country").agg(F.sum("amount").alias("revenue")).show()
spark.stop()
Build a pipeline
Use DataFrame operations and assign meaningful intermediate variables. Nothing expensive happens until an action.
Prove the result
Check row counts, nulls, duplicates, totals, and key business rules before writing output.
02 · Foundations
Get the mental model first
PySpark is the Python API for Apache Spark. It lets you describe data work in Python while Spark distributes that work across a machine or a cluster.
Data in chunks
A huge dataset is split into partitions so several machines can handle it together.
Lazy plan
Most PySpark lines build a plan. They do not process data yet.
Action starts work
Calling show(), count(), or write launches a job.
Workers cooperate
Spark schedules independent partitions on available CPUs and machines.
Driver, executors & partitions
The driver is your project manager: it reads your code, builds a plan, and assigns tasks. Executors are the workers: they keep data in memory and run those tasks. A partition is one slice of your dataset.
Non-technical analogy: the driver is a head chef, executors are cooks, and partitions are stations in the kitchen. More stations can speed service—until they create coordination overhead.
Driverbuilds plan · schedules tasks
↓ tasks
Executor 1partition A · partition B
Executor 2partition C · partition D
Executor 3partition E · partition F
Each executor processes its local partitions in parallel.
03 · Spark architecture
What happens when a PySpark job runs?
Spark separates planning, resource allocation, task scheduling, and actual data processing. Knowing those layers turns performance tuning from guesswork into diagnosis.
Your Python process: DriverSparkSession · query plan · DAG scheduler · result coordination
↔
Cluster managerStandalone · YARN · Kubernetes · Databricks-managed
↓ allocates CPU + memory
Worker / Executor AJVM · task slots · cached partitions
Worker / Executor BJVM · task slots · cached partitions
Worker / Executor CJVM · task slots · cached partitions
The driver coordinates; executors execute. Python code talks to the Spark JVM through Py4J.
The brain of the application
Creates SparkSession, converts your DataFrame calls to logical plans, requests executors, and schedules tasks. Keep large data out of the driver: collect() can crash it.
The resource allocator
It decides where executors run and how much CPU/memory they receive. It does not understand your business transformations; Spark's driver does.
The durable worker JVM
Runs many tasks over its lifetime, stores shuffled/cached blocks, and reports status. One executor processes multiple partitions over time according to available cores.
Job → stage → task
An action such as count() creates a job. Spark splits that job into stages at shuffle boundaries. Every stage is made of tasks; usually one task processes one partition.
A narrow transformation, like filter, lets each output partition be built from one input partition. A wide transformation, like groupBy, needs records redistributed by key—this is a shuffle and becomes a stage boundary.
One job is a delivery. Stages are delivery legs divided by a warehouse transfer. Tasks are individual parcels sent to drivers.
Read 200 files200 partitions / 200 tasks
→
Filter + selectnarrow: same stage
↓ shuffle by customer_id
groupBy / joinwide: new stage
→
Write outputone task per output partition
# Ask Spark to expose its physical execution plan
summary = orders.filter(F.col("status") == "paid") \
.groupBy("region").count()
summary.explain("formatted")
# Practical observability during development
summary.count() # action -> opens a Spark UI job
# Visit the Spark UI shown in console, often http://localhost:4040
Read the Spark UI like a map
Use the SQL tab to see query plans and the Stages tab to locate slow tasks. Large “shuffle read/write,” long task tails, spills to disk, and one oversized task point to the usual issues: expensive movement, skew, memory pressure, or unbalanced partitions.
Memory: executor heap is used for execution (sorts, joins, aggregations) and storage (cache). When it fills, Spark may spill intermediate data to disk; that is safer than failure but slower.
04 · DataFrames
Your primary tool: DataFrames
A DataFrame is a distributed table with named columns and a schema. Prefer it for nearly all structured data work: it is expressive, optimized, and easy to inspect.
| Choose | When it shines | Remember |
|---|---|---|
| DataFrame | Tables, files, SQL-like analysis | Default choice; Catalyst can optimize it. |
| SQL | Analysts, complex familiar queries | Same engine as DataFrame operations. |
| RDD | Very low-level / unstructured edge cases | More control, fewer automatic optimizations. |
| pandas API on Spark | Gradual pandas migration | Convenient, but learn native PySpark too. |
# Start Spark (often already available in notebooks)
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("sales-study").getOrCreate()
sales = spark.read.option("header", True).csv("sales.csv")
sales.printSchema()
sales.show(5)
Schema = the data contract
A schema says which columns exist and what their types mean—string, integer, date, array, and so on. Supplying a schema is safer and faster than asking Spark to guess.
Rule of thumb: inspect early with printSchema() and sample with show(). Bad types silently cause bad aggregations.
Think of a schema as labels on storage boxes. Without labels, “12” could be a quantity, a price, an ID, or a date fragment.
05 · Transformations
Shape the data, one intention at a time
Transformations return a new DataFrame; the original stays unchanged. Spark records the lineage and waits. An action is the moment you ask for an answer.
clean = (sales
.filter(F.col("amount") > 0)
.withColumn("day", F.to_date("sold_at"))
.select("customer_id", "day", "amount"))
Keep it columnar
Use built-in functions such as col, when, lower, and to_date. They become part of Spark’s optimized plan.
Avoid Python loops over rows. That takes work out of Spark’s fast distributed engine.
daily = (clean
.groupBy("day")
.agg(
F.sum("amount").alias("revenue"),
F.countDistinct("customer_id").alias("buyers")
))
Aggregation brings partitions together
Each worker first computes partial totals locally. Spark then shuffles matching group keys together and combines the partial totals.
A shuffle is like sorting scattered exam papers into one pile per student. Useful, but it moves data and can be expensive.
enriched = orders.join(
customers,
on="customer_id",
how="left"
)
Joining connects related tables
An inner join keeps only matches. A left join keeps every row on the left. Joins often shuffle both datasets by key.
If one table is tiny (for example, a country lookup), broadcast it so every executor gets a local copy instead of moving the big table.
w = Window.partitionBy("customer_id") \
.orderBy(F.col("sold_at").desc())
latest = orders.withColumn("rank", F.row_number().over(w)) \
.filter("rank = 1")
Windows calculate within a group
Unlike groupBy, a window keeps the original rows. Use it for rankings, running totals, previous/next values, and “latest record per entity.”
Imagine annotating every runner with their position within their own race—without merging runners into a single result row.
06 · Advanced patterns
The concepts that make pipelines scalable
Once the API feels familiar, performance is mostly about where data lives, where it moves, and how much of it exists.
Plan before execution
Spark combines compatible transformations, pushes filters close to reads, and only runs on an action. Use df.explain() to see the plan.
Data movement
groupBy, joins, sorts, and distinct often redistribute data. Reduce shuffles by filtering early and selecting only needed columns.
Reuse costly results
Use persist() when the same expensive DataFrame feeds multiple actions. Unpersist when done; memory is finite.
Right-size parallelism
Too few: workers idle. Too many: scheduling overhead. repartition() reshuffles; coalesce() reduces partitions cheaply.
Use sparingly
Python UDFs are custom escape hatches, but hide logic from the optimizer and add cross-language cost. Favor built-ins or pandas UDFs where appropriate.
Read less, faster
Parquet/Delta are columnar and retain types. Partition files by commonly filtered, low-cardinality columns such as event date—not user ID.
07 · Production toolkit
From notebook experiment to reliable system
Batch, streaming & ML
Batch processes a bounded dataset on a schedule. Structured Streaming handles new records continuously using the same DataFrame API. MLlib provides distributed feature pipelines and models.
For streaming, understand checkpoints (recovery memory), watermarks (late-data tolerance), and output modes. For all jobs, log metrics and make writes idempotent where possible.
A production pipeline is not just code that runs once. It is code that keeps giving trustworthy results when input volume, failures, and late data happen.
# A safe pattern for an optimized join
from pyspark.sql.functions import broadcast
result = (events
.filter(F.col("event_date") >= "2026-01-01")
.select("user_id", "event_type", "event_date")
.join(broadcast(country_lookup), "country_code", "left"))
result.write.mode("append").parquet("/gold/events")
Before you call it done
Quick check