■ Data

Python Data Libraries

NumPy • Pandas • Matplotlib • Seaborn • PySpark & Beyond

A compact, example-driven guide to the data & analytics stack

Every topic with code, line-by-line explanation, real output & projects

A hands-on companion — type every example, run every project, build real pipelines.
BEFORE YOU BEGIN

How to Use This Book

This is the data-analysis companion to a general Python course. It assumes you already know Python basics (variables, loops,
functions, lists, dictionaries) and focuses entirely on the libraries that turn Python into a data powerhouse. Every topic follows the
same teaching rhythm so you always know what to expect.

Section What it gives you

Concept Plain-language idea, why it exists, where it is used in real data work.

Syntax The exact form to type.

Example 1 & 2 A basic and a practical example, each explained line by line with real output.

Common Mistakes The traps that waste hours, so you skip them.

Exercise + Solution You try; then check against a fully explained answer.

Setup Install everything with one line: pip install numpy pandas matplotlib seaborn scipy . For big data, pip install pyspark
(needs Java 8+ installed). For the fast DataFrame alternative, pip install polars . Work in a Jupyter notebook or any .py file.
Every code block in this book was executed to capture its real output.

The mental model NumPy gives you fast arrays of numbers. Pandas builds labeled tables (DataFrames) on top of NumPy for
real-world messy data. Matplotlib/Seaborn turn tables into charts. PySpark does the same kind of table work but across many
machines when data is too big for one computer. Learn them in that order — each builds on the last.

Python Data Libraries | Page 2 of 30

CONTENTS

Table of Contents
Module 1 — NumPy: Fast Numerical Arrays
Creating arrays • Indexing/slicing • Vectorization & broadcasting • Aggregations • Reshaping • Boolean masking • Project: Exam
Scores Analyzer

Module 2 — Pandas: Labeled Data Tables

Series • DataFrames • Reading data • Selection (loc/iloc) • Filtering • Missing data • GroupBy • Merge/Join • New columns &
apply • Project: Sales Data Analyzer

Module 3 — Data Visualization (Matplotlib & Seaborn)

Line/bar/scatter/histogram • Customizing • Subplots • Seaborn statistical plots • Heatmaps • Project: Sales Dashboard

Module 4 — PySpark: Big Data Processing

What & why • SparkSession • DataFrames • Lazy transformations vs actions • select/filter/withColumn • groupBy/agg • Spark
SQL • Project: Big-Data Sales Aggregation

Module 5 — The Wider Ecosystem

SciPy • Polars • Dask • scikit-learn • choosing the right tool

Final Project — End-to-End Data Pipeline

Architecture • Ingest → clean → analyze → visualize → scale • every file & function explained

Python Data Libraries | Page 3 of 30

MODULE 1

NumPy — Fast Numerical Arrays

NumPy (Numerical Python) is the foundation of the entire scientific-Python stack. It provides the ndarray: a grid of numbers, all
the same type, stored compactly so operations run in fast C code instead of slow Python loops. Pandas, scikit-learn, and every
deep-learning framework sit on top of NumPy arrays. If you do anything numeric in Python, you are using NumPy underneath.

By convention it is imported as np : import numpy as np .

1.1 Creating Arrays

Concept
An array is NumPy's core object — like a Python list, but fixed-type and far faster for math. Why it exists: a list of a million
numbers is slow to add up; a NumPy array does it in one optimized operation. Where used: any numeric dataset — pixel grids,
sensor readings, financial series, model weights.

Syntax

[Link]([1, 2, 3]) # from a Python list

[Link](start, stop, step) # like range(), but an array
[Link](n) / [Link](n) # arrays pre-filled with 0 or 1
[Link](a, b, n) # n evenly spaced values from a to b

Example 1 — Basic

1 import numpy as np
2 a = [Link]([1, 2, 3, 4])
3 print(a)
4 print([Link], [Link], [Link])

Line 2: build an array from a list of four integers.

Line 4: every array carries metadata — dtype is the element type ( int64 ), shape is the size per dimension ( (4,) = 4 elements in one
dimension), ndim is the number of dimensions (1).

OUTPUT

[1 2 3 4]
int64 (4,) 1

Output explained: NumPy prints arrays without commas. The type is a 64-bit integer; the shape tuple has one number because it
is a 1-D array.

Example 2 — Practical

1 print([Link](0, 10, 2))

2 print([Link](3))
3 print([Link](0, 1, 5))

Line 1: arange(0,10,2) counts from 0 up to (not including) 10 in steps of 2.

Line 2: zeros(3) makes three 0.0 values — note they are floats by default, useful as an empty accumulator.
Line 3: linspace(0,1,5) gives 5 numbers evenly spread from 0 to 1 inclusive — the go-to for plotting axes and sampling.

OUTPUT

[0 2 4 6 8]
[0. 0. 0.]
[0. 0.25 0.5 0.75 1. ]

Common Mistakes • Mixing types in one array — [Link]([1, "a"]) silently makes everything a string.
• Expecting arange to include the stop value (it never does), or using it with floats (rounding surprises — prefer linspace ).
• Confusing shape (a tuple) with len() for multi-dimensional arrays.
Practice Exercise Create an array of the even numbers from 10 to 20 inclusive, and print its shape.
Solution

1 evens = [Link](10, 21, 2)

2 print(evens, [Link])

arange stops before 21, so 20 is included; step 2Python

keepsData
only evens. Output:
Libraries of 3012 14 16 18 20] (6,) .
| Page 4[10
 1.2 Indexing & Slicing
Concept
Reaching into an array to read or change values. 1-D works like lists; 2-D arrays (tables/matrices) use array[row, column] , which
is cleaner and faster than nested list indexing. Where used: selecting a column of data, cropping an image region, extracting a
time window.

Syntax

arr[i] # one element (1-D)

arr[start:stop] # a slice
mat[row, col] # one element (2-D)
mat[:, 1] # whole column 1 ; mat[0, :] = whole row 0

Example 1 — Basic (2-D access)

1 m = [Link]([[10, 20, 30],

2 [40, 50, 60]])
3 print(m[1, 2])
4 print(m[0])
5 print(m[:, 1])

Lines 1–2: a 2×3 matrix (2 rows, 3 columns).

Line 3: m[1, 2] — row index 1, column index 2 → 60 (indexes start at 0).
Line 4: a single index gives the whole first row.
Line 5: : means “all rows,” so m[:, 1] pulls out column 1 from every row.

OUTPUT
60
[10 20 30]
[20 50]

Example 2 — Practical (sub-block slice)

1 m = [Link]([[10, 20, 30],

2 [40, 50, 60]])
3 print(m[0:2, 1:3])

Line 3: two slices at once — rows 0–1 and columns 1–2 — carve out a rectangular sub-block. This is how you crop a region of a table or
image in one step.

OUTPUT
[[20 30]
[50 60]]

Common Mistakes • Using list-style double brackets m[1][2] — it works but is slower and won't accept slices like m[:, 1] .
• Forgetting slices are views, not copies: changing a slice changes the original. Use .copy() when you need independence.
Practice Exercise From the matrix above, extract the last column as a 1-D array.
Solution

1 print(m[:, -1])

: selects all rows and -1 the last column → [30 60] .

1.3 Vectorized Operations & Broadcasting

Concept
Vectorization means applying an operation to a whole array at once, with no Python loop. Broadcasting is NumPy's rule for
combining arrays of different shapes by automatically stretching the smaller one. Together they make NumPy both fast and
concise. Where used: scaling every price by tax, normalizing data, applying a discount per column.

Example 1 — Basic (vectorized scaling)

1 prices = [Link]([100, 200, 300])

2 with_tax = prices * 1.1
3 print(with_tax)
Python Data Libraries | Page 5 of 30
Line 2: prices * 1.1 multiplies every element by 1.1 in one optimized step — no loop. The single number 1.1 is “broadcast” across all
elements.

OUTPUT

[110. 220. 330.]

Example 2 — Practical (broadcasting a row across a matrix)

1 A = [Link]([[1, 2, 3],
2 [4, 5, 6]])
3 adjust = [Link]([10, 20, 30])
4 print(A + adjust)

Line 3: a 1-D array of three values.

Line 4: A is 2×3 and adjust is length 3. Broadcasting stretches adjust down to match every row, adding it to each. This is how you
apply per-column offsets to a whole table at once.

OUTPUT
[[11 22 33]
[14 25 36]]

Common Mistakes • Shape mismatch: broadcasting only works when dimensions are equal or one of them is 1 — otherwise you
get a ValueError .
• Falling back to for loops over arrays — almost always slower and less readable than a vectorized expression.
Practice Exercise Given temps_c = [Link]([0, 20, 37, 100]) , convert all to Fahrenheit (F = C×9/5 + 32) in one line.
Solution

1 temps_c = [Link]([0, 20, 37, 100])

2 print(temps_c * 9/5 + 32)

The whole formula is vectorized across every element at once. Output: [ 32. 68. 98.6 212. ] .

1.4 Aggregations & the axis Argument

Concept
Aggregation reduces many numbers to a summary: sum, mean, max, min, std. The axis argument controls the direction in a 2-D
array: axis=0 collapses rows (giving a result per column), axis=1 collapses columns (a result per row). Mastering axis is the
single most useful NumPy/Pandas skill.

Example 1 — Basic

1 g = [Link]([[80, 90],
2 [70, 60],
3 [100, 95]])
4 print([Link]())
5 print([Link](axis=0))
6 print([Link](axis=1))

Line 4: no axis → one grand total of all six numbers: 495.

Line 5: axis=0 averages down each column (across the three rows), giving a per-column mean.
Line 6: axis=1 takes the max across each row, giving one value per row.

OUTPUT

495
[83.33333333 81.66666667]
[ 90 70 100]

Output explained: think “axis=0 = go down, result per column; axis=1 = go across, result per row.”

Example 2 — Practical (per-student vs per-subject)

1 scores = [Link]([[85, 92, 78],

2 [70, 65, 80]])
3 print("Per student avg:", [Link](axis=1).round(1))
4 print("Per subject avg:", [Link](axis=0).round(1))

Line 3: each row is a student, so axis=1 averages their subjects → one average per student.
Python Data Libraries | Page 6 of 30
Line 4: each column is a subject, so axis=0 averages across students → one average per subject. .round(1) tidies the display.
 OUTPUT

Per student avg: [85. 71.7]

Per subject avg: [77.5 78.5 79. ]

Common Mistakes • Mixing up the axes — a quick sanity check on a tiny array saves big errors.
• Forgetting that aggregation ignores the array's labels (NumPy has none); use Pandas when you need named rows/columns.
Practice Exercise For scores above, print the highest score each student achieved in any subject.
Solution

1 print([Link](axis=1))

axis=1 scans across each student's row for their best subject → [92 80] .

1.5 Reshaping & Transpose

Concept
Reshaping rearranges the same data into a new layout (e.g. a flat list of 12 into a 3×4 grid) without changing the values.
Transpose ( .T ) flips rows and columns. Where used: preparing data for a model (which expects a specific shape), turning a row
into a column, image manipulation.

Example — Basic & Practical

1 r = [Link](1, 13)
2 grid = [Link](3, 4)
3 print(grid)
4 print("Transposed shape:", [Link])

Line 1: the numbers 1..12 in a flat array.

Line 2: reshape(3, 4) lays them into 3 rows of 4. The product of the new shape (3×4=12) must equal the element count.
Line 4: .T swaps axes, so a 3×4 becomes 4×3.

OUTPUT
[[ 1 2 3 4]
[ 5 6 7 8]
[ 9 10 11 12]]
Transposed shape: (4, 3)

Common Mistakes • Reshaping to incompatible dimensions (element count must match) → ValueError .
• Use reshape(-1, 4) to let NumPy infer the rows automatically when you only care about the column count.
Practice Exercise Turn [Link](1, 7) into a single column (6 rows, 1 column).
Solution

1 print([Link](1, 7).reshape(-1, 1))

-1 tells NumPy “work out the rows” given 1 column, producing a 6×1 column vector.

1.6 Boolean Masking & Filtering

Concept
A boolean mask is an array of True/False the same shape as your data; indexing with it keeps only the True positions. This is how
you filter numerically — “all values above the average,” “only the failing scores.” It is the NumPy ancestor of Pandas filtering and
SQL WHERE .

Example — Basic & Practical

1 data = [Link]([15, 42, 8, 23, 4, 16])

2 mask = data > 20
3 print(mask)
4 print(data[mask])
5 print("How many above 20:", (data > 20).sum())

Line 2: the comparison produces a boolean array — True wherever the value exceeds 20.
Line 4: indexing the data with that mask returns only the matching values.
Line 5: because True counts as 1, .sum() on the mask counts how many passed — a common one-liner.

Python Data Libraries | Page 7 of 30

 OUTPUT
[False True False True False False]
[42 23]
How many above 20: 2

Common Mistakes • Using Python's and / or between conditions — NumPy needs & and | , with each condition in parentheses:
(a > 1) & (a < 5) .
• Forgetting the parentheses, which causes an operator-precedence error.
Practice Exercise From data above, replace every value below 10 with 0 (leave the rest unchanged).
Solution

1 data[data < 10] = 0

2 print(data)

The mask data < 10 selects positions to overwrite; assigning 0 changes only those. Output: [15 42 0 23 0 16] .

1.7 Project — Exam Scores Analyzer

A complete NumPy mini-analysis of a class's scores stored as a matrix (rows = students, columns = subjects). It exercises array
creation, aggregation with axis , argmax , boolean masking, and broadcasting — the daily toolkit of numerical analysis.

1 import numpy as np
2
3 names = [Link](["Ali", "Bea", "Cy", "Dee", "Eve"])
4 scores = [Link]([[85, 92, 78],
5 [70, 65, 80],
6 [95, 88, 91],
7 [60, 72, 68],
8 [88, 79, 95]])
9
10 subject_avg = [Link](axis=0)
11 student_avg = [Link](axis=1)
12
13 top_idx = student_avg.argmax()
14 passed = student_avg >= 75
15 curved = [Link](scores + 5, 0, 100)
16
17 print("Subject averages:", subject_avg)
18 print("Student averages:", student_avg.round(1))
19 print("Top student:", names[top_idx], round(student_avg[top_idx], 1))
20 print("Passed:", names[passed].tolist(), "| count:", [Link]())
21 print("Curved scores (Ali):", curved[0])

Lines 3–8: a names array and a 5×3 score matrix — five students, three subjects each.
Line 10: axis=0 averages down each column → one average per subject.
Line 11: axis=1 averages across each row → one average per student.
Line 13: argmax() returns the index of the largest student average — i.e. the position of the top scorer.
Line 14: a boolean mask marking which students averaged at least 75 (passed).
Line 15: broadcasting adds 5 to every score (a “curve”), and [Link](..., 0, 100) keeps results within the valid 0–100 range so nobody
exceeds 100.
Line 19: use the top index to look up both the name and the score — array indexing links the two parallel arrays.
Line 20: names[passed] applies the mask to the names array, returning only those who passed; [Link]() counts them.

OUTPUT
Subject averages: [79.6 79.2 82.4]
Student averages: [85. 71.7 91.3 66.7 87.3]
Top student: Cy 91.3
Passed: ['Ali', 'Cy', 'Eve'] | count: 3
Curved scores (Ali): [90 97 83]

Output explained: Cy has the highest row average (91.3) and is named top; three students clear the 75 bar; Ali's 85/92/78
become 90/97/83 after the curve, with the clip preventing any from passing 100.

Practice Exercise Add a line that prints each student's best subject index (0, 1, or 2).
Solution

1 print("Best subject per student:", [Link](axis=1))

Python Data Libraries | Page 8 of 30
argmax(axis=1) returns, for each row, the column index of that student's highest score → [1 2 0 1 2] .

Python Data Libraries | Page 9 of 30

MODULE 2

Pandas — Labeled Data Tables

Pandas is the most-used data tool in Python. It adds labels and mixed types on top of NumPy, giving you the DataFrame — a
programmable spreadsheet with named columns and an index. Loading a CSV, cleaning messy values, filtering rows, grouping,
and joining tables are all one-liners. Every data analyst and data engineer lives in Pandas daily. Imported as pd : import pandas as
pd .

2.1 Series
Concept
A Series is a single labeled column — a 1-D array of values with an index (the labels). A DataFrame is really a dictionary of Series
sharing one index. Where used: one column of a table, a time series, the result of selecting a single column.

Example — Basic & Practical

1 import pandas as pd
2 s = [Link]([10, 20, 30, 40], index=["a", "b", "c", "d"])
3 print(s["b"])
4 print(s[s > 15].tolist())
5 print(round([Link](), 1))

Line 2: create a Series with custom string labels instead of default 0,1,2,3.
Line 3: look up by label — like a dictionary — returning 20.
Line 4: boolean filtering carries over from NumPy: keep only values above 15.
Line 5: Series have built-in stats like .mean() .

OUTPUT
20
[20, 30, 40]
25.0

2.2 Creating DataFrames

Concept
A DataFrame is a 2-D table: rows (with an index) and named columns, each column a Series that can hold its own type. The most
common way to build one by hand is from a dictionary where each key is a column name.

Example — Basic & Practical

1 df = [Link]({
2 "name": ["Ali", "Bea", "Cy", "Dee"],
3 "dept": ["IT", "HR", "IT", "HR"],
4 "salary": [50000, 45000, 60000, None]
5 })
6 print(df)

Lines 1–5: each dictionary key becomes a column; the lists become the column values. Rows get an automatic integer index 0–3. The None
in salary becomes NaN (“not a number”) — Pandas' marker for missing data.

OUTPUT
name dept salary
0 Ali IT 50000.0
1 Bea HR 45000.0
2 Cy IT 60000.0
3 Dee HR NaN

Output explained: the salary column became floats because NaN is a float; the leftmost column is the index.

2.3 Reading Data & Inspecting

Concept
Real data lives in files. pd.read_csv("[Link]") loads a CSV into a DataFrame in one line (there are siblings: read_excel ,
read_json , read_sql ). After loading, you inspectPython
before analyzing:
Data | Page ,[Link]
Libraries .head() of 30 , .info() , .describe() .
Example — Practical

1 df = pd.read_csv("[Link]")
2 print([Link]) # (rows, columns)
3 print([Link](2)) # first 2 rows
4 print(df["age"].describe()) # summary stats of a column

Line 1: read the file; column names come from the header row automatically.
Line 2: .shape reports size as (rows, columns) — your first sanity check.
Line 3: .head(n) previews the top rows without printing thousands.
Line 4: .describe() gives count, mean, std, min, quartiles, max for a numeric column.

OUTPUT (EXAMPLE [Link])

(3, 3)
name age city
0 Ali 25 NYC
1 Bea 30 LA
count 3.000000
mean 25.666667
min 22.000000
max 30.000000
...

Common Mistakes • Wrong path or delimiter — use sep=";" for semicolon files; check the file actually opens.
• Skipping inspection and analyzing dirty data — always .head() and .info() first.

2.4 Selecting Data: loc vs iloc

Concept
Two precise selectors: .loc[] selects by label (row index and column names); .iloc[] selects by integer position. Selecting a
single column is even simpler: df["col"] . Where used: grabbing specific cells, rows, or columns for analysis.

Example — Basic & Practical

1 print(df["name"]) # one whole column (a Series)

2 print([Link][1, "name"]) # row label 1, column "name"
3 print([Link][0, 2]) # row position 0, column position 2
4 print(df[["name", "dept"]]) # two columns (double brackets)

Line 1: single brackets with a column name return that column as a Series.
Line 2: .loc uses labels: row index 1 and column name "name" → "Bea".
Line 3: .iloc uses positions: row 0, third column (index 2).
Line 4: a list of columns in double brackets returns a smaller DataFrame.

OUTPUT (KEY LINES)

Bea
50000.0

Common Mistakes • Confusing loc (labels, end-inclusive in slices) with iloc (positions, end-exclusive).
• Single vs double brackets: df["a"] is a Series; df[["a"]] is a one-column DataFrame.

2.5 Filtering Rows

Concept
Keep only the rows that meet a condition — the equivalent of SQL WHERE . You build a boolean mask from a column comparison
and pass it back into the DataFrame. Combine conditions with & (and) / | (or), each in parentheses.

Example — Practical

1 high = df[df["salary"] > 48000]

2 print(high[["name", "salary"]])
3 it_high = df[(df["dept"] == "IT") & (df["salary"] > 48000)]
4 print(it_high["name"].tolist())

Line 1: df["salary"] > 48000 is a True/False mask; indexing with it keeps matching rows.
Line 3: two conditions combined with & ; each must be parenthesized because & binds tighter than == .
Python Data Libraries | Page 11 of 30
 OUTPUT
name salary
0 Ali 50000.0
2 Cy 60000.0
['Cy']

Common Mistakes • Using and / or instead of & / | on Series → raises an error.

• Forgetting parentheses around each condition.

2.6 Handling Missing Data

Concept
Real data has gaps ( NaN ). You must decide: drop the rows ( .dropna() ) or fill them ( .fillna(value) ). Filling with a sensible
statistic (mean/median) is common so you don't lose rows. Ignoring missing data corrupts averages and breaks models.

Example — Practical

1 print(df["salary"].isna().sum()) # how many missing

2 filled = df["salary"].fillna(df["salary"].mean())
3 print([Link]())

Line 1: .isna() flags each missing cell as True; .sum() counts them.
Line 2: .fillna() replaces every NaN with the column's mean of the present values.

OUTPUT
1
[50000.0, 45000.0, 60000.0, 51666.666666666664]

Output explained: one salary was missing; it was filled with the mean of the other three (≈51,667).

Common Mistakes • Filling categorical text with a numeric mean — pick a fitting value per column.
• Forgetting that most methods return a new object; reassign or use df["col"] = ... .

2.7 GroupBy & Aggregation

Concept
GroupBy implements “split → apply → combine”: split rows into groups by a column's value, apply an aggregation (sum, mean,
count) to each group, and combine the results. This is the heart of analytics — “total sales per region,” “average salary per
department.”

Example — Practical

1 print([Link]("dept")["salary"].mean())

Line 1: split rows by dept , take the salary column of each group, and average it. The result is a Series indexed by department.

OUTPUT

dept
HR 45000.0
IT 55000.0
Name: salary, dtype: float64

Output explained: HR's one present salary is 45,000; IT averages 50,000 and 60,000 → 55,000.

Common Mistakes • Forgetting to pick a column to aggregate, or expecting groups in input order (they come out sorted by key).
• Use .agg(["mean","sum","count"]) to compute several stats at once.

2.8 Merging & Joining

Concept
Merge combines two DataFrames on a shared key column — exactly like a SQL join. It is how you enrich data: attach department
details to employees, customer info to orders. [Link] instead stacks tables top-to-bottom or side-by-side.

Example — Practical

Python Data Libraries | Page 12 of 30

 1 dept_info = [Link]({"dept": ["IT", "HR"], "floor": [3, 2]})
2 merged = [Link](df, dept_info, on="dept")
3 print(merged[["name", "dept", "floor"]].head(2))

Line 1: a small lookup table mapping each department to a floor.

Line 2: merge(..., on="dept") matches rows where dept is equal and glues the columns together — each employee gains the matching
floor.

OUTPUT
name dept floor
0 Ali IT 3
1 Bea HR 2

Common Mistakes • Forgetting how= — default is an inner join (keeps only matching keys); use how="left" to keep all left rows.
• Duplicate keys multiply rows unexpectedly — check your key is unique where you expect it to be.

2.9 New Columns, apply , Sorting & value_counts

Concept
Create columns by assigning to a new name. .apply() runs a function on every value when there is no built-in vectorized way.
.sort_values() orders rows; .value_counts() tallies how often each category appears.

Example — Practical

1 df["bonus"] = df["salary"] * 0.1 # vectorized new column

2 df["band"] = df["salary"].apply(
3 lambda s: "high" if s and s > 50000 else "standard")
4 print(df.sort_values("salary", ascending=False)["name"].tolist())
5 print(df["dept"].value_counts().to_dict())

Line 1: arithmetic on a column makes a new column instantly (vectorized — prefer this).
Lines 2–3: .apply() with a lambda classifies each salary; use it for custom per-value logic.
Line 4: sort_values orders rows by salary, highest first.
Line 5: value_counts counts employees per department.

OUTPUT

['Cy', 'Ali', 'Bea', 'Dee']

{'IT': 2, 'HR': 2}

Practice Exercise Add a column tax equal to 20% of salary, then print the average tax per department.
Solution

1 df["tax"] = df["salary"] * 0.20

2 print([Link]("dept")["tax"].mean())

Line 1 creates the vectorized column; line 2 groups by department and averages the new column — combining two skills from
this module.

2.10 Project — Sales Data Analyzer

A realistic end-to-end Pandas analysis: load sales records, clean missing values, engineer a revenue column, then answer business
questions with grouping and ranking — the everyday job of a data analyst.

Python Data Libraries | Page 13 of 30

 1 import pandas as pd, numpy as np
2
3 data = {
4 "date": ["2026-01-05","2026-01-05","2026-01-06","2026-01-06","2026-01-07"],
5 "region": ["North","South","North","West","South"],
6 "product": ["Pen","Pen","Book","Pen","Book"],
7 "units": [100, 80, 40, 120, [Link]],
8 "price": [2, 2, 15, 2, 15]
9 }
10 df = [Link](data)
11
12 df["units"] = df["units"].fillna(df["units"].median())
13 df["revenue"] = df["units"] * df["price"]
14
15 by_region = [Link]("region")["revenue"].sum().sort_values(ascending=False)
16 top_product = [Link]("product")["revenue"].sum().idxmax()
17
18 print(df)
19 print("\nRevenue by region:\n", by_region)
20 print("\nTop product:", top_product)
21 print("Total revenue:", df["revenue"].sum())

Lines 3–10: build a DataFrame of five sales rows; one units value is missing ( [Link] ) to simulate dirty data.
Line 12: clean — fill the missing units with the column median (a robust middle value, less skewed by outliers than the mean).
Line 13: feature engineering — a vectorized revenue = units × price column for every row at once.
Line 15: group by region, sum revenue per region, and sort descending to rank regions.
Line 16: group by product, sum revenue, and idxmax() returns the label (product name) of the largest total — the best seller.
Lines 18–21: print the enriched table and the answers, including the grand total via a single .sum() .

OUTPUT

date region product units price revenue

0 2026-01-05 North Pen 100.0 2 200.0
1 2026-01-05 South Pen 80.0 2 160.0
2 2026-01-06 North Book 40.0 15 600.0
3 2026-01-06 West Pen 120.0 2 240.0
4 2026-01-07 South Book 90.0 15 1350.0

Revenue by region:
region
South 1510.0
North 800.0
West 240.0
Name: revenue, dtype: float64

Top product: Book

Total revenue: 2550.0

Output explained: the missing units became 90 (median of 100,80,40,120). South leads on revenue thanks to a large Book sale;
Book is the top product overall; everything sums to 2,550.

Practice Exercise Add a column flagging “big sale” when revenue exceeds 500, then count how many big sales occurred.
Solution

1 df["big_sale"] = df["revenue"] > 500

2 print("Big sales:", df["big_sale"].sum())

The comparison builds a boolean column; summing it counts the True rows. Here two rows (600 and 1350) exceed 500 → Big
sales: 2 .

Python Data Libraries | Page 14 of 30

MODULE 3

Data Visualization
Numbers in a table hide their story; a chart reveals it instantly. Matplotlib is the foundational plotting library — verbose but total
control. Seaborn sits on top of it for beautiful statistical charts in one line. You'll use Matplotlib for custom plots and Seaborn for
quick, attractive analysis. Standard imports: import [Link] as plt and import seaborn as sns . Every chart in this
module was generated by the code shown.

3.1 Matplotlib Basics — the Line Chart

Concept
The core workflow: call a plotting function, label it, then show() (or savefig() ). A line chart connects points in order — ideal for
trends over time (sales per month, temperature per day). Where used: dashboards, reports, any time-series.

Syntax

[Link](x, y) # draw the line

[Link](...); [Link](...); [Link](...)
[Link]() # display (or [Link]("[Link]"))

Example — Basic & Practical

1 import [Link] as plt

2 months = ["Jan","Feb","Mar","Apr","May","Jun"]
3 sales = [120, 135, 128, 160, 175, 168]
4 [Link](figsize=(6, 3.2))
5 [Link](months, sales, marker="o", color="#2f73c4", linewidth=2)
6 [Link]("Monthly Sales Trend")
7 [Link]("Month"); [Link]("Sales (k$)")
8 [Link](True, alpha=0.3)
9 plt.tight_layout(); [Link]("[Link]")

Line 4: figure(figsize=(w,h)) sets the canvas size in inches.

Line 5: plot x vs y; marker="o" dots each data point, and color/linewidth style the line.
Lines 6–7: a chart without a title and axis labels is unreadable — always add them.
Line 8: a faint grid ( alpha = transparency) helps read values.
Line 9: tight_layout() prevents labels being clipped; savefig writes the image (use [Link]() to view interactively).

Output: [Link] — a clear upward sales trend with a dip in March.

3.2 Bar, Histogram & Scatter

Concept
Different questions need different charts. Bar compares categories (sales per region). Histogram shows the distribution of one
Python Data Libraries | Page 15 of 30
numeric variable (how scores cluster). Scatter reveals the relationship between two numbers (ad spend vs revenue). Choosing
the right chart is half the skill.

Example 1 — Bar chart

1 regions = ["North","South","East","West"]
2 values = [420, 380, 510, 290]
3 [Link](regions, values, color="#0b3d91")
4 [Link]("Sales by Region"); [Link]("Sales (k$)")
5 [Link]("[Link]")

Line 3: bar(categories, heights) draws one bar per category — the eye compares heights instantly.

Output: [Link] — East leads, West trails.

Example 2 — Histogram & Scatter

1 import numpy as np
2 scores = [Link](70, 12, 300) # 300 fake exam scores
3 [Link](scores, bins=20, color="#2f73c4", edgecolor="white")
4 [Link]("Distribution of Exam Scores")
5 [Link]("[Link]")
6 # --- scatter ---
7 x = [Link](80) * 100
8 y = x * 1.3 + [Link](80) * 15
9 [Link](x, y, color="#0b3d91", alpha=0.6)
10 [Link]("Ad Spend vs Revenue"); [Link]("[Link]")

Line 3: hist buckets values into bins and counts how many fall in each — taller bars mean more common values.
Lines 9: scatter plots one dot per (x, y) pair; the upward cloud shows a positive relationship. alpha makes overlapping dots readable.

Python Data Libraries | Page 16 of 30

 Output: [Link] — a bell-shaped spread centered near 70.

Output: [Link] — revenue rises with ad spend (positive correlation).

Common Mistakes • Using a line chart for unordered categories — use a bar chart instead.
• Too few/many histogram bins, hiding or exaggerating the shape — tune bins .
• Forgetting [Link]() between plots, so charts overlap on one canvas.

3.3 Subplots — Multiple Charts in One Figure

Concept
A dashboard shows several charts together. [Link](rows, cols) returns a figure and a grid of axes you draw on
individually. Where used: any report or monitoring dashboard.

Example — Practical (2×2 dashboard)

1 fig, ax = [Link](2, 2, figsize=(7, 4.2))

2 ax[0,0].plot(months, sales, marker="o"); ax[0,0].set_title("Trend")
3 ax[0,1].bar(regions, values); ax[0,1].set_title("By Region")
4 ax[1,0].hist(scores, bins=15); ax[1,0].set_title("Scores")
5 ax[1,1].scatter(x, y, alpha=0.5); ax[1,1].set_title("Spend vs Rev")
6 [Link]("Sales Dashboard")
7 fig.tight_layout(); [Link]("[Link]")

Line 1: create a 2×2 grid; ax is a 2-D array of sub-plots addressed like ax[row, col] .
Lines 2–5: draw a different chart on each axis and give each its own title via .set_title() .
Python Data Libraries | Page 17 of 30
Line 6: suptitle adds one overall title above the grid.
 Output: [Link] — four views combined into one dashboard.

3.4 Seaborn — Statistical Charts in One Line

Concept
Seaborn understands DataFrames and makes polished statistical plots with single calls, automatically handling grouping, colors,
and legends. Two everyday workhorses: a grouped barplot (compare a value across categories and sub-categories via hue ), and
a heatmap (visualize a correlation matrix — which variables move together).

Example 1 — Grouped barplot

1 import seaborn as sns, pandas as pd

2 tips = [Link]({
3 "day": ["Thu","Fri","Sat","Sun"]*2,
4 "meal": ["Lunch"]*4 + ["Dinner"]*4,
5 "bill": [15,17,22,20, 28,30,35,33]})
6 [Link](data=tips, x="day", y="bill", hue="meal")
7 [Link]("Average Bill by Day & Meal"); [Link]("sns_bar.png")

Line 6: one call reads the DataFrame, groups bills by day, splits each day by meal (the hue ), and builds the legend automatically — work
that would take many Matplotlib lines.

Python Data Libraries | Page 18 of 30

 Output: sns_bar.png — dinners consistently cost more than lunches.

Example 2 — Correlation heatmap

1 corr = df_numeric.corr() # correlation matrix

2 [Link](corr, annot=True, cmap="Blues", fmt=".2f")
3 [Link]("Correlation Heatmap"); [Link]("[Link]")

Line 1: .corr() computes how strongly each pair of numeric columns moves together (1 = perfectly together, 0 = unrelated).
Line 2: heatmap colors each cell by its value; annot=True writes the number, cmap sets the color scale. Darker = stronger relationship.

Output: [Link] — ad spend and revenue are strongly correlated.

Common Mistakes • Calling .corr() on non-numeric columns — select numeric columns first.
• Forgetting [Link] must come before [Link]() (show clears the figure).

3.5 Project — Sales Dashboard

Combine Pandas analysis with a multi-panel Seaborn/Matplotlib figure to produce a shareable dashboard from raw sales data —
the deliverable a stakeholder actually sees.

Python Data Libraries | Page 19 of 30

 1 import pandas as pd, [Link] as plt, seaborn as sns
2
3 df = pd.read_csv("[Link]") # date, region, product, revenue
4 df["date"] = pd.to_datetime(df["date"]) # real dates for sorting
5
6 monthly = [Link](df["date"].[Link])["revenue"].sum()
7 by_region = [Link]("region")["revenue"].sum()
8
9 fig, ax = [Link](1, 2, figsize=(9, 3.5))
10 ax[0].plot([Link], [Link], marker="o", color="#2f73c4")
11 ax[0].set_title("Revenue by Month"); ax[0].set_xlabel("Month")
12 [Link](x=by_region.index, y=by_region.values, ax=ax[1], color="#0b3d91")
13 ax[1].set_title("Revenue by Region")
14 [Link]("Sales Dashboard"); fig.tight_layout()
15 [Link]("[Link]", dpi=130)

Line 3: load the raw sales file into a DataFrame.

Line 4: to_datetime converts the date text into real date objects so we can extract the month and sort chronologically.
Line 6: group by the month component ( .[Link] ) and sum revenue → a monthly trend Series.
Line 7: group by region for the comparison panel.
Line 9: a 1×2 figure: trend on the left, comparison on the right.
Lines 10–11: draw the monthly line on the first axis with titles.
Lines 12–13: draw the regional bars with Seaborn, telling it which axis to use via ax=ax[1] .
Lines 14–15: one overall title, tidy layout, and save at higher resolution for sharing.

RESULT
[Link] — a two-panel image: monthly revenue trend beside revenue-by-region bars, ready to drop into a report or
slide.

Practice Exercise Add a third panel showing revenue share per product as proportions. (Hint: [Link]("product")
["revenue"].sum() then a bar or pie.)
Solution

1 fig, ax = [Link](1, 3, figsize=(13, 3.5))

2 # ...panels 0 and 1 as before, using ax[0], ax[1]...
3 by_prod = [Link]("product")["revenue"].sum()
4 ax[2].pie(by_prod.values, labels=by_prod.index, autopct="%1.0f%%")
5 ax[2].set_title("Revenue Share by Product")

Widen the grid to 1×3, compute per-product totals, and use pie with autopct to label each slice's percentage.

Python Data Libraries | Page 20 of 30

MODULE 4

PySpark — Big Data Processing

Pandas loads all data into one machine's memory. When data grows to gigabytes or terabytes, it no longer fits. Apache Spark
solves this by spreading data and computation across a cluster of machines; PySpark is its Python interface. The good news: the
DataFrame API feels a lot like Pandas, so most of what you learned transfers. Install with pip install pyspark (requires Java 8+).
Outputs below are representative of what Spark prints when run on a cluster or locally.

4.1 What Spark Is & Why It Exists

Concept
Spark is a distributed engine: it splits a huge dataset into partitions, processes them in parallel across many machines (or many
CPU cores on one machine), and combines results. Why: speed and scale beyond a single computer's RAM. Where used:
processing logs, clickstreams, IoT data, ETL pipelines at companies handling billions of rows. Two core ideas: lazy evaluation
(Spark plans work but doesn't run it until forced) and fault tolerance (it can recompute lost partitions).

Concept Pandas PySpark

Scale One machine's memory Many machines / cluster

Execution Immediate (eager) Lazy until an action

Core object DataFrame DataFrame (distributed)

Best for < a few GB Large / huge data

4.2 SparkSession & Creating DataFrames

Concept
Everything starts with a SparkSession — your entry point to Spark, created once per program. From it you build DataFrames
from data or files.

Syntax

from [Link] import SparkSession

spark = [Link]("MyApp").getOrCreate()

Example — Basic

1 from [Link] import SparkSession

2 spark = [Link]("Intro").getOrCreate()
3
4 data = [("Ali", "IT", 50000),
5 ("Bea", "HR", 45000),
6 ("Cy", "IT", 60000)]
7 columns = ["name", "dept", "salary"]
8 df = [Link](data, columns)
9 [Link]()

Line 2: create (or reuse) the session; appName labels it in Spark's monitoring UI.
Lines 4–7: a list of row tuples and a list of column names.
Line 8: createDataFrame builds a distributed DataFrame from them.
Line 9: .show() is an action — it triggers computation and prints the table.

OUTPUT (REPRESENTATIVE)
+----+----+------+
|name|dept|salary|
+----+----+------+
| Ali| IT| 50000|
| Bea| HR| 45000|
| Cy| IT| 60000|
+----+----+------+

4.3 Transformations vs Actions (Lazy Evaluation)

Python Data Libraries | Page 21 of 30
Concept
PySpark operations split into two kinds. Transformations ( select , filter , groupBy , withColumn ) describe what you want but
run nothing — they just build a plan. Actions ( show , count , collect , write ) force Spark to execute the whole plan. This
laziness lets Spark optimize the entire chain before doing any work — crucial for performance at scale.

Key intuition Think of transformations as writing a recipe and actions as actually cooking. Spark reads the whole recipe first,
optimizes it, then cooks once when you ask to eat (an action). Nothing touches the data until then.

Transformations (lazy) Actions (trigger work)

select , filter , withColumn , groupBy , orderBy , join show , count , collect , first , write

4.4 select, filter & withColumn

Concept
The everyday transformations: select picks columns, filter (or where ) keeps rows, withColumn adds/derives a column. They
mirror Pandas selection, filtering, and column creation.

Example — Practical

1 from [Link] import col

2
3 [Link]("name", "salary").show()
4
5 [Link](col("salary") > 48000).show()
6
7 [Link]("bonus", col("salary") * 0.1).show()

Line 1: col("x") references a column by name in expressions.

Line 3: select keeps only the chosen columns.
Line 5: filter with a column condition keeps matching rows (like Pandas masking and SQL WHERE).
Line 7: withColumn("bonus", ...) returns a new DataFrame with an extra computed column; Spark DataFrames are immutable, so you
always get a new one.

OUTPUT OF LINE 7 (REPRESENTATIVE)

+----+----+------+------+
|name|dept|salary| bonus|
+----+----+------+------+
| Ali| IT| 50000|5000.0|
| Bea| HR| 45000|4500.0|
| Cy| IT| 60000|6000.0|
+----+----+------+------+

4.5 groupBy & Aggregation

Concept
Same split-apply-combine idea as Pandas, but distributed. Group by a column, then aggregate ( count , sum , avg , max ). This is
the backbone of analytics at scale.

Example — Practical

1 from [Link] import avg, count

2 [Link]("dept").agg(
3 avg("salary").alias("avg_salary"),
4 count("*").alias("headcount")
5 ).show()

Line 2: group rows by department.

Lines 3–4: .agg(...) computes several aggregates at once; .alias(...) names each output column. count("*") counts rows per
group.

OUTPUT (REPRESENTATIVE)
+----+----------+---------+
|dept|avg_salary|headcount|
+----+----------+---------+
| IT| 55000.0| 2|
| HR| 45000.0| 1|
+----+----------+---------+ Python Data Libraries | Page 22 of 30
 4.6 Spark SQL
Concept
If you know SQL, Spark lets you query DataFrames with it directly. Register the DataFrame as a temporary view, then run any
SQL string. Spark compiles SQL and the DataFrame API to the same optimized plan, so use whichever is clearer.

Example — Practical

1 [Link]("employees")
2 result = [Link]("""
3 SELECT dept, AVG(salary) AS avg_salary
4 FROM employees
5 GROUP BY dept
6 ORDER BY avg_salary DESC
7 """)
8 [Link]()

Line 1: expose the DataFrame under a table name usable in SQL.

Lines 2–7: a standard SQL query as a string; triple quotes allow multi-line.
Line 8: .show() runs it and prints — SQL and DataFrame code interchange freely.

OUTPUT (REPRESENTATIVE)
+----+----------+
|dept|avg_salary|
+----+----------+
| IT| 55000.0|
| HR| 45000.0|
+----+----------+

Common Mistakes • Calling .collect() on a huge DataFrame — it pulls everything to one machine and can crash it; prefer
.show() or aggregate first.
• Expecting transformations to run immediately — nothing happens until an action.
• Forgetting to [Link]() at the end of a script.

4.7 Project — Big-Data Sales Aggregation

Read a large sales file, clean and enrich it, then compute revenue per region and the top product — the same analysis as the
Pandas project, but written to scale to billions of rows.

1 from [Link] import SparkSession

2 from [Link] import col, sum as _sum
3
4 spark = [Link]("SalesAgg").getOrCreate()
5
6 df = [Link]("[Link]", header=True, inferSchema=True)
7
8 df = [Link]({"units": 0})
9 df = [Link]("revenue", col("units") * col("price"))
10
11 by_region = ([Link]("region")
12 .agg(_sum("revenue").alias("total_revenue"))
13 .orderBy(col("total_revenue").desc()))
14 by_region.show()
15
16 by_product = ([Link]("product")
17 .agg(_sum("revenue").alias("total"))
18 .orderBy(col("total").desc()))
19 print("Top product:", by_product.first()["product"])
20
21 by_region.[Link]("overwrite").csv("output/region_revenue")
22 [Link]()

Line 2: import column helpers; sum is renamed _sum to avoid clashing with Python's built-in sum .
Line 4: start the session.
Line 6: read the CSV; header=True uses the first row as column names, inferSchema=True detects numeric types automatically.
Line 8: [Link] replaces missing units with 0 — distributed data cleaning.
Line 9: derive a revenue column with withColumn (lazy — no work yet).
Lines 11–13: group by region, sum revenue, order descending. All still lazy — Spark is just building the plan.
Python Data Libraries | Page 23 of 30
Line 14: .show() is the action that finally executes the whole chain.
Lines 16–19: a second aggregation for products; .first() is an action returning the top row, from which we read the product name.
Line 21: .write...csv(...) saves results back to disk (distributed output); mode("overwrite") replaces any previous run.
Line 22: release the cluster resources.

OUTPUT (REPRESENTATIVE)
+------+-------------+
|region|total_revenue|
+------+-------------+
| South| 1510.0|
| North| 800.0|
| West| 240.0|
+------+-------------+
Top product: Book

Output explained: identical business answers to the Pandas project, but this code runs unchanged whether the file has 5 rows or
5 billion — Spark distributes the work.

Practice Exercise Add a count of distinct products sold per region. (Hint: countDistinct .)
Solution

1 from [Link] import countDistinct

2 [Link]("region").agg(
3 countDistinct("product").alias("distinct_products")
4 ).show()

countDistinct counts unique products within each region group — a common KPI in real reporting.

Python Data Libraries | Page 24 of 30

MODULE 5

The Wider Ecosystem

NumPy, Pandas, and Spark cover most data work, but four more libraries round out the toolkit. SciPy adds scientific math and
statistics. Polars is a blazing-fast modern alternative to Pandas. Dask scales Pandas/NumPy across cores and clusters with
familiar syntax. scikit-learn is the standard library for machine learning. This module gives each a working introduction and
shows when to reach for it.

5.1 SciPy — Scientific Computing & Statistics

Concept
SciPy builds on NumPy with modules for statistics, optimization, signal processing, and linear algebra. The most common
everyday use is statistical testing — deciding whether a difference in data is real or just noise. Where used: A/B test analysis,
research, engineering.

Example — Practical (one-sample t-test)

1 import numpy as np
2 from scipy import stats
3 data = [Link]([12, 15, 14, 10, 18, 20, 11])
4 t, p = stats.ttest_1samp(data, 13)
5 print("t =", round(t, 3), "p =", round(p, 3))

Line 4: tests whether the sample's mean differs significantly from 13. It returns a t-statistic and a p-value.
Line 5: a p-value above 0.05 means “not enough evidence of a real difference.” Here p ≈ 0.391, so the mean is not significantly different
from 13.

OUTPUT
t = 0.923 p = 0.391

5.2 Polars — A Faster DataFrame

Concept
Polars is a newer DataFrame library written in Rust. It is often many times faster than Pandas, uses less memory, and has lazy
evaluation like Spark — but runs on one machine with a clean, expressive API. Where used: medium-to-large data (millions of
rows) where Pandas is sluggish but you don't need a cluster.

Example — Practical

1 import polars as pl
2 df = [Link]({"city": ["A","B","A","B"], "sales": [100,200,150,50]})
3 out = (df.group_by("city")
4 .agg([Link]("sales").sum().alias("total"))
5 .sort("city"))
6 print(out)

Line 2: create a Polars DataFrame from a dictionary — familiar from Pandas.

Lines 3–5: group by city, sum sales per group, sort. The [Link](...) expression style is Polars' signature — readable and easy for the
engine to optimize.

OUTPUT
shape: (2, 2)
┌──────┬───────┐
│ city │ total │
│ --- │ --- │
│ str │ i64 │
├──────┾───────┤
│ A │ 250 │
│ B │ 250 │
└──────┴───────┘

5.3 Dask — Scaling Pandas & NumPy

Concept Python Data Libraries | Page 25 of 30
Dask mimics the Pandas and NumPy APIs but splits data into chunks and processes them in parallel across all your CPU cores —
or a cluster — with lazy evaluation. Where used: when your data is a bit too big for memory but you want to keep writing Pandas-
like code instead of switching to Spark.

Example — Practical

1 import [Link] as dd
2 ddf = dd.read_csv("huge_*.csv") # reads many files lazily
3 result = [Link]("region")["revenue"].sum()
4 print([Link]()) # .compute() triggers the work

Line 2: read many CSVs at once with a wildcard; nothing loads yet (lazy).
Line 3: the groupby looks exactly like Pandas but builds a task graph.
Line 4: .compute() is the action that runs everything in parallel and returns a normal Pandas result.

Mental model Dask = “Pandas that doesn't fit in memory.” Spark = “a full distributed cluster engine.” Polars = “a much faster
single-machine Pandas.” Pick the lightest tool that handles your data size.

5.4 scikit-learn — Machine Learning

Concept
scikit-learn is the standard library for classical machine learning: regression, classification, clustering. Every model follows the
same pattern — create the model, .fit(X, y) to learn from data, .predict(new_X) to forecast. Where used: price prediction,
churn detection, recommendations.

Example — Practical (linear regression)

1 from sklearn.linear_model import LinearRegression

2 import numpy as np
3 X = [Link]([[1],[2],[3],[4],[5]]) # feature (2-D)
4 y = [Link]([2, 4, 6, 8, 10]) # target
5 model = LinearRegression().fit(X, y)
6 print("slope:", round(model.coef_[0], 2))
7 print("predict(6):", round([Link]([[6]])[0], 1))

Lines 3–4: features X must be 2-D (rows = samples, columns = features); the target y is what we predict.
Line 5: create the model and .fit() it — it learns the relationship (here, y = 2x).
Line 6: the learned slope ( coef_ ) is 2.0, matching the pattern.
Line 7: .predict([[6]]) forecasts the target for a new input → 12.0.

OUTPUT
slope: 2.0
predict(6): 12.0

Common Mistakes • Passing 1-D features — scikit-learn expects X as 2-D; reshape with .reshape(-1, 1) .
• Evaluating a model on the same data it trained on — always hold out a test set ( train_test_split ).

5.5 Choosing the Right Tool

If you need to… Reach for

Fast math on numeric arrays NumPy

Clean & analyze labeled tables (< a few GB) Pandas

Same, but much faster on one machine Polars

Bigger-than-memory data, Pandas-style code Dask

Truly huge data across a cluster PySpark

Statistics & scientific math SciPy

Charts & dashboards Matplotlib / Seaborn

Machine learning models scikit-learn

Practice Exercise You have 50 GB of web logs to aggregate daily on a cluster, then train a model to predict tomorrow's traffic.
Which tools fit each step?
Solution PySpark for the 50 GB distributed aggregation
Python Data(too big for| one
Libraries machine),
Page 26 of 30 writing a small daily-summary table; then
Pandas to handle that small summary and scikit-learn to train the prediction model on it, with Matplotlib to chart the forecast.
Big data is reduced by Spark to a small table the single-machine tools can finish.

Python Data Libraries | Page 27 of 30

CAPSTONE

Final Project — End-to-End Data Pipeline

This capstone ties the whole book together into a real data pipeline: ingest raw sales data, clean it, analyze it for business KPIs,
visualize the results as a dashboard, and provide a PySpark version that scales to huge data. It is split into focused files — the way
data engineers structure production pipelines so each stage can be tested and reused.

Architecture & Design

The pipeline follows the standard ETL + analytics flow, with one stage per file (separation of concerns): Ingest → Clean →
Analyze → Visualize, orchestrated by [Link] . Data flows one way; each stage takes a DataFrame and returns a transformed
one, so any stage can be tested in isolation or swapped (e.g. read from a database instead of CSV) without touching the others. A
separate spark_job.py shows the same logic at cluster scale.

Folder Structure
sales_pipeline/
├── [Link] # orchestrates: ingest -> clean -> analyze -> visualize
├── [Link] # load raw CSV into a DataFrame
├── [Link] # fix missing values, types, duplicates
├── [Link] # compute KPIs (revenue by month/region/product)
├── [Link] # build the dashboard image
├── spark_job.py # big-data version of the same analysis (PySpark)
└── data/
├── [Link] # raw input
└── output/ # [Link] + summary tables

File 1 — [Link]

1 import pandas as pd
2
3 def load_sales(path="data/[Link]"):
4 """Read the raw sales CSV into a DataFrame."""
5 df = pd.read_csv(path)
6 df["date"] = pd.to_datetime(df["date"])
7 print(f"Ingested {len(df)} rows from {path}")
8 return df

Line 3: one function with a sensible default path; returning a DataFrame keeps the stage composable.
Line 5: read the CSV (header row becomes column names).
Line 6: convert the date column to real datetime objects immediately, so later stages can extract months and sort chronologically.
Lines 7–8: a log line for observability, then hand the DataFrame onward.

File 2 — [Link]

1 def clean_sales(df):
2 """Handle missing values, drop duplicates, add revenue."""
3 df = df.drop_duplicates()
4 df["units"] = df["units"].fillna(df["units"].median())
5 df = df[df["units"] > 0]
6 df["revenue"] = df["units"] * df["price"]
7 print(f"Cleaned: {len(df)} rows remain")
8 return df

Line 3: remove exact duplicate rows — a frequent data-quality issue.

Line 4: fill missing units with the median (robust to outliers) so no rows are lost.
Line 5: drop nonsensical rows where units are not positive — defensive validation.
Line 6: engineer the revenue feature once here, so every downstream stage can rely on it.
Line 8: return the cleaned DataFrame.

File 3 — [Link]

Python Data Libraries | Page 28 of 30

 1 def analyze_sales(df):
2 """Compute business KPIs and return them as a dict."""
3 kpis = {
4 "total_revenue": df["revenue"].sum(),
5 "by_month": [Link](df["date"].[Link])["revenue"].sum(),
6 "by_region": [Link]("region")["revenue"].sum().sort_values(ascending=False),
7 "by_product": [Link]("product")["revenue"].sum(),
8 "top_region": [Link]("region")["revenue"].sum().idxmax(),
9 "top_product": [Link]("product")["revenue"].sum().idxmax(),
10 }
11 return kpis

Line 3: bundle every metric into a dictionary — a clean contract for the next stage.
Line 4: the grand total revenue.
Line 5: monthly trend via grouping on the month component of the date.
Lines 6–7: revenue per region (ranked) and per product.
Lines 8–9: idxmax() returns the label of the biggest group — the leading region and best-selling product.
Line 11: return all KPIs together.

File 4 — [Link]

1 import [Link] as plt

2 import seaborn as sns
3
4 def build_dashboard(kpis, out="data/output/[Link]"):
5 fig, ax = [Link](1, 3, figsize=(13, 3.6))
6 m = kpis["by_month"]
7 ax[0].plot([Link], [Link], marker="o", color="#2f73c4", linewidth=2)
8 ax[0].set_title("Revenue by Month"); ax[0].grid(alpha=0.3)
9 r = kpis["by_region"]
10 [Link](x=[Link], y=[Link], ax=ax[1], color="#0b3d91")
11 ax[1].set_title("Revenue by Region")
12 p = kpis["by_product"]
13 ax[2].pie([Link], labels=[Link], autopct="%1.0f%%")
14 ax[2].set_title("Revenue Share by Product")
15 [Link]("Sales Pipeline Dashboard"); fig.tight_layout()
16 [Link](out, dpi=130)
17 print(f"Dashboard saved to {out}")

Line 5: a 1×3 panel layout for trend, comparison, and composition.

Lines 6–8: the monthly line chart from the KPI Series.
Lines 9–11: the regional bar chart via Seaborn, drawn onto the middle axis.
Lines 12–14: a pie of product share; autopct labels each slice's percentage.
Lines 15–16: overall title, tidy layout, and save the image for sharing.

Running the pipeline on six months of generated sales produced this real dashboard:

Output: data/output/[Link] — the pipeline's final deliverable.

File 5 — [Link] (Orchestrator)

Python Data Libraries | Page 29 of 30

 1 from ingest import load_sales
2 from clean import clean_sales
3 from analyze import analyze_sales
4 from visualize import build_dashboard
5
6 def run():
7 df = load_sales() # 1. INGEST
8 df = clean_sales(df) # 2. CLEAN
9 kpis = analyze_sales(df) # 3. ANALYZE
10 build_dashboard(kpis) # 4. VISUALIZE
11 print(f"\nTotal revenue: {int(kpis['total_revenue'])}")
12 print(f"Top region: {kpis['top_region']} | Top product: {kpis['top_product']}")
13
14 if __name__ == "__main__":
15 run()

Lines 1–4: import one function per stage — main only wires them together, holding no logic of its own.
Lines 7–10: the pipeline reads top to bottom exactly like its diagram: ingest → clean → analyze → visualize, each feeding the next.
Lines 11–12: print the headline numbers for a quick console summary.
Lines 14–15: the standard guard so the pipeline runs only when executed directly.

CONSOLE OUTPUT (SAMPLE RUN, 537 ROWS)

Ingested 537 rows from data/[Link]
Cleaned: 537 rows remain
Dashboard saved to data/output/[Link]

Total revenue: 753175

Top region: East | Top product: Pen

File 6 — spark_job.py (Scaling Up)

When the CSV grows past one machine's memory, this drop-in replaces ingest+clean+analyze with the same logic on Spark —
proving the design scales without changing the business questions.

1 from [Link] import SparkSession

2 from [Link] import col, sum as _sum
3
4 spark = [Link]("SalesPipeline").getOrCreate()
5 df = [Link]("data/[Link]", header=True, inferSchema=True)
6 df = [Link]({"units": 0}).filter(col("units") > 0)
7 df = [Link]("revenue", col("units") * col("price"))
8
9 ([Link]("region")
10 .agg(_sum("revenue").alias("total_revenue"))
11 .orderBy(col("total_revenue").desc())
12 .[Link]("overwrite").csv("data/output/region_revenue"))
13 [Link]()

Lines 5–7: read, clean, and enrich — the same three steps as the Pandas version, expressed in Spark's distributed API.
Lines 9–12: the regional aggregation, written straight to disk so results scale out too. Identical answers, unlimited data size.

How to run it 1) Create the folder structure and put your data in data/[Link] with columns date, region, product, units,
price . 2) pip install pandas matplotlib seaborn . 3) From the project folder run python [Link] — the dashboard appears in
data/output/ . 4) For big data, install PySpark and run python spark_job.py .
Capstone Challenges 1) Add a [Link] stage that writes the KPIs to a formatted .txt or Excel file.
2) Add a [Link] stage using scikit-learn to predict next month's revenue from the monthly trend.
3) Swap the Pandas ingest for Polars and benchmark the speed difference.
4) Parameterize [Link] to accept the input path from the command line ( [Link] ).

You've reached the end. You now hold the core data-engineering and analysis toolkit: arrays with NumPy, tables with Pandas,
charts with Matplotlib and Seaborn, scale with PySpark, and the wider ecosystem around them. The next step is the real one —
point these tools at a dataset you actually care about and build something.

Python Data Libraries | Page 30 of 30