Data Science with Python

Unit 1: The Data Science Landscape

Learning Objectives:
1. Differentiate between looking backward (Analytics) and looking forward
(Science).
2. Understand the "Data Kitchen": The tools, the ingredients, and the
process.
3. Recognize the ethical weight of algorithmic decision-making.

I. The Hook: "The Data Hierarchy of Needs"

Start by drawing or projecting a pyramid (Maslow’s style) on the board.
 Top (AI/Deep Learning): The flashy stuff everyone wants to do.
 Middle (Analytics/Metrics): Aggregating and counting things.
 Bottom (Collection/Infrastructure): Logging, sensors, moving bytes.
Faculty Note: Tell students: "You cannot do AI (the top) if you don't have solid
data collection (the bottom). Most data science failures happen because the
bottom of the pyramid is broken, not because the math at the top is wrong."

II. The Spectrum: Analytics vs. Science

Don't just define them; use the "Car Analogy."
1. Data Analytics (The Rearview Mirror)
 Function: Tells you where you have been.
 Key Question: "What happened?" or "Why did it happen?"
 Tools: Dashboards (Tableau/PowerBI), SQL, Excel.
  Example: A dashboard showing that sales dropped 15% last month in the
Northeast region.
2. Data Science (The Headlights & GPS)
 Function: Tells you where you are going and the best route to take.
 Key Question: "What will happen?" or "How can we make X happen?"
 Tools: Python, R, Machine Learning Models.
 Example: A model predicting that sales will drop next month unless we
lower the price by $5.

III. The Raw Material: Data Types & The "Hidden" Data
Move beyond the standard 3 types by introducing Metadata.

Type Definition The "Real World" Challenge

Structured Rows/Columns It’s rare. Only ~20% of world data is

(Excel, SQL). structured.

Unstructured Text, Audio, Video, Requires "Feature Extraction" (e.g.,

Images. turning an image into pixel values)
to be usable.

Semi- JSON, XML, Logs. Nested data. You have to "flatten"

Structured it to analyze it.

Metadata (Crucial Add) Data GPS location on a photo,

about data. timestamp on a tweet. Often more
valuable than the content itself.

Class Activity: "Open your phone's photo gallery. Pick a photo. The image is
Unstructured Data. Tell me 3 pieces of Metadata attached to it." (Answers:
Date, Location, Shutter speed, File size).
IV. The Toolkit: The "Data Kitchen" Analogy
Students struggle to understand how libraries fit together. Use a Kitchen
Analogy.
1. The Environment (Jupyter/Colab): This is the Kitchen itself. It's where
you work.
2. SQL: The Pantry. Where the bulk ingredients are stored. You go here to
fetch what you need.
3. Pandas: The Prep Station. You chop, peel, and clean the data here. This
is where you spend 80% of your time.
4. NumPy: The Knife. Sharp, fast, and mathematical. It handles the
underlying heavy lifting.
5. Scikit-Learn: The Oven. Where the magic (modeling/cooking) happens.
6. Matplotlib/Seaborn: The Plating. Making the result look good so the
customer (the stakeholder) wants to consume it.

V. The Process: OSEMN (With Reality Checks)

Walk through the acronym, but add the "Industry Reality" for each step.
Obtain
 Concept: Getting data (APIs, Scraping, SQL).
 Reality Check: "Access Denied." Often the hardest part is just getting
permission or finding the API key.
Scrub (Cleaning)
 Concept: Handling missing values, typos, duplicates.
 Reality Check: The 80/20 Rule. Warn students they will spend 80% of
their career here. It is not glamorous, but it is necessary. Garbage In,
Garbage Out.
Explore (EDA)
 Concept: Looking for patterns and outliers using graphs.
  Reality Check: "The intuition check." Does the data look right? (e.g., If a
customer's age is 200, stop now).
Model
 Concept: Training the algorithm.
 Reality Check: Start simple. A simple Linear Regression ($y = mx + c$) is
often better than a complex Neural Network because it's easier to
explain.
iNterpret
 Concept: Storytelling.
 Reality Check: The "So What?" Test. If you tell a CEO "The R-squared is
0.9," they won't care. If you say "We can save $1M by changing this
setting," they will listen.

VI. Sectors: The Value Proposition

Focus on High-Stakes examples to keep engagement high.
1. FinTech (The Money):
o Application: Fraud Detection.
o Concept: Anomaly Detection. "Is this credit card transaction normal for
this user, or is it a thief in another country?"
2. Healthcare (The Life):
o Application: Medical Imaging.
o Concept: Computer Vision. Detecting tumors in X-rays faster and more
accurately than human doctors.
3. E-Commerce (The Convenience):
o Application: Recommendation Engines.
o Concept: Collaborative Filtering. "Users like you also bought..."
VII. Ethics: The "Black Mirror" Moment
End the lecture on a serious note. Data Science has consequences.
1. Bias (The "Historical" Trap)
 Example: Amazon's failed AI hiring tool. It penalized resumes containing
the word "women's" (e.g., "Women's Chess Club captain") because it
was trained on 10 years of resumes from mostly men.
 Lesson: Data is not objective; it is a historical record of human behavior,
including our prejudices.
2. Privacy (The "Anonymity" Myth)
 Concept: Re-identification.
 Lesson: Even if you remove names, looking at "Location" + "Time" +
"Purchase History" can identify almost anyone.
3. Interpretability (The "Black Box")
 Example: In banking, you legally must explain why you denied a loan.
 Lesson: You cannot just say "The AI said no." If you can't explain it, you
often can't use it.

Unit II: Exploratory Data Analysis (The

"Detective" Phase)
Learning Objectives:
1. Understand EDA as a mindset, not just a checklist.
2. Master the "Two Pillars" of EDA: Summary Statistics (Quantitative) and
Visualization (Graphical).
3. Learn how to map specific Data Types to the correct Charts.
4. Bridge the gap between finding a pattern and telling a story.
I. EDA Concepts: The "First Date" with Data
Start by dispelling the myth that you jump straight to Machine Learning.
The Concept:
EDA is the critical first step where you get to know your data. You are looking
for patterns, anomalies, hypotheses, and assumptions.
The Three Goals of EDA:
1. Sanity Check: Is the data broken? (e.g., A student with a GPA of 5.0 on a
4.0 scale).
2. Assumption Testing: Does the data follow a normal distribution? (Crucial
for many statistical tests).
3. Feature Selection: Which columns actually matter? (If a column is 90%
empty, drop it).
Faculty Analogy: "EDA is like a doctor's check-up before surgery. You don't just
cut the patient open (Model). You take their pulse, check their blood pressure,
and ask where it hurts first (EDA)."

II. EDA Quantitative Techniques (The Numbers)

These are the hard numbers that summarize the data. Focus on the distinction
between Central Tendency and Spread.
1. Measures of Central Tendency (Where is the "Middle"?)
 Mean: The average. Warning: Highly sensitive to outliers (Bill Gates
walks into a bar, and the average wealth skyrockets).
 Median: The middle value. Key: Robust against outliers.
 Mode: The most frequent value. Useful for categorical data (e.g., "Most
sold shoe size").
2. Measures of Spread (How "Wide" is the data?)
 Range: Max - Min. (Too simple).
  Variance & Standard Deviation: How far, on average, are data points
from the mean?
 Interquartile Range (IQR): The spread of the middle 50% of data.
The "Five Number Summary": Teach students to always run this command
first.
Min | Q1 (25%) | Median (50%) | Q3 (75%) | Max

III. EDA Graphical Techniques (The Pictures)

Visuals reveal what statistics hide. A famous example is "Anscombe's
Quartet"—four datasets with identical means and variances but completely
different graphs.
1. Univariate Analysis (Looking at One Variable)
 Histogram: Shows the distribution of numerical data (e.g., Age of
students).
 Box Plot (Box-and-Whisker): The ultimate tool for detecting outliers.
o Anatomy: The box is the IQR. The "whiskers" are the range (excluding
outliers). The dots are the outliers.
2. Bivariate/Multivariate Analysis (Looking at Relationships)
 Scatter Plot: Relationship between two numerical variables (e.g., Study
Time vs. Exam Score).
 Correlation Heatmap: A color-coded grid showing how strongly variables
relate to each other (Red = Strong positive, Blue = Strong negative).

IV. Data Types and Plotting (The "Chart Chooser")

This is the most practical skill for students. If they choose the wrong chart, they
lie with data.
 Data Type Data Type Recommended Why?
A B Plot

Numerical (None) Histogram To see the shape (Bell

curve? Skewed?).

Categorical (None) Bar Chart To compare counts (e.g.,

Number of sales per
Country).

Numerical Numerical Scatter Plot To see correlation/trends.

Numerical Categorical Box Plot To compare distributions

across groups (e.g., Salary
vs. Gender).

Time Numerical Line Chart To see trends over time.

Series

Common Student Trap: Confusing Histograms and Bar Charts.

 Histogram: Bars touch (continuous data). X-axis is a range.
 Bar Chart: Gaps between bars (distinct categories). X-axis is a label.

V. Data Analytics Conclusion vs. Predictions

Help students distinguish between "reporting the news" and "forecasting the
weather."
1. Conclusion (Descriptive/Diagnostic)
 Definition: A statement of fact based on historical data.
 Example: "Sales dropped 15% because the server was down for 2 days."
 Certainty: 100% (assuming data is correct).
2. Prediction (Predictive/Prescriptive)
  Definition: A probabilistic statement about the future based on a model.
 Example: "If we fix the server, sales should rebound by 12% next month."
 Certainty: <100% (It is a probability, not a guarantee).

VI. Data Analytics Communication (The "So What?")

The best analysis fails if the communication is bad. Teach the "Pyramid
Principle."
1. Start with the Answer
 Don't walk the CEO through your code. Start with the insight.
 Bad: "I ran a regression, cleaned the nulls, used a scatter plot..."
 Good: "We are losing customers because our checkout page is too slow."
2. Know Your Audience
 Technical Audience: Show the code, the confidence intervals, and the
edge cases.
 Business Audience: Show the Actionable Insight and the Revenue
Impact.
3. Visual Ethics
 Don't truncate the Y-axis: Starting a graph at 50 instead of 0 exaggerates
small differences. This is "lying with statistics."

Unit III: Fundamentals of Statistics (The Grammar

of Science)
Learning Objectives:
1. Define Statistics not just as math, but as the "Science of Uncertainty."
2. Distinguish between facts (Non-statistical) and probabilities (Statistical).
 3. Master the critical difference between the Whole (Population) and the
Part (Sample).
4. Understand the two main branches: Descriptive (Summary) and
Inferential (Prediction).

I. Introduction to Statistics
The Definition:
Statistics is the science of collecting, analyzing, interpreting, and presenting
data. It is the tool we use to convert raw data into knowledge.
The "Why":
Without statistics, we are just people with opinions. Statistics allows us to
measure uncertainty.
 Opinion: "I think this medicine works."
 Statistics: "There is a 95% probability that this medicine reduces
symptoms by 50%."

II. Statistical vs. Non-statistical Analysis

This is often a confusion point. Help students distinguish between "Counting"
and "Inferring."

Feature Non-Statistical Analysis Statistical Analysis

Nature Deterministic (Facts). Probabilistic (Estimates).

Question "How many students are "How many students will likely
in this room?" pass the exam?"

Result Exact (e.g., 45 students). Range/Probability (e.g., 80% ±

5%).
 Use Case Accounting, Inventory Forecasting, Medical trials,
check. Quality Control.

Faculty Note: "If you have all the data (e.g., a list of every transaction made
yesterday), you are doing Non-Statistical analysis. If you only have some data
and want to guess the rest, you are doing Statistics."

III. Major Categories of Statistics

Use the "Snapshot vs. Crystal Ball" analogy.

1. Descriptive Statistics (The Snapshot)

 Goal: Organize and summarize the data you actually have.
 Tools: Mean, Median, Mode, Charts, Graphs.
 Example: "The average height of students in this class is 5'9"."
  Limitation: It tells you nothing about people outside the room.
2. Inferential Statistics (The Crystal Ball)
 Goal: Use the data you have to make predictions about data you don't
have.
 Tools: Hypothesis Testing, Confidence Intervals, Regression.
 Example: "Based on this class, the average height of all students in the
university is likely between 5'8" and 5'10"."
 Key Concept: Generalization.

IV. Population and Sample (The Core Concept)

This is the single most important concept in the unit. If the sample is bad, the
statistics are a lie.
1. Definitions
 Population (N): The entire group you want to study. (e.g., All voters in
India).
o Constraint: Usually too expensive or impossible to measure everyone.
 Sample (n): A subset of the population that you actually measure. (e.g.,
1,000 voters surveyed).
2. The "Soup Analogy" (Classroom Classic)
 Scenario: You are making a huge pot of soup (Population).
 Goal: You want to know if it needs more salt.
 Method: You don't drink the whole pot. You stir it and take one spoonful
(Sample).
 The Golden Rule: If you don't stir the soup (Randomize), your spoonful
might just be the salty top layer, and your conclusion about the whole
pot will be wrong.

V. Statistical Analysis Considerations

Before running the numbers, students must check for "Analytical Sins."
1. Sampling Bias
 Definition: When your sample doesn't look like your population.
 Example: Surveying "Internet Usage" by only asking people on Twitter.
(You miss everyone who doesn't use the internet).
2. Confounding Variables
 Definition: An outside influence that changes the effect of a dependent
and independent variable.
 Example: "Coffee drinkers have higher rates of lung cancer."
 The Confounder: Smoking. Coffee drinkers were historically more likely to
smoke. Coffee didn't cause the cancer; smoking did.
3. Validity vs. Reliability
 Validity: Are you measuring the right thing? (Accuracy).
 Reliability: Do you get the same result every time? (Consistency).

VI. Statistical Analysis Process

This is the workflow students should follow for any project.
1. Identify the Question: "Does studying more hours increase exam
scores?"
2. Define Population & Sample: "Pop: All students. Sample: 50 randomly
selected students."
3. Collect Data: Surveys, Observation, Experiments.
4. Descriptive Analysis: Calculate the mean study time and mean score of
your sample. Check for outliers.
5. Inferential Analysis: Run a correlation test or regression.
6. Interpret Results: "There is a strong positive correlation (r=0.8)."
7. Make Decision: "We should recommend longer library hours."
Unit IV: Advanced Statistical Analysis (The
Decision Engine)
Learning Objectives:
1. Understand the "Shape" of data (Distribution) and why the Bell Curve
rules the world.
2. Master the tools of variability (Dispersion) to measure risk.
3. Demystify "Hypothesis Testing" using the Courtroom Analogy.
4. Learn to distinguish between "Related" (Correlation) and "Caused by"
(Causation).

I. Data Distribution (The "Shape" of Reality)

Data isn't just a list of numbers; it has a shape. The shape determines which
statistical tests you can use.
1. The Normal Distribution (Gaussian)
 Concept: The "Bell Curve." Nature loves this shape. Heights, IQ scores,
and errors often follow it.
 Key Property: It is symmetric. The Mean, Median, and Mode are all in
the center.
 The 68-95-99.7 Rule:
o 68% of data falls within 1 Standard Deviation ($\sigma$) of the mean.
o 95% falls within 2 $\sigma$.
o 99.7% falls within 3 $\sigma$.
2. Skewness (The Lean)
 Left Skewed (Negative): The tail drags to the left (e.g., Age at death—
most are old, a few are young).
  Right Skewed (Positive): The tail drags to the right (e.g., Income—most
earn average, a few billionaires drag the tail way out).
Faculty Note: "If your data is heavily skewed (like salaries), the Mean is a liar.
Use the Median instead."

II. Dispersion (Measuring Risk)

Dispersion tells you how "reliable" the average is.
 Low Dispersion: Data is tight. (e.g., A reliable machine making screws).
 High Dispersion: Data is spread out. (e.g., A volatile stock market).
 Variance ($s^2$): The average squared distance from the mean.
(Mathematically useful, but hard to interpret).
 Standard Deviation ($s$): The square root of variance. This brings the
number back to the original units (e.g., "dollars" or "years").

III. Histograms (Visualizing the Shape)

The Histogram is the X-Ray machine for Distribution.
 How it works: It "bins" continuous data into buckets (e.g., 0-10, 10-20,
20-30) and counts how many fall in each.
 Why use it? To instantly see if data is Normal, Skewed, or Bimodal (two
humps).
 Bimodal Data: If you see two humps, you likely have two different
populations mixed together (e.g., Men's and Women's heights mixed in
one graph).

IV. Inferential Statistics (The Big Picture)

 Definition: Using a small sample to make a "best guess" about a larger
population.
  The Bridge: It connects "What we see" (Sample) to "What is true"
(Population).
 Tools: Hypothesis Testing, Confidence Intervals.

V. Testing (Hypothesis Testing)

This is the hardest part for students. Use the Courtroom Analogy.
1. The Setup
 Null Hypothesis ($H_0$): The "Status Quo." (e.g., "The defendant is
Innocent" or "The drug has no effect").
 Alternative Hypothesis ($H_1$): The "Claim." (e.g., "The defendant is
Guilty" or "The drug cures the disease").
2. The Trial (The Test)
You collect evidence (Data). You assume $H_0$ is true unless the evidence is
overwhelming.
3. The Verdict (The P-Value)
 Definition: The probability of seeing this evidence if the defendant were
innocent.
 Low P-Value (< 0.05): "It is highly unlikely this happened by chance." ->
Reject Null Hypothesis (Guilty).
 High P-Value (> 0.05): "This could easily be a coincidence." -> Fail to
Reject Null (Not Guilty).
Faculty Mantra: "If P is low, the Null must go."

VI. Correlation (Relationships)

Correlation measures how two variables move together.
1. The Coefficient ($r$)
 Range: -1.0 to +1.0.
  +1.0: Perfect Positive (As X goes up, Y goes up).
 -1.0: Perfect Negative (As X goes up, Y goes down).
 0: No relationship (Random noise).
2. The Trap: Causation
 Fact: High ice cream sales correlate with high drowning rates ($r=0.8$).
 False Conclusion: Eating ice cream causes drowning.
 Truth: Summer heat causes both. (This is a "Confounding Variable").

Unit V: Numerical and Mathematical Computing

(The Engine Room)
Learning Objectives:
1. Understand why Python lists are too slow for science and why NumPy is
the solution.
2. Master the ndarray object: The fundamental building block of Data
Science.
3. Differentiate between a "View" (looking at data) and a "Copy"
(duplicating data).
 4. Introduce SciPy as the advanced mathematical toolkit built on top of
NumPy.

I. Introduction to NumPy (The Foundation)

The Problem: Standard Python lists are slow. They store pointers to objects,
not the data itself.
The Solution: NumPy (Numerical Python).
 What is it? A library for high-performance mathematical computing.
 Key Feature: Vectorization. It performs operations on entire arrays at
once, without loops.
 Speed: NumPy is often 50x faster than standard Python lists because it
uses C-language optimization under the hood.
Faculty Demo: Show a timer comparison.
 Python List: Loop through 1 million numbers and add 5. (Time: ~0.5s)
 NumPy Array: arr + 5. (Time: ~0.01s)

II. Creating, Printing, and Attributes of ndarray

The ndarray (N-dimensional array) is the core object.
1. Creating Arrays
 [Link]([1, 2, 3]): Converts a list to an array.
 [Link]((3,3)): Creates a 3x3 matrix of zeros (crucial for initializing
weights in AI).
 [Link](0, 10, 2): Like Python's range(), but returns an array (0, 2, 4, 6,
8).
2. Attributes (The "ID Card" of the Array)
Every array has properties you must check before analysis:
 .ndim: Number of dimensions (1 = Vector, 2 = Matrix, 3 = Tensor).
  .shape: The dimensions (e.g., (3, 4) means 3 rows, 4 columns).
 .dtype: The data type (e.g., int64, float32). Note: NumPy arrays must be
homogeneous (all elements must be the same type).

III. Basic Operations, Copies, and Views

This is the most common source of bugs for beginners.
1. Element-wise Operations
 Math works element-by-element automatically.
 a * b: Multiplies corresponding elements (NOT matrix multiplication).
 a + 10: Adds 10 to every element.
2. Copies vs. Views (The "Sticky Note" Analogy)
 View ([Link]()): Just a window looking at the original data.
o Danger: If you change the view, you change the original array!
o Analogy: Looking at a document through a different window.
 Copy ([Link]()): A brand new, independent duplicate.
o Safety: Changing the copy does not affect the original.
o Analogy: Photocopying a document and writing on the copy.
Faculty Warning: "If you slice an array (b = a[0:5]), NumPy creates a View by
default to save memory. If you modify b, you modify a. Always use .copy() if
you need a separate object."

IV. Mathematical Functions of NumPy

NumPy replaces the standard math library.
 Aggregations: [Link](), [Link](), [Link](), [Link](), [Link]().
 Broadcasting: The magic ability to do math on arrays of different shapes.
 o Example: Adding a 1D vector [1, 2, 3] to a 3x3 matrix. NumPy
automatically "stretches" the vector to match the matrix dimensions.

V. Introduction to SciPy and Sub-Packages

If NumPy is the Foundation (Brick & Mortar), SciPy is the Architecture
(Blueprints).
What is SciPy?
A collection of mathematical algorithms built on top of NumPy. It adds high-
level science commands.
Key Sub-Packages (The Toolkit):
 [Link]: Calculus (Integration/ODEs).
 [Link]: Finding minimums/maximums (Essential for Machine
Learning).
 [Link]: Linear Algebra (Solvers, Decompositions).
 [Link]: Advanced Statistics (Probability distributions).

VI. Integration, Optimization, and Eigenvalues (The Heavy Lifting)

1. Integration ([Link])
 Use Case: Finding the area under a curve.
 Function: quad(func, a, b) computes the definite integral of func from a
to b.
 Real World: calculating total distance traveled given a velocity curve.
2. Optimization ([Link])
 Use Case: Finding the "Best" solution.
 Function: minimize(func, x0) finds the input x that makes func as small
as possible.
  Real World: Minimizing cost in a supply chain, or minimizing error in an
AI model.
3. Eigenvalues & Eigenvectors ([Link])
 Definition:
o Eigenvector: A vector that doesn't change direction during a
transformation, only length.
o Eigenvalue: How much that vector stretches or shrinks.
 Code: vals, vecs = eig(matrix)
 Why do we care? It is the math behind Google's PageRank algorithm
and PCA (Principal Component Analysis) for face recognition.