Machine Learning in Data Analysis

Overview & Industry Impact

Lesson Objective

In this lesson, we explore Machine Learning (ML) and its role in modern data analysis
applications across industries. You will learn:

• What machine learning is

• How it differs from traditional programming

• How ML creates business value

• Why ML is transforming data-driven decision-making

What is Machine Learning?

Machine Learning (ML) is a subset of Artificial Intelligence (AI) that focuses on building
systems that learn from data instead of being explicitly programmed with fixed rules.

Core Idea

Traditional programming:
Rules + Data → Output

Machine learning:
Data + Algorithms → Model → Output (Predictions/Decisions)

ML systems discover patterns, relationships, and trends in data and use them to make
predictions or decisions.

Definition

Machine Learning involves the development of mathematical models that allow

computers to make decisions based on patterns learned from data, without being directly
programmed for every scenario.

Key Characteristics of ML

pg. 1
 • Learns from historical data

• Improves performance as more data becomes available

• Adapts to changing conditions

• Handles complex relationships beyond manual analysis

Real-World Scenario: Machine Learning in Retail

Let’s understand ML through a practical industry example.

Retail Company Use Case

A retail company uses ML to:

Area ML Contribution

Inventory Management Predicts future product demand

Customer Insights Identifies buying behavior patterns

Marketing Personalizes promotions

Strategy Supports pricing and expansion decisions

What the ML Model Does

The model analyzes:

• Historical sales data

• Customer preferences

• Seasonal trends

• Supply chain logistics

It then predicts:

• Future product demand

• Optimal inventory levels

• Customer purchasing behavior

pg. 2
 Business Impact

• Minimizes overstock and understock

• Improves customer satisfaction

• Enhances marketing effectiveness

• Drives profitability and operational efficiency

How Machine Learning Enhances Data Analysis

Machine learning transforms how organizations use data. Below are the major
contributions of ML in data analysis industries.

1. Handling Large Datasets

ML algorithms can process and analyze massive volumes of data that humans cannot
handle manually.

Result: Organizations can extract insights from big data efficiently.

2. Predictive Analytics

ML uses past data to forecast future events.

Industry Example

Finance Stock market trends

Retail Product demand

Healthcare Disease outbreak prediction

ML moves organizations from reactive to proactive decision-making.

3. Automation of Routine Analysis

ML automates repetitive analytical tasks.

Benefits:

pg. 3
 • Faster decision-making

• Reduced human workload

• Focus on high-level strategy

4. Continuous Improvement

ML models learn from new incoming data and refine themselves.

More data → Better model accuracy → Better decisions

5. Real-Time Analysis

ML systems can analyze live data streams and generate immediate insights.

Industry Application

Finance Fraud detection

Cybersecurity Threat detection

Retail Real-time recommendations

6. Discovering Hidden Patterns

ML can uncover complex relationships that traditional methods might miss.

Leads to innovation, new opportunities, and competitive advantage.

7. Personalization

ML analyzes customer data to deliver personalized services and recommendations.

Examples:

• Netflix recommendations

• Amazon product suggestions

• Personalized marketing emails

pg. 4
 Improves customer satisfaction and loyalty

8. Operational Optimization

ML helps improve workflows and processes by identifying inefficiencies.

Results:

• Cost reduction

• Higher productivity

• Improved competitiveness

9. Innovative Problem Solving

ML enables organizations to approach problems in new and creative ways.

Traditional analysis may miss complex nonlinear patterns that ML can detect.

WARNING — ML Is Not Magic

Machine learning is powerful, but:

• It depends heavily on data quality

• Poor data → Poor model → Poor decisions

• Business understanding is still essential

TIP — ML Is a Decision Support Tool

ML does not replace human expertise.

It augments decision-making by providing deeper insights and predictions.

PITFALL — Ignoring Business Context

Using ML without understanding the business goal leads to:

• Overly complex solutions

pg. 5
 • Irrelevant predictions

• Wasted time and resources

Always align ML with business objectives.

Summary — Why ML Matters in Data Analysis

Machine learning is transforming data-driven industries by:

✔ Extracting insights from massive datasets

✔ Predicting future trends and behaviors
✔ Automating analysis and improving efficiency
✔ Enabling real-time decision-making
✔ Personalizing customer experiences
✔ Optimizing operations
✔ Driving innovation and competitiveness

Key Takeaway

Machine Learning is not just a technology — it is a strategic tool that enables organizations
to make smarter, faster, and more informed decisions.

pg. 6
Music by Jacob Sanz-Robinson: [Link]
Tired of the soundtrack? Send me your (own) recordings and I'll play them!

Supervised Classification
February 2, 2022
Tristan Glatard

Gina Cody School of Engineering

and Computer Science
Department of Computer Science and Software Engineering
1
2
1. Introduction
● Data
○ Every instance in a dataset is represented by the same set of features.
○ Features, a.k.a. attributes, may be continuous, categorical or binary.

● Supervised learning
○ Some instances are given with known labels
○ Goal: predict unknown labels

3
Related but different approaches
● Regression
○ Predict values, not labels

● Unsupervised classification
○ No labels are given
○ Classes have to be discovered

● Semi-supervised classification
○ Uses both labelled (less) and unlabelled (more) data

● Reinforcement learning
○ System receives hints (reward) on how it performs
○ Adjusts accordingly 4
2. General framework

● Data collection
○ Identify features
○ Use expert, common-sense or brute-force

● Irrelevant features often include

○ Noise
○ Missing values

● “Nothing good will come if the data is bad”

5
Data pre-processing

● Handle missing features

○ Ignore and discard
○ Replace with outlier value
○ Estimate (mean, interpolation)

● Instance selection
○ Remove noise
○ Sub-sample very large datasets
○ Without compromising performance

6
Data pre-processing (2)

● Feature selection (for all instances)

○ Remove irrelevant and redundant features
=> Classify faster and more effectively
○ Combine / transform features
=> More accurate classification, improved interpretation

7
Data pre-processing: time series

● Define time windows

○ Size is a parameter
○ May be overlapping or not

● Compute features over time windows

○ Statistics: mean, stdev, etc
○ Aggregates over previous time windows

● Classify time windows (from B. Dehghani)

8
Definition of training set

● Labelled dataset split in training vs test set

○ To estimate performance on unseen data
○ Sometimes a validation set is also used (why?)

● Split data randomly, by two thirds

○ Limitations?

● k-fold cross validation

○ Split data set in k equal-sized subsets
○ Pick k-1 sets to train, test on remaining 1
○ Repeat k times, measure average performance 9
Definition of training set (2)

● Leave-one-out validation
○ All tests subsets have only 1 instance
○ Repeat n times, n is the number of instances

10
Definition of the training set: mind the correlation!

● Cross-validation assumes that instances are i.i.d

○ Independent and Identically Distributed
○ This is in general not the case (why?)

● The i.i.d assumption should be evaluated for the target application

○ Does it make sense to have instances from the same <pick one depending
on context> in both the training and the test set?
○ Risk: performance is overestimated

11
Example: Human Activity Recognition (P.I.: Dr Shihab)

● Data
○ 3D acceleration
○ Multiple sensors on the body

● Goal: detect what people do

○ Walking, running, jogging
○ Biceps curl, jumping jacks, etc

● Use case
○ Build model from experiments
○ Embed model in device
12
Example: dataset with 17 subjects (B. Dehghani)

13
k-fold CV vs Subject CV (B. Dehghani)

K-fold CV Subject CV
14
Data set imbalance

● In an imbalanced dataset:
○ Not all the classes have
comparable numbers of
instances. Examples?
○ Classifier is biased toward the
majority class.

● Example: failure detection in

supply chains (M. Sharma, 2018)

15
Data set imbalance: solutions

● Undersample the majority class

○ Randomly and uniformly
○ According to the distribution of the data (e.g., NearMiss)

● Oversample the minority class

○ Simulate new instances (e.g., SMOTE)
○ Repeat existing samples, when classifier allows (e.g., RF, see Chen et al, 2004)
○ Don't put the same data points in the training and test sets!
● Many classifiers have a "class weight" parameter that allows to adjust
class weights based on data imbalance
16
The dangers of simulating new instances

● Simulated distribution should be

○ Realistic
○ Identical to the original distribution
○ Aware of continuous and categorical
features.

● Examples: locations of failed service

in supply chains (M. Sharma, 2018)

17
Evaluation with test set

18
 People who actually have the condition

Evaluation metrics

Evaluations metrics rely on

definitions of True Positives, True
Negatives, False Positives, False
Negatives.
Example: detect a medical
condition.

[Link]
People who are detected as having the condition 19
Accuracy

● Fraction of correctly classified instances

○ ACC = (TP + TN) / (TP + TN + FP + FN)

● Accuracy doesn’t reflect classifier performance properly

○ Why?

20
Precision and recall

We need two metrics to properly

evaluate a classifier.
Precision and recall are one
possibility, sensitivity and
specificity are another one.

21
F1-score

● Having two metrics isn’t convenient to compare classifiers

● The F1-score is a harmonic mean between precision and recall:
○ F1 = 2 (precision x recall) / (precision + recall)

22
Classification algorithms

23
3. Decision trees
● Decision trees classify instances by
sorting them based on feature values
● Each node is a feature, each branch is a
value range, each leaf is a label.

=>
24
Decision trees: classification

Instance〈at1 = a1, at2 = b2, at3 = a3, at4 = b4〉

isn’t in the training set. It is classified as “Yes”.

=>
25
Building the tree

● Main idea
○ Split dataset recursively so that classes are well separated

● Quality of 1 split can be measured by:

○ Gini index: 1 – Σi(pi)2
○ Entropy (information gain): -Σipilog(pi)
○ Both metrics are relevant and can be tried

● For n splits: take the average weighted by number of instances in split

26
Illustration: 2 classes, p, (1-p)

We want to split the dataset such that

resulting nodes have (pick one):

❐ High Entropy or Gini coefficient

❐ Low Entropy or Gini coefficient

27
Building the tree (2)

● Constructing optimal binary trees is an NP-complete problem.

● Pseudo-code of a common heuristics (binary tree):
○ For each attribute a
■ Sort instances according to a
■ Visit each instance in sorted order
■ Compute the impurity measure (Gini or Entropy) assuming the split is after the instance
■ Remember the point at which the minimum is reached
○ Split data using the attribute and value where minimum is reached

28
Example (J. Ullman)Jeffrey D. Ullman
● 1 attribute only
● M=Gini
● Class labels: +/-1

29
Example: complete design (J. Ullman)

30
Overfitting
● A classifier c is said to overfit training data
if c’ exists that:
○ Has a lower performance than c when
tested on training data
○ Has a higher performance than c when
tested on the entire dataset

● The classifier is:

○ “Too specific to the training set”.
○ “Too sensitive to noise”.
○ “Too focused on details”. [Link] 31
How to avoid overfitting?

● Stop the training algorithm when:

○ Max depth of the tree is reached
○ Information gain (difference in Entropy or Gini) is below threshold
○ Min number of instances per node is reached

● Use more than 1 tree (Ensemble methods)

○ Grow more than 1 tree, using randomness
○ Keep trees that perform well
○ Example: Random Forest

32
Computing time

● Computing time can be important with thousands of instances.

● Most time-consuming aspect: sorting features.
● Improvements
○ Convert features to discrete intervals
○ Determine thresholds from subsamples
○ Parallelize! e.g., use Spark.

33
Data-based parallelization

worker
worker
workers

workers
workers
workers

Map-reduce implementation?
34
Decision trees are usually univariate

● Split datasets based on a single feature at each node

● Partitioned regions are hyper-rectangles. Diagram?
● Methods exist to build multi-variate trees.

35
Conclusion: Decision Trees

● Most well-known algorithm: C4.5 and variants.

● Parallelize well
● Now mostly used in Random Forests (even more parallel)
● Among the top classifiers
● Available in Spark and scikit-learn

36
37
RF Overview

● Random forests are a combination of tree predictors

● Each tree depends on the values of a random vector sampled
independently and with the same distribution for all trees
○ Independence is good for parallelization

● Randomness governs
○ The sub-sample from which trees are grown
○ The set of features considered at each node

38
RF overview (2)

● For classification, trees vote for class labels

● RF converge, “overfitting is not a problem”
● RF provide a measure of feature importance
● In practice, one of the most accurate methods
● Available in Spark and scikit-learn.

39
k-Nearest Neighbour (kNN) (!= kmeans !!!)
● Main idea:
○ Close instances have similar labels.
○ An unclassified instance is labelled from the labels of its k nearest neighbors

● Pseudo-code:

Training step?

40
 Distances

● Instances are represented by

n-dimensional vectors.
● Several possible choices (see Table →)
● Caveats:
○ “Curse of dimensionality”
○ Attributes may be on different scales
○ Attributes may have different units
○ Mix of categorical and numerical attributes
41
Limitations of kNN

● Large storage requirements

● Sensitive to choice of distance
● Lack a way to choose k, except through cross-validation (expensive)
● Sensitive to noise

42
Computational time of k-NN

● Related to number of instances and number of features

● Solutions to reduce computational time
○ Sub-sampling
○ Instance filtering
■ 80% of cases are removed without substantial drop in accuracy (see references in paper)
○ Feature engineering
○ Parallelization! Map-reduce implementation?

43
In practice: kNNs are a very serious contender!

(B. Dehghani, 2018)

44
“The key question when dealing with ML classification is not whether a learning algorithm is superior to others, but under
which conditions a particular method can significantly outperform others on a given application problem.”

45
Conclusion

● Data preparation is KEY

● Experiment, experiment, experiment. With scikit-learn or Spark.
● Traditional classifiers: kNN, decision trees, SVMs.
● Important but not discussed here: (Deep) Neural Networks.

46
Types of Machine Learning Models in Data Analysis
Lesson Objective

In this lesson, we explore the major types of machine learning (ML) models used in data
analysis, how they work, and where they are applied. The focus is on understanding:

• Supervised learning

• Classification vs Regression

• Key algorithms

• Introduction to Unsupervised learning

Supervised Machine Learning

Definition

Supervised learning involves training a model using labeled data, where each input
example has a known output (target).

Goal: Learn the relationship between inputs and outputs so the model can predict
outcomes for new, unseen data.

Example

Predict sales based on advertising spending.

• Input → Advertising budget

• Output (Label) → Sales

Because the target (sales) is known during training, this is supervised learning.

Two Main Types of Supervised Learning

Type Output Type Example

Classification Categorical Spam vs Not Spam

Regression Continuous numeric Predict house price

pg. 1
 Classification Models

Used when the target variable represents categories or classes.

Examples

• Email spam detection (Spam / Not Spam)

• Customer churn prediction (Yes / No)

• Disease diagnosis (Positive / Negative)

Logistic Regression

A statistical classification method for binary outcomes.

• Output values range between 0 and 1

• Uses a logistic (S-shaped) function

• Interpreted as probability of belonging to a class

Scenario Example

Predict whether a patient will have a heart attack based on:

• Drug dosage

• Age

• Cholesterol

• Blood pressure

Output = 1 (Yes) or 0 (No)

Decision Trees (Classification)

A tree-like structure that makes decisions using a series of if-then splits.

How it works:

1. Ask a question about a feature

pg. 2
 2. Split data based on the answer

3. Repeat until a final category is assigned

Analogy

Choosing clothes:

• Is it raining?

o Yes → Raincoat

o No → Next question

• Is it cold? → Coat or T-shirt

Random Forest (Classification)

An ensemble method that combines many decision trees.

How it works:

• Each tree is trained on a random subset of data

• Each tree makes its own prediction

• Final output = majority vote

Analogy

Like asking multiple friends for advice and going with the most common answer.

PITFALL — Overfitting in Trees

Single decision trees may memorize training data.

Random Forest reduces this risk by combining many trees.

Regression Models

Used when predicting continuous numeric values.

Examples

• Predicting house prices

pg. 3
 • Forecasting sales revenue

• Estimating patient blood pressure

Linear Regression (ML Perspective)

Predicts a numeric value using a linear relationship between inputs and target.

Previously studied in statistics, now applied within ML workflows.

Decision Tree (Regression)

Instead of predicting categories, the tree predicts numeric values at its leaves.

Random Forest (Regression)

An ensemble of regression trees that:

• Improves prediction accuracy

• Reduces overfitting

TIP — When to Use Regression

Use regression when your target is:

✔ Measurable
✔ Numeric
✔ Continuous

Unsupervised Machine Learning

Unlike supervised learning, unsupervised learning uses unlabeled data.

Definition

The system tries to discover patterns and structures without knowing the correct outputs.

pg. 4
 Example

Customer segmentation:

• You have customer data

• No labels like “premium” or “regular”

• Model groups customers based on similarities

Key Idea

Instead of predicting outcomes, unsupervised learning focuses on:

• Pattern discovery

• Grouping similar data

• Understanding structure

WARNING — No Labels

Since there is no “correct answer,” evaluation is more subjective and requires domain
knowledge.

Summary

Learning Type Data Labels Goal Example

Supervised Yes Predict outputs Spam detection

Unsupervised No Find patterns Customer segmentation

Key Takeaways

✔ Supervised learning uses labeled data to predict outcomes

✔ Classification predicts categories
✔ Regression predicts numeric values
✔ Decision Trees split data using logical rules

pg. 5
✔ Random Forest improves performance using multiple trees
✔ Unsupervised learning finds hidden patterns without labels

pg. 6
 Step-by-Step Development of a ML Model
Lesson Objective

In this lesson, we study the complete lifecycle of developing a machine learning (ML)
model, from problem definition to deployment and monitoring. This structured process
ensures that ML solutions are reliable, accurate, and aligned with real-world needs.

Overview of the ML Development Workflow

Machine learning model development follows a systematic pipeline:

1. Define the problem

2. Collect & prepare data

3. Choose a model

4. Train the model

5. Evaluate performance

6. Deploy the model

7. Monitor & update

Each step directly impacts the final performance and usefulness of the model.

pg. 1
 Define the Core Problem

Before any coding begins, clearly identify:

• Goal – What decision or prediction is needed?

• Inputs (Features) – Variables used to make predictions

• Output (Target) – What you want to predict

Example

Predict whether a customer will churn

• Inputs → Usage frequency, subscription length, support calls

• Output → Churn (Yes/No)

NOTE: A poorly defined problem leads to wasted modeling effort.

Data Collection & Preparation

pg. 2
 Data Collection

Gather relevant data from:

• Databases

• Surveys

• Logs

• Sensors

• APIs

Data Cleaning

Ensure data quality by:

• Handling missing values

• Removing duplicates

• Correcting errors

• Detecting outliers

Feature Engineering

Create meaningful features from raw data to improve model performance.

TIP: High-quality data often improves model accuracy more than complex algorithms.

WARNING: Poor data quality → Poor model performance (Garbage In, Garbage Out).

Model Selection

Choose an algorithm based on the problem type:

Problem Type Example Algorithms

Classification Logistic Regression, Decision Trees, Random Forest

Regression Linear Regression, Decision Trees, Random Forest

Clustering K-Means, Hierarchical Clustering

pg. 3
Selection depends on:

• Dataset size

• Data quality

• Feature types

• Model complexity

Train the Model

Split dataset into:

Dataset Purpose

Training Set Used to train the model

Testing Set Used to evaluate performance

Example Split

Total observations = 1000

• Training = 700 (70%)

• Testing = 300 (30%)

Model learns from training data and predicts outcomes for the test set.

PITFALL: Never train and test on the same data — leads to overfitting.

Evaluate the Model

Classification Metric — Accuracy

Definition: Proportion of correct predictions.

𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =
𝑇𝑜𝑡𝑎𝑙 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠
Example:
70 correct out of 100 → 70% accuracy

NOTE: Accuracy works best for balanced datasets.

pg. 4
 Regression Metric — Mean Squared Error (MSE)

Measures average squared difference between predicted and actual values.

1
𝑀𝑆𝐸 = ∑(𝑦𝑖 − 𝑦̂𝑖 )2
𝑛

Where:

• 𝑦𝑖 = Actual value

• 𝑦̂𝑖 = Predicted value

• 𝑦ᵢ − 𝑦̂𝑖 = Prediction error

Lower MSE = Better model

Regression Metric — Mean Absolute Percentage Error (MAPE)

Measures prediction error as a percentage.

100 𝑦𝑖 − 𝑦̂𝑖
𝑀𝐴𝑃𝐸 = ∑∣ ∣
𝑛 𝑦𝑖
Lower MAPE = Better performance

TIP: Use MAPE when stakeholders prefer percentage-based errors.

Model Deployment

After achieving satisfactory performance:

• Integrate model into real systems

• Use it for real-time or batch predictions

• Connect it with applications, dashboards, or APIs

WARNING: A model that works in testing may fail in real-world conditions if deployment is
poorly planned.

pg. 5
 Monitoring & Updating

Machine learning models degrade over time due to data drift.

Continuous Monitoring Includes:

• Tracking prediction accuracy

• Detecting performance drops

• Updating with new data

TIP: Schedule periodic retraining to maintain performance.

NOTE: Deployment is not the end — it is the beginning of the maintenance cycle.

Summary

Step Purpose

Problem Definition Align ML with business goal

Data Preparation Ensure clean, useful data

Model Selection Choose appropriate algorithm

Training Learn patterns from data

Evaluation Measure performance

Deployment Use model in real world

Monitoring Maintain long-term accuracy

Key Takeaways

✔ Machine learning development is a structured process, not just model training

✔ Data quality strongly influences performance
✔ Evaluation metrics depend on the problem type
✔ Deployment makes the model useful
✔ Monitoring ensures long-term success

pg. 6
 Supervised ML Classification Models
Goal

In this lesson, we explore supervised classification models, which are used when the
target variable represents categories or labels rather than numeric values.

Examples of classification tasks:

• Spam vs Not Spam

• Fraud vs Legitimate

• Disease vs Healthy

• Buy vs Not Buy

We will cover the following models:

1. Logistic Regression

2. K-Nearest Neighbors (KNN)

3. Support Vector Machine (SVM)

4. Random Forest Classifier

5. Naive Bayes Classifier

What is Supervised Classification?

Supervised classification uses labeled training data where the output belongs to a finite
set of categories.

Output = Class label (0/1, Yes/No, A/B/C, etc.)

Logistic Regression

Theory

Logistic Regression is a method used to decide between two

categories.

It answers questions like:

pg. 1
 • Is this email spam or not spam?

• Will a customer buy or not buy?

• Is a tumor benign or malignant?

Instead of giving a direct Yes/No answer, it first gives a probability.

Logistic Regression calculates a number, then passes it through a special curve called the
Sigmoid Function:
1
𝜎(𝑧) =
1 + 𝑒 −𝑧

This function converts any real number into a value between 0 and 1. That value is
interpreted as a probability.

Example:

• Output = 0.85 → 85% chance it belongs to Class 1

• Output = 0.20 → 20% chance it belongs to Class 1

How the Final Decision Is Made

We choose a threshold (usually 0.5):

• Probability ≥ 0.5 → Predict Class 1

• Probability < 0.5 → Predict Class 0

So, the model says:

“If I’m more than 50% confident, I’ll choose Class 1.”

Example: Spam detection

You train the model using emails labeled:

• 1 = Spam

• 0 = Not Spam

For a new email:

• Model predicts 0.72

pg. 2
 • Since 0.72 ≥ 0.5 → It is classified as Spam

Step-by-Step

Collect labeled data (examples with correct answers)

Train the logistic regression model
Find the best weights (w₁, w₂, …) and bias (b) for this formula: 𝑧 = 𝑤1 𝑥1 + 𝑤2 𝑥2 +. . . +𝑏
Model outputs probabilities for new inputs
Apply threshold (like 0.5) to decide the class
Evaluate performance using:

o Accuracy (overall correctness)

o Precision (how many predicted positives were correct)
o Recall (how many real positives were found)
Strengths

✔ Simple and fast

✔ Probabilistic output
✔ Easy to interpret

Weaknesses

✖ Struggles with complex class boundaries

Tip: Great baseline model for binary classification.

Python Code Snippet

K-Nearest Neighbors (KNN)

Theory

K-Nearest Neighbors (KNN) is a method that makes predictions by looking at similar past
examples.

pg. 3
Instead of learning rules or building equations, KNN simply:

“Looks around” in the data and asks:

What do the closest examples look like?

How KNN Is Different

Most machine learning models go through a training phase where they learn patterns.

KNN is different:

• It does not build a model

• It stores all the training data

• It makes decisions only when a new example appears

So KNN is often called a lazy learning algorithm.

How Does It Decide What Is “Close”?

KNN measures distance between data points.

The most common method is Euclidean distance (like measuring straight-line distance
between two points).

If two customers have:

• Similar age

• Similar income

• Similar purchase history

Then their data points are close in distance.

Example

Imagine an online store. If a new customer buys:

• A gaming mouse

• A mechanical keyboard

KNN finds other customers with similar purchase behavior and recommends:

pg. 4
 “People like you also bought a gaming headset.”

Step-by-Step: How KNN Makes a Prediction

Choose a value for K (number of neighbors to look at)

Measure the distance from the new point to all training data
Find the K closest points
For classification → Take a majority vote among those neighbors

Example with K = 5:

• 3 neighbors are Class A

• 2 neighbors are Class B

➡ Final prediction = Class A

Strengths

✔ Simple and intuitive

✔ No training phase required

Weaknesses

✖ Slow for large datasets

✖ Sensitive to feature scaling

Pitfall: Choosing K

• K too small → Model is too sensitive to noise (overfitting)

• K too large → Model becomes too general (underfitting)

Choosing K carefully is very important.

pg. 5
 Python Code Snippet

Support Vector Machine (SVM) Classifier

Theory

Support Vector Machine (SVM) is a method that separates two groups of data with the
widest possible gap between them.

Imagine drawing a line between two groups of points.

SVM doesn’t just want any line — it wants the line that leaves the largest space (margin)
between the two groups.

Bigger margin = more confident separation.

What Is a “Margin”?

The margin is the distance between the separating boundary and the closest points from
each class.

SVM chooses the boundary that maximizes this distance.

What Are Support Vectors?

Only a few points really matter — the ones closest to the boundary.
These points are called support vectors.

They “support” or define where the boundary should be.

All the other points could move slightly, and the boundary would stay the same.

pg. 6
 Example: Handwritten Digits

Suppose we want to tell apart the digits 3 and 8 in handwritten images.

Each image becomes a data point with many features (pixel values).

SVM finds a boundary that best separates:

• Images of “3”

• Images of “8”

The most confusing examples (digits that look like both) become the support vectors.

Step-by-Step Example

Step 1 — Plot the data

Imagine your data points placed on a graph.

Each point belongs to one of two
categories.

Example:

• Red dots = Class A

• Blue dots = Class B

pg. 7
 Step 2 — Many lines could separate them

If the classes are separable, there are many possible lines (in 2D) or planes (in higher
dimensions) that could divide them.

SVM does not pick just any line.

Step 3 — Choose the widest margin

SVM finds the boundary that leaves the largest gap between the two classes.

That gap is called the margin.

Why?
A larger margin → better generalization → the model is less sensitive to noise.

Step 4 — The key points: Support Vectors

Only a few data points actually determine the boundary.

These are the points closest to the boundary.

They are called support vectors.

If you move other points, the boundary usually doesn’t change — but if you move a support
vector, the boundary shifts.

Step 5 — What if the data isn’t linearly separable?

SVM uses a trick called the kernel trick.

It mathematically transforms the data into a higher-dimensional space where a straight

boundary can separate them.

Common kernels:

• Linear

• Polynomial

• RBF (Gaussian)

SVM Strengths

pg. 8
✔ Strong performance in high-dimensional data
✔ Effective for complex boundaries

SVM Weaknesses

✖ Requires careful parameter tuning

✖ Slower with very large datasets

Tip: Works very well for image and text classification.

Random Forest Classifier

Theory

A Random Forest is like asking many decision trees to make a prediction, then choosing
the answer that most of them agree on.

Instead of trusting just one tree, we trust the group decision.

That’s why it’s called a forest — lots of trees working together.

How Does It Work?

Each tree is built a little differently to make sure they don’t all think the same way.

It uses two main ideas:

Random Data (Bootstrapping)

Each tree is trained on a random sample of the data.

Some records may appear more than once, others not at all.

This makes every tree see a slightly different version of the dataset.

Random Features

When a tree decides how to split the data, it doesn’t look at all features — only a random
subset.

This prevents one very strong feature from controlling all trees.

🗳 Final Prediction

pg. 9
Each tree gives its own class prediction.

The forest then takes a majority vote.

• If most trees say “Yes” → Final answer is “Yes”

• If most trees say “No” → Final answer is “No”

Example: Loan Default Prediction

A bank wants to predict if a person will default on a loan.

Features might include:

• Income

• Credit score

• Job status

• Existing debts

Each decision tree makes a guess.

The Random Forest combines all guesses and picks the most common result.

Step-by-Step

1. Set number of trees

2. Train multiple decision trees

3. Collect predictions from each tree

4. Use majority vote for final class

Strengths

pg. 10
✔ Handles messy and nonlinear data
✔ Reduces overfitting
✔ High accuracy

Weaknesses

✖ Slower on very large datasets

✖ Less interpretable

Note: Random Forest = “Wisdom of Crowds” in ML.

Many simple trees together make a smarter and more reliable decision than one alone.

Naive Bayes Classifier

Theory

Naive Bayes is a machine learning method that figures out the probability that something
belongs to a certain category.

It answers questions like:

“Based on these words, how likely is this email to be spam?”

It uses a math rule called Bayes’ Theorem, but the idea is simple:
Look at the evidence (features) and calculate which class is most likely.

Why Is It Called “Naive”?

Because it makes a very simple (and unrealistic) assumption:

It assumes all features are independent of each other once we know the class.

Example:
When classifying an email, it treats the words “free” and “money” as if they are unrelated —
even though in real life they often appear together.

This assumption is “naive,” but surprisingly, the method still works very well in many
situations.

How It Works (Simple Steps)

pg. 11
 Look at the features of the input (like words in a message)
Calculate how likely those features are for each class
Choose the class with the highest probability

Example: News Article Classification

Suppose we want to label news articles as:

• Sports
• Technology
• Politics

If an article contains words like:

“goal”, “team”, “match”

Naive Bayes calculates that these words are more common in Sports articles, so it
predicts Sports.

Strengths

pg. 12
✔ Extremely fast
✔ Works well with text data
✔ Efficient with high-dimensional features

Weaknesses

✖ Independence assumption may limit accuracy

Warning: Works best when features are not strongly correlated.

Quick Model Comparison

Model Best For Strength Weakness

Logistic Simple binary tasks Fast & Poor for complex

Regression interpretable boundaries

KNN Similarity-based Intuitive Slow at prediction time

problems

SVM High-dimensional Strong margins Needs tuning

complex data

Random Forest Messy real-world data Robust & Less interpretable

accurate

Naive Bayes Text classification Very fast Assumes

independence

Key Learning Points

✔ Classification predicts categories, not numbers

✔ Logistic Regression outputs probabilities
✔ KNN bases decisions on neighbor similarity
✔ SVM finds the widest separation margin
✔ Random Forest combines many decision trees
✔ Naive Bayes uses probabilistic reasoning

pg. 13
 EXAMPLE
Training Logistic Regression with a Small Dataset

Problem: Predict Pass or Fail

We want to predict:

• 1 = Pass

• 0 = Fail

Based on:

• Hours studied

Small Dataset

Hours Studied (x) Result (y)

1 0

2 0

3 0

4 1

5 1

We can see:

• Low hours → Fail

• High hours → Pass

Now let’s train Logistic Regression.

Step 1 — Model Formula

Logistic regression does:

𝑧 = 𝑤𝑥 + 𝑏

pg. 1
Then applies sigmoid:
1
𝜎(𝑧) =
1 + 𝑒 −𝑧

This gives a probability between 0 and 1.

Step 2 — Start with Random Weights

Let’s start simple:

• w=0

• b=0

Step 3 — Make First Predictions

For x = 1:

𝑧 = (0)(1) + 0 = 0
𝜎(0) = 0.5

Prediction = 0.5

But true label = 0

→ That’s not good.

For x = 5:

𝑧=0
𝜎(0) = 0.5

Prediction = 0.5

True label = 1
→ Also wrong.

pg. 2
 Problem:
Model predicts 0.5 for everything because weights are zero.

Step 4 — Adjust Weights

The algorithm now:

• Calculates error

• Slightly increases weight w

Suppose after one update:

• w=1

• b = -3

Now model becomes:

𝑧 = 1𝑥 − 3

Step 5 — New Predictions

For x = 1

𝑧 = 1(1) − 3 = −2
𝜎(−2) ≈ 0.12

Prediction ≈ 0.12
True label = 0
Good

For x = 2

𝑧 = 2 − 3 = −1
𝜎(−1) ≈ 0.27

True label = 0
Good

pg. 3
For x = 3

𝑧 =3−3=0
𝜎(0) = 0.5

Borderline case

For x = 4

𝑧 =4−3=1
𝜎(1) ≈ 0.73

True label = 1
Good

For x = 5

𝑧 =5−3=2
𝜎(2) ≈ 0.88

True label = 1
Good

What Just Happened?

The model learned:

• If hours < 3 → low probability

• If hours > 3 → high probability

The decision boundary is:

𝑤𝑥 + 𝑏 = 0
1𝑥 − 3 = 0
𝑥=3

pg. 4
So:

• Below 3 → Fail

• Above 3 → Pass

What Training Really Did

Training found:

• w=1

• b = -3

These values minimize prediction error.

The algorithm found:

“Around 3 hours is the turning point.”

In Real Life

Instead of one update:

• It adjusts weights thousands of times.

• Uses gradient descent.

• Minimizes log-loss.

But the idea is exactly what we did manually.

Intuition Summary

Training logistic regression means:

1. Start with random weights

2. Make predictions

3. Compare with true labels

4. Adjust weights

pg. 5
 5. Repeat until predictions are good

pg. 6
 KNN
Example

Problem: Predict Pass or Fail

We want to predict:

• 1 = Pass

• 0 = Fail

Based on:

• Hours studied

Small Dataset

Student Hours Studied Result

A 1 0 (Fail)

B 2 0 (Fail)

C 3 0 (Fail)

D 6 1 (Pass)

E 7 1 (Pass)

Now we have a new student:

• Hours studied = 5

• We want to predict Pass or Fail.

Step 1 — Choose K

Let’s choose:

K=3
(We will look at the 3 closest students.)

pg. 1
 Step 2 — Calculate Distance

We measure distance using:

𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 = |𝑁𝑒𝑤 − 𝐸𝑥𝑖𝑠𝑡𝑖𝑛𝑔|

New student studied 5 hours.

Student Hours Distance from 5

A 1 4

B 2 3

C 3 2

D 6 1

E 7 2

Step 3 — Find the 3 Closest

Smallest distances:

1 → Student D (Pass)
2 → Student C (Fail)
2 → Student E (Pass)

So, the 3 nearest neighbors are:

• D → Pass

• C → Fail

• E → Pass

🗳 Step 4 — Majority Vote

Among the 3 neighbors:

• Pass = 2

• Fail = 1

Final prediction:

pg. 2
 Pass

Why This Makes Sense

The new student (5 hours) is closer to:

• 6 hours (Pass)

• 7 hours (Pass)

So KNN says:

“Most similar students passed, so this student will likely pass.”

Why KNN is Called Lazy

Notice:

• We did NOT build a formula.

• We did NOT train a model.

• We simply stored the data.

• When a new example came, we compared distances.

That’s why it’s called lazy learning.

A Small 2D Example
(Slightly More Realistic)
Now let’s use two features:

• Hours studied

• Attendance %

Student Hours Attendance Result

A 2 50% Fail

B 3 60% Fail

pg. 3
 C 6 85% Pass

D 7 90% Pass

New student:

• 5 hours

• 80% attendance

Now we calculate Euclidean distance:

Distance formula:

√(𝑥1 − 𝑥2 )2 + (𝑦1 − 𝑦2 )2

The closest students will be C and D → both Pass.

So prediction = Pass.

Important Notes

TIP

Small K → model is sensitive to noise

Large K → model becomes too general

PITFALL

If features are on different scales (e.g., income vs age),

you MUST normalize them.

Otherwise, large numbers dominate distance.

Intuition Summary

KNN works like this:

“Show me who is most similar.

I will predict based on what they did.”

pg. 4
 Example
SVM (Support Vector Machine)

Problem: Classify Students as Pass or Fail

We want to separate students into:

• Fail (Class 0)

• Pass (Class 1)

Based on two features:

• Hours studied

• Practice problems solved

Small Dataset

Student Hours Problems Solved Result

A 1 1 Fail

B 2 1 Fail

C 2 2 Fail

D 6 5 Pass

E 7 6 Pass

F 6 6 Pass

If we plot this:

• Fail students are clustered near the bottom-left.

• Pass students are clustered near the top-right.

pg. 1
 Step 1 — Draw Possible Separating Lines

There are MANY lines that could separate Fail from Pass.

For example:

• A diagonal line

• A steeper line

• A flatter line

All of them might separate the groups.

But SVM does NOT choose just any line.

Step 2 — What Is the Margin?

The margin is the distance between:

pg. 2
 • The separating line

• The closest points from each class

SVM wants the largest possible gap between the two groups.

Bigger gap = safer separation.

Think of it like building a fence between two houses.

You don’t build it very close to one house.

You place it in the middle with the most space.

Step 3 — Which Points Matter?

Look at the points closest to the boundary.

Maybe:

• Student C (Fail)

• Student D (Pass)

These are the closest points to the separating line.

These are called:

Support Vectors

They determine where the line must be.

If you move other points slightly, the line may not change.
But if you move these support vectors, the boundary shifts.

Why Only These Points Matter?

Because SVM focuses only on:

The hardest-to-separate points.

Points far away don’t affect the boundary.

Step 4 — New Student Prediction

pg. 3
Now imagine a new student:

• 5 hours studied

• 5 problems solved

We check which side of the boundary this point falls on.

It falls on the Pass side.

So prediction = Pass.

What If Data Cannot Be Separated by a Straight Line?

Let’s imagine a different dataset:

• Fail students in the center

• Pass students around them (in a circle)

Now no straight line can separate them.

SVM uses something called:

Kernel Trick

It transforms the data into a higher dimension where separation becomes possible.

Think of it like lifting the data into 3D space so you can draw a flat plane.

Common kernels:

• Linear (straight line)

• Polynomial (curved boundary)

• RBF (flexible curved boundary)

Tiny Numeric Example

Imagine a simple 1D dataset:

Score Class

2 Fail

pg. 4
 3 Fail

8 Pass

9 Pass

Many boundaries are possible:

• At 4

• At 5

• At 6

SVM will choose the boundary exactly in the middle between:

• Closest Fail (3)

• Closest Pass (8)

So boundary ≈ 5.5

Margin = distance to 3 and 8.

Those two points (3 and 8) are the support vectors.

Why SVM Is Powerful

✔ It focuses on the most important points

✔ It creates confident separation
✔ It works well for high-dimensional data
✔ It works very well in text and image classification

Weaknesses

✖ Requires parameter tuning (C, kernel, gamma)

✖ Slower on very large datasets

Final Intuition

SVM asks: “What is the safest possible boundary that separates these two groups?”

pg. 5
 Random Forest
Example

Goal: Predict “Loan Default” (Yes/No)

We want to predict whether a customer will default on a loan.

Features (inputs)

• Income: High / Low

• Credit Score: Good / Poor

• Debt: High / Low

Target (output)

• Default: Yes / No

Small Training Dataset (6 people)

Person Income Credit Debt Default

P1 Low Poor High Yes

P2 Low Poor Low Yes

P3 Low Good High Yes

P4 High Good Low No

P5 High Good High No

P6 High Poor Low No

Random Forest Idea (3 Trees)

A Random Forest trains many decision trees, and each tree is slightly different because:

1. It sees a random sample of the data (bootstrapping)

2. At each split, it considers only some random features

pg. 1
Let’s pretend we trained 3 trees.

New Customer to Predict

New person (P7):

• Income = Low

• Credit = Good

• Debt = Low

We ask each tree to predict: Default?

Tree 1 (trained with a random sample)

Rule it learned (example):

• If Income = Low → Default = Yes

• Else → Default = No

So Tree 1 says: Yes

Tree 2 (different sample + different feature choices)

Rule it learned (example):

• If Credit = Poor → Default = Yes

• Else → Default = No

P7 has Credit = Good → Tree 2 says: No

Tree 3 (different sample)

Rule it learned (example):

• If Debt = High → Default = Yes

• Else → Default = No

P7 has Debt = Low → Tree 3 says: No

pg. 2
🗳 Majority Vote (Final Answer)

Tree 1: Yes
Tree 2: No
Tree 3: No

Most trees vote No → Final prediction = No Default

Why this helps

• A single decision tree can be “tricked” by small dataset quirks.

• Random Forest reduces that risk because it averages out mistakes across many
trees.

pg. 3
 Naive Bayes
Example
Goal: Classify Emails as Spam or Not Spam

We will build a tiny dataset and classify one new email.

Step 1 — Small Training Dataset

We have 4 training emails:

Email Words in Email Class

E1 free money Spam

E2 free prize Spam

E3 meeting schedule Not Spam

E4 project meeting Not Spam

So:

• 2 Spam emails

• 2 Not Spam emails

Step 2 — Calculate Basic Probabilities

Prior Probabilities (How common is each class?)

𝑃(𝑆𝑝𝑎𝑚) = 2/4 = 0.5

𝑃(𝑁𝑜𝑡𝑆𝑝𝑎𝑚) = 2/4 = 0.5

Both classes are equally likely at the start.

Step 3 — Count Word Frequencies

Let’s count how often words appear in each class.

Spam emails contain:

pg. 1
 • free (2 times)

• money (1 time)

• prize (1 time)

Total spam words = 4

Not Spam emails contain:

• meeting (2 times)

• schedule (1 time)

• project (1 time)

Total non-spam words = 4

Step 4 — Classify a New Email

New email:

“free meeting”

We must calculate:

𝑃(𝑆𝑝𝑎𝑚 ∣ 𝑓𝑟𝑒𝑒, 𝑚𝑒𝑒𝑡𝑖𝑛𝑔)

and

𝑃(𝑁𝑜𝑡𝑆𝑝𝑎𝑚 ∣ 𝑓𝑟𝑒𝑒, 𝑚𝑒𝑒𝑡𝑖𝑛𝑔)

Naive Bayes Idea

We compare:

𝑃(𝑆𝑝𝑎𝑚) × 𝑃(𝑓𝑟𝑒𝑒 ∣ 𝑆𝑝𝑎𝑚) × 𝑃(𝑚𝑒𝑒𝑡𝑖𝑛𝑔 ∣ 𝑆𝑝𝑎𝑚)

vs

𝑃(𝑁𝑜𝑡𝑆𝑝𝑎𝑚) × 𝑃(𝑓𝑟𝑒𝑒 ∣ 𝑁𝑜𝑡𝑆𝑝𝑎𝑚) × 𝑃(𝑚𝑒𝑒𝑡𝑖𝑛𝑔 ∣ 𝑁𝑜𝑡𝑆𝑝𝑎𝑚)

pg. 2
We do NOT calculate exact percentages — we only compare which is larger.

Step 5 — Compute for Spam

From spam emails:

• free appears 2 times out of 4 words → 2/4 = 0.5

• meeting appears 0 times → 0/4 = 0

So:

0.5 × 0.5 × 0 = 0

Spam score = 0

Step 6 — Compute for Not Spam

From not-spam emails:

• free appears 0 times → 0/4 = 0

• meeting appears 2 times → 2/4 = 0.5

So:

0.5 × 0 × 0.5 = 0

Not Spam score = 0

pg. 3
 Problem: We Got Zero!

Because one word did not appear in a class, the probability became 0.

In real systems, we fix this using Laplace smoothing (adding 1 to counts).

But even without math details, here’s the intuition:

• “free” is strongly linked to Spam

• “meeting” is strongly linked to Not Spam

If “free” appears more often in spam overall, the final probability usually leans toward
Spam.

So the model might classify it as:

Spam

Why This Works

pg. 4
Naive Bayes:

1. Looks at each word separately

2. Checks how common each word is in each class

3. Multiplies probabilities

4. Chooses the class with the higher score

Even though it assumes words are independent (which is not fully true), it works
surprisingly well for text classification.

Key Idea

Naive Bayes asks:

“Given these words, which class usually contains them more often?”

Then it picks the most likely class.

pg. 5
 Supervised ML Regression Models
Lesson Objective

In this lesson, we explore supervised regression algorithms, which are used when the
target variable is continuous (numerical). We focus on understanding how each model
works, when to use it, and its strengths and limitations.

What is Supervised Regression?

Supervised learning means the model is trained using labeled data, where the correct
output is known.

Regression models specifically predict continuous values, such as:

• House prices

• Sales revenue

• Temperature

• Exam scores

Linear Regression

Concept

The simplest regression model. It assumes a linear relationship between input 𝑥 and
output 𝑦.

𝑦 = 𝑎𝑥 + 𝑏

a = slope (how much 𝑦 changes when x changes)

b = intercept (value of y when x = 0 )

The model finds the best-fitting straight line by

minimizing the sum of squared errors (Least Squares
Method).

pg. 1
 Example

Predicting exam scores based on hours studied

Strengths

✔ Simple and easy to interpret

✔ Fast to train

Weaknesses

✖ Cannot capture curved (non-linear) patterns

PITFALL: Using linear regression for curved data leads to

underfitting.

Support Vector Regression (SVR)

Concept

Support Vector Regression is a way to predict numbers

(like prices or scores) without worrying too much about
tiny mistakes.

Instead of trying to make every prediction perfectly

accurate, SVR says:

“If my prediction is close enough, I’m happy.”

This “close enough” zone is called the epsilon (𝜀) margin.

SVR tries to fit a function within an 𝜺 margin of tolerance.

• Errors inside the margin are ignored

• Only larger errors affect the model

How It Works (Simple Idea)

Imagine drawing a line (or curve) that fits your data.

SVR creates a tube around that line.

• Points inside the tube → small errors → ignored

• Points outside the tube → big errors → model tries to fix these

pg. 2
So, SVR focuses only on the important mistakes, not the tiny ones.

What About Curved Data?

SVR can use something called a kernel function.

Think of a kernel as a trick that lets SVR:

✔ draw curves instead of just straight lines

✔ handle more complex patterns in data

So, SVR can use kernel functions to model non-linear relationships.

Example: Predicting house prices

Most houses follow normal pricing patterns.

But a few luxury mansions are extremely expensive.

A regular regression model might get pulled toward those extreme prices.

SVR is better because it:

✔ ignores small price differences

✔ doesn’t overreact to outliers

✔ focuses on the overall trend

Strengths

✔ Robust to noise
✔ Handles non-linear data via kernels

Weaknesses

✖ Requires careful parameter tuning

✖ Computationally intensive

TIP: Use SVR when you want a model that is less sensitive to small fluctuations.

Random Forest Regressor

Concept

pg. 3
A Random Forest Regressor is a machine learning method used to predict numbers (like
price, temperature, or crop yield).

Instead of using one decision tree, it builds many small decision trees and then
combines their answers.

Simple Idea

Think of it like asking a group of people for a prediction instead of just one person.

Each tree:

• Looks at a random part of the data

• Uses a random set of features
• Makes its own prediction

Then the model:

Averages all the predictions

Produces a final, more reliable answer

This is why it's called “Random Forest” — lots of random trees working together.

Example

Imagine predicting crop yield.

You have information about:

• Rainfall
• Soil type
• Temperature
• Fertilizer amount

Each tree might focus on different combinations of these factors.

Together, they give a more accurate and stable prediction.

Why It Works Well

✔ It can understand complex patterns

✔ It works well when relationships are not straight lines
✔ It avoids memorizing the training data too much (less overfitting)

pg. 4
 Strengths

✔ Handles complex, non-linear data

✔ Reduces overfitting compared to a single tree

Weaknesses

✖ Slower on very large datasets

✖ Less interpretable than linear models

NOTE: Random Forest = “Wisdom of Crowds” in machine learning.

The following content is excluded from the exam

(from this point to the end of the document)
Ridge Regression

Concept

pg. 5
Ridge regression is a safer version of linear regression.
It helps stop the model from becoming too complicated and memorizing the data
(overfitting).

The Main Idea

Normal linear regression tries to make predictions by drawing the best-fitting straight line
(or flat surface in many dimensions).

Ridge regression does the same thing but adds a rule:

“Don’t let the model use very large numbers for its weights.”

In other words, it penalizes big coefficients.

It adds a penalty term:

𝑳𝒐𝒔𝒔 = 𝑺𝑺𝑬 + 𝝀∑𝒘𝟐

• 𝑺𝑺𝑬 → The normal prediction error (how wrong the model is)
• 𝜮 𝒘² → The sum of the squared weights (how complex the model is)
• 𝝀 → A knob that controls how strong the penalty is

So, the model is trying to:

✔ Make accurate predictions

✔ Keep the weights small

What Lambda (λ) Does

• Small λ → Model focuses more on fitting the data

• Large λ → Model focuses more on keeping weights small

Bigger λ → simpler model

Example

You want to predict house prices using many features:

• Size
• Number of rooms
• Distance to city

pg. 6
 • Neighborhood score

Some of these features are related to each other (called correlated).

Regular regression can get confused and give huge weights.

Ridge regression shrinks those weights, making the model more stable.

Strengths

✔ Reduces overfitting
✔ Works well for high-dimensional data

Weaknesses

✖ Still assumes linear relationships

WARNING: Ridge regression reduces overfitting but cannot model curves.

Polynomial Regression

What Is It?

Polynomial regression is an extension of linear regression that helps the model follow
curved patterns, not just straight lines.

Instead of using only 𝒙, the model also uses:

• 𝑥2
• 𝑥³
• …and sometimes higher powers.

So, the equation looks like: 𝒚 = 𝒂𝟎 + 𝒂𝟏 𝒙 + 𝒂𝟐 𝒙𝟐 + 𝒂𝟑 𝒙𝟑 + ⋯

Even though it looks more complex, it is still called linear regression because it is linear in
the coefficients (𝑎₀, 𝑎₁, 𝑎₂ …).

Why Do We Need This?

Regular linear regression can only draw a straight line. But many real-life relationships are
curved. Polynomial regression allows the model to bend and follow those curves.

Example: Car speed over time

When a car starts moving:

• Speed increases slowly

pg. 7
 • Then increases faster
• Then slows down again when braking

This creates a curve, not a straight line. Polynomial regression can model this curved
motion much better than a straight line.

Strengths

✔ Captures smooth, curved trends

Weaknesses

✖ High-degree polynomials cause overfitting

TIP: Use when data shows curvature, not straight-line patterns.

PITFALL: High-degree polynomial = memorization, not generalization.

Model Comparison Summary

Model Best For Strength Weakness

Linear Regression Simple linear data Easy & fast Cannot model curves

SVR Noisy data, robustness Handles outliers Parameter sensitive

Random Forest Complex non-linear Powerful & Slower, less

data accurate interpretable

Ridge Regression High-dimensional linear Reduces Still linear

data overfitting

Polynomial Curved relationships Models non- Overfits at high

Regression linearity degree

Practical Workflow (Lab)

1. Plot the data first → Check if relationship looks linear or curved

2. Start with Linear Regression

3. If underfitting → Try Polynomial Regression

4. If overfitting → Try Ridge Regression

pg. 8
 5. If data is complex → Use Random Forest

6. If data is noisy with outliers → Use SVR

Key Takeaways

✔ Regression models predict continuous outcomes

✔ Linear Regression = simplest starting point
✔ SVR focuses on robustness
✔ Random Forest handles complex patterns
✔ Ridge controls overfitting in linear models
✔ Polynomial Regression captures curves

pg. 9
 Linear Regression
Example

Goal

Predict exam score based on hours studied.

This is regression because:

• The output (score) is a number

• Not a category like Yes/No

Step 1 — Small Training Dataset

Suppose we collect data from 4 students:

Hours Studied (x) Exam Score (y)

1 hour 50

2 hours 60

3 hours 70

4 hours 80

We can see something simple:

As study time increases, exam score increases.

Step 2 — What Linear Regression Tries to Do

Linear regression assumes a straight-line relationship:

𝑦 = 𝑎𝑥 + 𝑏

Where:

• a = slope (how much the score increases per hour)

• b = intercept (score when hours = 0)

pg. 1
 Step 3 — Find the Pattern

Look at the data:

From 1 → 2 hours
Score increases from 50 → 60 (+10)

From 2 → 3 hours
Score increases from 60 → 70 (+10)

From 3 → 4 hours
Score increases from 70 → 80 (+10)

So clearly:

Each extra hour adds about 10 points

So slope 𝑎 = 10

Now let’s find the intercept.

If:

𝑦 = 10𝑥 + 𝑏

Use one data point to find b.

Take (x = 1, y = 50):

50 = 10(1) + 𝑏
50 = 10 + 𝑏
𝑏 = 40

So, the model becomes:

𝑦 = 10𝑥 + 40

Step 4 — Make a Prediction

Now suppose a new student studies 5 hours.

pg. 2
Plug into the formula:

𝑦 = 10(5) + 40
𝑦 = 50 + 40 = 90

Predicted exam score = 9

What the Model Did

Linear regression:

• Looked at the pattern in the data

• Found the best straight line

• Used it to predict new values

What Does “Minimizing Squared Errors” Mean?

pg. 3
In real life, data is not perfectly straight.

The model:

1. Draws a line

2. Measures the error between real score and predicted score

3. Squares the errors

4. Adjusts the line to make total error as small as possible

That’s called the Least Squares Method.

When to Use Linear Regression

Use it when:

• The relationship looks roughly straight

• The output is numeric

• You want a simple, interpretable model

When It Fails

If the data looks curved like this:

Hours Score

1 40

2 55

3 75

4 95

Now the increase is not constant.

A straight line will not fit well.

That is called:

Underfitting

pg. 4
 Example
Support Vector Regression (SVR)

Goal

Predict house price based on house size (in square meters).

Step 1 — Small Dataset

Suppose we have 5 houses:

Size (m²) Price ($1000s)

50 150

60 180

70 210

80 240

90 400

Notice something:

The first four houses follow a clear pattern.

But the last one (90 m², 400k) is very expensive — maybe it's luxury.

That last point is an outlier.

What Linear Regression Would Do

Linear regression tries to reduce all errors.

Because of the expensive 400k house, the line will tilt upward more than it should.

That means:

• Smaller houses might get overestimated.

• The model gets pulled by the extreme value.

pg. 1
 What SVR Does Differently

SVR says:

“If my prediction is close enough, I don’t care.”

We define an epsilon (ε) value.

Let’s say:

𝜀 = 20

This means:

If the prediction is within ±20 (thousand dollars),

the model considers it “good enough.”

Step 2 — Imagine the “Tube”

SVR draws a line, just like linear regression.

But around that line, it builds a tube of width ±20.

Inside the tube:

✔ Errors are ignored
Outside the tube:
Errors matter and affect the model

Apply It to Our Data

Suppose SVR draws this approximate line:

𝑃𝑟𝑖𝑐𝑒 = 3 × 𝑆𝑖𝑧𝑒

So predictions are:

Size Predicted Price

50 150

60 180

pg. 2
 70 210

80 240

90 270

Now compare with actual prices.

For first four houses:

Predictions are exactly correct.

For the last house:

Actual = 400
Predicted = 270
Error = 130

That is outside the epsilon margin.

So this point affects the model.

But here’s the key:

SVR will not overreact too much.

It will not try to fully chase that 400 value.

It tries to balance things while keeping most data inside the tube.

pg. 3
 Why This Is Powerful

SVR:

• Ignores small noise

• Focuses only on big mistakes

• Is less sensitive to outliers

That’s why:

✔ It works well when there are extreme values

✔ It doesn’t get pulled heavily by luxury mansions
✔ It captures the overall trend

pg. 4
 What If the Pattern Is Curved?

Suppose house prices increase slowly at first,

then rise faster for larger houses.

A straight line won't work well.

SVR can use a kernel function.

A kernel allows SVR to:

• Draw curves instead of straight lines

• Capture non-linear patterns

For example:

Instead of:

𝑃𝑟𝑖𝑐𝑒 = 𝑎 × 𝑆𝑖𝑧𝑒 + 𝑏

It might learn something curved like:

Price = Size² pattern

But internally — without you writing the equation.

Big Intuition

Think of SVR as:

“Fit a line that keeps most points inside a comfortable tube.”

It does NOT try to perfectly hit every point.

It focuses only on points that really matter.

When to Use SVR

Use SVR when:

• You want robustness to noise

• You don’t want outliers to dominate

• The relationship may be non-linear

pg. 5
 • You care about general trend, not tiny errors

pg. 6
 Example
Random Forest Regressor

Goal

Predict house price using:

• Size (m²)

• Number of bedrooms

We want to predict a number, so this is regression.

Step 1 — Small Dataset

Suppose we have 6 houses:

House Size (m²) Bedrooms Price ($1000s)

H1 50 1 150

H2 60 2 180

H3 70 2 210

H4 80 3 250

H5 90 3 270

H6 100 4 320

Now we want to predict the price of:

A house with 75 m² and 2 bedrooms

Step 2 — Build Multiple Decision Trees

Instead of building one big decision tree, Random Forest builds many smaller trees.

Let’s imagine we build 3 trees.

pg. 1
 Tree 1

Tree 1 randomly selects some data and maybe focuses mostly on size.

It may learn a simple rule like:

• If size < 65 → price ≈ 165

• If size between 65–85 → price ≈ 230

• If size > 85 → price ≈ 295

For 75 m²:

➡ Prediction = 230

Tree 2

Tree 2 might randomly focus more on bedrooms.

It learns:

• 1–2 bedrooms → price ≈ 190

• 3 bedrooms → price ≈ 260

• 4 bedrooms → price ≈ 320

For 2 bedrooms:

➡ Prediction = 190

Tree 3

Tree 3 uses another random subset of data and features.

It might combine both features:

• If size < 80 AND bedrooms = 2 → price ≈ 220

• Else → other rule

For our house:

➡ Prediction = 220

pg. 2
🗳 Step 3 — Average the Predictions

Now we combine the three predictions:

• Tree 1 → 230

• Tree 2 → 190

• Tree 3 → 220

Final prediction:

(230 + 190 + 220)/3 = 213.3

➡ Final predicted price ≈ $213,000

pg. 3
 Why This Is Better Than One Tree

If we had used only Tree 2:

Prediction = 190 (maybe too low)

If we had used only Tree 1:

Prediction = 230 (maybe too high)

But averaging gives a balanced answer.

That’s the power of the “forest”.

Why Random Forest Works Well

Random Data (Bootstrapping)

Each tree sees a slightly different version of the data.

So they don’t all make the same mistake.

Random Features

Each tree considers only a random subset of features when splitting.

So:

• One tree may rely on size

• Another on bedrooms

• Another on both

This prevents one strong feature from controlling everything.

Intuition

Random Forest = “Wisdom of the Crowd”

Instead of trusting one opinion,

we trust the average of many opinions.

It:

pg. 4
✔ Handles complex patterns
✔ Works well with non-linear data
✔ Reduces overfitting
✔ Is usually more stable than a single tree

Big Picture

Single Tree → Can overfit

Random Forest → More stable, more accurate

It is one of the most powerful and commonly used regression models in practice.

pg. 5