Ch4

Supervised Learning
Introduction to Classification
● Classification is a Supervised Machine Learning technique.
● It is used to predict categorical output (Yes/No, Pass/Fail, Spam/Not Spam).
● The model learns from labeled training data.
Real-Life Examples:
● Email Spam Detection
● Disease Prediction
● Student Pass/Fail Prediction
● Loan Approval

: Types of Classification Algorithms Covered

1. 🌳 Decision Trees
2. 👥 K-Nearest Neighbors (KNN)
3. 📈 Support Vector Machine (SVM)
🌳 Decision Tree Algorithm
What is Decision Tree?
● A tree-like structure.
● Used for classification and regression.
● Splits data into branches based on conditions.
● Final output is given at leaf nodes.

Important Terms:
● Root Node
● Decision Node
● Leaf Node
● Branch

○
🟢 Root Node
● The top-most node of the tree.
● Represents the entire dataset.
● First point where splitting starts.
🔹 Example:Suppose we want to predict Student Pass/[Link] algorithm selects Study
Hours as best feature → 👉 Study Hours becomes Root Node
🔹 Why important?
● It gives maximum information gain.
● All other branches come from root.
●
 Decision Node
● Node where data is further split.
● Represents a condition or test.
● Also called Internal Node.
🔹 Example:
If:
● Study Hours = Low
→ Check Attendance
Here, Attendance becomes Decision Node
Rule:
● If Attendance = Poor → Fail
● If Attendance = Good → Pass
🔹 Key Point:
● One tree can have multiple decision nodes.
● They help in breaking complex problems step by step.
Leaf Node
● Final node of the tree.
● No further splitting.
● Represents final output/class label.
🔹 Example:
If:
● Study Hours = High → Pass
Here, Pass is Leaf Node
Similarly:
● Study Hours = Low & Attendance = Poor → Fail

👉 Fail is also Leaf Node

🔹 Important:
● Leaf nodes contain:
○ Final decision
○ Class label (Pass/Fail, Yes/No)
 Branch
● A connection between nodes.
● Represents outcome of a test.
● Shows decision path.
🔹 Example:
If Root Node = Study Hours
Branches:
● Study Hours = High → (Branch 1)
● Study Hours = Low → (Branch 2)
Complete Rule Example:
IF Study Hours = High
THEN Result = Pass
This path from Root → Decision → Leaf is called a Branch Path.
 Study Hours ← Root Node
/ \
High Low ← Branch
| |
Pass ← Leaf Attendance ← Decision Node
/ \
Good Poor ← Branch
| |
Pass Fail ← Leaf

🟢 How Decision Tree Works?

1. Select best feature (using Gini Index / Entropy).
2. Split dataset into subsets.
3. Repeat process recursively.
4. Stop when:
○ All data is pure
○ Maximum depth reached
Decision Rule Example:
● If Study Hours = High → Pass
● Else if Attendance = Poor → Fail

✅ Advantages:
● Easy to understand
● Works with categorical & numerical data
● No scaling required

❌ Disadvantages:
● Overfitting problem
● Sensitive to small data changes
Select best feature based on Entropy in Decision Trees
👉Entropy is one way to measure impurity.
🔹 Impurity measures how mixed the class labels are in a node(column).
● Mixed classes → Impure node (Entropy > 0)
● Single class → Pure node (Entropy = 0)
🔹 Where is Impurity Calculated?
✅ Calculated on Target/Output column only(Result)
❌ NOT calculated on feature columns (Study Hours, Attendance)
🔹 Example
Initial node (Result column is target column):
Pass = 2, Fail = 2 → Mixed → Impure
After good split:Pass = 2, Fail = 0 → Single class → Pure
🔹 Why Decision Trees Use Impurity :Reduce impurity ,Increase purity
👉 Best split = one that reduces impurity the most (maximum Information Gain)
⭐ Impurity is always measured using the target class labels, not input features.
 Entropy
Symbol Meaning

S Dataset

pᵢ Probability of class i

P(pass) =6/6 =1
P(Fail) =0/6 =0 P(pass) =3/6 =0.5
P(Fail) =3/6 =0.5
From Entropy to Information Gain (Complete Flow)
🔹 Step 1: Entropy — Measure of Impurity
Entropy tells how mixed the target values are in the dataset.

● High entropy → data mixed (impure)

● Low entropy → data pure

🔹 Step 2: Split the Dataset

Decision Tree tries different features to split the data.
Goal:👉 After split, each group should be more pure than before.

🔹 Step 3: Compute Weighted Entropy After Split

● Sv= subset after split

● ∣Sv∣ = size of subset
● ∣S∣ = total dataset size
🔹 Step 4: Information Gain — Reduction in Impurity

👉 Information Gain measures how much impurity decreased after the split.

🔹 Example
Study Attendance Result
Initial dataset: Hours
Pass = 3, Fail = 3
High Good Pass
Entropy(parent)=1
After splitting on Study Hours: High Poor Pass

● High → Entropy = 0 Low Good Fail

● Low → Entropy = 0 Low Poor Fail
Weighted Entropy = 0
High Good Pass
IG=1−0=1
Low Poor Fail
✅ Perfect split
Decision Tree – Step by Step Illustration
🎯 Problem:Predict whether a student will Pass or Fail based on:
● Study Hours
● Attendance

Student Study Hours Attendance Result

S1 High Good Pass

S2 High Poor Pass

S3 Low Good Fail

S4 Low Poor Fail

S5 High Good Pass

S6 Low Poor Fail

🟢 Step 1 – Calculate Initial Entropy
🔹 What is Entropy?
Entropy measures impurity in dataset.

Where:
● p₁ = Probability of Pass
● p₂ = Probability of Fail
📌 Count Values:
Pass = 3 , Fail = 3 , Total = 6
p(Pass)=3/6=0.5
p(Fail)=3/6=0.5

Entropy (S)=−0.5(−1)−0.5(−1)=1
👉 Initial Entropy = 1 (Maximum impurity)
🟢 Step 2 – Calculate Information Gain for "Study Hours"
Split by Study Hours
Case 1: Study Hours = High
Students: S1, S2, S5
All = Pass
Entropy = 0 (Pure)
Case 2: Study Hours = Low
Students: S3, S4, S6
All = Fail
Entropy = 0 (Pure)
📌 Information Gain Formula
IG=Entropy(S)−Weighted Entropy
Weighted Entropy:
=(3/6)∗0+(3/6)∗0= 0
IG=1−0=1
👉 Information Gain (Study Hours) = 1
🟢 Step 3 – Calculate Information Gain for "Attendance"
Case 1: Attendance = Good
Students: S1, S3, S5
Pass = 2 , Fail = 1

≈ 0.918
Case 2: Attendance = Poor
Students: S2, S4, S6
Pass = 1 , Fail = 2
Entropy ≈ 0.918
Weighted Entropy
=(3/6)∗0.918+(3/6)∗0.918= 0.918
IG=1−0.918= 0.082
👉 Information Gain (Attendance) = 0.082 and 👉 Highest IG = Study Hours
So, Study Hours becomes Root Node
🟢Python Implementation (Decision Tree)
import pandas as pd
from [Link] import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from [Link] import accuracy_score

# Sample Dataset
data = {
'StudyHours': [1,2,3,4,5,6],
'Attendance': [50,60,70,80,90,95],
'Result': [0,0,0,1,1,1] # 0=Fail, 1=Pass
}

df = [Link](data)

X = df[['StudyHours','Attendance']]
y = df['Result']
X_train, X_test, y_train, y_test =
train_test_split(X,y,test_size=0.3)

model = DecisionTreeClassifier()
[Link](X_train,y_train)

prediction = [Link](X_test)

print("Accuracy:", accuracy_score(y_test,prediction))
Movie Calculation Distance Genre

√((8.0−7.4)² +
Mission Impossible √(0.36 + 2401) = 49.01 Action
(163−114)²)

√((6.2−7.4)² +
Gadar 2 √(1.44 + 3136) = 56.03 Action
(170−114)²)

√((7.2−7.4)² +
Rocky Aur Rani √(0.04 + 2916) = 54.00 Comedy
(168−114)²)

√((8.2−7.4)² +
OMG 2 √(0.64 + 1681) = 41.00 Comedy
(155−114)²)
from [Link] import KNeighborsClassifier
from [Link] import StandardScaler

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

X_train, X_test, y_train, y_test =

train_test_split(X_scaled,y,test_size=0.3)

knn = KNeighborsClassifier(n_neighbors=3)
[Link](X_train,y_train)

prediction = [Link](X_test)

print("Accuracy:", accuracy_score(y_test,prediction))
Regression
🟦 What is Regression?
● Regression is a statistical technique used to predict continuous values
● It finds relationship between independent and dependent variables
👉 “Regression predicts numeric values based on relationships between variables”
📌 Key Idea:
● Learn from past data
● Predict future values

🟦 Goal of Regression
is to plot a line or curve that best fit the data and to estimate how one variable
affects another.
● To find best-fit line or curve
● To understand how one variable affects another
👉 Regression plots a line/curve that best fits the data
📘 2. Terminology
🟦Independent & Dependent Variables (Reference)
● Independent Variable (X):
○ input data feature ,Predictor / Feature
○ Used to estimate output
● Dependent Variable (Y):
○ output values ,Target variable
○ Value we want to predict
📌 “Independent variables predict dependent variables.”
🟦 Regression Line
● A regression line is a line that best fits the data
● It shows relationship between variables
👉 Regression line = best-fit line used for prediction
Overfitting and underfitting −
● Overfitting is when the regression model works well with the training dataset but
not with the testing dataset.
● It's also referred to as the problem of high variance. Underfitting is when the model
doesn't work well with training datasets.
● It's also referred to as the problem of high bias.
Outliers −
● These are data points that don't fit the pattern of the rest of the data. They are the
extremely high or extremely low values in the data set.
Multicollinearity −
● multicollinearity occurs when independent variables (features) have dependency
among them.
3. Working Principle
🟦 How Regression Works
1. Take labeled dataset (X, Y)
2. Train model to learn relationship
3. Fit best line
4. Predict new values
● Model learns relationship during training
● Uses it for prediction
📌 “Regression learns from training data and predicts new values.”
🟦 Best-Fit Line Concept
● Best-fit line minimizes error
● Uses concept of least squares
👉 Line minimizes difference between actual and predicted values
📌 “Regression finds a line that minimizes total error.”
Types of Regression in Machine Learning
Generally, the classification of regression methods is done based on the three metrics −
● the number of independent variables,
● type of dependent variables,
● shape of the regression line.
There are numerous regression techniques used in machine learning. However, the
following are commonly used types of regression −

● Linear Regression
● Logistic Regression
● Ridge Regression
📊 4. Linear Regression
● Linear regression models linear relationship between variables
👉 “It estimates linear relationship between dependent and independent variables”
🟦 Objective of Linear Regression
1. Understand relationship between variables
2. Predict future values
🟦 Real-Life Understanding (Added)
● Income vs Spending
● Experience vs Salary
Linear Regression Types
● Linear Regression is divided into:
1. Simple Linear Regression
2. Multiple Linear Regression (Multivariate)

●
🔹 Simple Linear Regression
● Uses one independent variable (X)
● Predicts one dependent variable (Y)
👉 Example: Salary prediction based on experience

🔹 Mathematical Representation
Y=mX+b
● Y → Dependent variable (output)
● X → Independent variable (input)
● m → Slope (effect of X on Y)
● b → Intercept (value of Y when X = 0)
Formula for Simple Linear Regression

Simple Example :
Hours studied (X): 1, 2, 3, 4
Marks scored (Y): 10, 20, 30, 40
Week(x) Sales(y) x*y x*x
1 10 1.2 1
2 20 3.6 4
3 30 7.8 9
4 40 12.8 16
sum 100 25.4 30

● Regression equation: Y = 0 + 10X

→ If a student studies for 5 hours, predicted marks = 10 × 5 = 50 marks.
Multiple Linear Regression
Definition:Multiple Linear Regression is an extension of linear regression .It uses two
or more independent variables (X₁, X₂, …) to predict one dependent variable (Y)
🔹 Mathematical Equation
Y=b0+b1X1+b2X2+⋯+bnXn
● Y → Dependent variable (output)
● X₁, X₂, … → Independent variables (inputs)
● b₀ → Intercept
● b₁, b₂, … → Coefficients (impact of each variable)
🔹 Example
👉 Predict house price based on:Area (X₁),Number of rooms (X₂),Location (X₃)

🔹 Key Features
● Considers multiple factors together
● More accurate than simple regression
● Models real-life problems better
📊 5. Logistic Regression
🔹 Definition
● Logistic Regression is a supervised learning algorithm
● Used for classification problems (not continuous prediction)
● It predicts probability (values between 0 and 1)
👉 Example: Spam (1) / Not Spam (0)
🔹 Why Logistic Regression?
● Linear regression gives values like 1.5, 2.3 ❌ (not valid for class)
● Logistic regression converts output into 0 or 1 (class labels) ✅
🔹 Sigmoid Function (Core Concept)

● Converts any value into range (0 to 1)

● Used to calculate probability
🔹 Working Principle
1. Input features are taken
2. Linear equation is applied
3. Sigmoid function converts output to probability
4. Apply threshold (usually 0.5):
○ ≥ 0.5 → Class 1
○ < 0.5 → Class 0
🔹 Real-Life Applications
● Email spam detection
● Disease prediction (Yes/No)
● Student pass/fail prediction
🔹 Advantages
● Simple and fast
● Works well for binary classification
● Gives probability output
🔹 Disadvantages
● Cannot handle complex non-linear data
● Sensitive to outliers
● Requires proper feature selection

Python Implementation
from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
[Link](X, y)
print([Link]([[value]]))
📊 6. Ridge Regression
🔹 Definition:Ridge Regression is a type of linear regression with regularization .It
adds a penalty term to reduce overfitting
🔹 Why Ridge Regression?
● In linear regression:Model may overfit (fits noise in data) ❌
● Ridge Regression:
○ Adds penalty to control model complexity ✅
🔹 Mathematical Concept

● λ (lambda) → Regularization parameter

● w → Model coefficients
👉 Penalizes large coefficient values
🔹 Key Idea
👉 Keeps coefficients small and balanced
👉 Improves generalization (performance on new data)
🔹 Example
● Predict house price using many features
● Ridge prevents model from depending too much on any one feature
🔹 Advantages
● Reduces overfitting
● Improves model accuracy
● Works well with multicollinearity
🔹 Disadvantages
● Adds complexity
● Does not perform feature selection
● Requires tuning of λ
🔹 Python Implementation
from sklearn.linear_model import Ridge
model = Ridge(alpha=1.0)
[Link](X, y)
print([Link]([[value]]))
Model Performance Evaluation
🔹 Need of Model Performance Evaluation
● To check how well the model is performing
● To measure accuracy of predictions
● To compare different models
● To detect problems like:
○ Overfitting
○ Underfitting
● To improve model performance
👉 Key Idea: “Evaluation helps in selecting the best model for real-world use”
🔹 Why It is Important?
● A model may work well on training data but fail on new data
● Evaluation ensures generalization ability
●
🔹 Criteria for Evaluation (Important Metrics)
📊 For Regression Models
● MAE (Mean Absolute Error) → Average error
● MSE (Mean Squared Error) → Penalizes large errors
● RMSE → Error in same unit
● R² Score → Model accuracy

📊 For Classification Models

● Accuracy → Overall correctness
● Precision → Correct positive predictions
● Recall → Detecting all positives
● F1-score → Balance of precision & recall
Confusion Matrix
🟦 1. What is Confusion Matrix?
● A Confusion Matrix is a performance evaluation tool for classification models
● It shows actual vs predicted values
👉 It helps us understand:
● Correct predictions
● Wrong predictions
2. Structure of Confusion Matrix
Predicted Positive Predicted Negative

Actual Positive True Positive (TP) False Negative (FN)

Actual Negative False Positive (FP) True Negative (TN)

3. Meaning of Terms (Very Important)

● True Positive (TP): Correctly predicted positive
👉 Example: Spam email correctly detected ,covid positive correctly detected
● True Negative (TN): Correctly predicted negative
👉 Example: Normal email correctly identified,covid Negative correctly detected
● False Positive (FP): Incorrectly predicted positive
👉 Example: Normal email marked as spam ,covid negative marked as positive
● False Negative (FN): Incorrectly predicted negative
👉 Example: Spam email missed ,covid positive marked as negative
5. Why Confusion Matrix is Important?
● Gives detailed performance analysis
● Helps calculate:
○ Accuracy
○ Precision
○ Recall
● Shows type of errors (FP & FN)
Accuracy
🔹 Definition:Accuracy is the ratio of correct predictions to total predictions
● It shows how often the model is correct
Formula
Accuracy=
Where:
● TP → True Positive ,TN → True Negative,FP → False Positive,FN → False Negative
Higher accuracy = better model
Advantages
● Simple and easy to understand
● Good when data is balanced
🔹 Disadvantages
● Misleading for imbalanced datasets
● Ignores type of errors (FP vs FN)
“Accuracy is the percentage of correctly predicted observations out of total
observations”
🔹 Advantages
● Simple and easy to understand
● Good when data is balanced

🔹 Disadvantages
● Misleading for imbalanced datasets
● Ignores type of errors (FP vs FN)

🔹 Python Implementation
from [Link] import accuracy_score

accuracy = accuracy_score(y_true, y_pred)

print(accuracy)
Precision
🔹 Definition:Precision measures how many predicted positive cases are actually
correct
● It focuses on quality of positive predictions
🔹 Formula

Where:
● TP → True Positive ,FP → False Positive
● Out of all predicted positives:
👉 How many are truly positive?
📌 High precision = fewer false positives
When to Use Precision?
● When false positives are costly
👉 Example:
● Email spam detection
● Fraud detection
🔹 Advantages
● Focuses on correctness of positive predictions
● Useful in critical applications

🔹 Disadvantages
● Ignores false negatives
● Not sufficient alone

🔹 Python Implementation
from [Link] import precision_score

precision = precision_score(y_true, y_pred)

print(precision)

“Precision is the ratio of correctly predicted positive observations to total

predicted positives.”
 Recall (Sensitivity / True Positive Rate)
🔹 Definition :Recall measures how many actual positive cases are correctly
identified
● It focuses on detecting all positive cases
🔹 Formula

Where:
● TP → True Positive ,FN → False Negative
● Out of all actual positives: 👉 How many did the model correctly predict?
📌 High recall = fewer false negatives
When to Use Recall?
● When missing positive cases is dangerous
👉 Example:
● Disease detection (missing patient is risky)
● Fraud detection
🔹 Advantages
● Detects maximum positive cases
● Important in critical systems

🔹 Disadvantages
● May increase false positives
● Not sufficient alone

🔹 Python Implementation
from [Link] import recall_score

recall = recall_score(y_true, y_pred)

print(recall)

📌 Exam Tip
👉 “Recall is the ratio of correctly predicted positive observations to all actual positives.”
Solved Problem (Step-by-Step)
📌 Given Data Predicted Predicted Not
A model predicts whether an email is spam or not: Spam Spam

Step 1: Identify Values Actual Spam 40 10

● TP = 40,FN = 10,FP = 5,TN = 45 Actual Not 5 45
🟦 Step 2: Calculate Accuracy Spam

🟦 Step 3: Calculate Precision 7. Interpretation of Result

● Accuracy = 85% (good overall)
● Precision = 88.89% (few false positives
🟦 Step 4: Calculate Recall ● Recall = 80% (some spam missed)
👉 Model is good but can improve recall
We have:

● Total patients = 100 with 50positive and

50 negative case
● Model predicts COVID Positive / Negative
●
●

Predicted Positive Predicted Negative Total

Actual Positive 40 10 50

Actual Negative 5 45 50

Total 45 55 100
Identify Confusion Matrix Values
From table: Predicted + Predicted -
● TP (True Positive) = 40
👉 Sick people correctly identified Actual + TP = 40 FN = 10
● FN (False Negative) = 10 Actual - FP = 5 TN = 45
👉 Sick people missed
● FP (False Positive) = 5
👉 Healthy marked as sick
● TN (True Negative) = 45
👉 Healthy correctly identified