Machine Learning-Unit 1

Contents
1 Introduction to Machine Learning 2

2 Importance of Machine Learning 2

3 Applications of Machine Learning 2

4 Advantages of Machine Learning 2

5 Disadvantages of Machine Learning 3

6 Machine Learning Lifecycle 3

7 Types of Machine Learning 3

7.1 1. Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
7.1.1 (a) Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
7.1.2 (b) Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
7.2 2. Unsupervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
7.2.1 (a) Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
7.2.2 (b) Association . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
7.3 3. Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

8 Important Terminologies 5

9 Conclusion 6

1
1 Introduction to Machine Learning
Machine Learning (ML) is a subset of Artificial Intelligence that enables systems to learn
from data and improve their performance without being explicitly programmed.
Definition: Machine Learning is the field of study that gives computers the ability to
learn from data and make predictions or decisions.
The main goal of Machine Learning is to develop models that can:
• Predict outcomes
• Classify data
• Discover hidden patterns
• Make intelligent decisions

2 Importance of Machine Learning

• Handles large volumes of data efficiently.
• Automates decision-making processes.
• Improves performance with more data.
• Solves complex real-world problems.
• Enables personalization in recommendation systems.

3 Applications of Machine Learning

• Healthcare: Disease prediction, medical image analysis.
• Finance: Fraud detection, credit scoring.
• Recommendation Systems: Product and movie recommendations.
• Self-driving Cars: Object detection and navigation.
• Spam Detection: Email filtering.
• Speech Recognition: Voice assistants.

4 Advantages of Machine Learning

• Can handle complex and large datasets.
• Improves accuracy over time.
• Reduces human intervention.
• Provides data-driven decisions.

2
5 Disadvantages of Machine Learning
• Requires large amount of data.

• High computational cost.

• Risk of bias and overfitting.

• Difficult to interpret some models (black-box).

6 Machine Learning Lifecycle

Machine Learning follows a structured process:

1. Problem Definition: Clearly define objective (classification or regression).

2. Data Collection: Gather relevant data from various sources.

3. Data Preprocessing: Handle missing values, remove noise, normalize data.

4. Feature Engineering: Select and create meaningful features.

5. Model Selection: Choose appropriate algorithm.

6. Model Training: Train model using training data.

7. Model Evaluation: Evaluate using metrics like Accuracy, Precision, Recall, MSE.

8. Deployment: Deploy model into real-world system.

9. Monitoring and Maintenance: Continuously monitor and retrain model.

7 Types of Machine Learning

Machine Learning is classified into three major types:

7.1 1. Supervised Learning

Supervised learning uses labeled data. Each input has a corresponding output.
Mathematical Representation:
Y = f (X)
Where:

• X = Input features

• Y = Output label

3
7.1.1 (a) Classification
Output is categorical (e.g., Spam/Not Spam).
Algorithms:

• Logistic Regression

• Decision Tree

• K-Nearest Neighbors

• Naive Bayes

• Support Vector Machine

7.1.2 (b) Regression

Output is continuous (e.g., House price prediction).
Algorithm:

• Linear Regression

Advantages:

• High accuracy with labeled data.

• Easy to evaluate.

Disadvantages:

• Requires labeled dataset.

• Data labeling can be expensive.

7.2 2. Unsupervised Learning

Unsupervised learning uses unlabeled data to find hidden patterns.

7.2.1 (a) Clustering

Groups similar data points.
Algorithms:

• K-Means

• Hierarchical Clustering

• Gaussian Mixture Model

Example: Customer segmentation.

4
7.2.2 (b) Association
Finds relationships between variables.
Algorithm:
• Apriori Algorithm
Example: Market basket analysis.
Advantages:
• No need for labeled data.

• Useful for pattern discovery.

Disadvantages:
• Hard to evaluate performance.

• May produce less accurate results.

7.3 3. Reinforcement Learning

Reinforcement Learning is based on reward and punishment mechanism.
An agent interacts with environment and learns optimal actions.
Applications:
• Robotics

• Gaming

• Autonomous systems

8 Important Terminologies
• Dataset: Collection of data.

• Feature: Input variable.

• Label: Output variable.

• Training Data: Data used to train model.

• Test Data: Data used to evaluate model.

• Overfitting: Model performs well on training but poorly on testing data.

• Underfitting: Model performs poorly on both training and testing.

• Bias: Error due to incorrect assumptions.

• Variance: Error due to sensitivity to small fluctuations in data.

5
9 Conclusion
Machine Learning is a powerful and rapidly growing field that enables systems to learn from
data and make intelligent decisions. It has wide applications across healthcare, finance, au-
tomation, and recommendation systems. The ML lifecycle ensures structured development,
and its major types—supervised, unsupervised, and reinforcement learning—serve different
real-world purposes.

6
 Classification, Linear Regression and
Logistic Regression

Contents
1 Classification 2
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Types of Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Linear Regression 2
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.2 Equation of Simple Linear Regression . . . . . . . . . . . . . . . . . . . . 2
2.3 Formulas for a and b . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.4 Fully Solved Numerical . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.5 Prediction Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Logistic Regression 4
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.2 Sigmoid Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.3 Fully Solved Numerical . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

4 Linear vs Logistic Regression 5

5 Conclusion 5

1
1 Classification
1.1 Introduction
Classification is a supervised learning technique in which the output variable is categorical
(class label). The model predicts the category to which the input data belongs.
Examples:

• Spam / Not Spam

• Pass / Fail

• Disease / No Disease

1.2 Types of Classification

1. Binary Classification (Two classes)

2. Multi-class Classification (More than two classes)

3. Multi-label Classification (Multiple labels per instance)

2 Linear Regression
2.1 Introduction
Linear Regression is used to predict continuous values. It establishes a linear relationship
between independent variable x and dependent variable y.

2.2 Equation of Simple Linear Regression

y = a + bx
Where:

• a = Intercept

• b = Slope

• x = Independent variable

• y = Dependent variable

2.3 Formulas for a and b

P P P
n xy − x y
b= P P
n x2 − ( x)2
P P
y−b x
a=
n

2
2.4 Fully Solved Numerical
Given Data:
x y
1 2
2 4
3 5
4 4
5 5

Step 1: Calculate Required Values

X
x = 15
X
y = 20
X
x2 = 55
X
xy = 66

n=5
Step 2: Calculate Slope b

5(66) − (15)(20)
b=
5(55) − (15)2
330 − 300
b=
275 − 225
30
b=
50

b = 0.6
Step 3: Calculate Intercept a

20 − (0.6)(15)
a=
5
20 − 9
a=
5
11
a=
5

a = 2.2
Final Regression Equation:

y = 2.2 + 0.6x

3
2.5 Prediction Example
If x = 6:

y = 2.2 + 0.6(6)

y = 2.2 + 3.6 = 5.8

Predicted value = 5.8

3 Logistic Regression
3.1 Introduction
Logistic Regression is used for classification problems, especially binary classification. It
predicts probability using the sigmoid function.

3.2 Sigmoid Function

1
P (y = 1) =
1 + e−z
Where:

z = a + bx

3.3 Fully Solved Numerical

Given:

z = −4 + 0.8x
Find probability when x = 6.
Step 1: Calculate z

z = −4 + 0.8(6)

z = −4 + 4.8

z = 0.8
Step 2: Apply Sigmoid Function
1
P =
1 + e−0.8
Since:

e−0.8 ≈ 0.449

4
 1
P =
1 + 0.449
1
P =
1.449

P ≈ 0.69
Final Answer:
Probability = 0.69 (69%)
If threshold = 0.5,
Since 0.69 ¿ 0.5,
Class = 1 (Positive Class)

4 Linear vs Logistic Regression

Feature Linear Regression Logistic Regression
Output Continuous Categorical
Equation y = a + bx Sigmoid Function
Used For Prediction Classification
Range (−∞, +∞) 0 to 1
Error Method Least Squares Maximum Likelihood

5 Conclusion
Classification predicts categories. Linear Regression predicts continuous values. Logistic
Regression predicts probabilities for classification problems.
Linear regression produces a straight line, whereas logistic regression produces an
S-shaped sigmoid curve.

5
 Confusion Matrix

1 Introduction
A Confusion Matrix is a performance evaluation tool used in classification problems to
measure the accuracy of a model. It compares the actual values with the predicted
values.

2 Structure of Confusion Matrix (Binary Classifica-

tion)
Predicted Positive Predicted Negative
Actual Positive True Positive (TP) False Negative (FN)
Actual Negative False Positive (FP) True Negative (TN)

3 Important Terminologies
• True Positive (TP): Correctly predicted positive cases.

• True Negative (TN): Correctly predicted negative cases.

• False Positive (FP): Incorrectly predicted positive cases (Type I Error).

• False Negative (FN): Incorrectly predicted negative cases (Type II Error).

4 Performance Metrics
1. Accuracy
TP + TN
Accuracy =
TP + TN + FP + FN

2. Precision
TP
P recision =
TP + FP

1
3. Recall (Sensitivity)
TP
Recall =
TP + FN

4. Specificity
TN
Specif icity =
TN + FP

5. F1-Score
2 × P recision × Recall
F1 =
P recision + Recall

5 Numerical Example
Given:
Total patients = 100

• 40 patients actually have the disease.

• 60 patients do not have the disease.

Model Predictions:

• 35 diseased patients predicted correctly.

• 5 diseased patients predicted as healthy.

• 10 healthy patients predicted as diseased.

• 50 healthy patients predicted correctly.

Step 1: Confusion Matrix

Predicted Positive Predicted Negative
Actual Positive 35 5
Actual Negative 10 50

Therefore,

T P = 35, F N = 5, F P = 10, T N = 50

2
Step 2: Accuracy
35 + 50
Accuracy =
100
85
Accuracy = = 0.85
100

Accuracy = 85%

Step 3: Precision
35
P recision =
35 + 10
35
P recision = = 0.777
45

Precision = 77.7%

Step 4: Recall
35
Recall =
35 + 5
35
Recall = = 0.875
40

Recall = 87.5%

Step 5: Specificity
50
Specif icity =
50 + 10
50
Specif icity = = 0.833
60

Specificity = 83.3%

3
Step 6: F1-Score
2 × 0.777 × 0.875
F1 =
0.777 + 0.875
1.359
F1 =
1.652

F 1 = 0.822

F1-Score = 82.2%

6 Important Observations
• High Precision means fewer False Positives.

• High Recall means fewer False Negatives.

• Accuracy may not be reliable for imbalanced datasets.

• F1-score balances Precision and Recall.

7 Conclusion
Confusion Matrix is an important evaluation tool in classification problems. It helps measure
model performance using metrics like Accuracy, Precision, Recall, Specificity, and F1-score.

4
 Evaluation Metrics of Regression Models

1 Introduction
Regression models are used to predict continuous values. To evaluate their performance, we
use error-based metrics such as MAE, MSE, RMSE, and R2 .

2 1. Mean Absolute Error (MAE)

Definition: MAE measures the average absolute difference between actual and predicted
values.
1X
M AE = |yi − ŷi |
n
Properties:
• Easy to interpret
• Less sensitive to outliers

3 2. Mean Squared Error (MSE)

Definition: MSE measures the average of squared differences.
1X
M SE = (yi − ŷi )2
n
Properties:
• Penalizes large errors more
• Used in optimization algorithms

4 3. Root Mean Squared Error (RMSE)

√
RM SE = M SE

• Same unit as output variable

• More interpretable than MSE

1
5 4. R-Squared (R2 Score)
Measures how much variance is explained by the model.
SSres
R2 = 1 −
SStot
Where,
X
SSres = (yi − ŷi )2

X
SStot = (yi − ȳ)2
Interpretation:

• R2 = 1 : Perfect model

• R2 = 0 : No explanatory power

• R2 < 0 : Poor model

6 Fully Solved Numerical Example

Given:

Actual (y) Predicted (ŷ)

10 12
20 18
30 29
40 41

Step 1: Calculate Errors

y ŷ y − ŷ |y − ŷ| (y − ŷ)2
10 12 -2 2 4
20 18 2 2 4
30 29 1 1 1
40 41 -1 1 1

Sum of absolute errors = 6

Sum of squared errors = 10

Step 2: MAE
6
M AE = = 1.5
4

2
Step 3: MSE
10
M SE = = 2.5
4

Step 4: RMSE
√
RM SE = 2.5 = 1.58

Step 5: R2 Calculation
Mean of actual values:
10 + 20 + 30 + 40
ȳ = = 25
4
Calculate SStot :

(10 − 25)2 = 225

(20 − 25)2 = 25
(30 − 25)2 = 25
(40 − 25)2 = 225

SStot = 225 + 25 + 25 + 225 = 500

We already have:

SSres = 10

10
R2 = 1 −
500

R2 = 1 − 0.02 = 0.98
R2 = 0.98 (98% variance explained)

7 Comparison of Metrics
Metric Formula Type Outlier Sensitivity Unit
MAE Absolute error Low Same as output
MSE Squared error High Squared unit
RMSE Square root of MSE High Same as output
R2 Variance ratio No Unitless

3
8 Conclusion
Regression evaluation metrics help measure prediction errors. MAE, MSE, RMSE, and
R2 are commonly used metrics. The choice depends on the problem type and business
requirement.

4
 Distance Measures in Machine Learning

1 Introduction
A Distance Measure is a mathematical formula used to calculate the similarity or dissim-
ilarity between two data points.

• Smaller distance ⇒ More similar

• Larger distance ⇒ Less similar

Distance measures are widely used in:

• Clustering Algorithms (K-Means)

• K-Nearest Neighbors (KNN)

• Pattern Recognition

• Recommendation Systems

2 1. Euclidean Distance
The most commonly used distance measure.

Formula (2-Dimension)
For two points A(x1 , y1 ) and B(x2 , y2 ):
p
d= (x2 − x1 )2 + (y2 − y1 )2

Formula (n-Dimension)
v
u n
uX
d = t (xi − yi )2
i=1

1
Example
Given A(2, 3) and B(5, 7):
p
d= (5 − 2)2 + (7 − 3)2
√ √
d= 9 + 16 = 25 = 5

3 2. Manhattan Distance (City Block Distance)

Formula (2-Dimension)
d = |x2 − x1 | + |y2 − y1 |

Formula (n-Dimension)
n
X
d= |xi − yi |
i=1

Example
Given A(2, 3) and B(5, 7):

d = |5 − 2| + |7 − 3|

d=3+4=7

4 3. Minkowski Distance
Generalized form of distance measure.

n
!1/p
X
d= |xi − yi |p
i=1
Where:
• p = 1 ⇒ Manhattan Distance
• p = 2 ⇒ Euclidean Distance

5 4. Chebyshev Distance
Maximum absolute difference between coordinates.

d = max(|xi − yi |)

2
Example
Given A(2, 3) and B(5, 7):

d = max(3, 4) = 4

6 5. Hamming Distance
Used for binary or categorical data.
It counts the number of positions where two strings differ.

Example
101110
100100

Difference positions = 2
Hamming Distance = 2

7 6. Cosine Similarity and Cosine Distance

Measures angle between two vectors.
A·B
Cosine Similarity =
|A||B|

Cosine Distance = 1 − Cosine Similarity

Applications:

• Text Mining

• NLP

• Recommendation Systems

8 7. Mahalanobis Distance
Accounts for correlation between variables.
p
d= (x − µ)T S −1 (x − µ)
Where:

• µ = Mean vector

• S = Covariance matrix

3
 Used in:

• Multivariate Outlier Detection

• Pattern Recognition

9 Comparison of Distance Measures

Distance Type Formula Type Applications
Euclidean Square root of squared differences K-Means
Manhattan Sum of absolute differences KNN
Minkowski General form General ML
Chebyshev Maximum difference Chess/Grid Problems
Hamming Binary mismatch count Text/Binary Data
Cosine Angle-based similarity NLP
Mahalanobis Covariance-based Outlier Detection

10 Conclusion
Distance measures are essential in machine learning for calculating similarity between data
points. The choice of distance metric depends on the type of data and the problem domain.

4
 Bias and Variance in Machine Learning

1 Overview
Bias and variance are two fundamental sources of error in machine learning models. Under-
standing these concepts helps in model selection, tuning, and avoiding underfitting/overfitting.
The total error of a model can be expressed as:

Total Error = Bias2 + Variance + Irreducible Error

Where:

• Bias2 : Error due to wrong assumptions in the model

• Variance: Error due to sensitivity to training data

• Irreducible Error: Noise in the data that cannot be modeled

2 Bias
2.1 Definition
Bias measures how far the predicted values are from the true values on average. High bias
leads to strong assumptions about the data, causing underfitting.

2.2 Example
Fitting a straight line to non-linear data:

• The line cannot capture the curve

• Predictions are systematically off

• High bias

1
2.3 Key Points
• High bias → underfitting

• Low bias → model captures complex patterns

• Bias depends on model capacity (complexity)

3 Variance
3.1 Definition
Variance measures how much predictions change when the model is trained on different
datasets. High variance leads to sensitivity to training data, causing overfitting.

3.2 Example
Fitting a 10th-degree polynomial to 5 data points:

• Perfect fit to training points

• Small changes in training data lead to large changes in predictions

• High variance

3.3 Key Points

• High variance → overfitting

• Low variance → stable predictions

• Variance depends on model complexity and data size

4 Bias-Variance Trade-off
Concept Effect on Error Model Behavior
High Bias High training and test error Underfitting
High Variance Low training error, high test error Overfitting
Balanced Moderate training and test error Good generalization

Trade-off: Increasing model complexity reduces bias but increases variance. Goal is to
find the sweet spot for best generalization.

2
4.1 Graphical Illustration
• Bias2 decreases as model complexity increases

• Variance increases as model complexity increases

• Total error follows a U-shaped curve

5 Mathematical Expression
For a target variable Y and prediction fˆ(X):

E[(Y − fˆ(X))2 ] = (Bias[fˆ(X)])2 + Var[fˆ(X)] + σ2

|{z}
| {z } | {z }
Bias squared Variance Irreducible error

Where:
Bias[fˆ(X)] = E[fˆ(X)] − f (X)

V ar[fˆ(X)] = E[(fˆ(X) − E[fˆ(X)])2 ]

—

6 Implications in Machine Learning

Scenario Example Remedy

High Bias Linear regression on non-linear data Use complex model, add features
High Variance Deep decision tree overfitting training data Regularization, prune tree, more data
Balanced Correctly fits and generalizes Optimal model selection
—

7 Strategies to Handle Bias and Variance

7.1 Reduce Bias
• Increase model complexity (non-linear models, more features)

• Use advanced algorithms (ensemble methods)

3
7.2 Reduce Variance
• Regularization (Ridge, Lasso, Dropout)

• Pruning trees

• Use more training data

• Ensemble methods (Bagging, Random Forests)

8 Summary
• Bias: Error from incorrect assumptions → underfitting

• Variance: Error from sensitivity to training data → overfitting

• Bias-Variance Trade-off: Balance needed for good generalization

• Total Error: Bias2 + Variance + Irreducible error

4
 Overfitting and Underfitting in Machine Learning

1 Overview
In machine learning, the goal is to build models that generalize well on unseen data. Poor
generalization usually occurs due to overfitting or underfitting:

Total Error = Bias2 + Variance + Irreducible Error

• Underfitting: High bias → model too simple

• Overfitting: High variance → model too complex

2 Underfitting
2.1 Definition
Underfitting occurs when a model is too simple to capture the underlying patterns in the
data. - Performs poorly on both training and test data - Associated with high bias

2.2 Causes
• Model too simple (e.g., linear model for non-linear data)

• Too few features

• Excessive regularization

2.3 Effects
• High training error

• High test error

• Poor predictive performance

1
2.4 Example
Using a linear regression model to fit a dataset that is clearly quadratic or non-linear.
—

3 Overfitting
3.1 Definition
Overfitting occurs when a model is too complex and fits the training data too closely,
including noise. - Performs very well on training data but poorly on unseen test
data - Associated with high variance

3.2 Causes
• Model too complex (e.g., deep decision trees, high-degree polynomials)
• Too many features with too few training samples
• Insufficient regularization

3.3 Effects
• Low training error
• High test error
• Poor generalization

3.4 Example
Fitting a 10th-degree polynomial to only 5 data points:
• Perfectly fits training points
• Small changes in data result in large prediction errors
• High variance
—

4 Detection Methods
Issue Training Error Test Error Bias/Variance Indicator
Underfitting High High High Bias
Overfitting Low High High Variance
Good Fit Low Low Balanced
- Learning Curves: Plot training vs validation error to detect underfitting/overfitting.
—

2
5 Remedies
5.1 For Underfitting
• Increase model complexity (e.g., move from linear to non-linear models)

• Add more features

• Reduce regularization

5.2 For Overfitting

• Reduce model complexity (prune trees, reduce polynomial degree)

• Use regularization techniques (Ridge, Lasso, Dropout)

• Collect more training data

• Use ensemble methods (Bagging, Random Forest, Boosting)

6 Summary Table

Aspect Underfitting Overfitting

Model Complexity Too low Too high
Training Error High Low
Test Error High High
Bias/Variance High Bias High Variance
Solution Increase complexity, add features Reduce complexity, regularization, more data
—

7 Visual Illustration
• Underfitting: Model fails to capture trend → line too flat

• Overfitting: Model captures all fluctuations → line too wiggly

• Good Fit: Model captures trend without following noise

3
 K-Nearest Neighbors (KNN) Algorithm

1 Overview
KNN is a supervised machine learning algorithm used for both classification and
regression tasks. It is an instance-based or lazy learning algorithm, meaning it doesn’t
build an explicit model; it stores training data and makes predictions on the fly.
Key idea: Predict the output of a new data point based on the majority label (for
classification) or average value (for regression) of its K nearest neighbors.

2 How KNN Works

2.1 Step 1: Choose K
• K is the number of nearest neighbors considered.
• Small K → sensitive to noise (overfitting).
• Large K → smoother decision boundary (underfitting).

2.2 Step 2: Compute Distance

Calculate distance between the query point and all points in the training set.
Common distance metrics:
• Euclidean Distance (continuous features):
v
u n
uX
d(p, q) = t (pi − qi )2
i=1

• Manhattan Distance: n
X
d(p, q) = |pi − qi |
i=1

• Minkowski Distance (generalization of Euclidean and Manhattan):

n
!1/p
X
p
d(p, q) = |pi − qi |
i=1

• Hamming Distance (categorical data)

1
2.3 Step 3: Find K Nearest Neighbors
Sort all distances from the query point to training points. Pick the K points with the smallest
distances.

2.4 Step 4: Make Prediction

• Classification: Majority voting among K neighbors. Weighted voting can be used.

• Regression: Average (or weighted average) of the neighbors’ values.

3 Advantages of KNN
• Simple and intuitive

• Non-parametric (no assumptions about data distribution)

• Can handle multi-class classification

• Flexible with distance metrics

4 Disadvantages of KNN
• Computationally expensive at prediction time

• Sensitive to irrelevant or correlated features

• Sensitive to feature scaling (needs normalization)

• Poor performance on high-dimensional data (curse of dimensionality)

5 Important Considerations
5.1 Feature Scaling
KNN relies on distance; features with larger ranges dominate. Apply standardization or
min-max scaling.

5.2 Choosing K
Use cross-validation to select the best K. Odd numbers are preferred for classification to
avoid ties.

2
5.3 Distance Metric Selection
• Continuous → Euclidean or Minkowski

• Categorical → Hamming

• Mixed → Gower distance

5.4 Handling Imbalanced Data

Weighted voting can give closer neighbors more influence. Consider distance-weighted KNN
to reduce bias.

6 Applications
• Classification: Handwriting recognition, disease prediction, spam detection

• Regression: House price prediction, stock price prediction

• Recommendation systems: Collaborative filtering based on user similarity

7 Summary
Aspect Description
Type Supervised, Instance-based
Use Classification & Regression
Key Parameter K (number of neighbors)
Distance Metric Euclidean, Manhattan, Minkowski, Hamming
Pros Simple, Non-parametric, Flexible
Cons Slow for large data, sensitive to scaling & noise, poor in high dimensions

3
 K-Nearest Neighbors (KNN) Numerical Example

1 Problem Statement
We have the following training data with 2 features and 2 classes:

Point x1 x2 Class
A 1 2 0
B 2 3 0
C 3 3 1
D 6 5 1

We want to predict the class of a new point P = (3, 4) using KNN with K = 3.

2 Step 1: Compute Distances

2.1 1a. Euclidean Distance
q
dE (P, Q) = (xP1 − xQ 2 P Q 2
1 ) + (x2 − x2 )

p √ √
dE (P, A) = (3 − 1)2 + (4 − 2)2 = 4 + 4 = 8 ≈ 2.828
p √ √
dE (P, B) = (3 − 2)2 + (4 − 3)2 = 1 + 1 = 2 ≈ 1.414
p √ √
dE (P, C) = (3 − 3)2 + (4 − 3)2 = 0+1= 1=1
p √ √
dE (P, D) = (3 − 6)2 + (4 − 5)2 = 9 + 1 = 10 ≈ 3.162

Point Euclidean Distance

A 2.828
B 1.414
C 1
D 3.162

1
2.2 1b. Manhattan Distance
dM (P, Q) = |xP1 − xQ P Q
1 | + |x2 − x2 |

dM (P, A) = |3 − 1| + |4 − 2| = 2 + 2 = 4
dM (P, B) = |3 − 2| + |4 − 3| = 1 + 1 = 2
dM (P, C) = |3 − 3| + |4 − 3| = 0 + 1 = 1
dM (P, D) = |3 − 6| + |4 − 5| = 3 + 1 = 4

Point Manhattan Distance

A 4
B 2
C 1
D 4

3 Step 2: Find K Nearest Neighbors (K = 3)

3.1 2a. Euclidean Distance
- Sorted distances: C (1), B (1.414), A (2.828), D (3.162) - 3 nearest neighbors: C, B, A -
Their classes: 1 (C), 0 (B), 0 (A)
Majority class: 0 ⇒ Predicted class = 0

3.2 2b. Manhattan Distance

- Sorted distances: C (1), B (2), A (4), D (4) - 3 nearest neighbors: C, B, A - Their classes:
1 (C), 0 (B), 0 (A)
Majority class: 0 ⇒ Predicted class = 0

4 Step 3: Conclusion
• Predicted class using Euclidean distance: 0

• Predicted class using Manhattan distance: 0

Both distance metrics give the same prediction in this example.

2
 Naı̈ve Bayes Algorithm

1 Introduction
Naı̈ve Bayes is a supervised classification algorithm based on Bayes’ Theorem with a strong
assumption that features are conditionally independent given the class.
It is called “naı̈ve” because it assumes:

P (X1 , X2 , X3 | C) = P (X1 | C) · P (X2 | C) · P (X3 | C)

Even though features may be dependent in real-world problems.

2 Bayes’ Theorem
P (X | C) · P (C)
P (C | X) =
P (X)
Where:

• C = Class label

• X = Feature vector

• P (C | X) = Posterior probability

• P (C) = Prior probability

• P (X | C) = Likelihood

• P (X) = Evidence

Since P (X) is constant for all classes:

P (C | X) ∝ P (X | C) · P (C)

1
3 Algorithm Steps
1. Calculate prior probabilities P (C)

2. Calculate conditional probabilities P (Xi | C)

3. Multiply likelihoods with prior

4. Compare posterior probabilities

5. Assign class with maximum probability

4 Fully Solved Numerical Example (3 Attributes)

Problem
Predict whether a person will Buy Laptop (Yes/No) based on:

• Age (Young / Middle / Senior)

• Income (High / Medium / Low)

• Student (Yes / No)

Training Dataset
Age Income Student Buy
Young High No No
Young High No No
Middle High No Yes
Senior Medium No Yes
Senior Low Yes Yes
Senior Low Yes No
Middle Low Yes Yes
Young Medium No No
Young Low Yes Yes
Senior Medium Yes Yes

Classify the Tuple

X = (Age = Y oung, Income = M edium, Student = Y es)

2
5 Step 1: Prior Probabilities
Total records = 10
Buy = Yes → 6
Buy = No → 4
6
P (Y es) = = 0.6
10
4
P (N o) = = 0.4
10

6 Step 2: Conditional Probabilities

For Class = Yes (6 records)
1
P (Y oung | Y es) =
6
2
P (M edium Income | Y es) =
6
4
P (Student = Y es | Y es) =
6

For Class = No (4 records)

3
P (Y oung | N o) =
4
1
P (M edium Income | N o) =
4
1
P (Student = Y es | N o) =
4

7 Step 3: Apply Naı̈ve Bayes Formula

Probability for YES
   
1 2 4
P (Y es | X) ∝ (0.6)
6 6 6

= (1/6)(1/3)(2/3)(0.6)

= 0.0222

3
Probability for NO
   
3 1 1
P (N o | X) ∝ (0.4)
4 4 4

= 0.01875

8 Step 4: Comparison
P (Y es | X) = 0.0222

P (N o | X) = 0.01875
Since:

0.0222 > 0.01875

Buy = Y es

9 Laplace Smoothing
If any probability becomes zero:
Count + 1
P (Xi | C) =
T otal + N umber of possible values
This prevents zero probability problems.

10 Types of Naı̈ve Bayes

• Gaussian Naı̈ve Bayes (Continuous data)

• Multinomial Naı̈ve Bayes (Text classification)

• Bernoulli Naı̈ve Bayes (Binary features)

11 Advantages
• Simple and easy to implement

• Fast computation

• Works well with high-dimensional data

• Good for text classification

4
12 Disadvantages
• Strong independence assumption

• Performs poorly if features are highly correlated

5
 Naı̈ve Bayes Classifier
Numerical Using Laplace Smoothing

Problem Statement
Predict whether a person will Buy Laptop (Yes/No) based on:
• Age = {Young, Middle, Senior}
• Income = {High, Medium, Low}
• Student = {Yes, No}

Training Dataset
Age Income Student Buy
Young High No No
Young High No No
Middle High No Yes
Senior Medium No Yes
Senior Low Yes Yes
Senior Low Yes No
Middle Low Yes Yes
Young Medium No No
Young Low Yes Yes
Senior Medium Yes Yes
Classify the tuple:

X = (Age = Y oung, Income = M edium, Student = Y es)

Step 1: Prior Probabilities

Total records = 10
6
P (Y es) = = 0.6
10
4
P (N o) = = 0.4
10
1
Step 2: Laplace Smoothing Formula
Count + 1
P (Xi |C) =
T otal in class + k
Where k is the number of possible values of that attribute.

• Age → 3 values

• Income → 3 values

• Student → 2 values

Step 3: Conditional Probabilities

For Class = Yes (6 records)
1+1 2
P (Y oung|Y es) = =
6+3 9
2+1 3
P (M edium|Y es) = =
6+3 9
4+1 5
P (Student = Y es|Y es) = =
6+2 8

For Class = No (4 records)

3+1 4
P (Y oung|N o) = =
4+3 7
1+1 2
P (M edium|N o) = =
4+3 7
1+1 2
P (Student = Y es|N o) = =
4+2 6

Step 4: Apply Naı̈ve Bayes Formula

P (C|X) ∝ P (Age|C) · P (Income|C) · P (Student|C) · P (C)

2
For YES
   
2 3 5
P (Y es|X) ∝ (0.6)
9 9 8
2×3×5
= × 0.6
9×9×8
30
= × 0.6
648

= 0.0463 × 0.6

= 0.02778

For NO
   
4 2 2
P (N o|X) ∝ (0.4)
7 7 6
4×2×2
= × 0.4
7×7×6
16
= × 0.4
294

= 0.0544 × 0.4

= 0.02176

Step 5: Comparison
P (Y es|X) = 0.02778

P (N o|X) = 0.02176
Since

0.02778 > 0.02176

Buy = Y es

3
Conclusion
After applying Laplace Smoothing, the predicted class for the given tuple is:

Y es
Laplace smoothing prevents zero probabilities and improves model robustness.

4
 Decision Tree (DT) Algorithm

1 Overview
Decision Tree (DT) is a supervised machine learning algorithm used for classification
and regression. It models decisions as a tree structure:
• Nodes: Features (attributes) or tests
• Edges/Branches: Outcomes of tests
• Leaves: Final predicted class (classification) or value (regression)
Key idea: Split data into subsets based on features to maximize information gain (or
minimize impurity).

2 How Decision Tree Works

2.1 Step 1: Select the Best Feature
Use a splitting criterion to select the best feature to split the dataset:
• Information Gain (IG) - based on Entropy
• Gini Index - measures impurity
• Variance Reduction - for regression

2.2 Step 2: Split the Dataset

Partition the dataset based on selected feature values and create child nodes.

2.3 Step 3: Repeat Recursively

Repeat feature selection and splitting for each child node until:
• All samples belong to the same class
• No features left
• Maximum tree depth reached
• Minimum samples per leaf reached

1
2.4 Step 4: Prediction
• Classification: Traverse tree using feature tests until a leaf node → return class

• Regression: Return mean value of the leaf node

3 Key Concepts
3.1 Entropy (Classification)
Measures uncertainty in a dataset:
c
X
Entropy(S) = − pi log2 (pi )
i=1

• pi = proportion of class i in dataset S

• c = number of classes

• Entropy = 0 → pure dataset, Entropy = 1 → maximum uncertainty (binary class)

3.2 Information Gain

Reduction in entropy after a dataset is split on feature A:
X |Sv |
IG(S, A) = Entropy(S) − Entropy(Sv )
|S|
v∈V alues(A)

3.3 Gini Index

Impurity measure of a node:
c
X
Gini(S) = 1 − p2i
i=1

Lower Gini → purer node. Commonly used in CART.

3.4 Variance Reduction (Regression)

Split feature that minimizes variance of target values in child nodes.

4 Advantages
• Easy to understand and interpret

• Handles numerical and categorical data

• Requires little preprocessing (no normalization needed)

2
 • Handles multi-class problems
• Non-parametric

5 Disadvantages
• Prone to overfitting (especially deep trees)
• Sensitive to small changes in data (high variance)
• Can be biased toward features with more levels
• Less effective for continuous variables if not discretized

6 Decision Tree Algorithms

• ID3 - uses Information Gain, categorical data only
• C4.5 - extension of ID3, handles continuous/missing values, uses Gain Ratio
• CART - uses Gini Index for classification, binary trees, can do regression

7 Pruning
Reduces overfitting by cutting branches that do not improve accuracy:
• Pre-pruning: Stop tree growth early (max depth, min samples per leaf)
• Post-pruning: Grow full tree, then remove branches

8 Example: Classification
Dataset:
Outlook Temp Humidity Wind Play Tennis
Sunny Hot High Weak No
Sunny Hot High Strong No
Overcast Hot High Weak Yes
Rain Mild High Weak Yes
Rain Cool Normal Weak Yes
Steps:
1. Compute Entropy of dataset
2. Compute Information Gain for features: Outlook, Temp, Humidity, Wind
3. Choose feature with highest IG → split
4. Repeat until leaves are pure

3
9 Implementation Notes

Listing 1: Decision Tree Classifier Example

from s k l e a r n . t r e e import D e c i s i o n T r e e C l a s s i f i e r
from s k l e a r n import t r e e

# Create D e c i s i o n Tree c l a s s i f i e r
c l f = D e c i s i o n T r e e C l a s s i f i e r ( c r i t e r i o n= ’ g i n i ’ , max depth=3)
c l f = c l f . f i t ( X train , y t r a i n )

# Predict
y pred = c l f . p r e d i c t ( X test )

# Optional : Visualize tree

t r e e . p l o t t r e e ( c l f , f e a t u r e n a m e s=f e a t u r e n a m e s , c l a s s n a m e s=c l a s s n a m e s , f i l l

10 Summary Table
Aspect Description
Type Supervised, Classification & Regression
Tree Nodes Decision Nodes (features) & Leaf Nodes (class/value)
Splitting Criteria Information Gain, Gini Index, Variance Reduction
Pros Easy to interpret, Handles categorical & numerical, Non-parametric
Cons Prone to overfitting, Sensitive to noise, Can be biased, High variance
Algorithms ID3, C4.5, CART
Pruning Pre-pruning, Post-pruning

4
 Decision Tree: Numerical Example

1 Problem Statement
We have a dataset of 5 days for playing tennis:

Day Outlook Temp Humidity Wind

Play Tennis
1 Sunny Hot High Weak
No
2 Sunny Hot High Strong
No
3 Overcast Hot High Weak
Yes
4 Rain Mild High Weak
Yes
5 Rain Cool Normal Weak
Yes

We want to build a Decision Tree to predict Play Tennis.

—

2 Step 1: Compute Entropy of the Dataset

Number of Yes = 3, No = 2

Entropy(S) = −pY es log2 (pY es ) − pN o log2 (pN o )

3 2
pY es = , pN o =
5 5
3 3 2 2
Entropy(S) = − log2 − log2 ≈ 0.971
5 5 5 5
—

1
3 Step 2: Compute Entropy for Each Feature
3.1 Feature: Outlook
Values: Sunny, Overcast, Rain

• Sunny (2 samples: No, No) → Entropy = 0

• Overcast (1 sample: Yes) → Entropy = 0

• Rain (2 samples: Yes, Yes) → Entropy = 0

Weighted entropy:
2 1 2
Entropy(S, Outlook) = ·0+ ·0+ ·0=0
5 5 5
Information Gain:
IG(S, Outlook) = 0.971 − 0 = 0.971
—

3.2 Feature: Temp

Values: Hot, Mild, Cool

• Hot (3 samples: No, No, Yes) → Yes=1, No=2

1 1 2 2
Entropy(Hot) = − log2 − log2 ≈ 0.918
3 3 3 3

• Mild (1 sample: Yes) → Entropy = 0

• Cool (1 sample: Yes) → Entropy = 0

Weighted entropy:
3 1 1
Entropy(S, T emp) = · 0.918 + · 0 + · 0 ≈ 0.551
5 5 5

IG(S, T emp) = 0.971 − 0.551 = 0.42

—

2
3.3 Feature: Humidity
Values: High, Normal

• High (4 samples: No, No, Yes, Yes) → Yes=2, No=2

2 2 2 2
Entropy(High) = − log2 − log2 = 1
4 4 4 4

• Normal (1 sample: Yes) → Entropy = 0

Weighted entropy:
4 1
Entropy(S, Humidity) = · 1 + · 0 = 0.8
5 5

IG(S, Humidity) = 0.971 − 0.8 = 0.171

—

3.4 Feature: Wind

Values: Weak, Strong

• Weak (4 samples: No, Yes, Yes, Yes) → Yes=3, No=1

3 3 1 1
Entropy(W eak) = − log2 − log2 ≈ 0.811
4 4 4 4

• Strong (1 sample: No) → Entropy = 0

Weighted entropy:
4 1
Entropy(S, W ind) = · 0.811 + · 0 ≈ 0.649
5 5

IG(S, W ind) = 0.971 − 0.649 = 0.322

—

4 Step 3: Choose the Best Feature

Feature Information Gain (IG)
Outlook 0.971
Temp 0.42
Humidity 0.171
Wind 0.322

Best feature: Outlook → root node

—

3
5 Step 4: Split by Outlook
• Sunny → All No → Leaf = No

• Overcast → All Yes → Leaf = Yes

• Rain → All Yes → Leaf = Yes

6 Step 5: Decision Tree Diagram

 Support Vector Machine (SVM)

1 Introduction
Support Vector Machine (SVM) is a supervised machine learning algorithm used for:

• Classification

• Regression (SVR)

It works by finding a hyperplane that best separates the classes, aiming to maximize
the margin between them.

2 Key Concepts
• Hyperplane: A line (2D), plane (3D), or n-dimensional surface that separates classes.

w·x+b=0

• Margin: Distance between the hyperplane and the nearest data points. SVM maxi-
mizes this.

• Support Vectors: Points closest to the hyperplane that define the margin.

• Linearly separable: Classes can be separated by a straight line (2D) or hyperplane

(nD).

• Kernel Trick: Allows SVM to handle non-linear data by mapping to higher dimen-
sions.

3 SVM Formulation
2
Maximize subject to yi (w · xi + b) ≥ 1
∥w∥
Where:

• w = weight vector

• b = bias

1
 • xi = feature vector

• yi ∈ {+1, −1} = class label

4 1D SVM Example (Fully Solved)

Dataset
x y
1 -1
3 -1
4 +1
6 +1

Step 1: Visualize Data

Class -1: 1, 3
Class +1: 4, 6

Step 2: Determine Hyperplane

Hyperplane lies midway between closest points: 3 (-1) and 4 (+1)
3+4
x= = 3.5
2

Step 3: Margin
Margin = 4 − 3.5 = 0.5
Support vectors = 3 and 4

Step 4: SVM Decision Function

f (x) = sign(x − 3.5)
- If x < 3.5 → y = −1 - If x > 3.5 → y = +1
—

2
5 2D SVM Example (Fully Solved)
Dataset
x1 x2 y
1 2 -1
2 3 -1
3 3 +1
5 4 +1

Step 1: Visualize Data

- Class -1: (1,2), (2,3) - Class +1: (3,3), (5,4)

Step 2: Assume Linear Hyperplane

Equation:
w 1 x1 + w 2 x2 + b = 0

Step 3: Use Support Vectors

Support vectors are closest points:
- -1 class: (2,3) - +1 class: (3,3)
Midpoint:  
2+3 3+3
, = (2.5, 3)
2 2
3−3
Slope of line connecting points = 3−2 = 0 (horizontal)
Perpendicular bisector → vertical line:

x1 = 2.5

Step 4: SVM Hyperplane

x1 = 2.5
Decision function:
f (x1 , x2 ) = sign(x1 − 2.5)
- If x1 < 2.5 → y = −1 - If x1 > 2.5 → y = +1

Step 5: Margin
Margin = |2.5 − 2| = 0.5
—

3
6 Key Points
- SVM works in 1D, 2D, and higher dimensions. - Support vectors determine the hyperplane.
- Margin is maximized for optimal separation. - For non-linear data, use kernel functions
(Linear, Polynomial, RBF).
—

7 Advantages
• Effective in high dimensions

• Works well with clear margin

• Robust to overfitting (with regularization)

8 Disadvantages
• Not suitable for very large datasets (computationally expensive)

• Poor performance if classes overlap

• Choosing the right kernel can be tricky

4
 Support Vector Machine (SVM)

Definition
Support Vector Machine (SVM) is a supervised machine learning algorithm used for
classification and regression. It finds an optimal hyperplane that separates data
points of different classes with the maximum margin. Only the points closest to the
hyperplane, called support vectors, define the decision boundary.
—

1. Working Principle
Hyperplane equation:
w·x+b=0
Classification constraint:

yi (w · xi + b) ≥ 1, yi ∈ {−1, +1}

Objective: Maximize margin

2
M=
∥w∥
Equivalently, minimize:
1
min ∥w∥2 s.t. yi (w · xi + b) ≥ 1
w,b 2

Soft margin SVM: Allows misclassification using slack variables ξi ≥ 0:

yi (w · xi + b) ≥ 1 − ξi
N
1 X
min ∥w∥2 + C ξi
w,b 2
i=1

C = penalty parameter controlling trade-off between margin and misclassification.

Kernel trick: Maps data to higher-dimensional space for non-linear separation:

K(xi , xj ) = ϕ(xi ) · ϕ(xj )

2. Types of SVM
—

1
Type Description
Linear SVM For linearly separable data; hyperplane is straight line/plane
Non-linear SVM Uses kernel functions to separate non-linear data
Soft Margin SVM Allows misclassification using slack variables ξi and penalty C
Support Vector Regression (SVR) Adaptation of SVM for regression (continuous outputs)

3. Advantages
• Effective in high-dimensional spaces

• Only support vectors needed → memory efficient

• Can handle non-linear data using kernels

• Robust to overfitting if parameters are tuned properly

4. Limitations
• Computationally expensive for very large datasets

• Sensitive to kernel choice, C, and γ

• Less interpretable compared to simpler models (e.g., logistic regression)

5. Applications of SVM
• Text classification: spam detection, sentiment analysis

• Image recognition: face detection, handwriting recognition

• Bioinformatics: gene classification, protein function prediction

• Medical diagnosis: cancer detection, disease classification

• Finance: fraud detection, stock trend prediction

• Other domains: object detection, speech recognition

2
6. Practical Tips
• Scale/normalize features before training

• Use cross-validation to select C and kernel parameters

• Start with linear kernel; use RBF for complex boundaries

• For imbalanced datasets, adjust class weights

Conclusion
SVM is a powerful supervised learning algorithm for classification and regression.
Its strength lies in maximizing the margin and using support vectors. With kernel
functions, it can handle both linear and non-linear data, making it widely applicable
in text, image, bioinformatics, finance, and medical fields.

3
 Random Forest Algorithm

1 Introduction
Random Forest is an ensemble learning algorithm used for classification and regression. It
builds multiple Decision Trees and combines their outputs to improve accuracy and reduce
overfitting.
It is based on:

• Bagging (Bootstrap Aggregation)

• Random Feature Selection

2 Why Random Forest?

A single Decision Tree:

• Has high variance

• Can overfit

Random Forest:

• Reduces variance

• Improves accuracy

• Handles large datasets well

• Works with missing values

3 Working of Random Forest

For classification:

1. Draw multiple bootstrap samples from dataset.

2. For each sample:

• Build a decision tree.

1
 • At each split, randomly select subset of features.

3. Each tree gives a class prediction.

4. Final prediction = Majority Voting.

4 Key Concepts
4.1 Bootstrap Sampling
Sampling with replacement.

4.2 Random Feature Selection

At each split, select a random subset of features instead of all features.

4.3 Out-of-Bag (OOB) Error

Data not selected in bootstrap sample is used for validation.

5 Algorithm Steps
1. Choose number of trees T .

2. For each tree:

• Take bootstrap sample.

• Grow decision tree using random feature subsets.

3. Aggregate predictions.

4. Use majority vote (classification).

6 Fully Solved Numerical Example

Problem
Classify whether a person will Play Game (Yes/No) based on:

• Weather (Sunny/Rainy)

• Temperature (Hot/Mild)

2
Dataset (6 Records)
ID Weather Temp Play
1 Sunny Hot No
2 Sunny Mild No
3 Rainy Hot Yes
4 Rainy Mild Yes
5 Sunny Hot No
6 Rainy Mild Yes

We build 3 trees.

7 Step 1: Bootstrap Samples

Tree 1 Sample
{1, 2, 3, 4, 4, 5}

Tree 2 Sample
{2, 3, 4, 5, 6, 6}

Tree 3 Sample
{1, 3, 3, 4, 5, 6}

8 Step 2: Build Decision Trees

Assume at each split we randomly choose only 1 feature.

Tree 1
Random feature: Weather

• Sunny → No

• Rainy → Yes

Tree 2
Random feature: Temperature

• Hot → No

• Mild → Yes

3
Tree 3
Random feature: Weather

• Sunny → No

• Rainy → Yes

9 Step 3: Prediction
Classify:

X = (W eather = Sunny, T emp = M ild)

Tree predictions:

• Tree 1 → No

• Tree 2 → Yes

• Tree 3 → No

10 Step 4: Majority Voting

Votes:

No = 2

Y es = 1

P lay = N o

11 Mathematical Insight
For classification:

ŷ = mode(T1 (x), T2 (x), . . . , Tn (x))

For regression:
n
1X
ŷ = Ti (x)
n i=1

4
12 Advantages
• High accuracy
• Reduces overfitting
• Handles large datasets
• Works with missing data
• Handles high-dimensional data

13 Disadvantages
• Computationally expensive
• Less interpretable
• Large memory usage

14 Important Parameters
• Number of trees (n estimators)
• Maximum depth
• Minimum samples split
• Maximum features

15 Comparison: Decision Tree vs Random Forest

Decision Tree Random Forest
High variance Low variance
Prone to overfitting Less overfitting
Easy to interpret Harder to interpret
Single model Ensemble model

16 Conclusion
Random Forest improves prediction accuracy by:
• Combining multiple decision trees
• Using bootstrap sampling
• Using random feature selection
• Applying majority voting

5
 Random Forest (Regression)

Problem Statement
Predict the House Price (in $1000s) based on Size ([Link].) and Bedrooms.

Training Dataset
ID Size ([Link].) Bedrooms Price ($1000s)
1 1000 2 200
2 1200 3 240
3 1500 3 300
4 1700 4 360
5 1300 2 260

Predict Price for:

X = (Size = 1400, Bedrooms = 3)

Step 1: Bootstrap Samples

Random Forest regression uses bootstrapping to build multiple trees. Assume 3 trees:

• Tree 1 Sample: {1, 2, 2, 3, 5}

• Tree 2 Sample: {2, 3, 4, 4, 5}

• Tree 3 Sample: {1, 1, 3, 4, 5}

Step 2: Build Regression Trees

We use simple decision tree splits based on minimizing mean squared error. Each leaf predicts
the average price of its records.

1
Tree 1
Sample: {1, 2, 2, 3, 5}
• Size ≤ 1250 → Average Price = (200 + 240 + 240)/3 = 226.7
• Size > 1250 → Average Price = (300 + 260)/2 = 280
Input X: Size=1400 → Size > 1250 → Prediction: 280

Tree 2
Sample: {2, 3, 4, 4, 5}
• Size ≤ 1500 → Average Price = (240 + 300)/2 = 270
• Size > 1500 → Average Price = (360 + 260)/2 = 310
Input X: Size=1400 → Size ≤ 1500 → Prediction: 270

Tree 3
Sample: {1, 1, 3, 4, 5}
• Size ≤ 1300 → Average Price = (200 + 200 + 300)/3 = 233.3
• Size > 1300 → Average Price = (360 + 260)/2 = 310
Input X: Size=1400 → Size > 1300 → Prediction: 310

Step 3: Aggregate Predictions

Random Forest regression aggregates predictions by averaging:
280 + 270 + 310 860
Ŷ = = ≈ 286.7
3 3

Step 4: Final Prediction

Predicted Price = $286.7k

Summary of Steps
1. Draw bootstrap samples.
2. Build regression tree for each sample.
3. Predict using each tree.
4. Aggregate predictions using average.

2
Advantages
• Reduces overfitting

• Handles nonlinear relationships

• Robust to outliers

• Can handle large datasets

Disadvantages
• Computationally expensive

• Harder to interpret

• Requires memory for multiple trees