ECE 208: AI/ML FOR

ELECTRONICS ENGINEERS

Presented by
Dr. Sibendu Samanta,
Assistant Professor,
Dept of ECE & Joint Faculty in CSE,
SRM University – AP.
AI, ML and DL:
Supervised ML:
• In supervised machine learning, we know about the data and the problem.

• There are two kinds of supervised learning: classification and regression.

• In a classification problem, we assign data to categories. For example,

given a client’s medical information, they test positive or negative for
diabetes. In classifications, our trained models, known as classifiers,
classify data points into different groups.

• If we instead wanted to solve a different problem, like predicting the

future value of GameStop stock given the stock market history, we’d turn
to a regression. In regression, we return numerical values. Given some
sentences, this is the percent likelihood the person is happy or sad.
Unsupervised ML:
• In unsupervised machine learning, our data is unlabeled.

• Clustering and Dimension reduction.

• In clustering, we learn more about data points as they are

clustered or grouped together.

• In dimension reduction, we plot data points across different

dimensions and feature sets to understand our data sets. This
allows for techniques like feature selection.
Reinforcement Learning:

• In reinforcement learning (RL), we are learning models over time.

• A common technique is to utilize deep learning with

reinforcement learning.

• Derive relationships between features of a data set that may not

otherwise be solved through human research.
Reinforcement Learning:
AI, ML and DL:
AI, ML and DL:

• [Link]
Statistics of AI in Everyday Life:
• According to a 2021 survey, 73% of consumers are already
using some form of AI in their daily lives.

• The global AI market size is expected to reach USD 266.92

billion by 2027.

• AI is being used in finance for fraud detection, with 75% of

banks planning to use AI in the next three years.

• AI is also being used in agriculture for crop monitoring, yield

prediction, and soil analysis, with the global market for AI in
agriculture expected to reach USD 1.4 billion by 2025.
Features of AI:
AI & ML Applications:
ML in Daily Lives:
1. Image recognition - Photograph face identification
2. Voice Assistant - Siri, Alexa, Echo, Google Home
3. Predictive Analysis
4. Medical Diagnosis
5. Traffic Alerts - Intelligent GPS technology of Google Maps

6. Video Surveillance - security and alarming situations

7. Chatbots for Support - Intelligent NLP algorithms and decision
trees
8. Google Translator
9. Stock Market Analysis
10. Self-Driving Cars - intelligent NLP algorithms and decision trees
11. Real Time Pricing
Threats of AI : Lack of Regulation

• Biased decision-making and discrimination.

• Displacement of jobs and lack of retraining support.

• Privacy and data security concerns.

• Spread misleading information through deepfake videos and social

media.
Key Takeaways:

• Artificial Intelligence (AI) simply means man-made Intelligence.

• Machine Learning is the study of Algorithms that achieves AI.

• Deep Learning is the study Deep Neural Networks, a learning

algorithm that is capable of achieving all cognitive tasks.
Activity:
Group 1: Linear Regression

Group 2: Logistic Regression

Group 3: Decision Tree and Random Forest

Group 4: Support Vector Machine

Group 5: Naive Bayes Classifier

Group 6: K-Nearest Neighbor

Group 7: K- Means Clustering and Hierarchical Clustering

Group 8: Artificial Neural Network

Scenario 1: Predicting Whether a Customer Will Buy a
Product

You have data on customer behavior and demographics, and you

want to predict whether a customer will purchase a new product
Scenario 2: Recommending Movies to Users Based on Their
Ratings

You have a user-item matrix of movie ratings and want to

recommend movies to users based on their preferences
• Scenario 3: Predicting Customer Churn in a Large Dataset

• You need to predict which customers are likely to leave your

service based on a large dataset with many features.
Scenario 4: Image Recognition for a Self-Driving Car

You need to identify objects in images (e.g., pedestrians, traffic

signs) to make real-time decisions in a self-driving car.
Scenario 5 : Predicting Customer Lifetime Value

You work for an e-commerce company and need to predict the

lifetime value of customers based on their purchasing history,
average order value, and frequency of purchases.
Scenario 6: Diagnosing a Medical Condition

You have a dataset with various symptoms and need to classify

whether a patient has a particular condition. The dataset is
relatively small.
Scenario 7: Customer Sentiment Analysis

You need to build a model that can understand and classify

customer sentiment in more nuanced ways, such as detecting
sarcasm or mixed feelings in text reviews.
Thank You
Unit 1 : Basics
Think Pair Share
• Think: Imagine you're using a very simple algorithm that
recommends movies to users based only on the genre of the last
movie they watched.
• How could this lead to high bias in the recommendations?

• Pair: Discuss with your partner what kind of movie recommendations

the algorithm might consistently fail to provide due to high bias and
how that might affect user satisfaction.

• Share: Share with the group your thoughts on how high bias can
limit the variety of recommendations and what other features could
be considered to reduce bias.
Bias – Variance:
Bias – Variance:
→Bias refers to erroneous assumptions made by the model about the
data to make the target function easier to learn. Mathematically, how
much predicted values differ from true values?

→Variance is the error, amount that the prediction (estimate of the

target function) will change if different training data sets were used. It
measures how scattered (inconsistent) are the predicted values from
the correct value due to different training data (or possibly with
different random seeds) sets. It is also known as Variance Error or
Error due to Variance.

Note: As the complexity of the model rises, the variance will increase,
and bias will decrease.
Bias-Variance & Underfitting -Overfitting

• →Underfitting or High bias means that the model

is not able to capture or learn the trend or pattern
in data.

• →Overfitting or High variance means that the

model fits the available data but does not
generalize well to predict on new data.
Training and Validation loss:
→The training loss goes down
over time, achieving low error
values.

→The validation loss goes down

until a turning point is found, and it
starts going up again. That point
represents the beginning of
overfitting.

Therefore, The training process

should be stopped when the
validation error trend changes
from descending to ascending.
Training and Validation loss:
Epochs: One Epoch is when an
ENTIRE dataset is passed forward
and backward through the neural
network only ONCE.

→Batch: You can’t pass the entire

dataset into the neural net at once.
So, you divide dataset into No. of
Batches or sets or parts.

→Iterations is the No. of Batches

needed to complete One Epoch.
How would you identify if your model is
overfitting?
• By analyzing the learning curves, you should be
able to spot whether the model is underfitting or
overfitting.

• The y-axis is some metric of learning (ex:,

classification accuracy) and the x-axis is
experience (time or No. of iteration).
Bias – Variance:
The goal of parameterization is to achieve a low bias and low
variance trade-off through methods such as:

→Cross-validation can be used to tune models so as to optimize the

trade-off

→Dimension reduction and feature selection

→Mixture models (probabilistic models) and ensemble learning.

Think Pair Share
• Think: Imagine you're using a very simple algorithm that
recommends movies to users based only on the genre of the last
movie they watched. How could this lead to high bias in the
recommendations?

• Pair: Discuss with your partner what kind of movie

recommendations the algorithm might consistently fail to provide
due to high bias and how that might affect user satisfaction.

• Share: Share with the group your thoughts on how high bias can
limit the variety of recommendations and what other features could
be considered to reduce bias.
Think Pair Share
• Think: Imagine you have a simple spam detection model that only
looks for specific keywords (like "win" or "free") to classify emails as
spam. How could this lead to high bias in the model?

• Pair: Discuss with your partner how relying solely on specific

keywords might cause the model to miss more subtle forms of spam
and incorrectly classify legitimate emails.

• Share: Share with the group how high bias in spam detection could
result in underfitting and suggest additional features that could be
included to improve the model's accuracy.
Linear Regression
Regression and Classification:
Assumptions of Linear Regression
Assumptions of Linear Regression
Linear Regression

• Linear regression is a supervised learning technique that makes an

attempt to predict the relationship between two variables,
the dependent and independent variables, by fitting a linear
equation to the observed data.

• Finding a linear relationship that exemplifies the given data actually

means finding a linear line that best represents the dataset.
Considering the simplest scenario, we can construct a basic linear line
using the following equation:
Linear Regression
Linear Regression

• Simple Linear Regression:

Simple Linear Regression
helps to find the linear
relationship between two
continuous variables, One
independent and one
dependent feature.

• The formula can be

represented as y=mx+b or
Multiple Linear Regression:

• Multilinear Regression: We
Often use Multiple Linear
Regression to do any kind
of predictive analysis as the
data we get has more than
1 independent feature to it.
• The formula can be
represented as

• Y=mX1+mX2+mX3…+b,
How Linear Regression Works:

The whole idea of linear Regression is to find the best-fit line, which has
a very low error (cost function). This line is also called the Least Square
Regression Line (LSRL).

Linear Regression learns data by fitting drawing straight Line. Linear

Regression will find the slope and intercept

Y= mx + c; where m=slope, c=intercept

How do you say model Best or
not?
Regression Metrics:
Linear Regression

• If we find the intercept and coefficient values using the dataset (X) and
labels (y), we can determine the most appropriate linear line. After
calculating these two unknowns, we have the basic information
necessary to make new predictions.

• All we need to do is to find the value that the new data intercepts with
the linear regression line. Sounds logical, doesn’t it?

• The most critical thing here is the selection of the intercept and
coefficient parameters to best represent the dataset.

• But how do we make this choice?

Ordinary Least Square: One-Step Solution

• We want to create a line such that the difference between the

obtained estimates and the actual values is minimal (or even zero if
possible), in other words, we want to find the best b values.

• OLS allows us to find the best b parameters by minimizing

the squared errors of the distance of each sample’s estimation from
the true value

• We are using the squared error since the negative and positive
distances should not impact the optimization differently.
Ordinary Least Square: One-Step Solution

• Slope (m): Represents the change in the dependent variable (y) for a one-unit change in the
independent variable (x). For example, if m=2, it means that for every one-unit increase in x, y
increases by 2 units.

• Intercept (c): Represents the value of y when x=0. It’s the point where the regression line crosses
the y-axis.
OLS Method:
• OLS method allows us to calculate all the parameters we are looking
for in a single step, we do not need to use any iterations. Additionally,
by solving the equation, the entire process becomes quite
straightforward and simple to understand.

• Besides the advantages, OLS also has some

disadvantages. Multicollinearity undermines this definition since a
change in one variable affects the other variables in a way that also
affects the dependent variable.

• As a result, the model is less effective and the coefficient estimates

are less precise. In addition, multiplying the entire matrix at once can
be a problem when the dataset is large.
Gradient Descent:
Gradient Descent:
Gradient Descent:

• GD is an iterative optimization algorithm that is widely used in many

different machine learning models. In contrast to OLS, gradient descent
avoids attempting to solve any closed-form equations.

• It starts with a random solution set and achieves the best parameters
by iteratively changing them. At each step, the parameters are updated
and the effect of this change on the error is evaluated.

• So how do we know how to update to get the best

parameter set?
GD Algorithm:
• For this, we need to define a loss and an
update function.

• The α in the update function, which is

called the learning rate, controls how big
steps we will take while updating. If we
select a small learning rate, it might
take too much time to reach the
convergence point, or we might stuck
into a local minimum point.

• Conversely, if we go with a large

learning rate, we might not reach the
minimum point. Therefore, careful
selection of the learning rate significantly
affects the performance of the algorithm.
How GD Works?
Learning Rates in GD:
Types of Linear Regression
What's this?
What's this?
What's this?
Logistic Regression
Logistic Regression

In Logistic regression,
instead of fitting a
regression line, we fit an
"S" shaped logistic
function, which predicts two
maximum values (0 or 1).
Sigmoid Function:

• It is a mathematical
function that is used to
• Suppose we have a data frame transform continuous
containing information about
customers of an online store, values into probability
including their age, gender, values that lies within
income, and whether or not they
made a purchase. the range of 0 to 1.
• We want to use binary logistic
regression to predict which
customers are likely to make a
purchase based on their age,
gender, income.
Linear Vs Logistic Regression
Linear Regression Vs Logistic Regression:
Linear Regression Vs Logistic Regression:
Linear Regression Vs Logistic Regression:
Linear Regression Vs Logistic Regression:
Why shouldn't we call Logistic Regression "Logistic
Classification"?
Why shouldn't we call Logistic Regression "Logistic
Classification"?

• It's not entirely accurate to call logistic regression "logistic

classification" is because, logistic regression is a statistical method
that estimates the probability of a binary outcome, while
classification is the task of predicting the category or class that a
new data point belongs to based on a set of features.

• While logistic regression can be used for classification tasks,

it is still a regression method at its core and is primarily used
for estimating probabilities rather than making direct
classifications.
Do logistic regression generates only binary
outcomes ?
Do logistic regression generates only binary
outcomes ?
• Yes, Binomial logistic regression is used to predict binary
outcomes or dependent variables that have only two possible
values or categories.
• As a result, the output of logistic regression should be categorical
or discrete. It could take on values such as Yes or No, 0 or 1, true
or false, and so on.
• However, rather than providing a precise value of 0 or 1, the
logistic regression model generates probability scores that
fall between 0 and 1.
• However, there are extensions of logistic regression, such as
multinomial logistic regression and ordinal logistic regression, that
can handle dependent variables with more than two categories.
What are the types of Logistic Regression?
[Link] Logistic Regression: This is the most common type of logistic regression
where the dependent variable has only two possible outcomes, such as yes or no,
pass or fail, and so on. For example, predicting whether a customer will buy a
product or not.
2. Multinomial Logistic Regression: This type of logistic regression is used when
the dependent variable has three or more categories that are not ordered. For
example, predicting whether a person will vote for one of five political parties.
[Link] Logistic Regression: This type of logistic regression is used when the
dependent variable has three or more ordered categories. For example, it can be
used to predict the level of customer satisfaction based on their feedback, such
as poor, average, and excellent.
 Types of Logistic Regression

• In summary, binary logistic regression is used for binary

outcomes, multinomial logistic regression is used for non-
ordered categorical outcomes, and ordinal logistic regression
is used for ordered categorical outcomes.
What happens when we use linear
regression to Logistic Regression
Problems?
Using linear regression for logistic regression problems in marks can lead to
inaccurate results because it assumes a linear relationship between the variables
and cannot handle binary outcomes.
Summary: Logistic Regression:
1.A category of Supervised Learning, commonly
used to solve classification problems.

[Link] Logistic regression, instead of fitting a

regression line, we fit an "S" shaped logistic
function.

[Link] output of logistic regression is a probability

score between 0 and 1, indicating the likelihood of
the binary outcome.

[Link] regression uses a sigmoid function to

convert the linear regression output to a
probability score.

[Link] of logistic Regression - Binomial,

Multinomial, Ordinal. The Binary logistic regression
is used for binary outcomes, multinomial logistic
regression is used for non-ordered categorical
What does that graph exactly convey to us?

• Logistic regression uses a sigmoid function to map the linear

regression output to a probability score, which is then used to
predict the binary outcome. The sigmoid function is an S-shaped
curve that ranges from 0 to 1.

• When we plot the sigmoid function on a graph, the X-axis

represents the linear regression output, while the Y-axis
represents the probability score. At the extreme ends of the graph,
the probability score is either 0 or 1, indicating a certain binary
outcome.
What does that graph exactly convey to us?
• As we move towards the middle of the graph, the probability
score approaches 0.5, indicating uncertainty about the binary
outcome. The point on the X-axis where the probability score
is exactly 0.5 is known as the decision boundary.

• The decision boundary separates the two classes in the binary

classification problem. Any input data point on one side of the
decision boundary is classified as one class (e.g., positive
outcome), while any input data point on the other side of the
decision boundary is classified as the other class (e.g.,
negative outcome).

• Thus, logistic regression uses the sigmoid function and the

decision boundary to predict the binary outcome based on the
input variables.
What happens when decision boundary is
0.5 in logistic regression?
What happens when decision boundary is 0.5 in
logistic regression?

• When the decision boundary is 0.5 in logistic regression, it

means that the model is equally uncertain about both classes.

• Therefore, data points that fall close to the decision boundary

may be misclassified, and the overall accuracy of the
model may be lower.

• To improve the performance of the model, one could consider

adjusting the threshold of the decision boundary or using a
different model altogether.
Problem Statement:

• Suppose we have a data frame containing information

about customers of an online store, including their age,
gender, income, and whether or not they made a purchase.

• We want to use binary logistic regression to predict which

customers are likely to make a purchase based on their
age, gender, income.
Steps:

Step 1: Import necessary libraries

Step 2 : Create a data frame with age, gender, income, and
purchase columns
Step 3 : Convert the gender column to a binary indicator
variable
Step 4 : Split the data into training and testing sets
Step 5 : Fit a logistic regression model to the training data
Step 6 : make predictions on the testing data
Step 7 : Evaluate the performance of the model
Step 8 : Make new prediction
Thank You
• We can use one-hot encoding to convert the categorical
variable "gender" into a numerical variable. We can use the
pandas get_dummies function to do this. ( This method is called
as one hot encoding)

• Finally, to predict the outcome for new customers, we can

create a new dataframe with the same columns as the original,
but with the values for age, gender, and income for the new
customers:
Decision Tree Algorithm:
• A Decision Tree algorithm is a type of Supervised machine learning
algorithm that uses a tree-like model to make decisions.

• The idea behind the algorithm is simple - you start with a question and based on
the answer, you move to the next question until you reach a conclusion.

• In Simple words, works by asking a series of questions and making

decisions based on the answers.

• Decision Tree is used for both regression and classification tasks. In

classification problems, the algorithm is used to predict which category
something belongs to, like whether an email is spam or not.

• In regression problems, the algorithm is used to predict a numerical value, like

the price of a house
Applications of Decision Tree Algorithm:
• Finance: Predicting loan defaults, assessing creditworthiness, and
making investment decisions.

• Healthcare: Diagnosing diseases, identifying high-risk patients, and

personalizing treatment plans.

• Marketing: Customer segmentation, targeted advertising campaigns,

and predicting customer churn.

• Retail: Optimizing inventory management, forecasting demand, and

identifying fraudulent transactions.
Types of Decision Tree Algorithm:

There are two main types of decision trees:

• Classification Trees: Used for predicting categorical outcomes (e.g., spam or not
spam).
• Regression Trees: Used for predicting continuous outcomes (e.g., housing
prices).

The choice between these depends on the nature of the target variable in your
dataset.
Decision Tree Algorithm: How it works?
Decision Tree Algorithm:
• Imagine you're trying to decide which movie to watch tonight. You've
got a bunch of movies to choose from, but you're not sure which one
you'll enjoy the most. That's where Decision Tree comes in!

• The algorithm works by breaking down the movies into smaller and
smaller groups based on a set of rules. For example, the first
question might be "Is the movie a comedy or a drama?" Based on
your answer, you would move down the corresponding branch of the
Decision Tree.

• If you said "comedy", then the next question might be "Is it a romantic
comedy or a slapstick comedy?" And so on, until you reach a
conclusion - in this case, the movie you want to watch tonight.
Decision Tree Algorithm:
Terminology in Decision Tree Algorithm:
• Node: It represents a question or a decision in the tree.

• Root Node: The topmost node in the tree, which represents the initial
question or decision.

• Leaf Node: The bottommost node in the tree, which represents the
conclusion or prediction.

• Branch: The path from one node to another node.

• Splitting: The process of dividing a node into two or more sub-nodes.

Terminology in Decision Tree Algorithm:
• Pruning: The process of removing unnecessary branches or nodes
from the tree to reduce overfitting.

• Information Gain: A measure used to determine the importance of a

feature in splitting a node.

• Gini Index: A measure of impurity used to split the data into

homogenous groups.

• Overfitting: When the model learns the training data too well, but
performs poorly on new data.
Splitting Criteria: Choosing the Best Split
A crucial aspect of building decision trees is selecting the best attribute and split
point at each node. Common splitting criteria include:

• Gini Impurity (Classification): Measures the "impurity" or randomness of a node,

favoring splits that maximize class separation.
• Entropy (Classification): Similar to Gini impurity but uses information theory to
measure randomness.
• Information Gain: Measures the reduction in uncertainty about the target variable
after a split.

These metrics guide the algorithm towards building a tree that effectively
separates data points belonging to different classes.
Gini Impurity, Entropy and Information Gain:
Example for Decision Tree:
Advantages: Decision Tree Algorithm:

• Interpretability: Easy to understand the decision-making logic through the tree

structure.

• Ease of Use: Relatively straightforward to implement and interpret compared to

other models.

• Handles Different Data Types: Can work with both numerical and categorical
data without extensive preprocessing.
Disadvantages: Decision Tree Algorithm:

• Overfitting: Prone to overfitting if not pruned carefully.

• Instability: Small changes in the data can lead to significant changes in the tree
structure.

• Black Box Nature: While the overall structure is interpretable, the specific
reasons behind each split might not be immediately clear.
Problem Statement:
• You're planning to watch a movie tonight and you want to decide
which movie to watch .

• You have a list of movies and their attributes, such as genre, rating,
and runtime.

• Using a Decision Tree algorithm, you want to build a model that can
help you make a decision on which movie to watch based on these
attributes.

• Google Colab Implementation

Random Forest Algorithm:

• Random Forest is a powerful machine learning algorithm that can be

used for both regression and classification tasks.

• It belongs to the family of ensemble methods, which combines

multiple models to improve the accuracy of predictions.
Random Forest Algorithm:

• Ensemble learning is a machine learning technique that combines

multiple individual models (called base models or weak learners) to
create a stronger, more accurate model.

• It is based on the idea that a group of weak learners can work

together to make better predictions than a single strong learner.

• The individual models in an ensemble can be trained using different

algorithms or with the same algorithm but on different subsets of the
training data.
Random Forest Algorithm:
Random Forest Algorithm:

Bagging– It creates a Boosting– It combines

different training subset weak learners into strong
from sample training data learners by creating
with replacement & the sequential models such that
final output is based on the final model has the
majority voting. For highest accuracy. For
example, Random example, ADA BOOST, XG
Forest. BOOST.
Random Forest Algorithm:
Random Forest Algorithm:
Random Forest Algorithm:
• Random Forest works by creating many decision trees,
• Each tree is trained on a random subset of the data and a random subset of
features.
• It reduces the risk of overfitting and improves the accuracy of the model.
Random Forest Algorithm:

❖ When making a prediction,

each tree in the forest
independently predicts the
outcome and the final
prediction is made by taking
the majority vote of all the
trees.
Random Forest Algorithm:

• Step-1: Select random K data points from the training set.

• Step-2: Build the decision trees associated with the selected data points
(Subsets).

• Step-3: Choose the number N for the decision trees that you want to build.

• Step-4: Repeat Steps 1 & 2.

• Step-5: For new data points, find the predictions of each decision tree, and
assign the new data points to the category that wins the majority votes.
Example: Random Forest Algorithm:
Advantages and Disadvantages of Random Forest
Algorithm:
• Random Forest is easy to use, fast to train, and can handle large
datasets with high-dimensional features.

• Random Forest is a powerful algorithm that works well on large

datasets with many features.

• When the dataset is small, the model may overfit to the training data,
resulting in poor performance on the testing data.
Applications of Random Forest Algorithm:
• Finance: It is used for credit scoring, fraud detection, and stock
market prediction.

• Healthcare: Random Forest aids in disease diagnosis, predicting

patient outcomes, and identifying risk factors.

• Image Processing: It is applied in image classification, object

recognition, and facial recognition systems.

• Environmental Sciences: Random Forest is used to predict species

distribution, land cover classification, and environmental monitoring.
Decision Tree Vs Random Forest
KNN:

• KNN is a type of machine learning algorithm that can be used for

classification or regression tasks.

• The KNN algorithm assumes that similar things exist in close

proximity (similar things are near to each other).

• In simple words, if two objects share similar features, they are

likely to be neighbors in the dataset.
What is K?
KNN:

“K" is a user-defined hyperparameter that

determines how many neighbors to consider.
KNN:
KNN:
KNN Example:

• Imagine you have a dataset of roses with their petal lengths and
widths, as well as their colors (red, pink, or white). You want to use
KNN to predict the color of a new rose based on its petal length and
width.

• To use KNN, you need to choose a value for k, which represents the
number of nearest neighbors to consider. For example, let's say you
choose k=5.
KNN Example:

• Now, suppose you have a new rose with a petal length of 4.8 cm and
a petal width of 1.5 cm. To predict its color using KNN, you need to
find the five nearest neighbors to this new rose based on their petal
lengths and widths.

• You can calculate the distance between the new rose and all the
roses in your dataset using a distance metric, such as Euclidean
distance. The five roses that are closest to the new rose (i.e., have
the shortest distance) are considered its nearest neighbors.
KNN Example:
Let's say the five nearest neighbors are:

• Rose A with petal length of 4.5 cm, petal width of 1.6 cm, and color of
pink
• Rose B with petal length of 5.2 cm, petal width of 1.8 cm, and color of
white
• Rose C with petal length of 4.9 cm, petal width of 1.7 cm, and color of
pink
• Rose D with petal length of 4.7 cm, petal width of 1.4 cm, and color of
red
• Rose E with petal length of 4.8 cm, petal width of 1.3 cm, and color of
red
KNN Example:

Since three of the five nearest neighbors are pink and two are
red, we would predict that the new rose is likely to be pink.
KNN (Another Example):
• In KNN, we believe similar things are close to each other.

• If you want to determine which class a new data point belongs to (e.g., red
or black), the first step in the K-Nearest Neighbors (KNN) algorithm is to
choose the value of k, which represents the number of neighbors to
consider.

• For example, if you choose k=3, the algorithm will select the three nearest
neighbors by calculating the distance between the new data point and all
other data points in the dataset.

• The three closest data points will be selected as neighbors and used to
make a prediction about the class of the new data point.
KNN (Another Example):

• KNN algorithm starts the voting process to determine the class of the
new data point.

• In your example where k=3 and two of the three neighbors are black;
the algorithm would classify the new data point as black since it is the
majority class among the three nearest neighbors.
How to calculate the distance in KNN?
KNN – Euclidean Distance:
The most common
distance metric used in
KNN is Euclidean distance.
For example, suppose you
have two data points A and
B, where A has features
(x1, y1) and B has features
(x2, y2). The Euclidean
distance between A and B
can be calculated using the
following formula:
How to choose right value of K in KNN?
How to choose right value of K in KNN?

• Consider the size of your dataset

• Choose an odd value for k
• Elbow Method
• Square root of N rule
• Experiment with different values of k
Do you Know KNN is a Lazy Learner?
KNN:

KNN is a type of lazy learning algorithm because it

doesn't build a model from the data, but instead stores
the entire dataset and calculates predictions based on
nearest neighbors.
Which algorithm will be most suitable
for the following scenarios?
Scenario 1 : Estimating the cost of groceries
based on the number of items purchased.
Scenario 2: Predicting the amount of time a
commute will take based on distance and
traffic conditions.
Scenario 3 : Deciding what movie to watch
based on genre, runtime, and rating.
Scenario 4: Finding a nearby gym based on
your location and preferred types of fitness
classes.
Scenario 5: Predicting the likelihood of a traffic jam based
on time of day, weather conditions, and traffic patterns.
Scenario 6: Choosing a restaurant based on
distance, type of cuisine, and price range.
• Scenario 7: Suggesting new books to read based on
your reading history and similar readers’ preferences.
Scenario 8: Identifying the type of exercise routine
(e.g., cardio, strength training) based on factors like
duration, intensity, and equipment used.
Scenario 9: Determining if an email is likely to be read
or ignored based on the subject line and time sent.
Naive Bayes Classifier:

• Naive Bayes is a machine learning algorithm that is used for

classification tasks.

• It is based on the Bayes theorem, which is a mathematical formula

that can be used to calculate the probability of an event occurring.
Naive Bayes Classifier:
❖The Naive Bayes
algorithm is based on a
simple idea: the
probability of an event
happening given some
evidence.

❖ In this case, the event

is whether an email is
spam or not, and the
evidence is the words
present in the email.
Naive Bayes Classifier:
Naive Bayes Classifier Assumption:

• The Naive Bayes algorithm makes the assumption that the features of
a data set are independent of each other.

• This means that the presence of one feature does not affect the
probability of another feature being present.

• The algorithm assumes that the occurrence of each word is independent

of the others, which is why it's called "naive."

• This assumption simplifies the calculations and makes the algorithm

computationally efficient.
Naive Bayes Classifier:
Naive Bayes Classifier:
Naive Bayes Classifier:
Naive Bayes Classifier: How it Works

• The algorithm starts by analyzing a set of training emails,

where you've already labeled each email as spam or not spam.

• It looks at the words in these emails and calculates the

probability of each word occurring in spam emails and in non-
spam emails.

• For example, it might find that the word "money" appears in 80%
of spam emails but only in 10% of non-spam emails.
Naive Bayes Classifier: How it Works
• Now, when a new email arrives, the Naive Bayes algorithm looks
at the words in that email and calculates the probability of it
being spam or not spam based on the probabilities it learned from
the training emails.

• To do this, it multiplies the probabilities of all the words

occurring in spam emails together and multiplies the
probabilities of all the words occurring in non-spam emails
together.

• It then compares these two probabilities and classifies the email

as spam if the probability of it being spam is higher, or as
non-spam otherwise.
Types of Naive Bayes Classifier:
Types of Naive Bayes Classifier:
Applications of Naive Bayes Classifier:
Strengths of Naive Bayes Classifier:

1. Simplicity

2. Efficiency

3. Good for Text Data

4. Handles High-Dimensional Data

Limitations of Naive Bayes Classifier:

1. Assumption of feature independence

2. Data Scarcity
SVM: Support Vector Machine
History of SVM:
• The original SVM algorithm was developed by Vladimir N Vapnik and
Alexey Ya. Chervonenkis in 1963.
• At that time, the algorithm was in early stages. The only possibility is
to draw hyperplanes for linear classifier.
• In 1992, Bernhard E. Boser, Isabelle M Guyon and Vladimir N Vapnik
suggested a way to create non-linear classifiers by applying the kernel
trick to maximum-margin hyperplanes.
• The current standard was proposed by Corinna Cortes and Vapnik in
1993 and published in 1995.
• Developed at AT&T Bell Laboratories by Vladimir Vapnik
SVM:
• It is a supervised machine learning algorithm which is used for both
classification and regression task and for outlier detection.
• It used for variety of tasks such as text classification, spam detection,
Handwritten identification etc.
• The main idea of SVM is to find a hyperplane that best separates the
data into two classes.
• In 2-D it is a line and in higher dimensions it is a hyperplane. There
can be many possible hyperplanes.
• Our objective is to find a plane that has maximum margin i.e the
maximum distance between the data points of both the classes.
SVM:
• In two dimensional space this hyperplane is a line dividing a plane in
two parts where in each class lay in either side, but in multi (n)
dimensional space the hyperplane is an (n-1) dimensional plane which
divide the data in multiple classes.
• Hyperplane: A hyperplane is a subspace whose dimension is one
less than that of its ambient space. If a space is 3-dimensional then its
hyperplanes are the 2-dimensional planes, while if the space is 2-
dimensional, its hyperplanes are the 1-dimensional lines. ... By its
nature, it separates the space into two half spaces.
• Margin: A margin is a separation gap between the two lines on the
closest data points. It is calculated as the perpendicular distance from
the line to support vectors or closest data points. In SVMs, we try to
maximize this separation gap so that we get maximum margin.
SVM:
SVM:
Support vectors: These are points that are closer to the hyperplane.

Margin: Perpendicular distance from the decision boundary to the nearest data
point of either class.

Hard margin : In hard margin SVM, hyperplane perfectly separated the two classes
without any misclassification. It works well when data is linearly separable. It is
sensitive to outliers.

Soft margin: In soft margin classification, algorithm allows some misclassifications.

It introduce a hyperparameter C which controls the trade-off between maximizing
margin and allowing misclassifications. Smaller the value of C results in larger
margin but allows more misclassifications, while a larger C results in a smaller
margin but fewer misclassifications.
How SVM Works?
The main objective of the SVM is to select the hyperplane with the
maximum possible margin between support vectors in the given
dataset. SVM searches for the maximum margin hyperplane in the
following 2 step process –

[Link] hyperplanes which segregates the classes in the best

possible way. There are many hyperplanes that might classify the
data. We should look for the best hyperplane that represents the
largest separation, or margin, between the two classes.

[Link], we choose the hyperplane so that distance from it to the support

vectors on each side is maximized. If such a hyperplane exists, it is
known as the maximum margin hyperplane and the linear classifier
it defines is known as a maximum margin classifier.
How SVM Works?
Two Main Types of SVMs:
• Linear SVM: It uses a linear kernel
function to create a linear decision
boundary to separate the two classes.
They work well when data is linearly
separable.
• Non-linear SVM: Non-linear SVM uses
non-linear kernel functions (polynomial,
radial basis function) which transforms
the data into higher dimensional space
where it becomes linearly separable. It
works well when there is non-linear data.
• Handles more complex data by using
the kernel trick. Imagine curving and
bending the line to adapt to more
intricate patterns in your data.
Kernel Trick:

• Sometimes, the data points are so dispersed or scattered that it is not

possible to separate them using a linear hyperplane. In such a
situation, SVMs uses something called as Kernel to transform the
input space to a higher dimensional space. It uses a mapping function
to consider the 2-D input space into the 3-D input space. So, it can
easily segregate the data points using linear separation.

• It is a costly operation to actually transform data points to a high-

dimensional feature space. The algorithm does not actually transform
the data points to a new, high dimensional feature space. Kernelized
SVM compute decision boundaries in terms of similarity measures in a
high-dimensional feature space without actually doing a
transformation. It is done by Kernel Trick
Kernel Trick:
Kernel Trick:
Kernel Trick:
Kernel Trick: Example
Kernel Trick:
Polynomial Kernel:
Kernel Trick:
Radial Basis Function (RBF) Kernel:
• The radial basis
function (RBF) kernel,
also known as the
Gaussian kernel, is the
default kernel for
Support Vector
Machines in scikit-learn.
• It measures similarity
between two data
points in infinite
dimensions and then
approaches
classification by
majority vote.
Sigmoid Kernel:

In the sigmoid
kernel, the
similarity between
two data points is
computed using
the hyperbolic
tangent function
SVM: Prediction on New Data:

• Once the SVM is trained on the labeled data (data with known
categories), it can be used to predict the class of new, unseen data
points.

• The model will determine which side of the hyperplane the new data
falls on, thus classifying it into one of the learned categories.

• Support Vector Machine emerges as a powerful tool in the arsenal of

machine learning algorithms, offering versatility, robustness, and
efficiency in solving complex classification and regression tasks.
Applications of SVM:

• Face detection

• Text and hypertext categorization

• Classification of images

• Bioinformatics

• Handwriting recognition
Advantages of SVM:
[Link] Dimensionality: SVM thrives in high-dimensional feature
spaces, making it suitable for tasks with a large number of features.

[Link]: SVM's margin maximization principle renders it robust

against outliers, leading to more reliable classification.

[Link]-linearity Handling: Through kernel trick, SVM can efficiently

handle non-linear relationships in data, enhancing its flexibility and
accuracy.

[Link] Optimization: The training of SVM involves solving a convex

optimization problem, ensuring convergence to the global optimum.
Disadvantages of SVM:
1. Choosing an appropriate Kernel function is difficult.

1. It requires Feature Scaling.

1. SVM takes a long training time on large datasets.

1. SVM doesn’t directly provide probability estimates, these are

calculated using an expensive five-fold cross-validation
Thank you
Questions on K-Means Clustering:

• How K-Mean Algorithm Works?

• How to measure distance?
• How to evaluate K-Mean Model?
• How to select the optimal value for K?
• Where to apply k-means clustering?
• How do you choose the initial centroid values in K-means?
K-Means Clustering:
• K-Means Clustering is an unsupervised learning algorithm that solves
clustering problems in machine learning or data science.
Classification Vs Clustering:
K-Means Clustering:
K-Means Clustering:
K-Means Clustering:
• Clustering in K-means groups
similar data points by minimizing
distances to cluster centroids.

• It’s widely used in tasks like

customer segmentation,
recommender systems, image
segmentation, and
dimensionality reduction, making
it a powerful tool for pattern
discovery in data.
K-Means Clustering:
• K-means is an unsupervised clustering algorithm
that groups unlabeled data into k clusters based
on similarity.

• The algorithm iteratively assigns data points to

clusters by minimizing the distance to centroids,
with each cluster represented by its centroid. The
process repeats until optimal clusters are found.

• The number of clusters, k, is predefined, and K-

means aims to minimize the sum of distances
between data points and their cluster centroids.
It’s widely used for discovering patterns in
unlabeled datasets.
How K-Means Clustering Works?
• Plot Data
• Select the number K to decide the number of clusters.
• Select random K points or centroids. (It can be other from the input
dataset).
• Assign each data point to their closest centroid, which will form the
predefined K clusters.
• Repeat the fourth step, which means reassigning each data point to
the new closest centroid of each cluster.
• Until you get a clearer cluster means no overlapping.
How K-Means Clustering Works?
K-Means Clustering: Measuring Distance?
How to Evaluate K-Means Clustering Model?

Silhouette Coefficient:

Silhouette Coefficient or
silhouette score is a metric
used to calculate the
goodness of a clustering
technique. Its value ranges
from -1 to 1.
How to Select Optimal Value for K?
• The Elbow method is one of the most popular ways to find the optimal number of clusters. This
method uses the concept of WCSS value. WCSS stands for Within Cluster Sum of Squares,
which defines the total variations within a cluster.
K-Means Clustering Algorithm Overview:
Applications of K-Means Clustering:
• Image Segmentation: Groups pixels based on color similarity, segmenting an image into
distinct regions.

• Customer Segmentation: Classifies customers by behavior or demographics to enable

targeted marketing.

• Anomaly Detection: Identifies outliers in data, useful for detecting fraud or network
intrusions.

• Document Clustering: Groups similar documents based on content, aiding in organization

and topic modeling.

• Recommendation Systems: Clusters users with similar preferences, enabling personalized

recommendations based on group behavior.
Disadvantages of K-Means Clustering:
• Fixed Number of Clusters (k)

• Sensitive to Initialization

• Only Detects Spherical

Clusters

• Sensitive to Outliers

• Equal Cluster Size Assumption

• Distance Metric Limitation

• Computationally Expensive
K-Means ++ Clustering:
• K-Means++ Initialization: K-Means++ is an improvement over the random
initialization method that aims to choose more representative initial centroid
values. The algorithm selects the first centroid randomly, and then for each
subsequent centroid, it chooses a data point that is farthest from the existing
centroids.
Hierarchical Clustering
Agglomerative Clustering:
Agglomerative Clustering:
Agglomerative Clustering:
Divisive Clustering:
Thank you
Ensemble Learning
Outline

• Introduction to Ensemble Methods

• Bagging and Boosting

• Random Forest

• Fixed Rule Fusion Techniques

• Trained Rule Fusion Techniques

What is Ensemble Learning?

• Combine multiple models to create a stronger predictive model.

• Aim to improve accuracy, reduce variance and bias.

• Inspired by the "wisdom of the crowd" concept.

Ensemble Learning
Why use Ensemble Learning?

• Improved Accuracy: Better performance than individual models.

• Robustness: More stable and reliable predictions.

• Reduced Overfitting: Combines models to generalize better.

Types of Ensemble Methods

• Bagging (Bootstrap Aggregating)

• Boosting

• Random Forest

• Fusion Techniques
Bagging (Bootstrap Aggregating)

Concept: Create multiple subsets of the data with replacement.

Process:

• Generate bootstrapped datasets.

• Train a model on each subset.
• Aggregate predictions (e.g., majority vote or averaging).
Bagging:
Bagging – Benefits and Example
Benefits

• Reduces variance and overfitting.

• Enhances model stability.

Example: Random Forest

• An ensemble of decision trees using bagging.

Boosting:
Concept: Sequentially build models that focus on correcting errors of
previous models.

Process:

[Link] initial model.

[Link] and focus on errors.
[Link] subsequent models to correct mistakes.
[Link] all models with weighted predictions.
Boosting:
Boosting:
Boosting – Benefits and Examples

Benefits

• Reduces both bias and variance.

• Improves accuracy on complex datasets.

Examples:

• AdaBoost
• Gradient Boosting
• XG Boost
Bagging Vs Boosting:
Random Forest

Concept: An ensemble of decision trees with added randomness.

Process:

• Create multiple decision trees using bootstrapped data.

• At each split, select a random subset of features.

• Aggregate tree predictions (majority vote or averaging).

Benefits of Random Forest:

• High accuracy and performance.

• Handles large datasets with numerous features.

• Resistant to overfitting due to feature randomness.

Fusion Techniques Overview
Fixed Rule Fusion Techniques

• Combine predictions using predefined, static rules.

• Common Methods: Majority Voting, Averaging, Weighted Voting

Examples:

• Majority Voting: Most models agree on the prediction.

• Averaging: Average the numerical predictions.
• Weighted Voting: Assign fixed weights to each model's prediction.
Fusion Techniques Overview

Trained Rule Fusion Techniques : Combine predictions by learning

the optimal combination from data.

Common Methods: Stacking (Stacked Generalization), Blending

Stacking (Stacked Generalization) &
Blending:
Stacking:

• Train multiple base models.

• Use their predictions as inputs for a meta-model.
• Meta-model learns the best way to combine base predictions.

Blending:

• Train base models on the training set.

• Use a separate validation set to train the meta-model.
• Combine predictions using the meta-model.
Stacking
Benefits of Trained Rule Fusion

• Learns optimal combination from data.

• Can capture complex relationships between models.

• Often achieves higher accuracy than fixed rules.

Fixed vs. Trained Rule Fusion
When to Use Ensemble Methods?

• When high accuracy is crucial.

• Handling complex and diverse datasets.

• Reducing overfitting and improving generalization.

• Combining different types of models for robust predictions.

 Real-World Example: Predicting Book Preferences with
Ensembles
Scenario: Predict whether someone will like a new book.

Approach:
• Bagging: Train multiple decision trees on different reader subsets.
• Boosting: Sequentially improve models by focusing on difficult
predictions.
• Random Forest: Use many trees with random feature selection.
• Fusion Techniques: Combine various models’ predictions via voting or
a meta-model.

Result: More accurate and reliable predictions on reader preferences.

Summary:

• Ensemble methods enhance model performance by leveraging

multiple models.
• Bagging reduces variance and overfitting.
• Boosting focuses on correcting errors to improve accuracy.
• Random Forest combines bagging with feature randomness for
robust predictions.
• Fusion Techniques effectively combine model outputs, either
through fixed rules or trained methods.
Thank you
Curse of Dimensionality
Why More Dimensions Cause Problems?
Curse of Dimensionality

• Happens when datasets have too many features (or dimensions).

• Makes it harder for models to work effectively.

• Increases complexity, makes data sparse, and confuses models.

Why More Dimensions Cause Problems

• Data gets too spread out (sparse).

• Patterns become harder to find.

• More irrelevant features lead to confusion.

• Computations take longer and become inefficient.

Problem 1: Data Sparsity

• In higher dimensions, data points are far apart.

• Makes it hard to find patterns or clusters.

• Machine learning models struggle to learn effectively.

Problem 2: Irrelevant Features

• Too many variables can distract the model.

• Some features are irrelevant and add "noise".

• Can cause the model to overfit (learn noise instead of patterns).

Problem 3: Distance Becomes Meaningless

• In high dimensions, everything looks equally far apart.

• Makes it hard to group or classify data points.

• Models based on distance (like K-Nearest Neighbors) struggle.

Problem 4: Increased Computation

• More dimensions = more calculations.

• Training models becomes slow and inefficient.

• Requires more memory and processing power.

Example : Face Recognition

• Low Dimensions: Simple features like distance between eyes and nose.

• High Dimensions: Too many details, like individual pores and hair strands,
confuse the system.

• Adding too many features doesn’t help, it hurts!

How to Handle the Curse of Dimensionality?
How to Handle the Curse of Dimensionality?

• Feature Selection: Pick the most important features.

• Dimensionality Reduction: Use techniques like PCA to reduce the

number of features.

• Regularization: Keep the model simple by penalizing too many features.

Feature Selection

• Choosing only the most important variables.

• Example: Predicting house prices—focus on location, size, etc., not paint

color.
Dimensionality Reduction

• Combines similar features to create fewer, more meaningful dimensions.

• Helps focus on key patterns without losing important information.

• Example: Reducing 100 features to just 10 important ones.

Regularization

• Adds a penalty for using too many features.

• Encourages the model to focus only on relevant features.

• Common techniques: Lasso, Ridge Regression.

Key Takeaways

• The Curse of Dimensionality occurs when we use too many features.

• It makes data sparse, adds noise, and slows down computation.

• Solutions: Feature selection, dimensionality reduction, and regularization.

Thank you
Feature Selection Techniques
Why Feature Selection?

• Reduce dimensionality.

• Improve model accuracy by eliminating irrelevant features.

• Reduce overfitting.

• Speed up model training and inference.

Types of Feature Selection

• Filter Methods

• Wrapper Methods

• Embedded Methods
Types of Feature Selection
Filter Methods
Filter Methods:

• Select features based on their individual relationship with the target variable.

• Independent of any machine learning algorithm.

Examples:

• Correlation Coefficient
• Chi-Square Test
• Mutual Information
• Variance Threshold
Correlation Coefficient
Chi-Square Test
Mutual Information:
Variance Threshold:
Wrapper Methods
Wrapper Methods:

• Evaluate subsets of features based on model performance.

• Computationally expensive but generally provides better results.

Techniques:

• Forward Selection: Starts with no features, adds features iteratively.

• Backward Elimination: Starts with all features, removes features iteratively.

• Recursive Feature Elimination (RFE): Removes features recursively based on importance

Wrapper Methods
Forward Selection & Backward Elimination:

Forward Selection:

[Link] with an empty set of features.

[Link] each feature, add it to the set and train the model.

[Link] the feature that improves performance the most.

Backward Elimination:

[Link] with all features.

[Link] the model and remove the feature whose removal increases performance the
least.
Recursive Feature Elimination (RFE)

[Link] features by importance (based on model coefficients or importance

scores).

[Link] the least important feature.

[Link] until the desired number of features remains.

Example of Wrapper Method (Recursive Feature
Elimination):

Let’s say you are working with a dataset containing 10 features, and you want to select the 5 most
important ones using Recursive Feature Elimination (RFE) with a decision tree model.

❖Start by training the decision tree model on all 10 features.

❖Rank the features based on their importance (for example, based on how well they split the
data).

❖Remove the least important feature.

❖Retrain the model with the remaining 9 features.

❖Repeat the process until only 5 features remain.

Embedded Methods
Embedded Methods: (Integrate the feature selection process into the model training algorithm itself.)

• Feature selection occurs during the model training process.

• Less computationally expensive than wrapper methods.

Examples:

• Lasso Regression (L1 Regularization)

• Ridge Regression (L2 Regularization)
• Decision Trees
Embedded Methods
Lasso (L1) Regression: Least Absolute Shrinkage and Selection
Operator

• Lasso adds a constraint on the sum of the absolute values of the model parameters (coefficients).
• This leads to shrinking some coefficients down to zero, effectively removing the associated features
from the model.
Lasso (L1) Regression: Least Absolute Shrinkage and Selection
Operator

• The L1 penalty encourages the model to set some coefficients to

exactly zero when they are not contributing significantly to the
prediction, effectively removing them.

• The larger the value of λ, the stronger the regularization and the
more features will be set to zero.
Ridge (L2) Regression
Impact of Regularization:
Decision Trees for Feature Selection
Lasso vs. Ridge

Feature Lasso (L1 Regularization) Ridge (L2 Regularization)

L1 penalty (sum of absolute L2 penalty (sum of squares of
Penalty
values of coefficients) coefficients)

Yes, some coefficients are set to No, all coefficients are shrunk
Feature Selection
zero (features eliminated) but remain in the model

When you expect sparsity (many When you expect most features
Use Case
irrelevant features) to contribute

Shrinks all coefficients but keeps

Effect on Coefficients Drives some coefficients to zero
them non-zero
Handling Multicollinearity Less effective More effective
Comparison of Feature Selection Methods

Method Advantages Disadvantages

Filter Fast, independent of model May miss feature interactions

Considers feature interactions,

Wrapper Computationally expensive
better results

Integrated into model training,

Embedded Dependent on the model
less costly
Summary:

• Use Filter Methods for initial screening.

• Apply Wrapper Methods if computation allows.

• Leverage Embedded Methods for models like Lasso or Trees.

Note: Always use domain knowledge and cross-validation to ensure feature selection is meaningful.
Thank you
PCA
What is PCA?

• Principal component analysis (PCA) is a dimensionality reduction and machine learning method used
to simplify a large data set into a smaller set while still maintaining significant patterns and trends.

• The idea of PCA is simple: reduce the number of variables of a data set, while preserving as
much information as possible.

• Geometrically speaking, principal components represent the directions of the data that explain
a maximal amount of variance, that is to say, the lines that capture most information of the data.

• The relationship between variance and information here, is that, the larger the variance carried by a
line, the larger the dispersion of the data points along it, and the larger the dispersion along a line, the
more information it has.

• To put all this simply, just think of principal components as new axes that provide the best angle to
see and evaluate the data, so that the differences between the observations are better visible.
What is PCA?
• Principal components are new variables
that are constructed as linear combinations
or mixtures of the initial variables.

• These combinations are done in such a way

that the new variables (i.e., principal
components) are uncorrelated and most of
the information within the initial variables
is squeezed or compressed into the first
components.

• So, the idea is 10-dimensional data gives

you 10 principal components, but PCA tries
to put maximum possible information in
the first component, then maximum
remaining information in the second and so
on, until having something like shown in
the scree plot below.
Why PCA?

• Smaller data sets are easier to explore and visualize, and thus
make analyzing data points much easier and faster for machine
learning algorithms without extraneous variables to process.
How PCA Works?

1. Standardize the range of continuous initial variables

1. Compute the covariance matrix to identify correlations

1. Compute the eigenvectors and eigenvalues of the covariance matrix to

identify the principal components

1. Create a feature vector to decide which principal components to keep

1. Recast the data along the principal components axes

Assumptions of PCA:

Linearity: Assumes relationships between variables are linear.

Variance Importance: High variance indicates significant features.

Uncorrelated Features: Features are uncorrelated post-transformation.

Mean-Centered Data: Data must be normalized by subtracting the mean.

Step 1: Standardization

• The aim of this step is to standardize the range of the continuous initial variables so that each one of
them contributes equally to the analysis.

• For example, a variable that ranges between 0 and 100 will dominate over a variable that ranges
between 0 and 1), which will lead to biased results.

• So, transforming the data to comparable scales can prevent this problem.
Step 2: Covariance Matrix Computation

• The aim of this step is to understand how the variables of the input data set are varying from the mean
with respect to each other.

• Because sometimes, variables are highly correlated in such a way that they contain redundant
information
What do the covariance matrix tell us?

It’s actually the sign of the covariance that matters:

• If positive then: the two variables increase or decrease together (correlated)

• If negative then: one increases when the other decreases (Inversely correlated)

• Covariance matrix summarizes the correlations between all the possible pairs of variables
Step 3: Compute the eigenvectors and eigenvalues

• It is eigenvectors and eigenvalues who are behind all the magic of principal components because the
eigenvectors of the Covariance matrix are actually the directions of the axes where there is the most
variance (most information) and that we call Principal Components.

• And eigenvalues are simply the coefficients attached to eigenvectors, which give the amount of
variance carried in each Principal Component.

• By ranking your eigenvectors in order of their eigenvalues, highest to lowest, you get the principal
components in order of significance.
Step 3: Example: Compute the eigenvectors and eigenvalues

• After having the principal components, to compute the percentage of variance (information)
accounted for by each component, we divide the eigenvalue of each component by the sum of
eigenvalues.

• If we apply this on the example above, we find that PC1 and PC2 carry respectively 96 percent and 4
percent of the variance of the data.
Step 4: Create a Feature Vector

• In this step, what we do is, to choose whether to keep all these components or discard those of lesser
significance (of low eigenvalues), and form with the remaining ones a matrix of vectors that we
call Feature vector.

• So, the feature vector is simply a matrix that has as columns the eigenvectors of the components that
we decide to keep.

• This makes it the first step towards dimensionality reduction, because if we choose to keep
only p eigenvectors (components) out of n, the final data set will have only p dimensions.
Step 4: Example: Create a Feature Vector
Step 5: Recast the Data Along the Principal Components Axes

• The aim is to use the feature vector formed using the eigenvectors of the covariance matrix, to reorient
the data from the original axes to the ones represented by the principal components (hence the name
Principal Components Analysis).

• This can be done by multiplying the transpose of the original data set by the transpose of the feature
vector.
Thank you
Linear Discriminant Analysis
Why LDA and Why not Logistic
regression?

• Two-Class Problems. Logistic regression is intended for two-class or binary

classification problems. It can be extended for multi-class classification but is
rarely used for this purpose.

• Unstable With Well Separated Classes. Logistic regression can become

unstable when the classes are well separated.
Why LDA?
Introduction to Linear Discriminant Analysis

• LDA is a generalization of Fisher's linear discriminant, a method used in statistics, pattern

recognition and machine learning to find a linear combination of features that characterizes or
separates two or more classes of objects or events.

• The resulting combination may be used as a linear classifier, or, more commonly,
for dimensionality reduction before later classification.

Objectives of LDA

• Maximize between-class variance.

• Minimize within-class variance.

• Dimensionality Reduction
Linear Discriminant Analysis

• Linear discriminant analysis is a simple classification

method.

• LDA is used as a preprocessing step in machine learning.

• It works well when between sum of square is more and

within sum of square is less.

• LDA uses fisher's method.

LDA = SSB/SSW

• It will project the features in higher dimensional space

onto a lower dimensional space.
Assumptions of Linear Discriminant Analysis

1. It assumes that the independent variables are normally distributed

2. There is equal variance or covariance for the class

3. When the number of samples is small compared to the number of features, LDA may not
perform well due to issues with matrix inversion.

When these assumptions are satisfied, LDA creates a linear decision boundary.
Intuition behind Linear Discriminant Analysis

• Goal: Find a linear direction that separates classes as much as possible.

• Applications: Image recognition, finance, medical diagnosis, etc.

• Comparison with PCA: LDA focuses on class separation, while PCA maximizes overall
variance without class information.
Linear Discriminant Analysis : Mathematical Overview
Linear Discriminant Analysis : Step-by-Step Process
LDA:
Within-Class Scatter Matrix
Between-Class Scatter Matrix
Objective Function of LDA
Finding the Projection Vector w:
Choosing a Threshold for Classification
LDA
LDA
Advantages and Limitations of LDA

Advantages:

• Simple to implement and interpret.

• Effective for linear class separation.

Limitations:

• Assumes normally distributed data with equal covariance among classes.

• Struggles with non-linear class boundaries and overlapping distributions.

Applications of LDA

• Facial Recognition: LDA can distinguish between different faces by maximizing separability.

• Bioinformatics: Identifying gene expressions that distinguish between health conditions.

• Marketing: Customer segmentation based on purchase behavior.

Summary: Linear Discriminant Analysis

• LDA is used for dimensionality reduction while preserving class separability.

• The core steps include computing scatter matrices and finding the optimal projection vector w.

• LDA is useful in several fields but has limitations due to assumptions on data distribution.
Thank you