Classification

• Classification in machine learning is a

supervised learning task where a
model learns from labeled data to
assign new, unseen data points into
predefined categories or classes.
• Predicts a discrete outcome, unlike
regression which predicts a
continuous value.
• Key applications include spam email
detection, image recognition, and
disease diagnosis.
Classification - Applications
• Email Spam Filtering
• Problem: Emails need to be classified as either "Spam" or "Not
Spam."
• Application: Machine learning algorithms, like Naive Bayes or
Support Vector Machines (SVM), are trained on a large dataset of
emails that are labeled as "spam" or "not spam." Once trained, the
model can classify new incoming emails into these two categories
based on their content.
Classification - Applications
• Medical Diagnosis
• Problem: Identifying diseases based on patient data (e.g.,
symptoms, lab results).
• Application: Classification can be used in predicting whether a
patient has a specific disease (such as diabetes, cancer, etc.)
based on their medical history, test results, and other input
features. Algorithms like Decision Trees, Random Forests, or
Logistic Regression can be applied to make these predictions,
helping doctors diagnose conditions faster and more accurately.
Classification - Applications
• Sentiment Analysis
• Problem: Classifying the sentiment behind text, such as whether a
review is positive, negative, or neutral.
• Application: In customer service, social media monitoring, or
marketing, sentiment analysis can automatically classify
customer feedback, product reviews, or social media posts into
different categories like "positive," "negative," or "neutral." This
helps businesses understand customer sentiment at scale.
Classification - Applications
• Image Recognition
• Problem: Classifying images into predefined categories (e.g.,
identifying animals in photos).
• Application: Image classification models like Convolutional
Neural Networks (CNNs) can classify images into different
categories (such as "dog," "cat," "car," etc.) based on pixel
patterns. This is widely used in facial recognition, autonomous
vehicles, and medical image analysis (e.g., detecting tumors in
Xrays).
Classification - Applications
• Fraud Detection
• Problem: Detecting fraudulent activity in transactions (e.g., credit
card fraud).
• Application: Classification models can be trained to detect
whether a given transaction is legitimate or fraudulent based on
features such as transaction amount, location, time, and user
behavior. Algorithms like Random Forests or Neural Networks are
often used for realtime fraud detection.
Classification - Applications
• Object Detection in Video Surveillance
• Problem: Detecting specific objects or activities in video footage
(e.g., people, cars, suspicious behavior).
• Application: Classification models can be trained to identify and
track objects in video surveillance footage, helping to detect
security breaches, monitor traffic flow, or track specific activities.
What are classification algorithms?
• A classification algorithm is a categorization-focused machine
learning algorithm that sorts input data into different classes or
categories.
• Artificial intelligence (AI) models use classification algorithms to
process input datasets against a specified classifier that sets the
criteria for how the data should be sorted.
• Classification algorithms are widely used in data science for
forecasting patterns and predicting outcomes.
How do classification models work?
• Two-step data classification process:
• Step 1 – Learning/Training - Classification has traditionally been a type of
supervised machine learning, which means it uses labeled data to train models.
• In supervised learning, each data point in the training data contains input variables
(also known as independent variables or features), and an output variable, or label.
• Model’s job is to understand the relationships between features and class labels,
then apply those criteria to future datasets.
• Purpose of training is to minimize errors during predictive modeling.
• Unsupervised learning approaches to classification problems have been a key focus
of recent research. Enable models to discover patterns in unlabeled data by
themselves.
• Lack of labels is what differentiates unsupervised learning and supervised learning.
• Situations where obtaining large datasets of labeled data is not feasible,
semisupervised learning is a viable alternative.
How do classification models work?
• Two-step data classification process:
• Step 2 – Classification - In this phase, users deploy the model on
a test set of new data.
• Previously unused data is used to evaluate model performance.
• Avoid overfitting: when a model leans too heavily on its training
data and becomes unable to make accurate predictions in the
real world.
• Model uses its learned predicted function to classify new data
across distinct classes according to each sample’s features.
Quiz
[Link]
B9
 Types of Classification
• Types of classification problems
depending on how many categories (or
classes) we are working with and how
they are organized.
• Binary classification - sorts data into
two exclusive categories.
• Simplest Form
• Email is spam or not-spam.
• Multiclass classification - sorts data
into more than two exclusive categories.
• Model picks the one that best
matches the input.
• Picture of animal categorized to
dog/cat/bird.
Types of Classification
• Multilabel classification - sorts data into nonexclusive
categories.
• In multi-label classification single piece of data can belong to multiple
categories at once.
• Unlike multiclass classification where each data point belongs to only
one class, multi-label classification allows datapoints to belong to
multiple classes.
• A movie recommendation system could tag a movie as both action and
comedy.
How does Classification in Machine Learning
Work?
• Classification involves training a model using a labeled dataset, where each input is
paired with its correct output label.
• The model learns patterns and relationships in the data, so it can later predict labels
for new, unseen inputs.
• Data Collection: Start with a dataset where each item is labeled with the correct
class.
• Feature Extraction: The system identifies features (like color, shape, or texture) that
help distinguish one class from another.
• Model Training: Classification - machine learning algorithm uses the labeled data to
learn how to map the features to the correct class.
• Model Testing: Once the model is trained, it's tested on new, unseen data to check
how accurately it can classify the items.
• Prediction: After being trained and tested, the model can be used to predict the
class of new data based on the features it has learned.
Discrete and continuous predictions
• Predictive models output two types of predictions:
• Discrete predictions definitively sort data into distinct categories.
• For example, a healthcare predictor can classify medical patients
as diabetic or nondiabetic based on health data.
• The classes diabetic and nondiabetic are the discrete categorical predictions.
• Continuous predictions assign class predictions as continuous
probabilities called confidence scores.
• Probabilities are values between 0 and 1, representing percentages.
• Example: The diabetes predictor model might classify a patient as diabetic with a 0.82
probability.
• The model believes that the patient has an 82% chance of having diabetes.
• Researchers typically evaluate models by using discrete predictions while
using continuous predictions as thresholds. A classifier ignores any
prediction under a certain threshold.
Lazy and Eager Learners
• Lazy Learners
• It first stores the training dataset before waiting for the test dataset to arrive.
• Classification is carried out using the training dataset's most appropriate data.
• Less time is spent on training, but more time is spent on predictions.
• Some of the examples are case-based reasoning and the KNN algorithm.
• Eager Learners
• Before obtaining a test dataset, eager learners build a classification model using
a training dataset.
• They spend more time studying and less time predicting.
• Some of the examples are ANN, naive Bayes, and Decision trees.
Classification vs. Regression
• The difference between classification and regression is that while
classification predicts a data point’s category, regression predicts
an associated real numerical value.
• Both classification and regression are types of predictive
modeling but with distinct use cases.
• Two are often closely related. For example, the logistic regression
algorithm uses regression to fulfill classification tasks.
Numerical
• Given X (input) = [2.5, 3.7, 1.8, 4.0, 5.2], Y (predictions) = [2.1, 3.9,
1.7, 3.8, 5.0]. Compute – MAE, MSE, R-Square. Ans: MAE=0.22,
MSE = 0.0579, R-Square=0.9588
Classification Algorithms
• Logistic Regression (Linear Classifier)
• Decision tree (Non-linear Classifier)
• Random forest (Non-linear Classifier)
• Support vector machine (SVM) (Non-linear Classifier)
• K-nearest neighbors (Non-linear Classifier)
• Naive Bayes (Non-linear Classifier)
Logistic Regression
• Logistic Regression is a supervised
machine learning algorithm used for
classification problems.
• Predicts the probability that an input
belongs to a specific class.
• Used for binary classification where the
output can be one of two possible
categories such as Yes/No, True/False or
0/1.
• Uses sigmoid function to convert inputs
into a probability value between 0 and 1.
• A threshold value is used in logistic
regression to make decisions based on
these probabilities. For instance, if
predicted probability is above threshold
0.5, it results to 1/True/Yes otherwise
0/False/No.
Logistic Regression - Types
• Binomial Logistic Regression: This type is used when the dependent
variable has only two possible categories. Examples include Yes/No,
Pass/Fail or 0/1. It is the most common form of logistic regression and
is used for binary classification problems.
• Multinomial Logistic Regression: This is used when the dependent
variable has three or more possible categories that are not ordered. For
example, classifying animals into categories like "cat," "dog" or "sheep."
It extends the binary logistic regression to handle multiple classes.
• Ordinal Logistic Regression: This type applies when the dependent
variable has three or more categories with a natural order or ranking.
Examples include ratings like "low," "medium" and "high." It takes the
order of the categories into account when modeling.
Logistic Regression – How to Perform?
• Import Libraries
• Load the dataset
• Convert the target variable to binary (e.g. 1 for diabetes, 0 for no diabetes)
• Split the data into training and testing sets
• Standardize features
• Train the Logistic Regression model
• Evaluate the model
• Visualize the decision boundary with accuracy information (Scatter Plot)
• [Link]
BXmY
Decision Tree
• Decision Tree is a supervised learning method used in data mining for
classification and regression methods.
• Helps us in decision making purposes
• Creates classification or regression models as a tree structure.
• A decision tree in machine learning is a non-parametric supervised
learning algorithm used for both classification and regression tasks.
• It operates by building a model in the form of a tree structure, where
each internal node represents a "test" on an attribute (a feature of the
data), each branch represents the outcome of that test, and each leaf
node (terminal node) represents a class label (for classification) or a
predicted value (for regression).
Key Components of a Decision Tree
• Root Node: The starting point of the tree, representing the entire
dataset and the initial decision point.
• Internal Nodes (Decision Nodes): Nodes where a decision is
made based on the value of a specific attribute, leading to further
branches.
• Branches: The connections between nodes, representing the
possible outcomes of a decision or test.
• Leaf Nodes (Terminal Nodes): The end points of the tree,
representing the final prediction or classification outcome.
Decision Tree – How it works?
• Recursively splits the dataset into subsets based on the values of
the attributes.
• At each internal node, the algorithm selects the "best" attribute to
split the data, aiming to maximize the information gain or minimize
impurity (e.g., using Gini impurity or entropy).
• Process continues until a stopping criterion is met, such as
reaching a maximum tree depth, having a minimum number of
samples in a node, or when further splitting does not significantly
improve the model
Choosing Best Attribute
• Two popular attribute selection measures used:
• Information Gain
• Gini Index
• Information Gain
• tells us how useful a question (or feature) is for splitting data into groups.
• measures how much the uncertainty decreases after the split.
• feature with the highest Information Gain is chosen to make the decision.
Information Gain
• Entropy - measures the impurity of the sample values. Defined
with by the following formula:

• Here, S represents the data set that entropy is calculated; c

represents the classes in set, S; p(c) represents the proportion of
data points that belong to class c to the number of total data
points in set, S.
• Entropy values can fall between 0 and 1.
Information Gain
• Entropy - If all samples in data set, S, belong to one class, then entropy
will equal zero.
• If half of the samples are classified as one class and the other half are
in another class, entropy will be at its highest at 1.
• To select the best feature to split on and find the optimal decision tree,
the attribute with the smallest amount of entropy should be used.
• Information gain represents the difference in entropy before and
after a split on a given attribute.
• The attribute with the highest information gain will produce the best
split as it’s doing the best job at classifying the training data according
to its target classification.
Information Gain
• Information gain is usually represented with the following formula:

• Here, a represents a specific attribute or class label; Entropy(S) is

the entropy of dataset, S; |Sv|/|S| represents the proportion of the
values in Sv to the number of values in dataset, S.
Decision Tree – Example Information Gain
• Sample Data – Total instance –
14, Play Tennis – Yes (9), No(5).
• Entropy (tennis) = 0.94

• Information gain for each

attribute is now computed.
Information Gain Example
Gain (Tennis, Humidity) = (0.94)-(7/14)*(0.985) – (7/14)*(0.592) =
0.151.
• 7/14 represents the proportion of values where humidity equals “high”
to the total number of humidity values. In this case, the number of
values where humidity equals “high” is the same as the number of
values where humidity equals “normal”.
• - 0.985 is the entropy when Humidity = “high”
• - 0.59 is the entropy when Humidity = “normal”
• Repeat the calculation for information gain for each attribute.
• Select the attribute with the highest information gain to be the first split
point in the decision tree.
Gini Index
• Gini Index is a metric to measure how often a randomly chosen
element would be incorrectly identified.
• Attribute with a lower Gini index should be preferred.
• Formula for Gini Index is given by :
• For example, if we have a group of people where all bought the
product (100% "Yes") the Gini Index is 0 indicate perfect purity.
• But if the group has an equal mix of "Yes" and "No" the Gini Index
would be 0.5 show high impurity or uncertainty.
Gini Index – Computation Example
• Dataset - target class label is “Buys_insurance” and it can take
two values “Yes” or “No”.
• Determine the best feature to use as the root node for the
decision tree.
• Calculate the Gini impurity for each feature and select the feature
with the lowest Gini impurity.
Age Gender Income Credit Score Buys_insurance

20 Male High Excellent Yes

25 Female High Fair No

30 Male High Excellent Yes

35 Female Medium Excellent Yes

40 Male Low Fair Yes

45 Female Low Poor No

Gini Index – Computation Example
• Calculate the Gini impurity of the target class label with following
formula:
• Gini impurity = 1 — (p(Yes))²- (p(No))²
• here p(Yes) and p(No) are the class probabilities.
• In this dataset, the class probabilities are:
• p(Yes) = 4/6 = 0.67 p(No) = 2/6 = 0.33
• so,
• Gini impurity = 1 — (0.67)² — (0.33)² = 0.48
Gini Index – Computation Example
• Calculate the Gini impurity for each feature using the formula:
• Gini impurity = 1 — (p(Feature=Value1))²- (p(Feature=Value2))²- …
-(p(Feature=ValueN))²
• Gini impurity for feature “Age”
• p(Age=20) = 1/6 = 0.17 p(Age=25) = 1/6 = 0.17
• p(Age=30) = 1/6 = 0.17 p(Age=35) = 1/6 = 0.17
• p(Age=40) = 1/6 = 0.17 p(Age=45) = 1/6 = 0.17
• Gini impurity for feature “Age” = 1 — (0.17)² — (0.17)² — (0.17)² —
(0.17)² — (0.17)² — (0.17)² = 1
Gini Index – Computation Example
• Gini impurity for feature “Gender”
• p(Gender=Male) = 3/6 = 0.5
• p(Gender=Female) = 3/6 = 0.5
• Gini impurity for feature “Gender” = 1 — (0.5)² — (0.5)² = 0.5
• Gini impurity for feature “Income”
• p(Income=High) = 3/6 = 0.5
• p(Income=Medium) = 1/6 = 0.17
• p(Income=Low) = 2/6 = 0.33
• Gini impurity for feature “Income” = 1 — (0.5)² — (0.17)² — (0.33)²
= 0.44
Gini Index – Computation Example
• Gini impurity for feature “Credit Score”
• p(Credit Score=Excellent) = 3/6 = 0.5
• p(Credit Score=Fair) = 2/6 = 0.33
• p(Credit Score=Poor) = 1/6 = 0.17
• Gini impurity for feature “Credit Score” = 1 — (0.5)² — (0.33)² —
(0.17)² = 0.44
• Feature “Income” and “Credit Score” have the lowest Gini
impurity of 0.44.
• Select either one of them as the root node for the decision
tree.
Random Forest Classification
• Machine learning algorithm that uses many decision trees during
training to make better predictions.
• Each tree looks at different random parts of the data, and their results
are combined by voting for classification or averaging for regression.
• It handles both classification and regression problems.
• Helps in improving accuracy and reducing errors.
• Ensemble Method - it combines multiple individual models (decision
trees) to achieve better predictive performance than a single model.
Popular ensemble methods include bagging, boosting, and stacking.
Ensemble Methods
• Bagging (Bootstrap Aggregating):
• Creates multiple training datasets by sampling with replacement from the original
dataset.
• Each model is trained independently on a different dataset, and predictions are
combined through averaging or voting.
• Boosting:
• Builds models sequentially, with each new model focusing on correcting the errors
made by the previous ones.
• Models are typically assigned weights based on their performance, and the final
prediction is a weighted combination of all models.
• Stacking:
• Combines models in a hierarchical manner.
• First, several diverse models (base learners) are trained on the data.
• Then, a new model (meta-learner) is trained to learn how to best combine the
predictions of these base learners to produce the final output.
Bagging
• Begins with any original random
data.
• Organised into samples known as
Bootstrap Sample (Bootstraping)
• Models are trained individually,
yielding different results known as
Aggregation
• Lastly, all the results are
combined, and the generated
output is based on majority
voting. Known as Bagging and is
done using an Ensemble
Classifier
Working of Random Forest
• Create Many Decision Trees
• Pick Random Features
• Helps to make multiple decision
trees
• Each Tree Makes a Prediction
• Combine the Predictions
• For classification use majority voting
• For regression use average value

• Helps avoid overfitting and makes more accurate predictions.

Features of Random Forest
• Handles Missing Data: Work even if some data is missing. So, you
don’t always need to fill in the gaps yourself.
• Shows Feature Importance: It tells you which features (columns)
are most useful for making predictions which helps you
understand your data better.
• Works Well with Big and Complex Data: It can handle large
datasets with many features without slowing down or losing
accuracy.
• Used for Different Tasks: You can use it for both classification like
predicting types or labels and regression like predicting numbers
or amounts.
Random Forest for Classification – How to
use?
• Import libraries (like pandas, scikit learn)
• Load Dataset
• Remove rows with missing target values
• Select features
• Split data with train and test
• Train random forest model
• Make Predictions

[Link]
FVtx7iLG
Random Forest based Regressor
• [Link]
7LQOXC54KUgjis
 Random Forest vs. Decision Tree
• Problem of overfitting doesn’t • Usually suffer from the problem of
happen. overfitting.
• It is slower. • Single decision tree is
comparatively faster in
computation.
• Random Forest randomly
selects observations, builds a • Use a particular set of rules when
decision tree and then the result a data set with features are taken
is obtained based on majority as input.
voting. No formulas are required
here.
Naïve Bayes Classifier
• The Naïve Bayes classifier is a supervised machine learning algorithm
that is used for classification tasks such as text classification.
• Use principles of probability to perform classification tasks.
• Assumes that all features are independent of each other. (why it is
called "naive.”).
• Calculates the probability of a sample belonging to a particular class
based on the probabilities of its features. E.g. a phone may be
considered as smart if it has touch-screen, internet facility, good
camera, etc.
• Naive Bayes Algorithm is used in spam filtration, Sentimental analysis,
classifying articles and many more.
Naïve Bayes Classifier
• In Bayesian classification, the main interest is to find the posterior
probabilities i.e. the probability of a label given some observed
features, P(L | features).
• Bayes theorem, we can express this in quantitative form as
follows:

P(y∣X): Posterior probability, probability of class 𝑦 given features 𝑋

𝑃 𝑋 ∣ 𝑦 :Likelihood, probability of features 𝑋 given class 𝑦
𝑃 𝑦 :Prior probability of class 𝑦
𝑃 𝑋 :Marginal likelihood or evidence
Naïve Bayes Working
• Terms
• Consider a classification problem (like predicting if
someone plays golf based on weather).
• y is the class label (e.g. "Yes" or "No" for playing golf)
• 𝑋 = 𝑥1 𝑥2 . . . 𝑥𝑛 is the feature vector (e.g.
Outlook, Temperature, Humidity, Windy).
• A sample row from the dataset:
𝑋 = (Rainy, Hot, High, False), 𝑦 = No
What is the probability that someone will not play golf
given that the weather is Rainy, Hot, High humidity,
and No wind?
Naïve Bayes Working
• Naïve Assumption
• The "naive" in Naive Bayes comes from the assumption that all
features are independent given the class.

• Use Bayes theorem:

• Construct Naïve classifier: compute the posterior for each
class 𝑦 and choose the class with the highest probability:
Naïve Bayes Classifier - Example
Take a dataset used for predicting if golf is played based on:
• Outlook: Sunny, Rainy, Overcast
• Temperature: Hot, Mild, Cool
• Humidity: High, Normal
• Windy: True, False

Example Input: 𝑋 = ( 𝑆𝑢𝑛𝑛𝑦, 𝐻𝑜𝑡, 𝑁𝑜𝑟𝑚𝑎𝑙, 𝐹𝑎𝑙𝑠𝑒)

Goal: Predict if golf will be played (Yes or No).
Naïve Bayes Classifier - Example
• Pre-computation from Dataset
• Class Probabilities: From dataset of 14 rows.
P(Yes)= 9/14
P(No)=5/14
•

• Conditional Probabilities
Naïve Bayes Classifier - Example
• Calculate Posterior Probabilities:
• For Class = Yes:

• For Class = No:

• Normalize Probabilities: To Compare

• Final Prediction:
Naive Bayes for Continuous Features
• For continuous features, assume a Gaussian distribution:

• Where:
• 𝜇𝑦 is the mean of feature 𝑥𝑖 for class 𝑦
• 𝜎𝑦2 is the variance of feature 𝑥𝑖 for class 𝑦
This leads to what is called Gaussian Naive Bayes.
Types of Naïve Bayes
• Gaussian Naive Bayes - continuous values associated with each
feature are assumed to be distributed according to a Gaussian
distribution.
• Also known as normal distribution.
• Gives a bell shaped curve which is symmetric.
• Multinomial Naive Bayes - used when features represent the
frequency of terms (such as word counts) in a document.
• Commonly applied in text classification, where term frequencies are important.
• Bernoulli Naive Bayes - deals with binary features, where each feature
indicates whether a word appears or not in a document.
• Suited for scenarios where the presence or absence of terms is more relevant
than their frequency.
Applications of Naïve Bayes
• Spam Email Filtering: Classifies emails as spam or non-spam
based on features.
• Text Classification: Used in sentiment analysis, document
categorization, and topic classification.
• Medical Diagnosis: Helps in predicting the likelihood of a disease
based on symptoms.
• Credit Scoring: Evaluates creditworthiness of individuals for loan
approval.
• Weather Prediction: Classifies weather conditions based on
various factors.
K-NN Classifier
• A simple, non-parametric machine learning algorithm that
categorizes new data points based on the majority class of their
"k" closest neighbors in the training data.
• Works by calculating the distance between a new data point and all
existing data points.
• Assigning the new point to the class that is most frequent among the "k"
nearest neighbors.
• It is a lazy-learning algorithm that stores the entire training dataset
and performs calculations only during the prediction phase.
K-NN Classifier – How it works?
• Store data: The algorithm stores the entire training dataset.
• Calculate distance: When a new data point arrives, it calculates
the distance (e.g., Euclidean or Manhattan) between this new
point and every other point in the dataset.
• Identify neighbors: It finds the "k" data points closest to the new
point based on the calculated distances.
• Vote for classification: It looks at the class labels of these "k"
neighbors and determines which class is the most frequent.
• Assign class: The new data point is assigned to the class that has
the majority of votes among its "k" neighbors.
What is 'K' in K Nearest Neighbour
• k is just a number that tells the algorithm how many nearby points
or neighbors to look at when it makes a decision.
• Choosing the right k is important for good results.
• If the data has lots of noise or outliers, using a larger k can make the
predictions more stable.
• Large value of k the model may become too simple and miss important
patterns (underfitting).
• k should be picked carefully based on the data.
Choosing K
• Cross-Validation: Good way to find the best value of k is by using k-fold cross-
validation.
• Divide the dataset into k parts.
• Model is trained on some of these parts and tested on the remaining ones.
• Repeated for each part.
• The k value that gives the highest average accuracy during these tests is usually the best one
to use.
• Elbow Method: Draw a graph showing the error rate or accuracy for different k
values.
• As k increases the error usually drops at first.
• After a certain point error stops decreasing quickly.
• The point where the curve changes direction and looks like an "elbow" is usually the best
choice for k.
• Odd Values for k: It’s a good idea to use an odd number for k especially in
classification problems.
• Helps to avoid ties when deciding the common class.
Distance Formula for K-NN
• Euclidean distance (p=2): This is the most commonly used
distance measure, and it is limited to real-valued vectors.
• Measures a straight line between the query point and the other
point being measured.

• Manhattan Distance: total distance you would travel if you could

only move along horizontal and vertical lines like a grid or city
streets. It’s also called "taxicab distance“.
Distance Formula for K-NN
• Minkowski Distance: includes both Euclidean and Manhattan
distances as special cases.

• when p=2, it becomes the same as the Euclidean distance

formula and when p=1, it turns into the Manhattan distance
formula.
• [Link]
kbT-5-SkUjS2ov
Support Vector Machine
Confusion
Matrix
• A confusion matrix is a
matrix that summarizes
the performance of a
machine learning model
on a set of test data.
• Used to measure the
performance of
classification models.
Terms in Confusion Matrix
• True Positive (TP): The model correctly predicted a positive
outcome (the actual outcome was positive).
• True Negative (TN): The model correctly predicted a negative
outcome (the actual outcome was negative).
• False Positive (FP): The model incorrectly predicted a positive
outcome (the actual outcome was negative). Also known as a
Type I error.
• False Negative (FN): The model incorrectly predicted a negative
outcome (the actual outcome was positive). Also known as a Type
II error.
Terms in Confusion Matrix
• Accuracy: how many predictions the model got right out of all the
predictions.
• Accuracy = (TP+TN)/(TP+FP+FN+TN)
• Precision: Measures the accuracy of positive predictions.
• Precision = TP/(TP+FP)
• Recall: Measures the completeness of positive predictions.
• Recall = TP/(TP+FN)
• F1score: combines precision and recall into a single metric to
balance their trade-off.
• F1 = 2(Precision*Recall)/(Precision+Recall)
Terms in Confusion Matrix
• Specificity: important metric in the evaluation of classification
models particularly in binary classification. Measures the ability of
a model to correctly identify negative instances.
• Specificity = TN / (TN + FP)
• Sensitivity: A performance metric that can be derived from a
confusion matrix. Measures a classification model's ability to
correctly identify all actual positive cases
• Sensitivity = TP / (TP + FN)
Example- Confusion Matrix
• Imagine a medical test designed to detect a disease. The
confusion matrix shows the following results for a group of 165
patients:
Numerical
• Compute Precision if TP=50, FP=10, FN=5, No TN provided. (ans:
83.3%).
• Compute Senstivity, Specificity, Recall, F1 Score, Precision,
Accuracy for the following data:
Clustering
• is a data analysis technique used to group similar data points
together based on their characteristics.
• Form of unsupervised learning, where the algorithm tries to
identify patterns without predefined labels.
• Widely used in various fields, including data mining, machine
learning, image processing, and market research.
Clustering Approaches/Methods
• Partition-based - divides the dataset into a predefined number of
clusters (k).
• Hierarchical - creates a treelike structure (dendrogram) to
represent data relationships, which can be either agglomerative
(bottomup) or divisive (topdown).
• Density-based - identifies clusters based on the density of data
points in the feature space. Clusters are formed by areas of high
density separated by areas of low density.
Partition-based Clustering
Example: KMeans is the most well-known algorithm in this category.
• Process: Choose the number of clusters (k).
• Randomly initialize k centroids.3
• Assign each data point to the nearest centroid.
• Update centroids by calculating the mean of the assigned points.
• Repeat the assignment and update steps until convergence.
• Pros – Efficient for large datasets, Easy implementation
• Cons – Need to mention number of clusters, Sensitive to Outliers,
May converge to a local minimum.
Hierarchical Clustering
Approaches
• Agglomerative Approach:
• Start with each data point as its own cluster. Merge the closest pair of
clusters iteratively until only one cluster remains or a stopping criterion is
met.
• Divisive Approach:
• Start with one cluster containing all data points. Split the cluster into
smaller clusters iteratively.
• Pros – No need to mention number of clusters, Represent
clustering visually as dendogram
• Cons – Computationally Intensive, Sensitive to noise and outliers.
Density-based Clustering
• Example: DBSCAN (DensityBased Spatial Clustering of
Applications with Noise) is a popular algorithm.
• Process:
• Define a radius (epsilon) and a minimum number of points (minPts).
• Identify core points (points in dense regions).
• Expand clusters from core points by connecting neighboring points that are within
epsilon.
• Mark points not belonging to any cluster as noise.
• Pros – Can find arbitrarily shaped clusters, Robust to outliers
• Cons – Performance degrade with varying density, Challenging to
choose right parameters.