Classification Algorithms: Introduction to supervised learning, Decision
trees and random forests, Naive Bayes classifier, Model evaluation metrics
(accuracy, precision, recall, F1-score)
classification algorithms :
Machine learning classification algorithms are essential tools used to
categorize data into predefined classes based on learned patterns. From
simple linear models to advanced neural networks, these algorithms are
used in applications like spam detection, image recognition, sentiment
analysis and medical diagnosis.
Includes Logistic Regression, Decision Trees, Random Forest, SVM,
KNN, Naive Bayes and more
Each algorithm has unique strengths suited for different datasets
and problem types
Widely used across AI systems for fast, accurate and scalable
prediction tasks
Supervised learning :
Supervised learning is a type of machine learning where a model learns
from labelled data, meaning each input has a correct output. The model
compares its predictions with actual results and improves over time to
increase accuracy.
Its main features are:
Labelled Data: Each input has a known output
Learning from Errors: Adjusts itself to reduce prediction errors
Goal: Make accurate predictions on new data
Example: Recognizing handwritten digits from trained data
Types of Supervised Learning
Now, Supervised learning can be applied to two main types of problems:
Classification: Where the output is a categorical variable (e.g.,
spam vs. non-spam emails, yes vs. no).
Regression: Where the output is a continuous variable (e.g.,
predicting house prices, stock prices).
Working of Supervised Machine Learning
The working of supervised machine learning follows these key steps:
1. Collect Labeled Data
Gather a dataset where each input has a known correct output
(label).
Example: Images of handwritten digits with their actual numbers as
labels.
2. Split the Dataset
Divide the data into training data (about 80%) and testing data
(about 20%).
The model will learn from the training data and be evaluated on the
testing data.
3. Train the Model
Feed the training data (inputs and their labels) to a suitable
supervised learning algorithm (like Decision Trees, SVM or Linear
Regression).
The model tries to find patterns that map inputs to correct outputs.
4. Validate and Test the Model
Evaluate the model using testing data it has never seen before.
The model predicts outputs and these predictions are compared
with the actual labels to calculate accuracy or error.
5. Deploy and Predict on New Data
Once the model performs well, it can be used to predict outputs for
completelySupervised Machine Learning
Its main features are:
Labelled Data: Each input has a known output
Learning from Errors: Adjusts itself to reduce prediction errors
Goal: Make accurate predictions on new data
Example: Recognizing handwritten digits from trained data
Types of Supervised Learning
Now, Supervised learning can be applied to two main types of
problems:
Classification: Where the output is a categorical variable (e.g.,
spam vs. non-spam emails, yes vs. no).
Regression: Where the output is a continuous variable (e.g.,
predicting house prices, stock prices).
Working of Supervised Machine Learning
The working of supervised machine learning follows these key steps:
1. Collect Labeled Data
Gather a dataset where each input has a known correct output
(label).
Example: Images of handwritten digits with their actual numbers as
labels.
2. Split the Dataset
Divide the data into training data (about 80%) and testing data
(about 20%).
The model will learn from the training data and be evaluated on the
testing data.
3. Train the Model
Feed the training data (inputs and their labels) to a suitable
supervised learning algorithm (like Decision Trees, SVM or Linear
Regression).
The model tries to find patterns that map inputs to correct outputs.
4. Validate and Test the Model
Evaluate the model using testing data it has never seen before.
The model predicts outputs and these predictions are compared
with the actual labels to calculate accuracy or error.
5. Deploy and Predict on New Data
Once the model performs well, it can be used to predict outputs for
completely new, unseen data.
Supervised Machine Learning Algorithms
Supervised learning can be further divided into several different
types, each with its own unique characteristics and applications.
Here are some of the most common types of supervised learning
algorithms:
Linear Regression: Linear regression is a type of supervised
learning regression algorithm that is used to predict a continuous
output value. It is one of the simplest and most widely used
algorithms in supervised learning.
Decision Trees : Decision tree is a tree-like structure that is used to
model decisions and their possible consequences. Each internal
node in the tree represents a decision, while each leaf node
represents a possible outcome.
Random Forests: Random forests again are made up of multiple
decision trees that work together to make predictions. Each tree in
the forest is trained on a different subset of the input features and
data. The final prediction is made by aggregating the predictions of
all the trees in the forest.
Naive Bayes Algorithm: The Naive Bayes algorithm is a
supervised machine learning algorithm based on applying Bayes'
Theorem with the “naive” assumption that features are independent
of each other given the class label.
Decision Trees :
A decision tree is a supervised learning algorithm used for both
classification and regression tasks. It has a hierarchical tree
structure which consists of a root node, branches, internal nodes
and leaf nodes. It works like a flowchart that helps in making step by
step decision, where:
Internal nodes represent attribute tests
Branches represent attribute values
Leaf nodes represent final decisions or predictions.
Decision trees are widely used due to their interpretability, flexibility
and low preprocessing needs.
How Does a Decision Tree Work
A decision tree splits the dataset based on feature values to create
pure subsets ideally all items in a group belong to the same class.
Each leaf node of the tree corresponds to a class label and the
internal nodes are feature-based decision points. Let’s understand
this with an example.
Let’s consider a decision tree for predicting whether a customer will
buy a product based on age, income and previous purchases: Here's
how the decision tree works:
1. Root Node (Income)
First Question: "Is the person’s income greater than
$50,000?"
If Yes, proceed to the next question.
If No, predict "No Purchase" (leaf node).
2. Internal Node (Age):
If the person’s income is greater than $50,000, ask: "Is the
person’s age above 30?"
If Yes, proceed to the next question.
If No, predict "No Purchase" (leaf node).
3. Internal Node (Previous Purchases):
If the person is above 30 and has made previous purchases, predict
"Purchase" (leaf node).
If the person is above 30 and has not made previous purchases,
predict "No Purchase" (leaf node).
Random Forest:
Random Forest is a machine learning algorithm that uses many
decision trees to make better predictions. Each tree looks at
different random parts of the data and their results are combined by
voting for classification or averaging for regression which makes it
as ensemble learning technique. This helps in improving accuracy
and reducing errors.
Working of Random Forest Algorithm
Create Many Decision Trees: The algorithm makes many decision
trees each using a random part of the data. So every tree is a bit
different.
Pick Random Features: When building each tree it doesn’t look at
all the features (columns) at once. It picks a few at random to
decide how to split the data. This helps the trees stay different from
each other.
Each Tree Makes a Prediction: Every tree gives its own answer or
prediction based on what it learned from its part of the data.
Combine the Predictions: For classification we choose a category
as the final answer is the one that most trees agree on i.e majority
voting and for regression we predict a number as the final answer is
the average of all the trees predictions.
Why It Works Well: Using random data and features for each tree
helps avoid overfitting and makes the overall prediction more
accurate and trustworthy.
Key Features of Random Forest
Handles Missing Data: It can work even if some data is missing so
you don’t always need to fill in the gaps yourself.
Shows Feature Importance: It tells you which features (columns)
are most useful for making predictions which helps you understand
your data better.
Works Well with Big and Complex Data: It can handle large
datasets with many features without slowing down or losing
accuracy.
Used for Different Tasks: You can use it for
both classification like predicting types or labels
and regression like predicting numbers or amounts.
Assumptions of Random Forest
Each tree makes its own decisions: Every tree in the forest
makes its own predictions without relying on others.
Random parts of the data are used: Each tree is built using
random samples and features to reduce mistakes.
Enough data is needed: Sufficient data ensures the trees are
different and learn unique patterns and variety.
Different predictions improve accuracy: Combining the
predictions from different trees leads to a more accurate final result.
Implementing Random Forest for Classification Tasks
Here we will predict survival rate of a person in titanic.
You can download dataset
Import libraries like pandas and scikit learn.
Load the Titanic dataset.
Remove rows with missing target values ('Survived').
Select features like class, sex, age, etc and convert 'Sex' to
numbers.
Fill missing age values with the median.
Split the data into training and testing sets, then train a Random
Forest model.
Predict on test data, check accuracy and print a sample prediction
result.
Advantages of Random Forest
Random Forest provides very accurate predictions even with large
datasets.
Random Forest can handle missing data well without compromising
with accuracy.
It doesn’t require normalization or standardization on dataset.
When we combine multiple decision trees it reduces the risk of
overfitting of the model.
Limitations of Random Forest
It can be computationally expensive especially with a large number
of trees.
It’s harder to interpret the model compared to simpler models like
decision trees.
Naive Bayes Classifiers
Naive Bayes is a machine learning classification algorithm that predicts the
category of a data point using probability. It assumes that all features are
independent of each other. Naive Bayes performs well in many real-world
applications such as spam filtering, document categorisation and sentiment
analysis.
here:
Original data has two classes: green circles (y = 1) and red squares (y = 2).
Estimate probability distribution along the first dimension i.e P(x_1 \mid y=1), \;
P(x_1 \mid y=2)
Estimate probability distribution along the second dimension i.e P(x_2 \mid y=1), \;
P(x_2 \mid y=2)
Combine both dimensions using conditional independence i.e P(x \mid y) = \prod_{\
alpha} P(x_{\alpha} \mid y)
Key Features of Naive Bayes Classifiers
The main idea behind the Naive Bayes classifier is to use Bayes' Theorem to
classify data based on the probabilities of different classes given the features of the
data. It is used mostly in high-dimensional text classification
The Naive Bayes Classifier is a simple probabilistic classifier and it has very few
number of parameters which are used to build the ML models that can predict at a
faster speed than other classification algorithms.
It is a probabilistic classifier because it assumes that one feature in the model is
independent of existence of another feature. In other words, each feature contributes
to the predictions with no relation between each other.
Naive Bayes Algorithm is used in spam filtration, Sentimental analysis, classifying
articles and many more
.
Why it is Called Naive Bayes?
It is named as "Naive" because it assumes the presence of one feature does not
affect other features. The "Bayes" part of the name refers to its basis in Bayes’
Theorem.
Consider a fictional dataset that describes the weather conditions for playing a game
of golf. Given the weather conditions, each tuple classifies the conditions as
fit(“Yes”) or unfit(“No”) for playing golf. Here is a tabular representation of our
dataset.
The dataset is divided into two parts i.e feature matrix and the response vector.
Feature matrix contains all the vectors(rows) of dataset in which each vector
consists of the value of dependent features. In above dataset, features are ‘Outlook’,
‘Temperature’, ‘Humidity’ and ‘Windy’.
Response vector contains the value of class variable (prediction or output) for each
row of feature matrix. In above dataset, the class variable name is ‘Play golf’.
Assumption of Naive Bayes
The fundamental Naive Bayes assumption is that each feature makes an:
Feature independence: This means that when we are trying to classify something,
we assume that each feature (or piece of information) in the data does not affect any
other feature.
Continuous features are normally distributed: If a feature is continuous, then it is
assumed to be normally distributed within each class.
Discrete features have multinomial distributions: If a feature is discrete, then it is
assumed to have a multinomial distribution within each class.
Features are equally important: All features are assumed to contribute equally to
the prediction of the class label.
No missing data: The data should not contain any missing values.
Advantages
Easy to implement and computationally efficient.
Effective in cases with a large number of features.
Performs well even with limited training data.
It performs well in the presence of categorical features.
For numerical features data is assumed to come from normal distributions
Disadvantages
Assumes that features are independent, which may not always hold in real-world
data.
Can be influenced by irrelevant attributes.
May assign zero probability to unseen events, leading to poor generalization.
Applications
Spam Email Filtering: Classifies emails as spam or non-spam based on features.
Text Classification: Used in sentiment analysis, document categorization and topic
classification.
Medical Diagnosis: Helps in predicting the likelihood of a disease based on
symptoms.
Credit Scoring: Evaluates creditworthiness of individuals for loan approval.
Weather Prediction: Classifies weather conditions based on various factors.
Evaluation metrics
Evaluation metrics are used to measure how well a machine learning model
performs. They help assess whether the model is making accurate predictions and
meeting the desired goals. This is important because:
Model Performance : Measures how well the model works
Different Tasks : Used for classification, regression and clustering
Right Metric Choice : Helps select the best way to evaluate a model
Better Decisions : Ensures the model meets its objectives
Classification Metrics
Classification problems aim to predict discrete categories. To evaluate the
performance of classification models, we use the following metrics:
1. Accuracy
Accuracy is a fundamental metric used for evaluating the performance of a
classification model. It tells us the proportion of correct predictions made by the
model out of all predictions.
NumberofCorrectPredictions
Accuracy=
TotalNumberofPredictions
While accuracy provides a quick snapshot, it can be misleading in cases of
imbalanced datasets. For example, in a dataset with 90% class A and 10% class B, a
model predicting only class A will still achieve 90% accuracy but it will fail to
identify any class B instances.
Accuracy is good but it gives a False Positive sense of achieving high accuracy. The
problem arises due to the possibility of misclassification of minor class samples
being very high.
2. Precision
It measures how many of the positive predictions made by the model are actually
correct. It's useful when the cost of false positives is high such as in medical
diagnoses where predicting a disease when it’s not present can have serious
consequences.
TP
Precision=
TP+ FP
Where:
TP = True Positives
FP = False Positives
Precision helps ensure that when the model predicts a positive outcome, it’s likely to
be correct.
3. Recall
Recall or Sensitivity measures how many of the actual positive cases were correctly
identified by the model. It is important when missing a positive case (false negative)
is more costly than false positives.
TP
Recall=
TP+ FN
Where:
FN = False Negatives
In scenarios where catching all positive cases is important (like disease detection),
recall is a key metric.
4. F1 Score
The F1 Score is the harmonic mean of precision and recall. It is useful when we
need a balance between precision and recall as it combines both into a single
number. A high F1 score means the model performs well on both metrics. Its range
is [0,1].
Lower recall and higher precision gives us great accuracy but then it misses a large
number of instances. More the F1 score better will be performance. It can be
expressed mathematically in this way:
Precision × Recall
F1 Score=2 ×
Precision +Recall
There are 4 terms we should keep in mind:
1. True Positives: It is the case where we predicted Yes and the real output was also
Yes.
2. True Negatives: It is the case where we predicted No and the real output was also
No.
3. False Positives: It is the case where we predicted Yes but it was actually No.
4. False Negatives: It is the case where we predicted No but it was actually Yes.