Machine Learning with Python

Unit – 2

Supervised Learning Algorithms:

Decision Tree
A decision tree is a supervised learning algorithm used for both classification and
regression tasks. It has a hierarchical tree structure which consists of a root node,
branches, internal nodes and leaf nodes. It It works like a flowchart help to make
decisions step by step where:
• Internal nodes represent attribute tests
• Branches represent attribute values
• Leaf nodes represent final decisions or predictions.
Decision trees are widely used due to their interpretability, flexibility and low
preprocessing needs.

How Does a Decision Tree Work?

A decision tree splits the dataset based on feature values to create pure subsets ideally
all items in a group belong to the same class. Each leaf node of the tree corresponds
to a class label and the internal nodes are feature-based decision points. Let’s
understand this with an example.
Here's how the decision tree works:
1. Root Node (Income)
First Question: "Is the person’s income greater than $50,000?"
• If Yes, proceed to the next question.
• If No, predict "No Purchase" (leaf node).
2. Internal Node (Age):
If the person’s income is greater than $50,000, ask: "Is the person’s age above 30?"
 • If Yes, proceed to the next question.
• If No, predict "No Purchase" (leaf node).
3. Internal Node (Previous Purchases):
• If the person is above 30 and has made previous purchases, predict "Purchase"
(leaf node).
• If the person is above 30 and has not made previous purchases, predict "No
Purchase" (leaf node).

Decision making with 2 Decision Tree

Example: Predicting Whether a Customer Will Buy a Product Using Two Decision
Trees
Tree 1: Customer Demographics
First tree asks two questions:
1. "Income > $50,000?"
• If Yes, Proceed to the next question.
• If No, "No Purchase"
2. "Age > 30?"21//2/1
• Yes: "Purchase"
 • No: "No Purchase"
Tree 2: Previous Purchases
"Previous Purchases > 0?"
• Yes: "Purchase"
• No: "No Purchase"
Once we have predictions from both trees, we can combine the results to make a
final prediction. If Tree 1 predicts "Purchase" and Tree 2 predicts "No Purchase", the
final prediction might be "Purchase" or "No Purchase" depending on the weight or
confidence assigned to each tree. This can be decided based on the problem context.
Information Gain and Gini Index in Decision Tree
Till now we have discovered the basic intuition and approach of how decision tree
works, so lets just move to the attribute selection measure of decision tree. We have
two popular attribute selection measures used:
1. Information Gain
Information Gain tells us how useful a question (or feature) is for splitting data into
groups. It measures how much the uncertainty decreases after the split. A good
question will create clearer groups and the feature with the highest Information Gain
is chosen to make the decision.
For example if we split a dataset of people into "Young" and "Old" based on age and
all young people bought the product while all old people did not, the Information
Gain would be high because the split perfectly separates the two groups with no
uncertainty left
• Suppose SS is a set of instances AA is an attribute, SvSv is the subset
of SS, vv represents an individual value that the attribute AA can take and
Values (AA) is the set of all possible values of AA then
Gain(S,A)=Entropy(S)−∑vA∣S∣∣Sv∣.Entropy(Sv)
 • Entropy: is the measure of uncertainty of a random variable it characterizes
the impurity of an arbitrary collection of examples. The higher the entropy
more the information content.
For example if a dataset has an equal number of "Yes" and "No" outcomes (like 3
people who bought a product and 3 who didn’t), the entropy is high because it’s
uncertain which outcome to predict. But if all the outcomes are the same (all "Yes"
or all "No") the entropy is 0 meaning there is no uncertainty left in predicting the
outcome
Suppose SS is a set of instances, AA is an attribute, SvSv is the subset
of SS with AA= vv and Values (AA) is the set of all possible values of AA, then
Gain(S,A)=Entropy(S)−∑vϵValues(A)∣Sv∣∣S∣.Entropy(Sv) Gain(S,A)=Entropy(S)−
∑vϵValues(A)∣S∣∣Sv∣.Entropy(Sv)
Example:
For the set X = {a,a,a,b,b,b,b,b}
Total instances: 8
Instances of b: 5
Instances of a: 3
Entropy H(X)=[(38)log⁡238+(58)log⁡258]
=−[0.375(−1.415)+0.625(−0.678)]
=−(−0.53−0.424)=0.954Entropy
H(X)=[(83)log283+(85)log285]
=−[0.375(−1.415)+0.625(−0.678)]
=−(−0.53−0.424)=0.954
Building Decision Tree using Information Gain the essentials
• Start with all training instances associated with the root node
• Use info gain to choose which attribute to label each node with
 • Recursively construct each subtree on the subset of training instances that
would be classified down that path in the tree.
• If all positive or all negative training instances remain, the label that node
“yes" or “no" accordingly
• If no attributes remain label with a majority vote of training instances left at
that node
• If no instances remain label with a majority vote of the parent's training
instances.
Example: Now let us draw a Decision Tree for the following data using Information
gain. Training set: 3 features and 2 classes

X Y Z C

1 1 1 I

1 1 0 I

0 0 1 II

1 0 0 II

Here, we have 3 features and 2 output classes. To build a decision tree using
Information gain. We will take each of the features and calculate the information for
each feature.

Split on attribute Z
From the above images we can see that the information gain is maximum when we
make a split on feature Y. So, for the root node best-suited feature is feature Y. Now
we can see that while splitting the dataset by feature Y, the child contains a pure
subset of the target variable. So we don't need to further split the dataset. The final
tree for the above dataset would look like this:
information gain on attribute Y
2. Gini Index
Gini Index is a metric to measure how often a randomly chosen element would be
incorrectly identified. It means an attribute with a lower Gini index should be
preferred. Sklearn supports “Gini” criteria for Gini Index and by default it takes
“gini” value.
For example if we have a group of people where all bought the product (100% "Yes")
the Gini Index is 0 indicate perfect purity. But if the group has an equal mix of "Yes"
and "No" the Gini Index would be 0.5 show high impurity or uncertainty. Formula
for Gini Index is given by :
Gini=1−∑i
Some additional features of the Gini Index are:
1. It is calculated by summing the squared probabilities of each outcome in a
distribution and subtracting the result from 1.
2. A lower Gini Index indicates a more homogeneous or pure distribution while
a higher Gini Index indicates a more heterogeneous or impure distribution.
3. In decision trees the Gini Index is used to evaluate the quality of a split by
measuring the difference between the impurity of the parent node and the
weighted impurity of the child nodes.
4. Compared to other impurity measures like entropy, the Gini Index is faster to
compute and more sensitive to changes in class probabilities.
5. One disadvantage of the Gini Index is that it tends to favour splits that create
equally sized child nodes, even if they are not optimal for classification
accuracy.
 6. In practice the choice between using the Gini Index or other impurity
measures depends on the specific problem and dataset and requires
experimentation and tuning.
Understanding Decision Tree with Real life use case:
Till now we have understand about the attributes and components of decision tree.
Now lets jump to a real life use case in which how decision tree works step by step.
Step 1. Start with the Whole Dataset
We begin with all the data which is treated as the root node of the decision tree.
Step 2. Choose the Best Question (Attribute)
Pick the best question to divide the dataset. For example ask: "What is the outlook?"
Possible answers: Sunny, Cloudy or Rainy.
Step 3. Split the Data into Subsets
Divide the dataset into groups based on the question:
• If Sunny go to one subset.
• If Cloudy go to another subset.
• If Rainy go to the last subset.
Step 4. Split Further if Needed (Recursive Splitting)
For each subset ask another question to refine the groups. For example If the Sunny
subset is mixed ask: "Is the humidity high or normal?"
• High humidity → "Swimming".
• Normal humidity → "Hiking".
Step 5. Assign Final Decisions (Leaf Nodes)
When a subset contains only one activity, stop splitting and assign it a label:
• Cloudy → "Hiking".
• Rainy → "Stay Inside".
• Sunny + High Humidity → "Swimming".
• Sunny + Normal Humidity → "Hiking".
Step 6. Use the Tree for Predictions
To predict an activity follow the branches of the tree. Example: If the outlook is
Sunny and the humidity is High follow the tree:
• Start at Outlook.
• Take the branch for Sunny.
• Then go to Humidity and take the branch for High Humidity.
• Result: "Swimming".
A decision tree works by breaking down data step by step asking the best possible
questions at each point and stopping once it reaches a clear decision. It's an easy and
understandable way to make choices. Because of their simple and clear structure
decision trees are very helpful in machine learning for tasks like sorting data into
categories or making predictions.

Decision tree pruning is a technique used to prevent decision trees

from overfitting the training data. Pruning aims to simplify the decision tree by
removing parts of it that do not provide significant predictive power, thus improving
its ability to generalize to new data.
Decision Tree Pruning removes unwanted nodes from the overfitted decision tree to
make it smaller in size which results in more fast, more accurate and more effective
predictions.
Types Of Decision Tree Pruning
There are two main types of decision tree pruning: Pre-Pruning and Post-Pruning.
Pre-Pruning (Early Stopping)
Sometimes, the growth of the decision tree can be stopped before it gets too complex,
this is called pre-pruning. It is important to prevent the overfitting of the training
data, which results in a poor performance when exposed to new data.
Some common pre-pruning techniques include:
 • Maximum Depth: It limits the maximum level of depth in a decision tree.
• Minimum Samples per Leaf: Set a minimum threshold for the number of
samples in each leaf node.
• Minimum Samples per Split: Specify the minimal number of samples needed
to break up a node.
• Maximum Features: Restrict the quantity of features considered for splitting.
By pruning early, we come to be with a simpler tree that is less likely to overfit the
training facts.
Post-Pruning (Reducing Nodes)
After the tree is fully grown, post-pruning involves removing branches or nodes to
improve the model's ability to generalize. Some common post-pruning techniques
include:
• Cost-Complexity Pruning (CCP): This method assigns a price to each subtree
primarily based on its accuracy and complexity, then selects the subtree with
the lowest fee.
• Reduced Error Pruning: Removes branches that do not significantly affect the
overall accuracy.
• Minimum Impurity Decrease: Prunes nodes if the decrease in impurity (Gini
impurity or entropy) is beneath a certain threshold.
• Minimum Leaf Size: Removes leaf nodes with fewer samples than a specified
threshold.
Post-pruning simplifies the tree while preserving its Accuracy. Decision tree pruning
helps to improve the performance and interpretability of decision trees by reducing
their complexity and avoiding overfitting. Proper pruning can lead to simpler and
more robust models that generalize better to unseen data.
Decision Tree Implementation in Python
Here we are going to create a decision tree using preloaded dataset breast_cancer in
sklearn library.
The Decision Tree model is using pre-pruning technique, specifically, the default
approach of scikit-learn's DecisionTreeClassifier, which employs the Gini impurity
criterion for making splits. This is evident from the
parameter criterion="gini" passed to the DecisionTreeClassifier() constructor. Gini
impurity is a measure of how often a randomly chosen element from the set would
be incorrectly labeled if it were randomly labeled according to the distribution of
labels in the set.
from [Link] import load_breast_cancer
from sklearn.model_selection import train_test_split
from [Link] import DecisionTreeClassifier
from [Link] import plot_tree
import [Link] as plt

# Load breast cancer dataset

X, y = load_breast_cancer(return_X_y=True)

# Separating Training and Testing data

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.2,
random_state=42)

# Train decision tree model

model = DecisionTreeClassifier(criterion="gini")
[Link](X_train, y_train)

# Plot original tree

[Link](figsize=(15, 10))
plot_tree(model, filled=True)
[Link]("Original Decision Tree")
[Link]()

# Model Accuracy before pruning

accuracy_before_pruning = [Link](X_test, y_test)
print("Accuracy before pruning:", accuracy_before_pruning)
Output:

Accuracy before pruning: 0.8793859649122807

Decision Tree Pre-Pruning Implementation
In the implementation, we pruning technique is hyperparameter tuning through
cross-validation using Grid Search CV. Hyperparameter tuning involves searching
for the optimal hyperparameters for a machine learning model to improve its
performance. It does not directly prune the decision tree, but it helps in finding the
best combination of hyperparameters, such as max_depth, max_features, criterion,
and splitter, which indirectly controls the complexity of the decision tree and
prevents overfitting. Therefore, it's a form of post-pruning technique.
from [Link] import Decision Tree Classifier
parameter = {
'criterion' :['entropy','gini','log_loss'],
'splitter':['best','random'],
'max_depth':[1,2,3,4,5],
'max_features':['auto','sqrt','log2']
}
model = DecisionTreeClassifier()
from sklearn.model_selection import GridSearchCV
cv = GridSearchCV(model,param_grid = parameter,cv = 5)
[Link](X_train,Y_train)
Visualizing
from [Link] import export_graphviz
import graphviz
best_estimator = cv.best_estimator_
feature_names = features

dot_data = export_graphviz(best_estimator, out_file=None, filled=True,

rounded=True,
feature_names=feature_names, class_names=['0', '1', '2'])
graph = [Link](dot_data)
[Link]("decision_tree", format='png', cleanup=True)
graph
Output:

Best Parameters
[Link](X_test,Y_test)
cv.best_params_
Output:
0.9736842105263158
{'criterion': 'gini',
'max_depth': 4,
'max_features': 'sqrt',
'splitter': 'best'}
Decision Tree Post-Pruning Implementation
# Cost-complexity pruning (Post-pruning)
path = model.cost_complexity_pruning_path(X_train, y_train)
ccp_alphas, impurities = path.ccp_alphas, [Link]

# Train a series of decision trees with different alpha values

pruned_models = []
for ccp_alpha in ccp_alphas:
pruned_model = DecisionTreeClassifier(criterion="gini", ccp_alpha=ccp_alpha)
pruned_model.fit(X_train, y_train)
pruned_models.append(pruned_model)

# Find the model with the best accuracy on test data

best_accuracy = 0
best_pruned_model = None
for pruned_model in pruned_models:
accuracy = pruned_model.score(X_test, y_test)
if accuracy > best_accuracy:
best_accuracy = accuracy
best_pruned_model = pruned_model
# Model Accuracy after pruning
accuracy_after_pruning = best_pruned_model.score(X_test, y_test)
print("Accuracy after pruning:", accuracy_after_pruning)
Output:
Accuracy after pruning: 0.918859649122807
# Plot pruned tree
[Link](figsize=(15, 10))
plot_tree(best_pruned_model, filled=True)
[Link]("Pruned Decision Tree")
[Link]()
Output:

Why Pruning decision trees is Important?

Decision Tree Pruning has an important role in optimizing the decision tree model.
It involves the removal of certain parts of the tree which can potentially reduce its
performance. Here is why decision tree pruning is important:
1. Prevents Overfitting: Decision trees are prone to overfitting, where the model
memorizes the training data rather than learning generalizable patterns.
Pruning helps prevent overfitting by simplifying the tree structure, removing
branches that capture noise or outliers in the training data.
2. Improves Generalization: By reducing the complexity of the decision tree,
pruning enhances the model's ability to generalize to unseen data. A pruned
decision tree is more likely to capture underlying patterns in the data rather
than memorizing specific instances, leading to better performance on new
data.
3. Reduces Model Complexity: Pruning results in a simpler decision tree with
fewer branches and nodes. This simplicity not only makes the model easier to
interpret but also reduces computational requirements during both training
and inference. A simpler model is also less prone to overfitting and more
robust to changes in the data.
4. Enhances Interpretability: Pruning produces decision trees with fewer
branches and nodes, which are easier to interpret and understand. This is
particularly important in applications where human insight into the decision-
making process is valuable, such as in medical diagnosis or financial decision-
making.
 5. Speeds Up Training and Inference: Pruned decision trees require less
computational resources during both training and inference phases. With
fewer branches and nodes, the decision-making process becomes more
efficient, resulting in faster predictions without sacrificing accuracy.
6. Facilitates Model Maintenance: Pruning helps maintain decision tree models
over time by keeping them lean and relevant. As new data becomes available
or the problem domain evolves, pruned decision trees are easier to update and
adapt compared to overly complex, unpruned trees.
Conclusion
Decision tree pruning plays a crucial role in optimizing decision tree models by
preventing overfitting, improving generalization, and enhancing model
interpretability. Post-Pruning is used generally for small datasets whereas Pre-
Pruning is used for larger ones. Pre-Pruning is considered more efficient and
effective as it considered multiple parameters and choose best ones from them.
Rule-based classifiers are just another type of classifier which makes the class
decision depending by using various "if..else" rules. These rules are easily
interpretable and thus these classifiers are generally used to generate descriptive
models. The condition used with "if" is called the antecedent and the predicted class
of each rule is called the consequent. Properties of rule-based classifiers:
• Coverage: The percentage of records which satisfy the antecedent conditions
of a particular rule.
• The rules generated by the rule-based classifiers are generally not mutually
exclusive, i.e. many rules can cover the same record.
• The rules generated by the rule-based classifiers may not be exhaustive, i.e.
there may be some records which are not covered by any of the rules.
 • The decision boundaries created by them is linear, but these can be much more
complex than the decision tree because the many rules are triggered for the
same record.
An obvious question, which comes into the mind after knowing that the rules are not
mutually exclusive is that how would the class be decided in case different rules with
different consequent cover the record. There are two solutions to the above problem:
• Either rules can be ordered, i.e. the class corresponding to the highest priority
rule triggered is taken as the final class.
• Otherwise, we can assign votes for each class depending on some their
weights, i.e. the rules remain unordered.
Example: Below is the dataset to classify mushrooms as edible or poisonous:

Cap
Cap Surfac Bruis Stalk Populati
Class Shape e es Odour Shape on Habitat

edible flat scaly yes anise tapering scattered grasses

poisono conve punge enlargeni

scaly yes several grasses
us x nt ng

conve smoot almon enlargeni numerou

edible yes grasses
x h d ng s

conve almon meado

edible scaly yes tapering scattered
x d ws
 Cap
Cap Surfac Bruis Stalk Populati
Class Shape e es Odour Shape on Habitat

fibrou enlargeni
edible flat yes anise several woods
s ng

fibrou enlargeni
edible flat no none several urban
s ng

poisono conic punge enlargeni

scaly yes scattered urban
us al nt ng

smoot enlargeni numerou meado

edible flat yes anise
h ng s ws

poisono conve smoot punge enlargeni

yes several urban
us x h nt ng

Rules:
• Odour = pungent and habitat = urban -> Class = poisonous
• Bruises = yes -> Class = edible : This rules covers both negative and positive
records.

The given rules are not mutually exclusive.

How to generate a rule:
Sequential Rule Generation Rules can be generated either using general-to-
specific approach or specific-to-general approach. In the general-to-
specific approach, start with a rule with no antecedent and keep on adding conditions
to it till we see major improvements in our evaluation metrics. While for the other
we keep on removing the conditions from a rule covering a very specific case. The
evaluation metric can be accuracy, information gain, likelihood ratio etc. Algorithm
for generating the model incrementally: The algorithm given below generates a
model with unordered rules and ordered classes, i.e. we can decide which class to
give priority while generating the rules.
A <-Set of attributes T <-Set of training records Y <-Set of classes Y' <-Ordered Y
according to relevance R <-Set of rules generated, initially to an empty list for each
class y in Y' while the majority of class y records are not covered generate a new
rule for class y, using methods given above Add this rule to R Remove the records
covered by this rule from T end while end for Add rule {}->y' where y' is the default
class
Classifying a record: The classification algorithm described below assumes that the
rules are unordered and the classes are weighted.
R <-Set of rules generated using training Set T <-Test Record W <-class name to
Weight mapping, predefined, given as input F <-class name to Vote mapping,
generated for each test record, to be calculated for each rule r in R check if r covers
T if so then add W of predicted_class to F of predicted_class end for Output the class
with the highest calculated vote in F
Naive Bayes is a classification algorithm that uses probability to predict which
category a data point belongs to, assuming that all features are unrelated. This article
will give you an overview as well as more advanced use and implementation of
Naive Bayes in machine learning.

Illustration behind the Naive Bayes algorithm. We estimate P(xα∣y) independently in

each dimension (middle two images) and then obtain an estimate of the full data
distribution by assuming conditional independence P(x∣y)=∏αP(xα∣y) (very right
image).
Key Features of Naive Bayes Classifiers
The main idea behind the Naive Bayes classifier is to use Bayes' Theorem to classify
data based on the probabilities of different classes given the features of the data. It
is used mostly in high-dimensional text classification
• The Naive Bayes Classifier is a simple probabilistic classifier and it has very
few number of parameters which are used to build the ML models that can
predict at a faster speed than other classification algorithms.
 • It is a probabilistic classifier because it assumes that one feature in the model
is independent of existence of another feature. In other words, each feature
contributes to the predictions with no relation between each other.
• Naïve Bayes Algorithm is used in spam filtration, Sentimental analysis,
classifying articles and many more.
Why it is Called Naive Bayes?
It is named as "Naive" because it assumes the presence of one feature does not affect
other features. The "Bayes" part of the name refers to its basis in Bayes’ Theorem.
Consider a fictional dataset that describes the weather conditions for playing a game
of golf. Given the weather conditions, each tuple classifies the conditions as
fit(“Yes”) or unfit(“No”) for playing golf. Here is a tabular representation of our
dataset.

Outlook Temperature Humidity Windy Play Golf

0 Rainy Hot High False No

1 Rainy Hot High True No

2 Overcast Hot High False Yes

3 Sunny Mild High False Yes

4 Sunny Cool Normal False Yes

5 Sunny Cool Normal True No

6 Overcast Cool Normal True Yes

 Outlook Temperature Humidity Windy Play Golf

7 Rainy Mild High False No

8 Rainy Cool Normal False Yes

9 Sunny Mild Normal False Yes

10 Rainy Mild Normal True Yes

11 Overcast Mild High True Yes

12 Overcast Hot Normal False Yes

13 Sunny Mild High True No

The dataset is divided into two parts, namely, feature matrix and the response vector.
• Feature matrix contains all the vectors(rows) of dataset in which each vector
consists of the value of dependent features. In above dataset, features are
‘Outlook’, ‘Temperature’, ‘Humidity’ and ‘Windy’.
• Response vector contains the value of class variable(prediction or output) for
each row of feature matrix. In above dataset, the class variable name is ‘Play
golf’.
Assumption of Naive Bayes
The fundamental Naive Bayes assumption is that each feature makes an:
• Feature independence: This means that when we are trying to classify
something, we assume that each feature (or piece of information) in the data
does not affect any other feature.
 • Continuous features are normally distributed: If a feature is continuous, then
it is assumed to be normally distributed within each class.
• Discrete features have multinomial distributions: If a feature is discrete, then
it is assumed to have a multinomial distribution within each class.
• Features are equally important: All features are assumed to contribute equally
to the prediction of the class label.
• No missing data: The data should not contain any missing values.
Introduction to Bayes' Theorem
Bayes’ Theorem provides a principled way to reverse conditional probabilities. It is
defined as:
P(y∣X)=P(X∣y)⋅P(y)P(X)
Where:
• P(y∣X) Posterior probability, probability of class yy given features XX
• P(X∣y): Likelihood, probability of features XX given class yy
• P(y): Prior probability of class yy
• P(X): Marginal likelihood or evidence
Naive Bayes Working
1. Terminology
Consider a classification problem (like predicting if someone plays golf based on
weather). Then:
• y is the class label (e.g. "Yes" or "No" for playing golf)
• X=(x1,x2,...,xn) is the feature vector (e.g. Outlook, Temperature, Humidity,
Wind)
A sample row from the dataset:
X=(Rainy, Hot, High, False),y=No
This represents:
What is the probability that someone will not play golf given that the weather is
Rainy, Hot, High humidity, and No wind?
2. The Naive Assumption
The "naive" in Naive Bayes comes from the assumption that all features are
independent given the class. That is:
P(x1,x2,...,xn∣y)=P(x1∣y)⋅P(x2∣y)⋯P(xn∣y)
Thus, Bayes' theorem becomes:
P(y∣x1,...,xn)=P(y)⋅∏i=1nP(xi∣y)P(x1)P(x2)...P(xn)
Since the denominator is constant for a given input, we can write:
P(y∣x1,...,xn)∝P(y)⋅∏i=1nP(xi∣y)
3. Constructing the Naive Bayes Classifier
We compute the posterior for each class yy and choose the class with the highest
probability:
y^=arg⁡max⁡yP(y)⋅∏i=1nP(xi∣y))
This becomes our Naive Bayes classifier.
4. Example: Weather Dataset
Let’s take a dataset used for predicting if golf is played based on:
• Outlook: Sunny, Rainy, Overcast
• Temperature: Hot, Mild, Cool
• Humidity: High, Normal
• Wind: True, False
 Example
Tables for Naive Bayes
Example Input: X=(Sunny, Hot ,Normal, False)
Goal: Predict if golf will be played (Yes or No).
5. Pre-computation from Dataset
Class Probabilities:
From dataset of 14 rows:
• P(Yes)=914P(Yes)=149
 • P(No)=514P(No)=145
Conditional Probabilities (Tables 1–4):

Feature Value P (Value | Yes) P (Value | No)

Outlook Sunny 2/9 3/5

Temperature Hot 2/9 2/5

Humidity Normal 6/9 1/5

Wind False 6/9 2/5

6. Calculate Posterior Probabilities

For Class = Yes:
P(Yes | today)∝29⋅29⋅69⋅69⋅914
P(Yes | today)≈0.02116
For Class = No:
P(No | today)∝35⋅25⋅15⋅25⋅514
P(No | today)≈0.0068
7. Normalize Probabilities
To compare:
P(Yes | today)=0.021160.02116+0.0068≈0.756
P(No | today)=0.00680.02116+0.0068≈0.244
8. Final Prediction
Since:
P(Yes | today)>P(No | today)
The model predicts: Yes (Play Golf)
Naive Bayes for Continuous Features
For continuous features, we assume a Gaussian distribution:
P(xi∣y)=12πσy2exp⁡(−(xi−μy)22σy2)
Where:
• μy is the mean of feature xi for class y
• σy2 is the variance of feature xi for class y
This leads to what is called Gaussian Naive Bayes.
Types of Naive Bayes Model
There are three types of Naive Bayes Model :
1. Gaussian Naive Bayes
In Gaussian Naive Bayes, continuous values associated with each feature are
assumed to be distributed according to a Gaussian distribution. A Gaussian
distribution is also called Normal distribution When plotted, it gives a bell shaped
curve which is symmetric about the mean of the feature values as shown below:
2. Multinomial Naive Bayes
Multinomial Naive Bayesis used when features represent the frequency of terms
(such as word counts) in a document. It is commonly applied in text classification,
where term frequencies are important.
3. Bernoulli Naive Bayes
Bernoulli Naive Bayes deals with binary features, where each feature indicates
whether a word appears or not in a document. It is suited for scenarios where the
presence or absence of terms is more relevant than their frequency. Both models are
widely used in document classification tasks
Advantages of Naive Bayes Classifier
• Easy to implement and computationally efficient.
• Effective in cases with a large number of features.
• Performs well even with limited training data.
 • It performs well in the presence of categorical features.
• For numerical features data is assumed to come from normal distributions
Disadvantages of Naive Bayes Classifier
• Assumes that features are independent, which may not always hold in real-
world data.
• Can be influenced by irrelevant attributes.
• May assign zero probability to unseen events, leading to poor generalization.
Applications of Naive Bayes Classifier
• Spam Email Filtering: Classifies emails as spam or non-spam based on
features.
• Text Classification: Used in sentiment analysis, document categorization, and
topic classification.
• Medical Diagnosis: Helps in predicting the likelihood of a disease based on
symptoms.
• Credit Scoring: Evaluates creditworthiness of individuals for loan approval.
• Weather Prediction: Classifies weather conditions based on various factors.
Bayesian Belief Network (BBN) is a graphical model that represents the
probabilistic relationships among variables. It is used to handle uncertainty and make
predictions or decisions based on probabilities.
• Graphical Representation: Variables are represented as nodes in a directed
acyclic graph (DAG), and their dependencies are shown as edges.
• Conditional Probabilities: Each node's probability depends on its parent
nodes, expressed as P(Variable | Parent)P(Variable | Parent).
• Probabilistic Model: Built from probability distributions, BBNs apply
probability theory for tasks like prediction and anomaly detection.
Bayesian Belief Networks are valuable tools for understanding and solving problems
involving uncertain events. They are also known as Bayes networks, belief
networks, decision networks, or Bayesian models.
(Note: A classifier assigns data in a collection to desired categories.)
• Consider this example:

• In the above figure, we have an alarm 'A' - a node, say installed in a house of
a person 'gfg', which rings upon two probabilities i.e burglary 'B' and fire 'F',
which are - parent nodes of the alarm node. The alarm is the parent node of
two probabilities P1 calls 'P1' & P2 calls 'P2' person nodes.
• Upon the instance of burglary and fire, 'P1' and 'P2' call person 'gfg',
respectively. But, there are few drawbacks in this case, as sometimes 'P1' may
forget to call the person 'gfg', even after hearing the alarm, as he has a
tendency to forget things, quick. Similarly, 'P2', sometimes fails to call the
person 'gfg', as he is only able to hear the alarm, from a certain distance.
Calculating Conditional Probability of Events in a Bayesian Network
Find the probability that 'P1' is true (P1 has called 'gfg'), 'P2' is true (P2 has called
'gfg') when the alarm 'A' rang, but no burglary 'B' and fire 'F' has occurred.
=> P ( P1, P2, A, ~B, ~F) [ where- P1, P2 & A are 'true' events and '~B' & '~F' are
'false' events]
[ Note: The values mentioned below are neither calculated nor computed. They have
observed values ]
Burglary 'B' -
• P (B=T) = 0.001 ('B' is true i.e burglary has occurred)
• P (B=F) = 0.999 ('B' is false i.e burglary has not occurred)
Fire 'F' -
• P (F=T) = 0.002 ('F' is true i.e fire has occurred)
• P (F=F) = 0.998 ('F' is false i.e fire has not occurred)
Alarm 'A' -

B F P (A=T) P (A=F)

T T 0.95 0.05

T F 0.94 0.06

F T 0.29 0.71

F F 0.001 0.999

• The alarm 'A' node can be 'true' or 'false' ( i.e may have rung or may not have
rung). It has two parent nodes burglary 'B' and fire 'F' which can be 'true' or
'false' (i.e may have occurred or may not have occurred) depending upon
different conditions.
Person 'P1' -
 A P (P1=T) P (P1=F)

T 0.95 0.05

F 0.05 0.95

• The person 'P1' node can be 'true' or 'false' (i.e may have called the person
'gfg' or not) . It has a parent node, the alarm 'A', which can be 'true' or 'false'
(i.e may have rung or may not have rung ,upon burglary 'B' or fire 'F').
Person 'P2' -

A P (P2=T) P (P2=F)

T 0.80 0.20

F 0.01 0.99

• The person 'P2' node can be 'true' or false' (i.e may have called the person 'gfg'
or not). It has a parent node, the alarm 'A', which can be 'true' or 'false' (i.e
may have rung or may not have rung, upon burglary 'B' or fire 'F').
Solution: Considering the observed probabilistic scan -
With respect to the question — P ( P1, P2, A, ~B, ~F) , we need to get the probability
of 'P1'. We find it with regard to its parent node - alarm 'A'. To get the probability of
'P2', we find it with regard to its parent node — alarm 'A'.
We find the probability of alarm 'A' node with regard to '~B' & '~F' since burglary
'B' and fire 'F' are parent nodes of alarm 'A'.
From the observed probabilistic scan, we can deduce -
P ( P1, P2, A, ~B, ~F)
= P (P1/A) * P (P2/A) * P (A/~B~F) * P (~B) * P (~F)
= 0.95 * 0.80 * 0.001 * 0.999 * 0.998
= 0.00075
Support Vector Machine (SVM) is a supervised machine learning algorithm used for
classification and regression tasks. It tries to find the best boundary known as
hyperplane that separates different classes in the data. It is useful when you want to
do binary classification like spam vs. not spam or cat vs. dog.
The main goal of SVM is to maximize the margin between the two classes. The
larger the margin the better the model performs on new and unseen data.

Key Concepts of Support Vector Machine

• Hyperplane: A decision boundary separating different classes in feature space
and is represented by the equation wx + b = 0 in linear classification.
• Support Vectors: The closest data points to the hyperplane, crucial for
determining the hyperplane and margin in SVM.
• Margin: The distance between the hyperplane and the support vectors. SVM
aims to maximize this margin for better classification performance.
• Kernel: A function that maps data to a higher-dimensional space enabling
SVM to handle non-linearly separable data.
• Hard Margin: A maximum-margin hyperplane that perfectly separates the data
without misclassifications.
• Soft Margin: Allows some misclassifications by introducing slack variables,
balancing margin maximization and misclassification penalties when data is
not perfectly separable.
• C: A regularization term balancing margin maximization and misclassification
penalties. A higher C value forces stricter penalty for misclassifications.
 • Hinge Loss: A loss function penalizing misclassified points or margin
violations and is combined with regularization in SVM.
• Dual Problem: Involves solving for Lagrange multipliers associated with
support vectors, facilitating the kernel trick and efficient computation.
How does Support Vector Machine Algorithm Work?
The key idea behind the SVM algorithm is to find the hyperplane that best separates
two classes by maximizing the margin between them. This margin is the distance
from the hyperplane to the nearest data points (support vectors) on each side.

Multiple hyperplanes separate the data from two classes

The best hyperplane also known as the "hard margin" is the one that maximizes the
distance between the hyperplane and the nearest data points from both classes. This
ensures a clear separation between the classes. So from the above figure, we choose
L2 as hard margin. Let's consider a scenario like shown below:
Selecting hyperplane for data with outlier
Here, we have one blue ball in the boundary of the red ball.
How does SVM classify the data?
The blue ball in the boundary of red ones is an outlier of blue balls. The SVM
algorithm has the characteristics to ignore the outlier and finds the best hyperplane
that maximizes the margin. SVM is robust to outliers.

Hyperplane which is the most optimized one

A soft margin allows for some misclassifications or violations of the margin to
improve generalization. The SVM optimizes the following equation to balance
margin maximization and penalty minimization:
Objective Function=(1margin)+λ∑penalty Objective Function=(margin1
)+λ∑penalty
The penalty used for violations is often hinge loss which has the following behavior:
• If a data point is correctly classified and within the margin there is no penalty
(loss = 0).
• If a point is incorrectly classified or violates the margin the hinge loss
increases proportionally to the distance of the violation.
Till now we were talking about linearly separable data that seprates group of blue
balls and red balls by a straight line/linear line.
What if data is not linearly separable?
When data is not linearly separable i.e it can't be divided by a straight line, SVM
uses a technique called kernels to map the data into a higher-dimensional space
where it becomes separable. This transformation helps SVM find a decision
boundary even for non-linear data.

Original 1D dataset for classification

A kernel is a function that maps data points into a higher-dimensional space without
explicitly computing the coordinates in that space. This allows SVM to work
efficiently with non-linear data by implicitly performing the mapping. For example
consider data points that are not linearly separable. By applying a kernel function
SVM transforms the data points into a higher-dimensional space where they become
linearly separable.
• Linear Kernel: For linear separability.
• Polynomial Kernel: Maps data into a polynomial space.
• Radial Basis Function (RBF) Kernel: Transforms data into a space based on
distances between data points.
Mapping 1D data to 2D to become able to separate the two classes
In this case the new variable y is created as a function of distance from the origin.
Mathematical Computation of SVM
Consider a binary classification problem with two classes, labeled as +1 and -1. We
have a training dataset consisting of input feature vectors X and their corresponding
class labels Y. The equation for the linear hyperplane can be written as:
wTx+b=0
Where:
• ww is the normal vector to the hyperplane (the direction perpendicular to it).
• bb is the offset or bias term representing the distance of the hyperplane from
the origin along the normal vector w.

Distance from a Data Point to the Hyperplane

The distance between a data point xixiand the decision boundary can be calculated
as:
di=wTxi+b∣∣w∣∣
where ||w|| represents the Euclidean norm of the weight vector w.
Linear SVM Classifier
Distance from a Data Point to the Hyperplane:
y^={1: wTx+b≥00: wTx+b <0
Where y^ is the predicted label of a data point.
Optimization Problem for SVM
For a linearly separable dataset the goal is to find the hyperplane that maximizes the
margin between the two classes while ensuring that all data points are correctly
classified. This leads to the following optimization problem:
minimizew,b12∥w∥2w,bminimize21∥w∥2
Subject to the constraint:
yi(wTxi+b)≥1fori=1,2,3,⋯,m
Where:
• yi is the class label (+1 or -1) for each training instance.
 • xi is the feature vector for the ii-th training instance.
• mm is the total number of training instances.
The condition yi(wTxi+b)≥1yi(wTxi+b)≥1 ensures that each data point is correctly
classified and lies outside the margin.

Soft Margin in Linear SVM Classifier

In the presence of outliers or non-separable data the SVM allows some
misclassification by introducing slack variables ζiζi. The optimization problem is
modified as:
minimize w,b12∥w∥2+C∑i=1mζiw,bminimize 21∥w∥2+C∑i=1mζi
Subject to the constraints:
yi(wTxi+b)≥1−ζiandζi≥0for i=1,2,…,myi(wTxi+b)≥1−ζiandζi≥0for i=1,2,…,m
Where:
• C is a regularization parameter that controls the trade-off between margin
maximization and penalty for misclassifications.
• Ζi are slack variables that represent the degree of violation of the margin by
each data point.
Dual Problem for SVM
The dual problem involves maximizing the Lagrange multipliers associated with the
support vectors. This transformation allows solving the SVM optimization using
kernel functions for non-linear classification.
The dual objective function is given by:
maximize α12∑i=1m∑j=1mαiαjtitjK(xi,xj)−∑i=1mαiα
Where:
• αiαi are the Lagrange multipliers associated with the ith training sample.
• titi is the class label for the ith-th training sample.
 • K(xi)K(xi) is the kernel function that computes the similarity between data
points xi and xj. The kernel allows SVM to handle non-linear classification
problems by mapping data into a higher-dimensional space.
The dual formulation optimizes the Lagrange multipliers αiαi and the support
vectors are those training samples where αi>0αi>0.
SVM Decision Boundary
Once the dual problem is solved, the decision boundary is given by:
w=∑i=1mαitiK(xi,x)+b
Where ww is the weight vector, xx is the test data point and bb is the bias term.
Finally the bias term bb is determined by the support vectors, which satisfy:
ti(wTxi−b)=1⇒b=wTxi
Where xi is any support vector.
This completes the mathematical framework of the Support Vector Machine
algorithm which allows for both linear and non-linear classification using the dual
problem and kernel trick.
Types of Support Vector Machine
Based on the nature of the decision boundary, Support Vector Machines (SVM) can
be divided into two main parts:
• Linear SVM: Linear SVMs use a linear decision boundary to separate the data
points of different classes. When the data can be precisely linearly separated,
linear SVMs are very suitable. This means that a single straight line (in 2D)
or a hyperplane (in higher dimensions) can entirely divide the data points into
their respective classes. A hyperplane that maximizes the margin between the
classes is the decision boundary.
• Non-Linear SVM: Non-Linear SVM can be used to classify data when it
cannot be separated into two classes by a straight line (in the case of 2D). By
using kernel functions, nonlinear SVMs can handle nonlinearly separable
 data. The original input data is transformed by these kernel functions into a
higher-dimensional feature space where the data points can be linearly
separated. A linear SVM is used to locate a nonlinear decision boundary in
this modified space.
Implementing SVM Algorithm Using Scikit-Learn
We will predict whether cancer is Benign or Malignant using historical data about
patients diagnosed with cancer. This data includes independent attributes such as
tumor size, texture, and others. To perform this classification, we will use an SVM
(Support Vector Machine) classifier to differentiate between benign and malignant
cases effectively.
• load_breast_cancer(): Loads the breast cancer dataset (features and target
labels).
• SVC(kernel="linear", C=1): Creates a Support Vector Classifier with a linear
kernel and regularization parameter C=1.
• [Link](X, y): Trains the SVM model on the feature matrix X and target labels
y.
• DecisionBoundaryDisplay.from_estimator(): Visualizes the decision
boundary of the trained model with a specified color map.
• [Link](): Creates a scatter plot of the data points, colored by their labels.
• [Link](): Displays the plot to the screen.
from [Link] import load_breast_cancer
import [Link] as plt
from [Link] import Decision Boundary Display
from [Link] import SVC

cancer = load_breast_cancer()
X = [Link][:, :2]
y = [Link]

svm = SVC(kernel="linear", C=1)

[Link](X, y)

DecisionBoundaryDisplay.from_estimator(
svm,
X,
response_method="predict",
alpha=0.8,
cmap="Pastel1",
xlabel=cancer.feature_names[0],
ylabel=cancer.feature_names[1],
)

[Link](X[:, 0], X[:, 1],

c=y,
s=20, edgecolors="k")
[Link]()
Output:
SVM
Advantages of Support Vector Machine (SVM)
1. High-Dimensional Performance: SVM excels in high-dimensional spaces,
making it suitable for image classification and gene expression analysis.
2. Nonlinear Capability: Utilizing kernel functions like RBF and polynomial
SVM effectively handles nonlinear relationships.
 3. Outlier Resilience: The soft margin feature allows SVM to ignore outliers,
enhancing robustness in spam detection and anomaly detection.
4. Binary and Multiclass Support: SVM is effective for both binary classification
and multiclass classification suitable for applications in text classification.
5. Memory Efficiency: It focuses on support vectors making it memory efficient
compared to other algorithms.
Disadvantages of Support Vector Machine (SVM)
1. Slow Training: SVM can be slow for large datasets, affecting performance in
SVM in data mining tasks.
2. Parameter Tuning Difficulty: Selecting the right kernel and adjusting
parameters like C requires careful tuning, impacting SVM algorithms.
3. Noise Sensitivity: SVM struggles with noisy datasets and overlapping classes,
limiting effectiveness in real-world scenarios.
4. Limited Interpretability: The complexity of the hyperplane in higher
dimensions makes SVM less interpretable than other models.
5. Feature Scaling Sensitivity: Proper feature scaling is essential, otherwise
SVM models may perform poorly.

K-Nearest Neighbors (KNN) is a supervised machine learning algorithm generally

used for classification but can also be used for regression tasks. It works by finding
the "k" closest data points (neighbors) to a given input and makes a predictions based
on the majority class (for classification) or the average value (for regression). Since
KNN makes no assumptions about the underlying data distribution it makes it a non-
parametric and instance-based learning method.
K-Nearest Neighbors is also called as a lazy learner algorithm because it does not
learn from the training set immediately instead it stores the dataset and at the time
of classification it performs an action on the dataset.
For example, consider the following table of data points containing two features:

KNN Algorithm working visualization

The new point is classified as Category 2 because most of its closest neighbors are
blue squares. KNN assigns the category based on the majority of nearby points. The
image shows how KNN predicts the category of a new data point based on its closest
neighbours.
• The red diamonds represent Category 1 and the blue squares represent
Category 2.
• The new data point checks its closest neighbors (circled points).
 • Since the majority of its closest neighbors are blue squares (Category 2) KNN
predicts the new data point belongs to Category 2.
KNN works by using proximity and majority voting to make predictions.
What is 'K' in K Nearest Neighbour?
In the k-Nearest Neighbours algorithm k is just a number that tells the algorithm how
many nearby points or neighbors to look at when it makes a decision.
Example: Imagine you're deciding which fruit it is based on its shape and size. You
compare it to fruits you already know.
• If k = 3, the algorithm looks at the 3 closest fruits to the new one.
• If 2 of those 3 fruits are apples and 1 is a banana, the algorithm says the new
fruit is an apple because most of its neighbors are apples.
How to choose the value of k for KNN Algorithm?
• The value of k in KNN decides how many neighbors the algorithm looks at
when making a prediction.
• Choosing the right k is important for good results.
• If the data has lots of noise or outliers, using a larger k can make the
predictions more stable.
• But if k is too large the model may become too simple and miss important
patterns and this is called underfitting.
• So k should be picked carefully based on the data.
Statistical Methods for Selecting k
• Cross-Validation: Cross-Validation is a good way to find the best value of k is
by using k-fold cross-validation. This means dividing the dataset into k parts.
The model is trained on some of these parts and tested on the remaining ones.
This process is repeated for each part. The k value that gives the highest
average accuracy during these tests is usually the best one to use.
 • Elbow Method: In Elbow Method we draw a graph showing the error rate or
accuracy for different k values. As k increases the error usually drops at first.
But after a certain point error stops decreasing quickly. The point where the
curve changes direction and looks like an "elbow" is usually the best choice
for k.
• Odd Values for k: It’s a good idea to use an odd number for k especially in
classification problems. This helps avoid ties when deciding which class is the
most common among the neighbors.

Distance Metrics Used in KNN Algorithm

KNN uses distance metrics to identify nearest neighbor, these neighbors are used for
classification and regression task. To identify nearest neighbor we use below
distance metrics:
1. Euclidean Distance
Euclidean distance is defined as the straight-line distance between two points in a
plane or space. You can think of it like the shortest path you would walk if you were
to go directly from one point to another.
distance(x,Xi)=∑j=1d(xj−Xij)2]
2. Manhattan Distance
This is the total distance you would travel if you could only move along horizontal
and vertical lines like a grid or city streets. It’s also called "taxicab distance" because
a taxi can only drive along the grid-like streets of a city.
d(x,y)=∑i=1n∣xi−yi∣
3. Minkowski Distance
Minkowski distance is like a family of distances, which includes both Euclidean and
Manhattan distances as special cases.
d(x,y)=(∑i=1n(xi−yi)p)1p
From the formula above, when p=2, it becomes the same as the Euclidean distance
formula and when p=1, it turns into the Manhattan distance formula. Minkowski
distance is essentially a flexible formula that can represent either Euclidean or
Manhattan distance depending on the value of p.
Working of KNN algorithm
Thе K-Nearest Neighbors (KNN) algorithm operates on the principle of similarity
where it predicts the label or value of a new data point by considering the labels or
values of its K nearest neighbors in the training dataset.

Step 1: Selecting the optimal value of K

• K represents the number of nearest neighbors that needs to be considered
while making prediction.
Step 2: Calculating distance
• To measure the similarity between target and training data points Euclidean
distance is used. Distance is calculated between data points in the dataset and
target point.
Step 3: Finding Nearest Neighbors
• The k data points with the smallest distances to the target point are nearest
neighbors.
Step 4: Voting for Classification or Taking Average for Regression
 • When you want to classify a data point into a category like spam or not spam,
the KNN algorithm looks at the K closest points in the dataset. These closest
points are called neighbors. The algorithm then looks at which category the
neighbors belong to and picks the one that appears the most. This is called
majority voting.
• In regression, the algorithm still looks for the K closest points. But instead of
voting for a class in classification, it takes the average of the values of those
K neighbors. This average is the predicted value for the new point for the
algorithm.
It shows how a test point is classified based on its nearest neighbors. As the test point
moves the algorithm identifies the closest 'k' data points i.e. 5 in this case and assigns
test point the majority class label that is grey label class here.
Python Implementation of KNN Algorithm
1. Importing Libraries
Counter is used to count the occurrences of elements in a list or iterable. In KNN
after finding the k nearest neighbor labels Counter helps count how many times each
label appears.
import numpy as np
from collections import Counter
2. Defining the Euclidean Distance Function
euclidean_distance is to calculate euclidean distance between points.
def euclidean_distance(point1, point2):
return [Link]([Link](([Link](point1) - [Link](point2))**2))
3. KNN Prediction Function
• [Link] saves how far each training point is from the test point,
along with its label.
• [Link] is used to sorts the list so the nearest points come first.
 • k_nearest_labels picks the labels of the k closest points.
• Uses Counter to find which label appears most among those k labels that
becomes the prediction.
def knn_predict(training_data, training_labels, test_point, k):
distances = []
for i in range(len(training_data)):
dist = euclidean_distance(test_point, training_data[i])
[Link]((dist, training_labels[i]))
[Link](key=lambda x: x[0])
k_nearest_labels = [label for _, label in distances[:k]]
return Counter(k_nearest_labels).most_common(1)[0][0]
4. Training Data, Labels and Test Point
training_data = [[1, 2], [2, 3], [3, 4], [6, 7], [7, 8]]
training_labels = ['A', 'A', 'A', 'B', 'B']
test_point = [4, 5]
k=3
5. Prediction
prediction = knn_predict(training_data, training_labels, test_point, k)
print(prediction)
Output:
A
The algorithm calculates the distances of the test point [4, 5] to all training points
selects the 3 closest points as k = 3 and determines their labels. Since the majority
of the closest points are labelled 'A' the test point is classified as 'A'.
In machine learning we can also use Scikit Learn python library which has in built
functions to perform KNN machine learning model and for that you refer to
Implementation of KNN classifier using Sklearn.
Applications of KNN
• Recommendation Systems: Suggests items like movies or products by finding
users with similar preferences.
• Spam Detection: Identifies spam emails by comparing new emails to known
spam and non-spam examples.
• Customer Segmentation: Groups customers by comparing their shopping
behavior to others.
• Speech Recognition: Matches spoken words to known patterns to convert
them into text.
Advantages of KNN
• Simple to use: Easy to understand and implement.
• No training step: No need to train as it just stores the data and uses it during
prediction.
• Few parameters: Only needs to set the number of neighbors (k) and a distance
method.
• Versatile: Works for both classification and regression problems.
Disadvantages of KNN
• Slow with large data: Needs to compare every point during prediction.
• Struggles with many features: Accuracy drops when data has too many
features.
• Can Overfit: It can overfit especially when the data is high-dimensional or not
clean.
Ensemble learning is a method where we use many small models instead of just one.
Each of these models may not be very strong on its own, but when we put their
results together, we get a better and more accurate answer. It's like asking a group of
people for advice instead of just one person—each one might be a little wrong, but
together, they usually give a better answer.
Types of Ensembles Learning in Machine Learning
There are three main types of ensemble methods:
1. Bagging (Bootstrap Aggregating):
Models are trained independently on different random subsets of the training
data. Their results are then combined—usually by averaging (for regression)
or voting (for classification). This helps reduce variance and prevents
overfitting.
2. Boosting:
Models are trained one after another. Each new model focuses on fixing the
errors made by the previous ones. The final prediction is a weighted
combination of all models, which helps reduce bias and improve accuracy.
3. Stacking (Stacked Generalization):
Multiple different models (often of different types) are trained, and their
predictions are used as inputs to a final model, called a meta-model. The
meta-model learns how to best combine the predictions of the base models,
aiming for better performance than any individual model.
1. Bagging Algorithm
Bagging classifier can be used for both regression and classification tasks. Here is
an overview of Bagging classifier algorithm:
• Bootstrap Sampling: Divides the original training data into ‘N’ subsets and
randomly selects a subset with replacement in some rows from other subsets.
This step ensures that the base models are trained on diverse subsets of the
data and there is no class imbalance.
• Base Model Training: For each bootstrapped sample we train a base model
independently on that subset of data. These weak models are trained in parallel
 to increase computational efficiency and reduce time consumption. We can
use different base learners i.e. different ML models as base learners to bring
variety and robustness.
• Prediction Aggregation: To make a prediction on testing data combine the
predictions of all base models. For classification tasks it can include majority
voting or weighted majority while for regression it involves averaging the
predictions.
• Out-of-Bag (OOB) Evaluation: Some samples are excluded from the training
subset of particular base models during the bootstrapping method. These “out-
of-bag” samples can be used to estimate the model’s performance without the
need for cross-validation.
• Final Prediction: After aggregating the predictions from all the base models,
Bagging produces a final prediction for each instance.
Python pseudo code for Bagging Estimator implementing libraries:
1. Importing Libraries and Loading Data
• BaggingClassifier: for creating an ensemble of classifiers trained on different
subsets of data.
• DecisionTreeClassifier: the base classifier used in the bagging ensemble.
• load_iris: to load the Iris dataset for classification.
• train_test_split: to split the dataset into training and testing subsets.
• accuracy_score: to evaluate the model’s prediction accuracy.
from [Link] import BaggingClassifier
from [Link] import DecisionTreeClassifier
from [Link] import load_iris
from sklearn.model_selection import train_test_split
from [Link] import accuracy_score
2. Loading and Splitting the Iris Dataset
 • data = load_iris(): loads the Iris dataset, which includes features and target
labels.
• X = [Link]: extracts the feature matrix (input variables).
• y = [Link]: extracts the target vector (class labels).
• train_test_split(...): splits the data into training (80%) and testing (20%) sets,
with random_state=42 to ensure reproducibility.
data = load_iris()
X = [Link]
y = [Link]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
3. Creating a Base Classifier
Decision tree is chosen as the base model. They are prone to overfitting when trained
on small datasets making them good candidates for bagging.
• base_classifier = DecisionTreeClassifier(): initializes a Decision Tree
classifier, which will serve as the base estimator in the Bagging ensemble.
base_classifier = DecisionTreeClassifier()
4. Creating and Training the Bagging Classifier
• A BaggingClassifier is created using the decision tree as the base classifier.
• n_estimators = 10 specifies that 10 decision trees will be trained on different
bootstrapped subsets of the training data.
bagging_classifier = BaggingClassifier(base_classifier, n_estimators=10,
random_state=42)
bagging_classifier.fit(X_train, y_train)
5. Making Predictions and Evaluating Accuracy
• The trained bagging model predicts labels for test data.
 • The accuracy of the predictions is calculated by comparing the predicted
labels (y_pred) to the actual labels (y_test).
y_pred = bagging_classifier.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print("Accuracy:", accuracy)
Output:
Accuracy: 1.0
2. Boosting Algorithm
Boosting is an ensemble technique that combines multiple weak learners to create a
strong learner. Weak models are trained in series such that each next model tries to
correct errors of the previous model until the entire training dataset is predicted
correctly. One of the most well-known boosting algorithms is AdaBoost (Adaptive
Boosting). Here is an overview of Boosting algorithm:
• Initialize Model Weights: Begin with a single weak learner and assign equal
weights to all training examples.
• Train Weak Learner: Train weak learners on these dataset.
• Sequential Learning: Boosting works by training models sequentially where
each model focuses on correcting the errors of its predecessor. Boosting
typically uses a single type of weak learner like decision trees.
• Weight Adjustment: Boosting assigns weights to training datapoints.
Misclassified examples receive higher weights in the next iteration so that
next models pay more attention to them.
Python pseudo code for boosting Estimator implementing libraries:
1. Importing Libraries and Modules
• AdaBoostClassifier from [Link]: for building the AdaBoost
ensemble model.
 • DecisionTreeClassifier from [Link]: as the base weak learner for
AdaBoost.
• load_iris from [Link]: to load the Iris dataset.
• train_test_split from sklearn.model_selection: to split the dataset into training
and testing sets.
• accuracy_score from [Link]: to evaluate the model’s accuracy.
from [Link] import AdaBoostClassifier
from [Link] import DecisionTreeClassifier
from [Link] import load_iris
from sklearn.model_selection import train_test_split
from [Link] import accuracy_score
2. Loading and Splitting the Dataset
• data = load_iris(): loads the Iris dataset, which includes features and target
labels.
• X = [Link]: extracts the feature matrix (input variables).
• y = [Link]: extracts the target vector (class labels).
• train_test_split(...): splits the data into training (80%) and testing (20%) sets,
with random_state=42 to ensure reproducibility.
data = load_iris()
X = [Link]
y = [Link]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
3. Defining the Weak Learner
We are creating the base classifier as a decision tree with maximum depth 1 (a
decision stump). This simple tree will act as a weak learner for the AdaBoost
algorithm, which iteratively improves by combining many such weak learners.
base_classifier = DecisionTreeClassifier(max_depth=1)
4. Creating and Training the AdaBoost Classifier
• base_classifier: The weak learner used in boosting.
• n_estimators = 50: Number of weak learners to train sequentially.
• learning_rate = 1.0: Controls the contribution of each weak learner to the final
model.
• random_state = 42: Ensures reproducibility.
adaboost_classifier = AdaBoostClassifier(
base_classifier, n_estimators=50, learning_rate=1.0, random_state=42
)
adaboost_classifier.fit(X_train, y_train)
5. Making Predictions and Calculating Accuracy
We are calculating the accuracy of the model by comparing the true
labels y_test with the predicted labels y_pred. The accuracy_score function returns
the proportion of correctly predicted samples. Then, we print the accuracy value.
accuracy = accuracy_score(y_test, y_pred)
print("Accuracy:", accuracy)
Output:
Accuracy: 1.0
Benefits of Ensemble Learning in Machine Learning
Ensemble learning is a versatile approach that can be applied to machine learning
model for: -
• Reduction in Overfitting: By aggregating predictions of multiple model's
ensembles can reduce overfitting that individual complex models might
exhibit.
• Improved Generalization: It generalizes better to unseen data by minimizing
variance and bias.
 • Increased Accuracy: Combining multiple models gives higher predictive
accuracy.
• Robustness to Noise: It mitigates the effect of noisy or incorrect data points
by averaging out predictions from diverse models.
• Flexibility: It can work with diverse models including decision trees, neural
networks and support vector machines making them highly adaptable.
• Bias-Variance Tradeoff: Techniques like bagging reduce variance, while
boosting reduces bias leading to better overall performance.
There are various ensemble learning techniques we can use as each one of them has
their own pros and cons.
Ensemble Learning Techniques

Technique Category Description

Random forest constructs multiple decision trees on

bootstrapped subsets of the data and aggregates
Bagging
their predictions for final output, reducing
Random Forest overfitting and variance.

Random Trains models on random subsets of input features

Subspace Bagging to enhance diversity and improve generalization
Method while reducing overfitting.

Gradient Gradient Boosting Machines sequentially builds

Boosting decision trees, with each tree correcting errors of
Boosting
Machines the previous ones, enhancing predictive accuracy
(GBM) iteratively.
Technique Category Description

Extreme
XGBoost do optimizations like tree pruning,
Gradient
Boosting regularization, and parallel processing for robust
Boosting
and efficient predictive models.
(XGBoost)

AdaBoost focuses on challenging examples by

AdaBoost assigning weights to data points. Combines weak
Boosting
(Adaptive classifiers with weighted voting for final
Boosting) predictions.

CatBoost specialize in handling categorical features

natively without extensive preprocessing with high
Boosting
predictive accuracy and automatic overfitting
CatBoost handling.

Random Forest is a machine learning algorithm that uses many decision trees to
make better predictions. Each tree looks at different random parts of the data and
their results are combined by voting for classification or averaging for regression.
This helps in improving accuracy and reducing errors.

Working of Random Forest Algorithm

• Create Many Decision Trees: The algorithm makes many decision trees each
using a random part of the data. So every tree is a bit different.
 • Pick Random Features: When building each tree it doesn’t look at all the
features (columns) at once. It picks a few at random to decide how to split the
data. This helps the trees stay different from each other.
• Each Tree Makes a Prediction: Every tree gives its own answer or prediction
based on what it learned from its part of the data.
• Combine the Predictions:
o For classification we choose a category as the final answer is the one
that most trees agree on i.e majority voting.
o For regression we predict a number as the final answer is the average
of all the trees predictions.
• Why It Works Well: Using random data and features for each tree helps avoid
overfitting and makes the overall prediction more accurate and trustworthy.
Random forest is also a ensemble learning technique which you can learn more
about from: Ensemble Learning
Key Features of Random Forest
• Handles Missing Data: It can work even if some data is missing so you don’t
always need to fill in the gaps yourself.
• Shows Feature Importance: It tells you which features (columns) are most
useful for making predictions which helps you understand your data better.
• Works Well with Big and Complex Data: It can handle large datasets with
many features without slowing down or losing accuracy.
• Used for Different Tasks: You can use it for both classification like predicting
types or labels and regression like predicting numbers or amounts.
Assumptions of Random Forest
• Each tree makes its own decisions: Every tree in the forest makes its own
predictions without relying on others.
 • Random parts of the data are used: Each tree is built using random samples
and features to reduce mistakes.
• Enough data is needed: Sufficient data ensures the trees are different and learn
unique patterns and variety.
• Different predictions improve accuracy: Combining the predictions from
different trees leads to a more accurate final result.
Implementing Random Forest for Classification Tasks
Here we will predict survival rate of a person in titanic.
• Import libraries and load the Titanic dataset.
• Remove rows with missing target values ('Survived').
• Select features like class, sex, age, etc and convert 'Sex' to numbers.
• Fill missing age values with the median.
• Split the data into training and testing sets, then train a Random Forest model.
• Predict on test data, check accuracy and print a sample prediction result.
import pandas as pd
from sklearn.model_selection import train_test_split
from [Link] import RandomForestClassifier
from [Link] import accuracy_score, classification_report
import warnings
[Link]('ignore')

url =
"[Link]
titanic_data = pd.read_csv(url)

titanic_data = titanic_data.dropna(subset=['Survived'])
X = titanic_data[['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare']]
y = titanic_data['Survived']

[Link][:, 'Sex'] = X['Sex'].map({'female': 0, 'male': 1})

[Link][:, 'Age'].fillna(X['Age'].median(), inplace=True)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,

random_state=42)

rf_classifier = RandomForestClassifier(n_estimators=100, random_state=42)

rf_classifier.fit(X_train, y_train)

y_pred = rf_classifier.predict(X_test)

accuracy = accuracy_score(y_test, y_pred)

classification_rep = classification_report(y_test, y_pred)

print(f"Accuracy: {accuracy:.2f}")
print("\nClassification Report:\n", classification_rep)

sample = X_test.iloc[0:1]
prediction = rf_classifier.predict(sample)

sample_dict = [Link][0].to_dict()
print(f"\nSample Passenger: {sample_dict}")
print(f"Predicted Survival: {'Survived' if prediction[0] == 1 else 'Did Not
Survive'}")

Random Forest for Classification Tasks

We evaluated model's performance using a classification report to see how well it
predicts the outcomes and used a random sample to check model prediction.
Implementing Random Forest for Regression Tasks
We will do house price prediction here.
• Load the California housing dataset and create a DataFrame with features and
target.
• Separate the features and the target variable.
• Split the data into training and testing sets (80% train, 20% test).
• Initialize and train a Random Forest Regressor using the training data.
• Predict house values on test data and evaluate using MSE and R² score.
• Print a sample prediction and compare it with the actual value.
import pandas as pd
from [Link] import fetch_california_housing
from sklearn.model_selection import train_test_split
from [Link] import RandomForestRegressor
from [Link] import mean_squared_error, r2_score

california_housing = fetch_california_housing()
california_data = [Link](california_housing.data,
columns=california_housing.feature_names)
california_data['MEDV'] = california_housing.target

X = california_data.drop('MEDV', axis=1)
y = california_data['MEDV']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,

random_state=42)

rf_regressor = RandomForestRegressor(n_estimators=100, random_state=42)

rf_regressor.fit(X_train, y_train)

y_pred = rf_regressor.predict(X_test)

mse = mean_squared_error(y_test, y_pred)

r2 = r2_score(y_test, y_pred)

single_data = X_test.iloc[0].[Link](1, -1)

predicted_value = rf_regressor.predict(single_data)
print(f"Predicted Value: {predicted_value[0]:.2f}")
print(f"Actual Value: {y_test.iloc[0]:.2f}")

print(f"Mean Squared Error: {mse:.2f}")

print(f"R-squared Score: {r2:.2f}")

We evaluated the model's performance using Mean Squared Error and R-squared
Score which show how accurate the predictions are and used a random sample to
check model prediction.
Advantages of Random Forest
• Random Forest provides very accurate predictions even with large datasets.
 • Random Forest can handle missing data well without compromising with
accuracy.
• It doesn’t require normalization or standardization on dataset.
• When we combine multiple decision trees it reduces the risk of overfitting of
the model.
Limitations of Random Forest
• It can be computationally expensive especially with a large number of trees.
• It’s harder to interpret the model compared to simpler models like decision
trees.