MIT School of Computing
Department of Computer Science & Engineering
Third Year Engineering
23CSE3006 -MACHINE LEARNING
Class - T.Y. AIA (SEM-II)
Unit IV: Baysian Learning
Name Of the Course Coordinator:
Prof. Aarti Pimpalkar
Team Members
1. Prof. Dr. Nilima Kulkarni
2. Prof. Abhishek Das
3. Prof. Dattatray Kale
4. Prof. Nilesh Kulal
AY 2025-2026 SEM-II
Ensemble Learning
• Ensemble learning is a method where we use many small models instead of just one.
• Each of these models may not be very strong on its own, but when we put their results
together, we get a better and more accurate answer.
• It's like asking a group of people for advice instead of just one person—each one might
be a little wrong, but together, they usually give a better answer.
Methods for creating ensemble classifier
• By manipulating the training set
• multiple training sets are created by
resampling the original data according to
some sampling distribution
• A classifier is then built from each training set
using a particular learning algorithm. Bagging
and boosting are two examples of ensemble
methods that manipulate their training sets
• By manipulating the input features
• In this approach, a subset of input features is
chosen to form each training set.
• The subset can be either chosen randomly or
based on the recommendation of domain
experts
• E.g. Random Forest
Methods for creating ensemble classifier
• By manipulating the class labels
• This method can be used when the number of classes is sufficiently large.
• The training data is transformed into a binary class problem by randomly
partitioning the class labels into two disjoint subsets, A0 and A1.
• Training examples whose class label belongs to the subset A0 are assigned
to class 0, while those that belong to the subset A1 are assigned to class 1.
• The relabeled examples are then used to train a base classifier
• E.g. error-correcting output coding method
1. Voting
• Voting is the simplest ensemble method: you train multiple different models
and then combine their predictions.
• Hard voting → majority wins (like elections).
• Soft voting → average the predicted probabilities and pick the highest.
The idea is: even if some models are wrong, the majority is usually right.
• Imagine a group of friends deciding where to eat:
• Each friend suggests a restaurant (each friend = one model).
• In the end, the group goes to the restaurant that gets the most votes.
• If you did soft voting, you’d also consider how strongly each friend prefers
their choice (probability scores).
1. Voting
2. Bagging
• Bagging, which is also known as bootstrap aggregating
• It is a technique that repeatedly samples (with replacement) from a
data set according to a uniform probability distribution
• Each bootstrap sample has the same size as the original data.
• Because the sampling is done with replacement, some instances may
appear several times in the same training set, while others may be
omitted from the training set
• On average, a bootstrap sample Di; contains approximately 63% of the
original training data because each sample has a probability
1 - (1 - 1/N)N of being selected in each Di,
2. Bagging -Intuition
•Instead of relying on one model,
Bagging trains several models on
different random samples.
•Each model gives its own prediction.
•The final output is decided by majority
vote (classification) or averaging
(regression).
•This reduces variance and makes
predictions more stable.
•Instead of trusting one weather app,
you check 10 different apps. Each gives
a prediction (rainy, cloudy, sunny).You
take the majority opinion as the final
forecast. This way, you reduce the risk
of relying on one wrong prediction.
Boosting Intuition
• Boosting is an iterative procedure used to adaptively change the
distribution of training examples so that the base classifiers will focus on
examples that are hard to classify.
• Boosting assigns a weight to each training example and may adaptively
change the weight at the end of each boosting round
• Suppose that as a patient, you have certain symptoms. Instead of
consulting one doctor, you choose to consult several.
• Suppose you assign weights to the value or worth of each doctor’s
diagnosis, based on the accuracies of previous diagnoses they have made.
• The final diagnosis is then a combination of the weighted diagnoses.
Boosting
• Initially the examples are assigned equal weights, 1/N, so that they are
equally likely to be chosen for training.
• A sample is drawn according to the sampling distribution of the training
examples to obtain a new training set
• Next, a classifier is induced from the training set and used to classify all
the examples in the original data.
• The weights of the training examples are updated at the end of each
boosting round
• Examples that are classified incorrectly will have their weights increased,
while those that are classified correctly will have their weights decrease
• This forces the classifier to focus on examples that are difficult to classify
in subsequent iterations
3. Boosting
• Each time a tuple is selected, it is
equally likely to be selected again and
re-added to the training set.
• For instance, imagine a machine that
randomly selects tuples for our
training set.
• In sampling with replacement, the
machine is allowed to select the same
tuple more than once
• Each tuple has a probability of 1/d of
being selected, so the probability of
not being chosen is (1-1/d).
• We have to select d times, so the
probability that a tuple will not be
chosen during this whole time is
• (1-1/d)d
Bootstrapping
• This statistical technique consists in generating samples of size B (called
bootstrap samples) from an initial dataset of size N by randomly drawing with
replacement B observations
Random Forest Intuition
• Diversity through randomness – Instead of relying on one decision tree,
Random Forest builds many trees using random subsets of data and features,
which reduces overfitting.
• Wisdom of the crowd – Each tree gives a "vote," and the majority
(classification) or average (regression) decision is taken, leading to more robust
and accurate results.
• Stability and generalization – Combining multiple trees smooths out noise
and makes predictions more reliable on unseen data.
• In plant disease detection, individual decision trees may misclassify due to
variations in leaf color, lighting, or background. A Random Forest combines
many trees trained on different subsets of leaf images, making the overall
prediction (healthy/diseased type) much more accurate and reliable.
4. Random Forests
Stacked Generalization Intuition
• Stacking combines the strengths of multiple different models (base
learners) by training another model (meta-learner) to learn how to best
blend their predictions.
• Instead of simple voting or averaging, the meta-learner finds the optimal
way to weigh and combine base learners.
• Suppose a company is hiring and 3 different panels (technical, HR, and
management) each give scores for candidates.
• Instead of directly averaging the scores, the company uses a final hiring
committee (meta-learner) that learns over time which panel is more
reliable for predicting good hires.
• This way, the final decision is smarter than just voting.
5. Stacked Generalisation (Blending)
Base How They
Method Key Idea Strengths Weaknesses Example
Learners Combine
Train multiple models Heterogene Majority vote Not adaptive,
(homogeneous or ous or (classification) or Simple, easy to weak models may Voting
Voting
heterogeneous) and combine homogeneo averaging implement reduce Classifier
predictions us (regression) performance
Homogeneo Average
Train models on random Less effective if
us (usually (regression) or Reduces variance, Bagged
Bagging subsets of data (with base learners are
decision majority vote avoids overfitting Trees
replacement) weak
trees) (classification)
Strong
Majority vote
Extension of bagging with Homogeneo performance, Can be slower Random
Random (classification) or
decision trees; adds feature us (decision handles high with many trees, Forest
Forest randomness at splits trees)
averaging
dimensionality, less interpretable Classifier
(regression)
reduces variance
Homogeneo
Sequentially train models, each Weighted vote / Reduces bias, Sensitive to noise,
us (usually AdaBoost,
Boosting correcting errors of the sum based on strong predictive prone to
weak XGBoost
previous one learner accuracy performance overfitting
learners)
Stacked Meta-learner
Heterogene Leverages More complex,
Generaliz Train diverse models and learns best
ous strengths of higher risk of Stacked
ation combine their outputs via a combination of
(different different models, overfitting, needs Classifier
(Stacking meta-learner algorithms)
base model
flexible careful tuning
) predictions