Ensemble Methods in Machine

Learning

Presented BY:
Dr. Amran Hossain
Associate Professor
CSE,DUET
 Ensemble Methods, what are they?
• Ensemble method is a machine learning
technique that combines several base models in
order to produce one optimal predictive model. To
better understand this definition lets take a step
back into ultimate goal of machine learning and
model building.
• Ensemble learning is a general meta approach to
machine learning that seeks better predictive
performance by combining the predictions from
multiple models.
• Ensemble methods are techniques that aim at
improving the accuracy of results in models by
combining multiple models instead of using a
single model. The combined models increase the
accuracy of the results significantly
Types of Ensemble Methods
 BAGGING
• Bagging, the short form for bootstrap aggregating, is mainly applied in classification
and regression.
• It increases the accuracy of models through decision trees, which reduces variance to
a large extent. The reduction of variance increases accuracy, eliminating overfitting,
which is a challenge to many predictive models.
• Bagging is classified into two types, i.e., bootstrapping and aggregation.
• Bootstrapping is a sampling technique where samples are derived from the whole population (set)
using the replacement procedure. The sampling with replacement method helps make the
selection procedure randomized. The base learning algorithm is run on the samples to complete
the procedure.
• Aggregation in bagging is done to incorporate all possible outcomes of the prediction and
randomize the outcome. Without aggregation, predictions will not be accurate because all
outcomes are not put into consideration. Therefore, the aggregation is based on the probability
bootstrapping procedures or on the basis of all outcomes of the predictive models.
 BAGGING
• BAGGing, or Bootstrap AGGregating.
BAGGing gets its name because it
combines Bootstrapping and
Aggregation to form one ensemble
model.
• Given a sample of data, multiple
bootstrapped subsamples are pulled.
• A Decision Tree is formed on each of
the bootstrapped subsamples.
• After each subsample Decision Tree
has been formed, an algorithm is used
to aggregate over the Decision Trees to
form the most efficient predictor. The
image below will help explain:
 BOOSTING
• Boosting is an ensemble technique that learns from previous predictor
mistakes to make better predictions in the future.
• The technique combines several weak base learners to form one strong
learner, thus significantly improving the predictability of models.
• Boosting works by arranging weak learners in a sequence, such that weak
learners learn from the next learner in the sequence to create better
predictive models.
• Boosting takes many forms, including gradient boosting, Adaptive Boosting
(AdaBoost), and XGBoost (Extreme Gradient Boosting).
• AdaBoost uses weak learners in the form of decision trees, which mostly
include one split that is popularly known as decision stumps.
• AdaBoost’s main decision stump comprises observations carrying similar
weights.
 BOOSTING
• Gradient boosting adds predictors sequentially to the
ensemble, where preceding predictors correct their
successors, thereby increasing the model’s accuracy. New
predictors are fit to counter the effects of errors in the
previous predictors.
• The key property of boosting ensembles is the idea of
correcting prediction errors. The models are fit and added to
the ensemble sequentially such that the second model
attempts to correct the predictions of the first model, the
third corrects the second model, and so on.
• The gradient of descent helps the gradient booster identify
problems in learners’ predictions and counter them
accordingly.
• XGBoost makes use of decision trees with boosted gradient,
providing improved speed and performance.
• It relies heavily on the computational speed and the
performance of the target model. Model training should
follow a sequence, thus making the implementation of
gradient boosted machines slow.
BOOSTING
 Stacking
• Stacking, another ensemble method, is often referred to as
stacked generalization. This technique works by allowing a
training algorithm to ensemble several other similar learning
algorithm predictions. Stacking has been successfully
implemented in regression, density estimations, distance
learning, and classifications. It can also be used to measure
the error rate involved during bagging.
• Stacked Generalization, or stacking for short, is an ensemble
method that seeks a diverse group of members by varying
the model types fit on the training data and using a model to
combine predictions.
• Stacking is a general procedure where a learner is trained to
combine the individual learners. Here, the individual learners
are called the first-level learners, while the combiner is called
the second-level learner, or meta-learner.