Ensemble learning in machine learning combines multiple individual models to create a
stronger, more accurate predictive model. By leveraging the diverse strengths of different
models, ensemble learning aims to mitigate errors, enhance performance, and increase the
overall robustness of predictions, leading to improved results across various tasks in
machine learning and data analysis.
Bagging and Boosting are two types of Ensemble Learning. These two decrease the
variance of a single estimate as they combine several estimates from different models. So
the result may be a model with higher stability. Let's understand these two terms in a
glimpse.
1. Bagging: It is a homogeneous weak learners’ model that learns from each other
independently in parallel and combines them for determining the model average.
2. Boosting: It is also a homogeneous weak learners’ model but works differently from
Bagging. In this model, learners learn sequentially and adaptively to improve model
predictions of a learning algorithm.
Bagging
Bootstrap Aggregating, also known as bagging, is a machine learning ensemble
meta-algorithm designed to improve the stability and accuracy of machine learning
algorithms used in statistical classification and regression. It decreases the variance and
helps to avoid overfitting. It is usually applied to decision tree methods. Bagging is a special
case of the model averaging approach.
Description of the Technique
Suppose a set D of d tuples, at each iteration i, a training set Di of d tuples is selected via
row sampling with a replacement method (i.e., there can be repetitive elements from
different d tuples) from D (i.e., bootstrap). Then a classifier model Mi is learned for each
training set D < i. Each classifier Mi returns its class prediction. The bagged classifier M*
counts the votes and assigns the class with the most votes to X (unknown sample).
Implementation Steps of Bagging
Step 1: Multiple subsets are created from the original data set with equal tuples, selecting
observations with replacement.
Step 2: A base model is created on each of these subsets.
Step 3: Each model is learned in parallel with each training set and independent of each
other.
Step 4: The final predictions are determined by combining the predictions from all the
models.
Example of Bagging
The Random Forest model uses Bagging, where decision tree models with higher variance
are present. It makes random feature selection to grow trees. Several random trees make a
Random Forest.
To read more refer to this article: Bagging classifier
Boosting
Boosting is an ensemble modeling technique designed to create a strong classifier by
combining multiple weak classifiers. The process involves building models sequentially,
where each new model aims to correct the errors made by the previous ones.
Initially, a model is built using the training data.
Subsequent models are then trained to address the mistakes of their predecessors.
boosting assigns weights to the data points in the original dataset.
Higher weights: Instances that were misclassified by the previous model receive higher
weights.
Lower weights: Instances that were correctly classified receive lower weights.
Training on weighted data: The subsequent model learns from the weighted dataset,
focusing its attention on harder-to-learn examples (those with higher weights).
This iterative process continues until:
The entire training dataset is accurately predicted, or
A predefined maximum number of models is reached.
Boosting Algorithms
There are several boosting algorithms. The original ones, proposed by Robert Schapire and
Yoav Freund were not adaptive and could not take full advantage of the weak learners.
Schapire and Freund then developed AdaBoost, an adaptive boosting algorithm that won the
prestigious Gödel Prize. AdaBoost was the first really successful boosting algorithm
developed for the purpose of binary classification. AdaBoost is short for Adaptive Boosting
and is a very popular boosting technique that combines multiple “weak classifiers” into a
single “strong classifier”.
Algorithm:
1. Initialise the dataset and assign equal weight to each of the data point.
2. Provide this as input to the model and identify the wrongly classified data
points.
3. Increase the weight of the wrongly classified data points and decrease
the weights of correctly classified data points. And then normalize the
weights of all data points.
4. if (got required results)
Goto step 5
else
Goto step 2
5. End
Differences Between Bagging and Boosting
S.N Bagging Boosting
O
1. The simplest way of combining A way of combining predictions that
predictions that belong to the different types.
belong to the same type.
2. Aim to decrease variance, not bias. Aim to decrease bias, not variance.
3. Each model receives equal weight. Models are weighted according to
their performance.
4. Each model is built independently. New models are influenced
by the performance of previously built
models.
5. Different training data subsets are Iteratively train models, with each
selected using row sampling with new model focusing on correcting the
replacement and random sampling errors (misclassifications or high
methods from the entire training residuals) of the previous models
dataset.
6. Bagging tries to solve the Boosting tries to reduce bias.
over-fitting problem.
7. If the classifier is unstable (high If the classifier is stable and simple
variance), then apply bagging. (high bias) the apply boosting.
8. In this base classifiers are trained In this base classifiers are trained
parallelly. sequentially.
9 Example: The Random forest model Example: The AdaBoost uses
uses Bagging. Boosting techniques