Ensemble Classifier
Ensemble learning helps improve machine learning results by combining
several models. This approach allows the production of better predictive
performance compared to a single model. Basic idea is to learn a set of
classifiers (experts) and to allow them to vote.
Advantage : Improvement in predictive accuracy.
Disadvantage : It is difficult to understand an ensemble of classifiers.
Why do ensembles work?
Dietterich(2002) showed that ensembles overcome three problems –
Statistical Problem –
The Statistical Problem arises when the hypothesis space is too large for
the amount of available data. Hence, there are many hypotheses with
the same accuracy on the data and the learning algorithm chooses only
one of them! There is a risk that the accuracy of the chosen hypothesis is
low on unseen data!
Computational Problem –
The Computational Problem arises when the learning algorithm cannot
guarantees finding the best hypothesis.
Representational Problem –
The Representational Problem arises when the hypothesis space does
not contain any good approximation of the target class(es).
As we know, Ensemble learning helps improve machine learning results by
combining several models. This approach allows the production of better
predictive performance compared to a single model. Basic idea is to learn a set
of classifiers (experts) and to allow them to vote. Bagging and Boosting are
two types of Ensemble Learning. These two decrease the variance of a single
estimate as they combine several estimates from different models. So the
result may be a model with higher stability.
Bagging
Bootstrap Aggregating, also known as bagging, is a machine learning ensemble
meta-algorithm designed to improve the stability and accuracy of machine
learning algorithms used in statistical classification and regression. It decreases
the variance and helps to avoid overfitting. It is usually applied to decision tree
methods. Bagging is a special case of the model averaging approach.
Description of the Technique
Suppose a set D of d tuples, at each iteration i, a training set Di of d tuples is
selected via row sampling with a replacement method (i.e., there can be
repetitive elements from different d tuples) from D (i.e., bootstrap). Then a
classifier model Mi is learned for each training set D < i. Each classifier
Mi returns its class prediction. The bagged classifier M* counts the votes and
assigns the class with the most votes to X (unknown sample).
Implementation Steps of Bagging
Step 1: Multiple subsets are created from the original data set with
equal tuples, selecting observations with replacement.
Step 2: A base model is created on each of these subsets.
Step 3: Each model is learned in parallel with each training set and
independent of each other.
Step 4: The final predictions are determined by combining the
predictions from all the models.
Example of Bagging: Random Forest
Boosting
Boosting is an ensemble modeling technique designed to create a strong
classifier by combining multiple weak classifiers. The process involves building
models sequentially, where each new model aims to correct the errors made
by the previous ones.
The Explanation for Training the Boosting Model:
The above diagram explains the AdaBoost algorithm in a very simple way. Let’s
try to understand it in a stepwise process:
B1 consists of 10 data points which consist of two types namely plus(+)
and minus(-) and 5 of which are plus(+) and the other 5 are minus(-) and
each one has been assigned equal weight initially. The first model tries
to classify the data points and generates a vertical separator line but it
wrongly classifies 3 plus(+) as minus(-).
B2 consists of the 10 data points from the previous model in which the 3
wrongly classified plus(+) are weighted more so that the current model
tries more to classify these pluses(+) correctly. This model generates a
vertical separator line that correctly classifies the previously wrongly
classified pluses(+) but in this attempt, it wrongly classifies three
minuses(-).
B3 consists of the 10 data points from the previous model in which the 3
wrongly classified minus(-) are weighted more so that the current model
tries more to classify these minuses(-) correctly. This model generates a
horizontal separator line that correctly classifies the previously wrongly
classified minuses(-).
B4 combines together B1, B2, and B3 in order to build a strong
prediction model which is much better than any individual model used.
Boosting Algorithms
There are several boosting algorithms. The original ones, proposed by Robert
Schapire and Yoav Freund were not adaptive and could not take full advantage
of the weak learners. Schapire and Freund then developed AdaBoost, an
adaptive boosting algorithm that won the prestigious Gödel Prize. AdaBoost
was the first really successful boosting algorithm developed for the purpose of
binary classification. AdaBoost is short for Adaptive Boosting and is a very
popular boosting technique that combines multiple “weak classifiers” into a
single “strong classifier”.
Types of Boosting Algorithms
There are several types of boosting algorithms some of the most famous and
useful models are as:
Gradient Boosting – It is a boosting technique that builds a final model from
the sum of several weak learning algorithms that were trained on the same
dataset. It operates on the idea of stagewise addition. The first weak learner in
the gradient boosting algorithm will not be trained on the dataset; instead, it
will simply return the mean of the relevant column. The residual for the first
weak learner algorithm’s output will then be calculated and used as the output
column or target column for the next weak learning algorithm that will be
trained. The second weak learner will be trained using the same methodology,
and the residuals will be computed and utilized as an output column once
more for the third weak learner, and so on until we achieve zero residuals. The
dataset for gradient boosting must be in the form of numerical or categorical
data, and the loss function used to generate the residuals must be differential
at all times.
XGBoost – In addition to the gradient boosting technique, XGBoost is another
boosting machine learning approach. The full name of the XGBoost algorithm is
the eXtreme Gradient Boosting algorithm, which is an extreme variation of the
previous gradient boosting technique. The key distinction between XGBoost
and GradientBoosting is that XGBoost applies a regularisation approach. It is a
regularised version of the current gradient-boosting technique. Because of
this, XGBoost outperforms a standard gradient boosting method, which
explains why it is also faster than that. Additionally, it works better when the
dataset contains both numerical and categorical variables.
Adaboost – AdaBoost is a boosting algorithm that also works on the principle
of the stagewise addition method where multiple weak learners are used for
getting strong learners. The value of the alpha parameter, in this case, will be
indirectly proportional to the error of the weak learner, Unlike Gradient
Boosting in XGBoost, the alpha parameter calculated is related to the errors of
the weak learner, here the value of the alpha parameter will be indirectly
proportional to the error of the weak learner.
CatBoost – The growth of decision trees inside CatBoost is the primary
distinction that sets it apart from and improves upon competitors. The decision
trees that are created in CatBoost are symmetric. As there is a unique sort of
approach for handling categorical datasets, CatBoost works very well on
categorical datasets compared to any other algorithm in the field of machine
learning. The categorical features in CatBoost are encoded based on the
output columns. As a result, the output column’s weight will be taken into
account while training or encoding the categorical features, increasing its
accuracy on categorical datasets.
Advantages of Boosting
Improved Accuracy – Boosting can improve the accuracy of the model
by combining several weak models’ accuracies and averaging them for
regression or voting over them for classification to increase the accuracy
of the final model.
Robustness to Overfitting – Boosting can reduce the risk of overfitting
by reweighting the inputs that are classified wrongly.
Better handling of imbalanced data – Boosting can handle the
imbalance data by focusing more on the data points that are
misclassified
Better Interpretability – Boosting can increase the interpretability of the
model by breaking the model decision process into multiple processes.
Disadvantages of Boosting Algorithms
Boosting algorithms also have some disadvantages these are:
Boosting Algorithms are vulnerable to the outliers
It is difficult to use boosting algorithms for Real-Time applications.
It is computationally expensive for large datasets
Boosting vs Bagging
Boosting Bagging
In Boosting we combine predictions Bagging is a method of combining the
that belong to different types same type of prediction
The main aim of boosting is to The main aim of bagging is to
decrease bias, not variance decrease variance not bias
At every successive layer Models are All the models have the same
weighted according to their weightage
performance.
New Models are influenced by the All the models are independent of
accuracy of previous Models each other
Suggestion:
[Link]
learning-a-comprehensive-guide-bdeaa1167a6