Ensemble Machine Learning Models
[Link]
Ensemble Modeling
• Combines multiple machine learning models to improve overall predictive
performance
• The basic idea is that a group of weak learners can come together to form
one strong learner.
• An ensemble model typically consists of two steps:
• Multiple machine learning models are trained independently.
• Their predictions are aggregated in some way, such as by voting, averaging, or
weighting.
• ensemble is then used to make the overall prediction.
• Ensembles tend to yield better results because the different models complement
each other and overcome their individual weaknesses.
• ensemble methods are bagging, boosting, and stacking.
• improve predictions using the stacking approach.
Simple Ensemble Models
Ensemble Modeling
Why ensembles ?
• There are two main reasons to use an ensemble over a single model,
and they are related; they are:
• Performance: An ensemble can make better predictions and achieve better
performance than any single contributing model.
• Robustness: An ensemble reduces the spread or dispersion of the predictions
and model performance.
What is Bagging?
• Bagging (bootstrap aggregating) is an ensemble method that involves
training multiple models independently on random subsets of the data,
and aggregating their predictions through voting or averaging.
• bagging reduces the variance of the individual
models by training models on different
bootstraps.
• It also avoids overfitting by exposing the
constituent models to different parts of the
dataset.
• The predictions from all the sampled models
are then combined through a simple averaging
to make the overall prediction.
Bagging
Random Forest
Advantages of Bagging
• Advantages of Bagging
• Here are some key advantages of bagging:
• Reduces overfitting: It can reduce the chance of an overfit model, resulting in improved model accuracy on
unseen data.
• Decreases model variance: Multiple models trained on different subsets of data average out their
predictions, leading to lower variance than a single model.
• Improves stability: Changes in the training dataset have less impact on bagged models, making the overall
model more stable.
• Handles high variability: Especially effective for algorithms like decision trees, which tend to have high
variance.
• Parallelizable computation: Each model in the ensemble can be trained independently, allowing for parallel
processing and efficient use of computational resources.
• Easy to understand and implement: The concept behind bagging is straightforward and can be implemented
without complex modifications to the learning algorithm.
• Good with noisy data: The averaging process helps in reducing the noise in the final prediction.
• Handles imbalanced data: Bagging can help in scenarios where the dataset is imbalanced, improving the
performance of the model in such situations.
Boosting
AdaBoosting
• Bagging: Models are trained Boosting: Models are trained
Bagging independently in parallel on
different random subsets of the
sequentially, with each model learning
from the errors of the previous one.
Vs. data. Boosting assigns weights based on
accuracy.
• Involves simple averaging of
Boosting models boosting reduces bias
boosting works better for stable
• Bagging reduces variance models like linear regression.
• Bagging can be used with unstable
models like decision trees