RAGHU ENGINEERING

COLLEGE
Autonomous
(Approved by AICTE, New Delhi, Accredited by NBA (CIV, ECE, MECH, CSE), NAAC with ‘A’ grade
& Permanently Affiliated to JNTU-GV Vizianagaram)
Dakamarri, Bheemunipatnam Mandal, Visakhapatnam Dist. – 531 162 (A.P.)
Ph: +91-8922-248001, 248002 Fax: + 91-8922-248011
e-mail: principal@[Link] website: [Link]

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

(Common to AI&ML,DATA SCINECE)

UNIT 5
Ensemble Models: Ensemble Definition, Bootstrapped Aggregation (Bagging) Intuition,
Random Forest and their construction, Extremely randomized trees, Gradient Boosting,
Regularization by Shrinkage, XGBoost, AdaBoost.

We train machine learning models to make the best predictions possible for a given problem. On
the other hand, a single model may not make the best predictions and may be prone to errors such
as variance and bias.
We can combine multiple models into a single model to reduce these errors and improve
predictions. This is called ensemble learning. We will investigate ensemble learning techniques
for improving the machine learning process.
What is Ensemble Learning?
Ensemble learning is the process of strategically creating and combining multiple models, such as
classifiers or experts, to solve a specific computational intelligence problem.
Ensemble learning is primarily used to improve the performance of a model (classification,
prediction, function approximation, etc.) or to reduce the likelihood of an unintentional selection
of a poor one.
Other uses for ensemble learning include assigning a confidence level to the model's decision,
selecting optimal (or near-optimal) features, data fusion, incremental learning, nonstationary
learning, and error correction.
This article focuses on classification-related ensemble learning applications; however, all of the
principles described below can be easily generalized to function approximation or prediction-type
problems as well.
How does Ensemble Learning Work?
Assume you want to create a machine learning model that predicts inventory stock orders for
your company based on historical data from previous years.
Assume you want to create a machine learning model that predicts inventory stock orders for
your company based on historical data from previous years. You train four machine learning
models using different algorithms:
 linear regression, support vector machine, regression decision tree, and basic artificial neural
network. However, even after extensive tweaking and configuration, none of them achieve the
desired 95 percent prediction accuracy. These machine learning models are referred to as "weak
learners" because they fail to converge to the desired level.
However, being weak does not imply being useless. You can put them together to form an
ensemble. For each new prediction, you run your input data through all four models and then
average the results.
When you examine the new result, you will notice that the aggregate results provide 96 percent
accuracy, which is more than satisfactory. Because your machine learning models work
differently, ensemble learning is efficient. Each model may perform well on some data but not on
others. When all of them are combined, their weaknesses cancel each other out.
Ensemble methods can be applied to both prediction problems, such as the inventory prediction
example we just saw, and classification problems, such as determining whether a picture contains
a specific object.
Also Read | 5 Machine Learning Techniques to Solve Overfitting
Why use Ensemble Learning?
For several reasons, an ensemble is preferable to a single model:

1. Performance: As described in the preceding section, the outcome of ensemble learning is

a strong learner. The strong learner is the result of weak learners. As a result, models'
predictive capabilities improve. Better performance is achieved when compared to a single
model.
2. Error reduction: Machine learning model prediction errors can be described by bias and
variance. Bias is defined as the difference between a prediction and the actual outcome.
Variance is defined as a model's sensitivity to small changes in the training set. A model
with low bias and variance is preferable, but it is difficult to achieve in practice.
Ensemble methods can help to reduce prediction variance. For example, the spread of
performance can be contained by averaging the performance of various models. The bagging
technique helps to reduce variance.
Types of Ensemble Learning Methods
Below are the advanced types of ensemble learning methods :
Types of Ensemble Learning
1. Bagging

Bagging, short for bootstrap aggregation, is primarily used in classification and regression. It
improves model accuracy by using decision trees, which greatly reduces variance. The elimination
of overfitting, which is a challenge for many predictive models, increases accuracy by reducing
variance.
There are two types of bagging: bootstrapping and aggregation.
Bootstrapping is a sampling technique that uses the replacement procedure to obtain samples
from the entire population (set).
In bagging, aggregation is used to incorporate all possible outcomes of the prediction and
randomize the outcome. As a result, the aggregation is based on probability bootstrapping
procedures or on all predictive model outcomes.

2. Boosting

Boosting is an ensemble technique that learns from previous predictor errors in order to make
better predictions in the future. The technique combines several weak base learners to form one
strong learner, significantly improving model predictability.
 Boosting works by arranging weak learners in a sequence so that weak learners can learn from
the next learner in the sequence to create better predictive models.
Gradient boosting, adaptive boosting (AdaBoost), and XGBoost are all types of boosting
(Extreme Gradient Boosting).
AdaBoost employs weak learners in the form of decision trees, which typically include only one
split, also known as decision stumps. New predictors are fitted to compensate for errors in
previous predictors. The gradient of descent aids the gradient booster in identifying and correcting
errors in learners' predictions.
XGBoost improves speed and performance by utilizing decision trees with boosted gradients. It
heavily relies on the target model's computational speed and performance. Model training should
be done in a specific order, making gradient boosted machine implementation time-consuming.

3. Stacking

Another ensemble method is stacking, which is also known as stacked generalization. This
technique works by allowing a training algorithm to combine the predictions of several other
similar learning algorithms.
Stacking has been used successfully in regression, density estimation, distance learning, and
classification. It can also be used to calculate the error rate during bagging.

Basic Types of Ensemble Learning

Below are the basic Types of Ensemble Learning :
1. Averaging
This is the simplest ensemble technique to define because it simply averages all of the models'
predictions. The final prediction is made using the average of the predictions.
This method is applicable to classification and regression problems. Consider the previous
section's scenario, where we want to forecast car prices. Our models forecast $ 6000, 7000, 5000,
and 6000.
Since we’re averaging, the final prediction is:
6000+7000+5000+6000 ÷ 4=6000
2. Max Voting Classifier
This method is similar to the average, but it is best suited for classification problems. Its
explanation is straightforward.
Several models make predictions known as "votes"; each prediction counts as one vote. As is
often the case with voting, decisions are frequently made in favor of the majority of votes. The
same is true here. The majority of classifiers' predictions become the final prediction.
For example, we try to forecast car prices. The models involved yield prices (in dollars) such as
6000, 5000, 8000, 5000, 6000, 5000, 7000, 5000, and 7000. We notice that the majority of the
models forecast $5000. 5000 is the final prediction using maximum voting.
3. Weighted Averaging
This method is a variant of the averaging method. The difference is that, whereas averaging gives
equal weight to all models, a weighted average gives more weight to a model with greater
predictive power. This significance is represented by weights. These weights can be represented
as decimals with a total of one.

Also Read | What is AdaBoost Algorithm in Ensemble Learning?

Challenges of Ensemble Learning
While ensemble learning is a powerful tool, and as fascinating as it can look, it still does have
some drawbacks. So below are some of the challenges faced by ensemble learning.
 When you use an ensemble, you must devote more time and resources to training your machine
learning models. A random forest with 500 trees, for example, produces much better results than a
single decision tree, but it also takes much longer to train. Running ensemble models can also be
difficult if the algorithms you use require a large amount of memory.
Explainability is another issue with ensemble learning. While adding new models to an ensemble
can improve its overall accuracy, it makes it more difficult to investigate the decisions of the AI
algorithm.
A single machine learning model, such as a decision tree, is simple to trace. However, when
hundreds of models contribute to an output, it becomes much more difficult to understand the
logic behind each decision.
Ensemble, like most things in machine learning, is one of many tools available to you for solving
complex problems. It can help you get out of difficult situations, but it is not a panacea. Make
good use of it.
Ensemble Learning Applications
Ensemble learning is a fairly common deep learning strategy that has been used to solve a wide
range of problems. It has aided in the completion of complex pattern recognition tasks that require
computers to learn high-level semantic information from digital images or videos, such as object
detection, which requires the formation of bounding boxes around the objects of interest, and
image classification.
Here are some examples of Ensemble Learning in action.

1. Remote Sensing
Monitoring the physical characteristics of a target area without coming into physical contact,
known as remote sensing, is a difficult task because the data acquired by different sensors has
varying resolutions, resulting in data distribution incoherence.
Ensemble Learning has also been used to perform tasks such as landslide detection and scene
classification.

2. Emotion recognition in Speech

Ensemble learning is also used in speech emotion recognition, particularly in multilingual
environments. Instead of selecting one classifier and compromising the accuracy of a specific
language corpus, the technique allows for the combining of all classifiers' effects.

3. Detection of diseases
Ensemble learning has aided in disease classification and localization for simplified and rapid
prognosis, such as in cardiovascular disease detection from X-Ray and CT scans.

4. Detection of Fraud
Because very fine precision is required to automate the process, detecting digital fraud is an
important and difficult task. Ensemble Learning has been shown to be effective in detecting credit
card and impression fraud.
Ensemble learning is a common machine learning technique that involves combining the
predictions of multiple experts (classifiers). The need for ensemble learning arises in a variety of
problematic situations that can be both data-centric and algorithm-centric, such as data
scarcity/excess, problem complexity, computational resource constraints, and so on.
Several methods developed over the years have proven useful in addressing a variety of such
issues. Nonetheless, researchers are developing newer ensemble approaches that address the
shortcomings of traditional ensembles.
  Discreditization: Discreditiization pools data into smaller intervals. It’s somewhat similar
to binning, but usually happens after data has been cleaned. For example, when calculating
average daily exercise, rather than using the exact minutes and seconds, you could join
together data to fall into 0-15 minutes, 15-30, etc.
 Concept hierarchy generation: Concept hierarchy generation can add a hierarchy within
and between your features that wasn’t present in the original data. If your analysis contains
wolves and coyotes, for example, you could add the hierarchy for their genus: canis.
4. Data reduction
The more data you’re working with, the harder it will be to analyze, even after cleaning and
transforming it. Depending on your task at hand, you may actually have more data than you need.
Especially when working with text analysis, much of regular human speech is superfluous or
irrelevant to the needs of the researcher. Data reduction not only makes the analysis easier and
more accurate, but cuts down on data storage.
It will also help identify the most important features to the process at hand.
 Attribute selection: Similar to discreditization, attribute selection can fit your data into
smaller pools. It, essentially, combines tags or features, so that tags
like male/female and professor could be combined into male professor/female professor.
 Numerosity reduction: This will help with data storage and transmission. You can use a
regression model, for example, to use only the data and variables that are relevant to your
analysis.
 Dimensionality reduction: This, again, reduces the amount of data used to help facilitate
analysis and downstream processes. Algorithms like K-nearest neighbors use pattern
recognition to combine similar data and make it more manageable.
Data Preprocessing Examples
Take a look at the table below to see how preprocessing works. In this example, we have three
variables: name, age, and company. In the first example we can tell that #2 and #3 have been
assigned the incorrect companies.
Name AgeCompany
Karen Lynch57 CVS Health
Elon Musk 49 Amazon
Jeff Bezos 57 Tesla
Tim Cook 60 Apple
We can use data cleaning to simply remove these rows, as we know the data was improperly
entered or is otherwise corrupted.
Name AgeCompany
Karen Lynch57 CVS Health
Tim Cook 60 Apple
Or, we can perform data transformation, in this case, manually, in order to fix the problem:
Name AgeCompany
Karen Lynch57 CVS Health
Elon Musk 49 Tesla
Name AgeCompany
Jeff Bezos 57 Amazon
Tim Cook 60 Apple
Once the issue is fixed, we can perform data reduction, in this case by descending age, to choose
which age range we want to focus on:
Name AgeCompany
Tim Cook 60 Apple
Karen Lynch57 CVS Health
Jeff Bezos 57 Amazon
Elon Musk 49 Tesla

Introduction
In this tutorial, we discuss bagging in machine learning. Bagging, or bootstrap aggregation, is the
ensemble getting-to-know method generally used to lessen variance within a loud dataset. In
Bagging, a random pattern of statistics in this study set is selected with replacement, meaning that
the character statistics factors may be chosen more soon as possible. After numerous facts samples
are generated, those susceptible fashions are trained independently. For example, the common or
Majority of these predictions yield a correct estimate depending on the sort of task- regression or
type. As a note, the random woodland set of rules is considered an extension of the bagging
approach, using both bagging and function randomness to create an uncorrelated wooded area of
selection trees.
The Bagging is an assembling approach that tries to resolve overfitting for class or the regression
problems. Bagging pursuits to improve the accuracy and overall performance of gadget mastering
algorithms. It does this by taking random subsets of an original dataset, with substitute, and fits
either a classifier (for classification) or regressor (for regression) to each subset. Bagging is also
known as Bootstrap aggregating. It is an ensemble learning approach that enhances the overall
performance and accuracy of the gadget for learning algorithms. It is miles used to address bias-
variance alternate-off increases and decreases the variance of a prediction version. The Bagging
avoids overfitting of data and is used for each regression and classification of the class, in
particular for the decision tree algorithms.
What is Ensemble Learning?
Ensemble learning gives us credence to the idea of the "wisdom of crowds," it suggests that the
choice-making for a more extensive organization of humans is usually higher than that of an
individual professional. Another side, ensemble learning refers to a collection (or ensemble) of
base newbies or fashions, which are paintings collectively to attain a better very last of the
prediction. A single model, also called a base or susceptible learner, may not perform well due to
high variance or bias. But, while vulnerable learners are aggregated, they could shape a sturdy
learner, as their combination reduces bias or variance, yielding higher model performance.
Ensemble learning is a widely used and desired tool learning technique in which more than one
person models, often referred to as base models, are blended to produce a powerful ideal of the
prediction version. An example of ensemble learning is the Random Forest algorithm.
Ensemble learning is frequently illustrated using selection timber as this algorithm may be liable
to overfitting (excessive variance and low bias) when it has not been pruned. It could additionally
lend itself to underfitting (low variance and extreme bias) when it is very small, like a decision
stump, a decision tree with one stage. While an algorithm overfits or fits its education set, it
cannot generalize nicely to new datasets, so ensemble strategies are used to counteract this
conduct to allow for the generalization of the model to new datasets. While selection timber can
showcase excessive variance or high bias, it is worth noting that it is not the best modelling
approach that leverages ensemble learning to find the "sweet spot" in the bias-variance trade-off.
Backward Skip 10sPlay VideoForward Skip 10s
What is the difference between Bagging and Boosting?
There are some differences between Bagging and boosting. These are two principal forms of
ensemble studying strategies. The main difference between these two learning strategies is the
way they are skilled. In the bagging technique, it is vulnerable newcomers trained in parallel. But
in the boosting, they are trained sequentially. This means that a sequence of fashions is
constructed, and with each new version generation, the weights of the misclassified information in
the preceding version are improved. This redistribution of weights enables the algorithm to
perceive the parameters it wishes to the consciousness of to enhance its performance. AdaBoost,
which stands for "adaptative boosting set of rules," is onemost maximum famous boosting
algorithmbecamened into one of the first of its kind. Different varieties of boosting algorithms
consist of XGBoost, GradientBoost, and BrownBoost.
Another difference between Bagging and boosting is the scenarios wherein they may be used. For
example, bagging strategies or techniques are usually used on susceptible novices, mainly
showcasing excessive variance and occasional bias. But the boosting plans are leveraged while
low friction and high tendency are located.
Difference between bagging and boosting are:
Bagging Boosting
The most effective manner of mixing A manner of mixing
predictions that belong to the same type. predictions that belong to
different sorts.
The main task of it is decrease the The main task of it is
variance but not bias. decrease the bias but not
variance.
Here each of the model is different Here each of the model is
weight. same weight.
Each of the model is built here Each of the model is built
independently. here dependently.
This training records subsets are decided Each new subset consists of
on using row sampling with alternative the factors that were
and random sampling techniques from misclassified through
the whole training dataset. preceding models.
It is trying to solve by over fitting It is trying to solve by
problem. reducing the bias.
If the classifier is volatile (excessive If the classifier is stable and
variance), then apply bagging. easy (excessive bias) the
practice boosting.
In the bagging base, the classifier is In the boosting base, the
works parallelly. classifier is works
sequentially.
Example is random forest model by Example is AdaBoost using
using bagging. the boosting technique.
What are the similarities between Bagging and Boosting?
The similarities between Bagging and boosting are the commonly used strategies with a general
similarity of being labelled as ensemble strategies. Now here we will briefly explain the
similarities between Bagging and boosting.
1. They both are ensemble techniques to get the N novices from 1 learner.
2. Each generates numerous training statistics sets through random sampling.
3. They each make the very last decision by averaging the N number of beginners (or they
take most of the people of them, i.e., the Majority of voting).
4. The Bagging and boosting are exact at reducing the variance and offer better stability.
Describe the Bagging Technique:
Assume the set D of d tuples, at each iteration I, a schooling set Di of d tuples is selected thru row
sampling with a substitute approach (i.e., there may be repetitive factors from distinct d tuples)
from D (i.e., bootstrap). Then a classifier version Mi is discovered for each training set D < i.
every classifier Mi returns its class prediction. The bagged classifier M* counts the votes and
assigns the class with the most votes to X (unknown pattern).
What are the Implementation Steps of Bagging?
o Step 1: Multiple subsets are made from the original information set with identical tuples,
deciding on observations with replacement.
o Step 2: A base model is created on all subsets.
o Step 3: Every version is found in parallel with each training set and unbiased.
o Step 4: The very last predictions are determined by combining the forecasts from all
models.
Application of the Bagging:
There are various applications of Bagging, which are given below -
1. IT:
Bagging can also improve the precision and accuracy of IT structures, together with network
intrusion detection structures. In the meantime, this study seems at how Bagging can enhance the
accuracy of network intrusion detection and reduce the rates of fake positives.
2. Environment:
Ensemble techniques, together with Bagging, were carried out inside the area of far-flung sensing.
This study indicates how it has been used to map the styles of wetlands inside a coastal landscape.
3. Finance:
Bagging has also been leveraged with deep gaining knowledge of models within the finance
enterprise, automating essential tasks, along with fraud detection, credit risk reviews, and option
pricing issues. This research demonstrates how Bagging amongst different device studying
techniques was leveraged to assess mortgage default hazard. This highlights how Bagging limits
threats by saving you from credit score card fraud within the banking and economic institutions.
4. Healthcare:
The Bagging has been used to shape scientific data predictions. These studies (PDF, 2.8 MB)
show that ensemble techniques had been used for various bioinformatics issues, including gene
and protein selection, to perceive a selected trait of interest. More significantly, this study mainly
delves into its use to expect the onset of diabetes based on various threat predictors.
What are the Advantages and Disadvantages of Bagging?
Advantages of Bagging are -
There are many advantages of Bagging. The benefit of Bagging is given below -
1. Easier for implementation:
Python libraries, including scikit-examine (sklearn), make it easy to mix the predictions of base
beginners or estimators to enhance model performance. Their documentation outlines the
available modules you can leverage for your model optimization.
2. Variance reduction:
The Bagging can reduce the variance inside a getting to know set of rules which is especially
helpful with excessive-dimensional facts, where missing values can result in better conflict,
making it more liable to overfitting and stopping correct generalization to new datasets.
Disadvantages of Bagging are -
There are many disadvantages of Bagging. The disadvantages of Bagging are given below -
1. Flexible less:
As a method, Bagging works particularly correctly with algorithms that are much less solid. One
which can be more stable or a problem with high amounts of bias does now not provide an awful
lot of gain as there is less variation in the dataset of the version. As noted within the hands-On
guide for machine learning, "the bagging is a linear regression version will efficaciously just
return the original predictions for huge enough b."
2. Loss of interpretability:
The Bagging slows down and grows extra in depth because of the quantity of iterations growth.
accordingly, it is no longer adequately suitable for actual-time applications. Clustered structures
or large processing cores are perfect for quickly growing bagged ensembles on massive look-at
units.
3. Expensive for computation:
The Bagging is tough to draw unique business insights via Bagging because of the averaging
concerned throughout predictions. While the output is more precise than any person's information
point, a more accurate or whole dataset may yield greater precision within a single classification
or regression model.

Random Forest Algorithm

Random Forest is a popular machine learning algorithm that belongs to the supervised learning
technique. It can be used for both Classification and Regression problems in ML. It is based on
the concept of ensemble learning, which is a process of combining multiple classifiers to solve a
complex problem and to improve the performance of the model.
As the name suggests, "Random Forest is a classifier that contains a number of decision trees
on various subsets of the given dataset and takes the average to improve the predictive accuracy
of that dataset." Instead of relying on one decision tree, the random forest takes the prediction
from each tree and based on the majority votes of predictions, and it predicts the final output.
The greater number of trees in the forest leads to higher accuracy and prevents the problem
of overfitting.
The below diagram explains the working of the Random Forest algorithm:

Note: To better understand the Random Forest Algorithm, you should have knowledge of the
Decision Tree Algorithm.
Assumptions for Random Forest
Since the random forest combines multiple trees to predict the class of the dataset, it is possible
that some decision trees may predict the correct output, while others may not. But together, all the
trees predict the correct output. Therefore, below are two assumptions for a better Random forest
classifier:
o There should be some actual values in the feature variable of the dataset so that the
classifier can predict accurate results rather than a guessed result.
o The predictions from each tree must have very low correlations.
Why use Random Forest?
Below are some points that explain why we should use the Random Forest algorithm:
<="" li="">
o It takes less training time as compared to other algorithms.
o It predicts output with high accuracy, even for the large dataset it runs efficiently.
o It can also maintain accuracy when a large proportion of data is missing.
How does Random Forest algorithm work?
Random Forest works in two-phase first is to create the random forest by combining N decision
tree, and second is to make predictions for each tree created in the first phase.
The Working process can be explained in the below steps and diagram:
Step-1: Select random K data points from the training set.
Step-2: Build the decision trees associated with the selected data points (Subsets).
Step-3: Choose the number N for decision trees that you want to build.
Step-4: Repeat Step 1 & 2.
Step-5: For new data points, find the predictions of each decision tree, and assign the new data
points to the category that wins the majority votes.
The working of the algorithm can be better understood by the below example:
Example: Suppose there is a dataset that contains multiple fruit images. So, this dataset is given
to the Random forest classifier. The dataset is divided into subsets and given to each decision tree.
During the training phase, each decision tree produces a prediction result, and when a new data
point occurs, then based on the majority of results, the Random Forest classifier predicts the final
decision. Consider the below image:

Applications of Random Forest

There are mainly four sectors where Random forest mostly used:
1. Banking: Banking sector mostly uses this algorithm for the identification of loan risk.
2. Medicine: With the help of this algorithm, disease trends and risks of the disease can be
identified.
3. Land Use: We can identify the areas of similar land use by this algorithm.
 4. Marketing: Marketing trends can be identified using this algorithm.
Advantages of Random Forest
o Random Forest is capable of performing both Classification and Regression tasks.
o It is capable of handling large datasets with high dimensionality.
o It enhances the accuracy of the model and prevents the overfitting issue.
Disadvantages of Random Forest
o Although random forest can be used for both classification and regression tasks, it is not
more suitable for Regression tasks.

Extremely Randomized Trees

Extremely Randomized Trees, also known as Extra Trees, construct multiple trees like RF
algorithms during training time over the entire dataset. During training, the ET will construct
trees over every observation in the dataset but with different subsets of features.
It is important to note that although bootstrapping is not implemented in ET’s original structure,
we can add it in some implementations. Furthermore, when constructing each decision tree, the
ET algorithm splits nodes randomly.
4.1. Advantages and Disadvantages
The main advantage of Extra Trees is the reduction in bias. This is in terms of sampling from the
entire dataset during the construction of the trees. Different subsets of the data may introduce
different biases in the results obtained, hence Extra Trees prevents this by sampling the entire
dataset.
Another advantage of Extra Trees is that they reduce variance. This is a result of the randomized
splitting of nodes within the decision trees, hence the algorithm is not heavily influenced by
certain features or patterns in the dataset.
4.2. Applications
Similarly, we can apply Extra Trees to classification and regression tasks like Random Forests. In
some instances, Extra Trees are also used for feature selection. Here, an Extra Trees classifier is
used to pick features that matter the most.
5. Differences and Similarities
RFs and ETs are similar in that they both construct multiple decision trees to use for the task at
hand, whether classification or regression. However, subtle differences exist between the two.
4. Discuss the concept of gradient boosting and its use in ensemble models.
Ans:
Gradient Boosting is a popular boosting algorithm in machine learning used for classification
and regression tasks. Boosting is one kind of ensemble Learning method which trains the model
sequentially and each new model tries to correct the previous model. It combines several weak
learners into strong learners. There is two most popular boosting algorithm i.e
1. AdaBoost
2. Gradient Boosting
Gradient Boosting
Gradient Boosting is a powerful boosting algorithm that combines several weak learners into
strong learners, in which each new model is trained to minimize the loss function such as mean
squared error or cross-entropy of the previous model using gradient descent. In each iteration, the
algorithm computes the gradient of the loss function with respect to the predictions of the current
ensemble and then trains a new weak model to minimize this gradient. The predictions of the new
model are then added to the ensemble, and the process is repeated until a stopping criterion is met.
In contrast to AdaBoost, the weights of the training instances are not tweaked, instead, each
predictor is trained using the residual errors of the predecessor as labels. There is a technique
called the Gradient Boosted Trees whose base learner is CART (Classification and Regression
Trees). The below diagram explains how gradient-boosted trees are trained for regression
problems.

Gradient Boosted Trees for Regression

The ensemble consists of M trees. Tree1 is trained using the feature matrix X and the labels y. The
predictions labeled y1(hat) are used to determine the training set residual errors r1. Tree2 is then
trained using the feature matrix X and the residual errors r1 of Tree1 as labels. The predicted
results r1(hat) are then used to determine the residual r2. The process is repeated until all the M
trees forming the ensemble are trained. There is an important parameter used in this technique
known as Shrinkage. Shrinkage refers to the fact that the prediction of each tree in the ensemble
is shrunk after it is multiplied by the learning rate (eta) which ranges between 0 to 1. There is a
trade-off between eta and the number of estimators, decreasing learning rate needs to be
compensated with increasing estimators in order to reach certain model performance. Since all
trees are trained now, predictions can be made. Each tree predicts a label and the final prediction
is given by the formula,
y(pred) = y1 + (eta * r1) + (eta * r2) + ....... + (eta * rN)
Difference between Adaboost and Gradient Boosting
The difference between AdaBoost and gradient boosting are as follows:
AdaBoost Gradient Boosting

During each iteration in AdaBoost, the weights of Gradient Boosting updates the weights by
incorrectly classified samples are increased, so that computing the negative gradient of the loss
the next weak learner focuses more on these function with respect to the predicted
samples. output.

Gradient Boosting can use a wide range of

AdaBoost uses simple decision trees with one split
base learners, such as decision trees, and
known as the decision stumps of weak learners.
linear models.

Gradient Boosting is generally more robust,

AdaBoost is more susceptible to noise and outliers
as it updates the weights based on the
in the data, as it assigns high weights to
gradients, which are less sensitive to
misclassified samples
outliers.
Gradient Boosting Algorithm
Step 1:
Let’s assume X, and Y are the input and target having N samples. Our goal is to learn the function
f(x) that maps the input features X to the target variables y. It is boosted trees i.e the sum of trees.
The loss function is the difference between the actual and the predicted variables.
L(f)= \sum ^{N}_{i=1}L(y_i,f(x_i))
Step 2: We want to minimize the loss function L(f) with respect to f.
f_0 (x) = arg\underset{f}{min} \;L(f) = arg\underset{f}{min} \sum ^{N}_{i=1}L(y_i,f(x_i))
If our gradient boosting algorithm is in M stages then To improve the the algorithm can
add some new estimator as hm having 1\le m \le M
y_i = F_{m+1}(x_i) = F_m(x_i) + h_m(x_i)
Step 3: Steepest Descent
For M stage gradient boosting, The steepest Descent finds h_m = -\rho_m where is constant and
known as step length and is the gradient of loss function L(f)
g_{im} =-\left[\frac{\partial L(y_i,f(x_i))}{\partial f(x_i)} \right]_{f(x_i)=f_{m-1}(x_i)}
Step 4: Solution
The gradient Similarly for M trees:
f_m (x) = f_{m-1} (x) + \left(\underset{h_{m}\epsilon H}{argmin} \left[ \sum
^{N}_{i=1}L(y_i,f_{m-1}(x_i)+h_m(x_i)) \right]\right)(x)
The current solution will be
f_m = f_{m-1}-\rho_m g_m

Regularization in Machine Learning

What is Regularization?
Regularization is one of the most important concepts of machine learning. It is a technique to
prevent the model from overfitting by adding extra information to it.
Sometimes the machine learning model performs well with the training data but does not perform
well with the test data. It means the model is not able to predict the output when deals with unseen
data by introducing noise in the output, and hence the model is called overfitted. This problem can
be deal with the help of a regularization technique.
This technique can be used in such a way that it will allow to maintain all variables or features in
the model by reducing the magnitude of the variables. Hence, it maintains accuracy as well as a
generalization of the model.
It mainly regularizes or reduces the coefficient of features toward zero. In simple words, "In
regularization technique, we reduce the magnitude of the features by keeping the same number of
features."
How does Regularization Work?
Regularization works by adding a penalty or complexity term to the complex model. Let's
consider the simple linear regression equation:
y= β0+β1x1+β2x2+β3x3+⋯+βnxn +b
In the above equation, Y represents the value to be predicted
X1, X2, …Xn are the features for Y.
β0,β1,…..βn are the weights or magnitude attached to the features, respectively. Here represents
the bias of the model, and b represents the intercept.
Linear regression models try to optimize the β0 and b to minimize the cost function. The equation
for the cost function for the linear model is given below:

Now, we will add a loss function and optimize parameter to make the model that can predict the
accurate value of Y. The loss function for the linear regression is called as RSS or Residual sum
of squares.
Techniques of Regularization
There are mainly two types of regularization techniques, which are given below:
o Ridge Regression
 o Lasso Regression
Ridge Regression
o Ridge regression is one of the types of linear regression in which a small amount of bias is
introduced so that we can get better long-term predictions.
o Ridge regression is a regularization technique, which is used to reduce the complexity of
the model. It is also called as L2 regularization.
o In this technique, the cost function is altered by adding the penalty term to it. The amount
of bias added to the model is called Ridge Regression penalty. We can calculate it by
multiplying with the lambda to the squared weight of each individual feature.
o The equation for the cost function in ridge regression will be:

o In the above equation, the penalty term regularizes the coefficients of the model, and
hence ridge regression reduces the amplitudes of the coefficients that decreases the
complexity of the model.
o As we can see from the above equation, if the values of λ tend to zero, the equation
becomes the cost function of the linear regression model. Hence, for the minimum
value of λ, the model will resemble the linear regression model.
o A general linear or polynomial regression will fail if there is high collinearity between the
independent variables, so to solve such problems, Ridge regression can be used.
o It helps to solve the problems if we have more parameters than samples.
Lasso Regression:
o Lasso regression is another regularization technique to reduce the complexity of the
model. It stands for Least Absolute and Selection Operator.
o It is similar to the Ridge Regression except that the penalty term contains only the absolute
weights instead of a square of weights.
o Since it takes absolute values, hence, it can shrink the slope to 0, whereas Ridge
Regression can only shrink it near to 0.
o It is also called as L1 regularization. The equation for the cost function of Lasso
regression will be:

o Some of the features in this technique are completely neglected for model evaluation.
o Hence, the Lasso regression can help us to reduce the overfitting in the model as well as
the feature selection.
Key Difference between Ridge Regression and Lasso Regression
o Ridge regression is mostly used to reduce the overfitting in the model, and it includes all
the features present in the model. It reduces the complexity of the model by shrinking the
coefficients.
o Lasso regression helps to reduce the overfitting in the model as well as feature selection.

XGBoost stands for “Extreme Gradient Boosting”. XGBoost is an optimized distributed gradient
boosting library designed to be highly efficient, flexible and portable. It implements Machine
Learning algorithms under the Gradient Boosting framework. It provides a parallel tree boosting
to solve many data science problems in a fast and accurate way.
Boosting
Boosting is an ensemble learning technique to build a strong classifier from several weak
classifiers in series. Boosting algorithms play a crucial role in dealing with bias-variance trade-off.
Unlike bagging algorithms, which only controls for high variance in a model, boosting controls
both the aspects (bias & variance) and is considered to be more effective.
Below are the few types of boosting algorithms:
1. AdaBoost (Adaptive Boosting)
2. Gradient Boosting
3. XGBoost
4. CatBoost
5. Light GBM
XGBoost
XGBoost stands for eXtreme Gradient Boosting. It became popular in the recent days and is
dominating applied machine learning and Kaggle competition for structured data because of its
scalability.
XGBoost is an extension to gradient boosted decision trees (GBM) and specially designed to
improve speed and performance.
AdaBoost
AdaBoost is short for Adaptive Boosting. AdaBoost was the first successful boosting algorithm
developed for binary classification. Also, it is the best starting point for understanding boosting
algorithms. It is adaptive in the sense that subsequent classifiers built are tweaked in favour of
those instances misclassified by previous classifiers. It is sensitive to noisy data and outliers.
AdaBoost uses multiple iterations to generate a single composite strong learner. It creates a strong
learner by iteratively adding weak learners. During each phase of training, a new weak learner is
added to the ensemble, and a weighting vector is adjusted to focus on examples that were
misclassified in previous rounds. The result is a classifier that has higher accuracy than the weak
learner classifiers.
Gradient Boosting
Gradient boosting is one of the most powerful techniques for building predictive models, and it is
called a Generalization of AdaBoost. The main objective of Gradient Boost is to minimize the loss
function by adding weak learners using a gradient descent optimization algorithm. The
generalization allowed arbitrary differentiable loss functions to be used, expanding the technique
beyond binary classification problems to support regression, multi-class classification and more.
Gradient Boost has three main components.
 Loss Function: The role of the loss function is to estimate how best is the model in
making predictions with the given data. This could vary depending on the type of the
problem.
 Weak Learner: Weak learner is one that classifies the data so poorly when compared to
random guessing. The weak learners are mostly decision trees, but other models can be
used in GBM.
 Additive Model: It is an iterative and sequential process in adding the decision trees one
step at a time. Each iteration should reduce the value of loss function. A fixed number of
trees are added, or training stops once loss reaches an acceptable level or no longer
improves on an external validation dataset.
Improvements to Gradient Boosting
Gradient boosting is a greedy algorithm and can overfit a training dataset quickly. So
regularization methods are used to improve the performance of the algorithm by reducing
overfitting.
 Subsampling: This is the simplest form of regularization method introduced for GBM’s.
This improves the generalization properties of the model and reduces the computation
efforts. Subsampling introduces randomness into the fitting procedure. At each learning
 iteration, only a random part of the training data is used to fit a consecutive base-learner.
The training data is sampled without replacement.
 Shrinkage: Shrinkage is commonly used in ridge regression where it shrinks regression
coefficients to zero and, thus, reduces the impact of potentially unstable regression
coefficients. In GBM’s, shrinkage is used for reducing the impact of each additionally
fitted base-learner. It reduces the size of incremental steps and thus penalizes the
importance of each consecutive iteration. The intuition behind this technique is that it is
better to improve a model by taking many small steps than by taking fewer large steps. If
one of the boosting iterations turns out to be erroneous, its negative impact can be
corrected easily in subsequent steps.
 Early Stopping: One important practical consideration that can be derived from Decision
Tree is that early stopping or tree pruning. This means that if the ensemble was trimmed
by the number of trees, corresponding to the validation set minima on the error curve, the
overfitting would be circumvented at the minimal accuracy expense. Another observation
is that the optimal number of boosts, at which the early stopping is considered, varies
concerning the shrinkage parameter λ. Therefore, a trade-off between the number of boosts
and λ should be considered.
XGBoost Features
 Regularized Learning: Regularization term helps to smooth the final learnt weights to
avoid over-fitting. The regularized objective will tend to select a model employing simple
and predictive functions.
 Gradient Tree Boosting: The tree ensemble model cannot be optimized using traditional
optimization methods in Euclidean space. Instead, the model is trained in an additive
manner.
 Shrinkage and Column Subsampling: Besides the regularized objective, two additional
techniques are used to further prevent overfitting. The first technique is shrinkage
introduced by Friedman. Shrinkage scales newly added weights by a factor η after each
step of tree boosting. Similar to a learning rate in stochastic optimization, shrinkage
reduces the influence of each tree and leaves space for future trees to improve the model.
The second technique is the column (feature) subsampling. This technique is used in Random
Forest. Column sub-sampling prevents over-fitting even more so than the traditional row sub-
sampling. The usage of column sub-samples also speeds up computations of the parallel
algorithm.
SPLITTING ALGORITHMS
 Exact Greedy Algorithm: The main problem in tree learning is to find the best split. This
algorithm enumerates over all the possible splits on all the features. It is computationally
demanding to enumerate all the possible splits for continuous features.
 Approximate Algorithm: The exact greedy algorithm is very powerful since it
enumerates over all possible splitting points greedily. However, it is impossible to
efficiently do so when the data does not fit entirely into memory. Approximate Algorithm
proposes candidate splitting points according to percentiles of feature distribution. The
algorithm then maps the continuous features into buckets split by these candidate points,
aggregates the statistics and finds the best solution among proposals based on the
aggregated statistics.
 Weighted Quantile Sketch: One important step in the approximate algorithm is to
propose candidate split points. XGBoost has a distributed weighted quantile sketch
algorithm to effectively handle weighted data.
 Sparsity-aware Split Finding: In many real-world problems, it is quite common for the
input x to be sparse. There are multiple possible causes for sparsity:
1. Presence of missing values in the data
 2. Frequent zero entries in the statistics
3. Artifacts of feature engineering such as one-hot encoding
It is important to make the algorithm aware of the sparsity pattern in the data. XGBoost handles
all sparsity patterns in a unified way.
Also Read: What is Cross-Validation in ML?
System Features
 Parallelization of tree construction using all of your CPU cores during training. Collecting
statistics for each column can be parallelized, giving us a parallel algorithm for split
finding.
 Cache-aware Access: XGBoost has been designed to make optimal use of hardware. This
is done by allocating internal buffers in each thread, where the gradient statistics can be
stored.
 Blocks for Out-of-core Computation for very large datasets that don’t fit into memory.
 Distributed Computing for training very large models using a cluster of machines.
 Column Block for Parallel Learning. The most time-consuming part of tree learning is to
get the data into sorted order. In order to reduce the cost of sorting, the data is stored in the
column blocks in sorted order in compressed format.
Goals of XGBoost
 Execution Speed: XGBoost was almost always faster than the other benchmarked
implementations from R, Python Spark and H2O and it is really faster when compared to
the other algorithms.
 Model Performance: XGBoost dominates structured or tabular datasets on classification
and regression predictive modelling problems.

Gradient
Algorithms Boosting AdaBoost XGBoost CatBoost LightGBM

Year – 1995 2014 2017 2017

May
require Automatica
preprocessi lly handles
No NO No
Handling ng like categorical
Categorical one-hot variables
Variables encoding

Speed/
Moderate Fast Fast Moderate Fast
Scalability

Memory
Moderate Low Moderate High Low
Usage

Regularizatio
NO No Yes Yes Yes
n

Parallel
No No Yes Yes Yes
Processing

GPU Support No No Yes Yes Yes

 Gradient
Algorithms Boosting AdaBoost XGBoost CatBoost LightGBM

Feature
Available Available Available Available Available
Importance

In this article, we will learn about the AdaBoost classifier and its practical implementation over a
[Link] algorithm falls under ensemble boosting techniques, as we will discuss it
combines multiple models to produce more accurate results and this is done in two phases:
1. Multiple weak learners are allowed to learn on training data
2. Combining these models to generate a meta-model, this meta-model aims to resolve the
errors as predicted by the individual weak learners.
Note: For more information, refer Boosting ensemble models
What is AdaBoost
AdaBoost short for Adaptive Boosting is an ensemble learning used in machine learning for
classification and regression problems. The main idea behind AdaBoost is to iteratively train the
weak classifier on the training dataset with each successive classifier giving more weightage to
the data points that are misclassified. The final AdaBoost model is decided by combining all the
weak classifier that has been used for training with the weightage given to the models according
to their accuracies. The weak model which has the highest accuracy is given the highest
weightage while the model which has the lowest accuracy is given a lower weightage.
Institution Behind AdaBost Algorithm
AdaBoost techniques combine many weak machine-learning models to create a powerful
classification model for the output. The steps to build and combine these models are as
Step1 – Initialize the weights
 For a dataset with N training data points instances, initialize N W_{i} weights for each
data point with W_{i} = \frac{1}{N}
Step2 – Train weak classifiers
 Train a weak classifier Mk where k is the current iteration
 The weak classifier we are training should have an accuracy greater than 0.5 which means
it should be performing better than a naive guess
Step3 – Calculate the error rate and importance of each weak model Mk
 Calculate rate error_rate for every weak classifier Mk on the training dataset
 Calculate the importance of each model α_k using formula \alpha_k = \frac{1}{2} \ln{\
frac{1 – error_k}{error_k}}
Step4 – Update data point weight for each data point Wi
 After applying the weak classifier model to the training data we will update the weight
assigned to the points using the accuracy of the model. The formula for updating the
weights will be w_i = w_i \exp{(-\alpha_k y_i M_k(x_i))} . Here yi is the true output and
Xi is the corresponding input vector
Step5 – Normalize the Instance weight
 We will normalize the instance weight so that they can be summed up to 1 using the
formula W_i = W_i / sum(W)
Step6 – Repeat steps 2-5 for K iterations
 We will train K classifiers and will calculate model importance and update the instance
weights using the above formula
 The final model M(X) will be an ensemble model which is obtained by combining these
weak models weighted by their model weights
Boosting Algorithms

Overfitting and Regularization

Overfitting of the model occurs when the model learns just ‘too-well’ on the train data. This
would sound like an advantage but it is not. When a model is overtrained on training data, it
performs worst on the test data or any new data provided. Technically, the model learns the details
as well as the noise of the train data. This would hinder the performance of any new data provided
to the model as the learned details and noise cannot be applied to the new data. This is the case
when we say the performance of the model is not adequate. There are several ways of avoiding
the overfitting of the model such as K-fold cross-validation, resampling, reducing the number of
features, etc. One of the ways is to apply Regularization to the model. Regularization is a better
technique than Reducing the number of features to overcome the overfitting problem as in
Regularization we do not discard the features of the model.
Regularization is a technique that penalizes the coefficient. In an overfit model, the coefficients
are generally inflated. Thus, Regularization adds penalties to the parameters and avoids them
weigh heavily. The coefficients are added to the cost function of the linear equation. Thus, if the
coefficient inflates, the cost function will increase. And Linear regression model will try to
optimize the coefficient in order to minimize the cost function.

Practically, you can check if the regression model is overfitting or not by RMSE. A good model
has a similar RMSE for the train and test sets. If the difference is too large, we can say the model
is overfitting to the training set. There are two kinds of techniques for adding penalities to the cost
function, L1 Norm or LASSO term and L2 Norm or Ridge Term.
L1 Regularization or LASSO
L1 Regularization technique is also known as LASSO or Least Absolute Shrinkage and Selection
Operator. In this, the penalty term added to the cost function is the summation of absolute values
of the coefficients. Since the absolute value of the coefficients is used, it can reduce the
coefficient to 0 and such features may completely get discarded in LASSO. Thus, we can say,
LASSO helps in Regularization as well as Feature Selection.
Following is the equation of Cost function with L1 penalty term:
Cost Function after adding L1 Penalty (Source – Personal Computer)
Here, alpha is the multiplier term.
L2 Regularization or Ridge
L2 Regularization technique is also known as Ridge. In this, the penalty term added to the cost
function is the summation of the squared value of coefficients. Unlike the LASSO term, the Ridge
term uses squared values of the coefficient and can reduce the coefficient value near to 0 but not
exactly 0. Ridge distributes the coefficient value across all the features.
Following is the equation of Cost function with L2 penalty term:

Cost Function after adding L2 Penalty (Source – Personal Computer)