Week 7 Classification using
Ensemble Machine
Learning
Assessmen
ts
• Machine learning classification involves
predicting a discrete class label
• For example : disease vs. no disease,
fraud vs. legitimate transactions.
• While single models like decision trees,
logistic regression, or SVMs can perform
well, they often suffer from limitations
such as high variance, high bias, or
sensitivity to noise.
• Ensemble learning addresses these
issues by combining multiple models to
produce a more robust, accurate, and
stable classifier.
Ensemble learning is the process of
combining predictions from
multiple base models (called weak
What Is learners) to produce a strong learner.
Ensembl
e Why ensembles work
• Reduction in variance (e.g.,
Learning Random Forest)
? • Reduction in bias (e.g., Boosting)
• Reduction in noise
• Better generalization
[Link]
Advantages of Ensemble Disadvantages of Ensemble
Learning Learning
✔ Improved predictive ✖ Higher computational cost
performance ✖ Harder to interpret than simple
✔ Better generalization models
✔ Robust to noisy data ✖ Risk of overfitting if not tuned
✔ Can handle complex (especially boosting)
nonlinear patterns ✖ Requires careful hyperparameter
✔ Works well with tuning
imbalanced datasets
(boosting especially)
Types of Ensemble Techniques
Type 1 : Bagging (Bootstrap Aggregating)
Method: Train multiple models on different bootstrap samples and
average their predictions
Base learners: Usually high-variance models like decision trees
Key Example: Random Forest
• Builds many decision trees using:
• Bootstrapped data
• Random subsets of features
• Final prediction:
• Classification: Majority vote
• Regression: Average
Advantages:
• Handles high-dimensional data
• Robust to outliers and noise
• Reduces overfitting
Type 2 : Boosting
Method: Sequentially train weak learners, each focusing on correcting
mistakes of the previous ones
Popular Boosting Algorithms
1. AdaBoost
• Assigns higher weights to misclassified samples
• Final prediction is a weighted vote
2. Gradient Boosting Machines (GBM)
• Each new model fits the residual errors of the previous model
• Uses gradient descent to optimize loss
3. XGBoost : Enhanced GBM with Regularization, Faster training, Missing value
handling, Parallelism
4. LightGBM : (1) Uses leaf-wise tree growth, (2) Very efficient for large datasets
5. CatBoost : (1) Excellent for categorical data, (2) Avoids extensive preprocessing
Type 3 : Stacking (Stacked Generalization)
• Combines different models (heterogeneous learners)
• A meta-learner (e.g., logistic regression) learns to combine their predictions
• Architecture:
• Train base models (e.g., SVM, decision tree, NN)
• Use their outputs as features
• Train a meta-model on these outputs
• Strength: Learns optimal combination patterns, often outperforming
bagging/boosting.
[Link]
Random Forest
Method
• The random forest is a classification algorithm consisting of many decisions trees.
• It uses feature randomness when building each individual tree to try to create an
uncorrelated forest of trees whose prediction by committee is more accurate than that of
any individual tree.
A Decision Tree is a single, tree-like model that makes decisions by splitting data
based on feature values. It works by creating a set of rules in a hierarchical, flow-
chart-like structure. Each internal node represents a test on an attribute, each
branch represents an outcome of the test, and each leaf node represents a class
label or a predicted value.
RANDOM FOREST IS CREATED EACH DECISION TREE IS EACH TREE IS DEPENDENT ON CONSIDERING IT TO BE A IF ITS REGRESSION, THE
BY RANDOMLY SPLITTING THE FORMED USING FEATURE AN INDEPENDENT SAMPLE. CLASSIFICATION PROBLEM, AVERAGE OF ALL THE TREE'S
DATA. SELECTION INDICATORS LIKE THEN EACH TREE COMPUTES OUTPUTS IS DECLARED AS THE
INFORMATION GAIN, GAIN VOTES AND THE HIGHEST RESULT.
RATIO OF EACH FEATURE. VOTES CLASS IS CHOSEN.
[Link]
[Link]
Assumptions for Random Forest
• There should be some actual values in
the feature variable of the dataset so
that the classifier can predict accurate
results rather than a guessed result.
• The predictions from each tree must
have very low correlations.
Random Forest works in two-phases:
1) first is to create the random forest by combining N decision tree,
2) second is to make predictions for each tree created in the first phase.
The Working process can be explained in the below steps and diagram:
• Step-1: Select random K data points from the training set.
• Step-2: Build the decision trees associated with the selected data
points (Subsets).
• Step-3: Choose the number N for decision trees that you want to
build.
• Step-4: Repeat Step 1 & 2.
• Step-5: For new data points, find the predictions of each decision tree,
and assign the new data points to the category that wins the majority
[Link]
votes.
Four sectors where Random forest mostly used:
Banking: Medicine: Land Use: Marketing:
With the
Banking
help of this We can
sector mostly Marketing
algorithm, identify the
uses this trends can be
disease areas of
algorithm for identified
trends and similar land
the using this
risks of the use by this
identification algorithm.
disease can algorithm.
of loan risk.
be identified.
Applications The Random forest regression has a wide
range of real-world problems including:
of Random • Predicting continuous numerical
values: Predicting house prices, stock
Forest
prices or customer lifetime value.
• Identifying risk factors: Detecting risk
factors for diseases, financial crises or
other negative events.
• Handling high-dimensional
data: Analyzing datasets with a large
number of input features.
• Capturing complex
relationships: Modeling complex
relationships between input features
and the target variable.
[Link]
Advantages of Random Forest
• Handles Non-Linearity: It can capture complex, non-linear
relationships in the data that other models might miss.
• Reduces Overfitting: By combining multiple decision trees and
averaging predictions it reduces the risk of overfitting compared to
a single decision tree.
• Robust to Outliers: Random Forest is less sensitive to outliers as it
aggregates the predictions from multiple trees.
• Works Well with Large Datasets: It can efficiently handle large
datasets and high-dimensional data without a significant loss in
performance.
• Handles Missing Data: Random Forest can handle missing values by
using surrogate splits and maintaining high accuracy even with
incomplete data.
• No Need for Feature Scaling: Unlike many other algorithms
Random Forest does not require normalization or scaling of the
data.
Disadvantages of Random Forest
• Complexity: It can be computationally expensive and slow to train
especially with a large number of trees and high-dimensional data.
Due to this it may not be suitable for real-time predictions especially
with a large number of trees.
• Less Interpretability: Since it uses many trees it can be harder to
interpret compared to simpler models like linear regression or decision
trees.
• Memory Intensive: Storing multiple decision trees for large datasets
require significant memory resources.
• Overfitting on Noisy Data: While Random Forest reduces overfitting, it
can still overfit if the data is highly noisy especially with a large
number of trees.
• Sensitive to Imbalanced Data: It may perform poorly if the dataset is
highly imbalanced like one class is significantly more frequent than
another.