CS2250 Fundamentals of Machine Learning
Module 1 – Fundamentals of Machine Learning
Ensemble Learning, Bagging: Concept and use of Random Forests, boosting: AdaBoost,
Gradient Boosting, Stacking and Voting Classifiers
What is Ensemble Learning?
It is a machine learning paradigm in which multiple models (base learners) are trained and
combined to produce a single, more robust predictive model.
• Instead of depending on one model, ensemble learning takes the outputs of many models
and combines them to make a single final prediction.
• This improves accuracy because different models may make different mistakes, and
combining them helps cancel out errors.
Why do we do this?
One model can make mistakes, but many models together make fewer mistakes
Ensemble learning improves model performance by:
• Reducing variance (e.g., bagging)
• Reducing bias (e.g., boosting)
• Improving generalization on unseen data
• Increasing robustness to noise and data fluctuations
CS2250 Fundamentals of Machine Learning
Ensemble Learning Methods
CS2250 Fundamentals of Machine Learning
Bagging and Boosting
Bagging:
• Train many independent models at the same time (parallel) and average them.
• In bagging, each model is independent of the others.
• Reduce variance and overfitting
Boosting:
• Train models sequentially, fixing mistakes step by step.
• In boosting, each model depends on the errors of the previous model.
• Reduce bias
Bagging is parallel because its models are independent and trained on different bootstrap
samples, whereas boosting is sequential because each model depends on the errors made by
the previous model.
CS2250 Fundamentals of Machine Learning
Bagging (Learning Together)
It is an ensemble learning method where many models are trained independently on different
random bootstrap samples of the same dataset, and their predictions are combined using voting
or averaging.
• The original training data is randomly sampled (with repetition) to create multiple
datasets.
• Each dataset trains a separate model.
• All models work independently.
• The final answer is chosen by majority voting (classification) or average (regression).
Bagging mainly helps to reduce variance and avoid overfitting.
Use bagging when:
• The model overfits
• Data is small to medium
• Base learners have high variance
• Noise exists but bias is already low
How Bagging Works (Step-by-Step)
1. From the original dataset, create multiple bootstrap samples (sampling with replacement).
2. Train a separate base model (often decision trees) on each sample.
3. Aggregate predictions:
o Classification → Majority voting
o Regression → Average of predictions
Common Algorithm Using Bagging
• Random Forest
o Uses bagging + feature randomness
o Very effective for high-variance data
CS2250 Fundamentals of Machine Learning
Problem
Aggregation in Classification (Majority Voting):
Example: Predict whether a student Passes (P) or Fails (F).
Consider a bagging / random forest model consists of 5 decision trees.
Predictions from Individual Models
Model (Tree) Prediction
Tree 1 Pass (P)
Tree 2 Fail (F)
Tree 3 Pass (P)
Tree 4 Pass (P)
Tree 5 Fail (F)
Majority Voting Rule
Count the predictions: Pass (P) = 3 and Fail (F) = 2
Final Aggregated Prediction = Pass
CS2250 Fundamentals of Machine Learning
Aggregation in Regression (Average of Predictions)
Example: Predict the house price (in lakhs) using an ensemble of 4 regression trees.
Predictions from Individual Models
Model (Tree) Predicted Price (₹ Lakhs)
Tree 1 52
Tree 2 55
Tree 3 50
Tree 4 53
Averaging Rule
52 + 55 + 50 + 53 210
Final Prediction = = = 52.5
4 4
Final Aggregated Prediction: 52.5 Lakhs
CS2250 Fundamentals of Machine Learning
Problem
Predict whether a student Passes (P) or Fails (F) based on Study Hours.
Dataset (D)
Student Study Hours Result
S1 1 F
S2 2 F
S3 3 F
S4 4 P
S5 5 P
S6 6 P
Total records = 6
Step 1: Bootstrap Sampling (Bagging)
Generate multiple datasets of size 6, sampled with replacement.
Bootstrap Sample 1 (D₁) Bootstrap Sample 2 (D₂) Bootstrap Sample 3 (D₃)
Study Hours Result Study Hours Result Study Hours Result
2 F 1 F 1 F
3 F 2 F 1 F
3 F 4 P 3 F
4 P 5 P 4 P
6 P 5 P 5 P
6 P 6 P 6 P
Each dataset is slightly different, even though all are derived from the same original data.
Step 2: Train Models on Each Bootstrap Sample
Train three decision trees:
• Tree 1 trained on D₁
• Tree 2 trained on D₂
• Tree 3 trained on D₃
Each tree learns. slightly different decision boundaries
Step 3: Aggregation (Voting)
New Student: Study Hours = 3.5
Predictions
Model Prediction
Tree 1 F
Tree 2 P
Tree 3 F
Final Bagging Prediction: Majority vote = Fail (F)
CS2250 Fundamentals of Machine Learning
Problem
Random Forests: Bagging + Feature Randomness
A Random Forest is an improved form of bagging where:
1. Bootstrap sampling is applied (same as bagging)
2. Random subset of features is used at each split
Dataset (D)
Student Study Hours Attendance (%) Result
S1 1 60 F
S2 2 65 F
S3 3 70 F
S4 4 75 P
S5 5 80 P
S6 6 85 P
Feature Selection in Random Forest
At each split:
• Tree 1 may consider only Study Hours
• Tree 2 may consider only Attendance
• Tree 3 may consider both
This reduces correlation between trees, making the ensemble stronger.
Prediction Using Random Forest
For a student with:
• Study Hours = 3.5
• Attendance = 72%
Tree Used Feature Prediction
Tree 1 Study Hours F
Tree 2 Attendance F
Tree 3 Both P
Final Random Forest Prediction: Majority vote = Fail (F)
CS2250 Fundamentals of Machine Learning
Boosting
Boosting is an ensemble technique that reduces bias by training models sequentially, where
each new model focuses more on the previous model’s errors.
How Boosting Works (Step-by-Step)
1. Train the first model on the original dataset.
2. Identify misclassified or high-error samples.
3. Increase the importance (weights) of these samples.
4. Train the next model focusing more on difficult cases.
5. Combine all models using weighted voting or weighted sum.
Example
• First classifier misclassifies some data points.
• Second classifier pays more attention to those mistakes.
• Third classifier focuses even more on hard samples.
• Final model is a weighted combination of all classifiers.
Common Boosting Algorithms
• AdaBoost (Adaptive Boosting)
• Gradient Boosting
• XGBoost / LightGBM / CatBoost
When to Use Boosting
Use boosting when:
• The model underfits
• Bias is high
• Complex patterns exist
• Large datasets are available
• Noise level is controlled