Module – 2
Ensemble Methods
Table of Contents
1. Introduction to Ensemble Methods
2. Bagging
3. Random Forest
4. Bootstrap Method
5. Bootstrap Aggregation
6. Variable Importance
7. Boosting
8. AdaBoost
9. CatBoost
10. Learning with Ensembles
11. Majority Vote Classifier
12. Weak Learners via Adaptive Boosting
13. Summary Comparison Table
14. References
Module – 2: Ensemble Methods
1. Bagging (Bootstrap Aggregating)
• Definition: Bagging is an ensemble method that reduces variance by training multiple
models (weak learners) on different subsets of the data and averaging their
predictions.
• Process:
1. Create multiple bootstrap samples (random samples with replacement) from
the dataset.
2. Train a base learner (e.g., Decision Tree) on each sample.
3. Aggregate predictions (average for regression, majority vote for
classification).
• Advantages:
o Reduces overfitting.
o Improves stability and accuracy.
• Practical Example (Python):
from [Link] import BaggingClassifier
from [Link] import DecisionTreeClassifier
from [Link] import load_iris
from sklearn.model_selection import train_test_split
from [Link] import accuracy_score
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
random_state=42)
bagging = BaggingClassifier(DecisionTreeClassifier(), n_estimators=50,
random_state=42)
[Link](X_train, y_train)
y_pred = [Link](X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))
2. Random Forest
• Definition: An extension of bagging where each tree is trained not only on a
bootstrap sample but also with a random subset of features at each split.
• Key Idea: Adds extra randomness to reduce correlation between trees.
• Advantages:
o Better accuracy than bagging.
o Provides feature importance.
o Handles missing data and categorical variables effectively.
• Practical Example:
from [Link] import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=100, random_state=42)
[Link](X_train, y_train)
y_pred = [Link](X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))
print("Feature Importances:", rf.feature_importances_)
3. Bootstrap Method
• Definition: Statistical resampling method where samples are drawn with
replacement from the dataset.
• Purpose: Helps estimate statistics (mean, variance, confidence intervals).
• Use in ML: Provides diverse training datasets for bagging.
• Mini Example:
import numpy as np
data = [5, 7, 8, 7, 6, 9]
bootstrap_sample = [Link](data, size=len(data), replace=True)
print("Original:", data)
print("Bootstrap sample:", bootstrap_sample)
4. Bootstrap Aggregation (Bagging)
• Already covered above — combination of bootstrap sampling + aggregation of weak
learners.
5. Variable Importance
• Definition: Measures how much each feature contributes to prediction.
• Random Forest approach:
o Gini importance (impurity reduction).
o Permutation importance (shuffling a feature and measuring accuracy drop).
• Example (Permutation Importance):
from [Link] import permutation_importance
result = permutation_importance(rf, X_test, y_test, n_repeats=10,
random_state=42)
print("Permutation Importances:", result.importances_mean)
6. Boosting
• Definition: Sequential ensemble method where each new model corrects the errors of
the previous ones.
• Key Idea: Focus on misclassified samples.
• Difference from Bagging:
o Bagging = parallel learners (independent).
o Boosting = sequential learners (dependent).
7. AdaBoost (Adaptive Boosting)
• Definition: Boosting method that assigns weights to samples. Misclassified samples
get higher weights in the next iteration.
• Base Learner: Usually decision stumps (one-level decision trees).
• Process:
1. Train weak learner.
2. Compute error rate.
3. Update sample weights.
4. Repeat.
• Practical Example:
from [Link] import AdaBoostClassifier
ada = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1),
n_estimators=50, random_state=42)
[Link](X_train, y_train)
y_pred = [Link](X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))
8. CatBoost
• Definition: Gradient boosting algorithm optimized for categorical features (hence
“Cat”-Boost).
• Advantages:
o Handles categorical data without preprocessing (like one-hot encoding).
o Fast and accurate.
• Installation: pip install catboost
• Example:
from catboost import CatBoostClassifier
cat = CatBoostClassifier(iterations=100, learning_rate=0.1, depth=6,
verbose=0)
[Link](X_train, y_train)
y_pred = [Link](X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))
9. Learning with Ensembles
• Idea: Combining multiple models improves prediction accuracy compared to a single
model.
• Types:
o Bagging (reduces variance).
o Boosting (reduces bias).
o Stacking (meta-model learns from base learners).
10. Implementing a Simple Majority Vote Classifier
• Definition: A simple ensemble where each classifier votes, and the majority class
wins.
• Example:
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from [Link] import SVC
from [Link] import VotingClassifier
clf1 = LogisticRegression()
clf2 = GaussianNB()
clf3 = SVC(probability=True)
voting = VotingClassifier(estimators=[('lr', clf1), ('nb', clf2), ('svc',
clf3)], voting='hard')
[Link](X_train, y_train)
y_pred = [Link](X_test)
print("Majority Vote Accuracy:", accuracy_score(y_test, y_pred))
11. Leveraging Weak Learners via Adaptive Boosting
• Weak Learners: Simple models (e.g., decision stumps) with slightly better than
random accuracy.
• AdaBoost improves them by:
o Assigning higher weights to misclassified samples.
o Combining weak learners into a strong classifier.
• Key Formula:
Final prediction = weighted sum of weak learners’ outputs.
✅ Summary
• Bagging & Random Forest → Reduce variance by parallel learners.
• Bootstrap & Aggregation → Sampling + combining predictions.
• Variable Importance → Identifies key features.
• Boosting & AdaBoost → Sequential correction of errors.
• CatBoost → Boosting optimized for categorical data.
• Majority Vote Classifier → Simple ensemble by voting.
• Adaptive Boosting → Turns weak learners into strong learners.
Summary Comparison Table
Method Key Idea Strengths Weaknesses
Bagging Bootstrap sampling + aggregation Reduces variance May not reduce bias
Random Forest Bagging + random feature selection Handles high dimensions Less interpretable
Boosting Sequential error correction High accuracy Prone to overfitting
AdaBoost Weighted boosting Improves weak learners Sensitive to noise
CatBoost Boosting for categorical data No preprocessing needed Computationally heavy
14. References
- Hastie, Tibshirani, Friedman. The Elements of Statistical Learning.
- Scikit-learn Documentation ([Link]
- CatBoost Documentation ([Link]