1.
Ensemble Learning
Ensemble learning is a technique where we combine multiple models (often called weak
learners or base estimators) to build a stronger predictive model.
The core idea:
A single model might be biased, noisy, or unstable.
By combining many models, we reduce variance, bias, and improve generalization.
Types of ensembles:
1. Bagging (Bootstrap Aggregating) → Reduces variance by averaging predictions over
diverse models.
2. Boosting → Reduces bias by training models sequentially, where each model fixes
mistakes of the previous one.
3. Stacking → Combines different types of models and uses another "meta-learner" to
blend predictions.
Example: Random Forest is an ensemble of decision trees built using Bagging.
2. Bagging (Bootstrap Aggregating)
Bagging is short for Bootstrap Aggregating. It’s a technique to reduce variance of a model by
training multiple learners on different bootstrapped samples of the dataset.
How it works:
1. Given a dataset of size N, generate M new datasets, each also of size N.
o Each dataset is created by sampling with replacement (bootstrap sampling).
o This means some samples will appear multiple times, others might not appear at
all.
2. Train a base model (e.g., decision tree) on each bootstrapped dataset.
3. Combine predictions:
o Regression → take the average of predictions.
o Classification → take a majority vote.
Why it helps:
Each model sees slightly different data, so they make different errors.
Averaging or voting smooths out the noise, improving stability.
Formula for Bagging Regression:
M
^f bag ( x )= 1 ∑ f^ m (x)
M m=1
where ^f m ( x) is the prediction from the m-th model.
Random Forest is Bagging applied to Decision Trees + random feature selection.
3. Sub-Bagging (Subsample Aggregating)
Sub-Bagging (or Subsampling without replacement) is a variant of Bagging.
How it differs from Bagging:
Instead of bootstrap sampling (with replacement) → Sub-Bagging uses subsamples
without replacement.
Each dataset used to train a base learner has fewer than N samples (say n < N).
Since there’s no replacement, each subsample is strictly a subset of the original dataset.
Why use Sub-Bagging?
Efficiency → Faster training since we use fewer samples per learner.
Less correlation between models than bagging, sometimes improving performance.
Useful when the dataset is very large, and full bootstrapping would be computationally
expensive.
Example: Pasting (Breiman, 1999) is similar to Sub-Bagging, where subsets are sampled without
replacement.
4. Bagging vs Sub-Bagging
Aspect Bagging Sub-Bagging
Sampling With replacement (bootstrap) Without replacement (subsample)
Sample size per learner Equal to original dataset (N) Smaller subset (n < N)
Aspect Bagging Sub-Bagging
Diversity of learners Higher (due to duplicate samples) Lower, but less redundancy
Computation More expensive More efficient
Use case When variance reduction is crucial When dataset is very large
Summary
Ensemble = combining multiple models for stronger performance.
Bagging = build learners on bootstrap samples (with replacement) → reduces variance.
Sub-Bagging = build learners on subsamples (without replacement) → efficient, useful
for large datasets.
Random Forest = a Bagging-based ensemble of decision trees + random feature
selection.
Problem 1: Conceptual
Explain the main difference between Bagging and Sub-Bagging.
Why does Bagging usually use bootstrap sampling with replacement?
What advantage does Sub-Bagging provide when working with very large datasets?
Problem 2: Numerical Sampling
You have a dataset of 10 samples:
D={1,2,3,4,5,6,7,8,9,10}D = \{1,2,3,4,5,6,7,8,9,10\}D={1,2,3,4,5,6,7,8,9,10}
1. If you create a bagging sample of size 10, give one possible bootstrap sample (sampling
with replacement).
2. If you create a sub-bagging sample of size 6, give one possible subsample (sampling
without replacement).
Problem 3: Bias-Variance
Suppose you train a Decision Tree as a base learner:
Single Decision Tree has high variance and low bias.
Bagging multiple Decision Trees reduces variance.
Question:
Explain how Bagging improves prediction stability.
Does Bagging reduce bias? Why or why not?
Problem 4: Application
You are building a spam email classifier using Decision Trees.
You decide to use Bagging with 50 trees.
Each tree outputs whether the email is spam (1) or not spam (0).
Suppose for a new email, 30 trees output 1 (spam) and 20 trees output 0 (not spam).
What will the final Bagging classifier prediction be?
Problem 5: Advanced – Sub-Bagging Efficiency
Imagine you have a dataset with 1 million samples.
Bagging requires training each base learner on 1 million bootstrapped samples.
Sub-Bagging trains each learner on 100,000 randomly chosen samples without
replacement.
Questions:
1. Why might Sub-Bagging be faster in this case?
2. How might using smaller subsets affect prediction accuracy compared to Bagging?