.

Introduction & Plan of Attack

Video Overview: This video introduces Bagging, an important ensemble machine learning technique
(0:01).

What will be covered:

Core idea, intuition, and introduction to Bagging (0:28).

Bagging for Classification (0:33) and Regression (0:46) (covered in subsequent videos).

Important points and reasons to use Bagging (1:03).

When to apply Bagging (1:11).

Code demo (1:19).

Types of Bagging techniques (1:29).

2. What is Bagging? (Bootstrap Aggregating)

Definition: Bagging is formed from two words: Bootstrapping and Aggregation (1:50).

Bootstrapping: A technique where you randomly sample data with replacement to create subsets (2:14).

How Bagging Works:

Start with a large dataset (e.g., 10,000 rows) (2:29).

Create multiple "base models" (e.g., M1, M2, M3) (2:47). These models typically use the same
algorithm (e.g., all Decision Trees or all SVMs), unlike boosting (3:01).

Bootstrapping Step: Instead of training all models on the same data, you create different subsets of your
original dataset by sampling with replacement (3:31).

For each model, a new dataset (e.g., D1, D2, D3) is created by randomly sampling rows from the original
dataset (e.g., 1000 rows each) (3:40).

This sampling can be with replacement (most common in bagging) or without replacement (3:51).

Each base model (M1, M2, M3) is then trained on its respective unique bootstrapped dataset (D1, D2,
D3) (4:01 - 4:37). This step is called Bootstrapping (4:38).

Aggregation Step: Once all base models are trained, for a new data point (X_new) you want to predict
(4:51):

Feed X_new to all base models (5:01).

Each model makes its own prediction (e.g., 0 or 1 for classification) (5:08).

For classification, the final output is determined by majority voting (mode) among the base models'
predictions (5:17 - 5:38).

For regression, the final output is typically the average of the base models' predictions.
This combining step is called Aggregation (5:48).

Core Idea: Bagging aims to create variation among base models by providing them with different subsets
of data. This variation, when combined, leads to a more robust and better-performing overall model
(6:06).

3. Why Use Bagging? (The Bias-Variance Tradeoff)

Dependence on Bias-Variance Tradeoff: Bagging's effectiveness is explained by its impact on bias and
variance (7:16).

Bias: Inherent error of a machine learning algorithm to miss the true relationships in the data
(underfitting) (7:41). Low bias means the model performs well on training data (7:55).

Variance: How much the model's performance changes if the training data is slightly different
(overfitting) (8:04). High variance means inconsistent results across different datasets (8:12).

The Tradeoff: Bias and variance are inversely correlated; reducing one often increases the other (8:35).

Most ML algorithms are either "low bias, high variance" or "high bias, low variance" (9:04). Bagging
helps achieve low bias and low variance (9:16).

How Bagging Reduces Variance:

Bagging typically uses base algorithms that are low bias and high variance (prone to overfitting) (9:26).
Examples include:

Fully grown Decision Trees (max_depth=None) (9:42).

Support Vector Machines (SVMs) (10:06).

k-Nearest Neighbors (kNN) (10:09).

When these high-variance models are trained on different bootstrapped subsets of data, the variations
in the data lead to slightly different models.

When their predictions are aggregated (e.g., through majority voting), the individual errors and
variations tend to cancel each other out (10:50 - 12:00).

This leads to a final ensemble model that is consistent (low variance) and accurate (low
bias) (12:12 - 12:46).

When to Use: Bagging is highly recommended, especially for algorithms that are low bias and high
variance (13:16) (i.e., those that tend to overfit the training data). Random Forest, a very popular
algorithm, is an implementation of bagging (13:27).

4. Code Demo (14:32)

Dataset: Iris dataset is used for demonstration (14:35).

Preprocessing:

Drop 'Id' column (14:53).

Simplify the problem by removing certain columns and converting it to a binary classification problem
(15:20).

Shuffle the dataset to ensure randomness (16:02).

Split data into df_train, df_val, and df_test (16:55).

Bagging Implementation Steps:

Bootstrapping:

Create bootstrapped datasets (df_bag1, df_bag2, df_bag3) by randomly

Extract X and Y for each bootstrapped dataset (19:54).

Base Model Training:

Initialize a DecisionTreeClassifier (with max_depth=None for low bias/high variance) (20:17).

Train each decision tree on its respective bootstrapped dataset (20:41).

Evaluate each individual base model's accuracy on the validation set (X_val, y_val) (20:51).

Aggregation (Prediction):

Select a point from the test dataset (df_test) (23:29).

Each trained base model makes a prediction on this point (23:50).

The final prediction is determined by majority voting among the base models' predictions
(24:57 - 25:05). The example shows how multiple "2" predictions outweigh a single "1" prediction.

5. Types of Bagging Techniques (25:39)

Bagging types depend on how the subsets are created (25:49).

1. Bagging (Standard):

Row Sampling: Uses row sampling with replacement (25:57 - 26:03).

Example: df_train.sample(frac=0.8, replace=True).

2. Pasting:

Row Sampling: Uses row sampling without replacement (26:31 - 26:45).

Example: df_train.sample(n=8, replace=False).

3. Random Subspace:

Column Sampling: Uses column sampling (27:42) (with or without replacement) while taking all rows.

Example: df_train.sample(n=2, axis=1, replace=True).

Both Row and Column Sampling: Uses a combination of both row and column sampling (29:21 - 29:27).

Example: First sample rows with replacement, then from that subset, sample columns with replacement.

Goal of all types: To create variety in the base models (30:12) by feeding them different data, leading to
improved overall model performance.

Additional Resources Shared in Video:

Code used: [Link]

Bias Variance Tradeoff video: