Ensemble Learning Methods
Feature Bagging (e.g., Random Forest) Boosting (e.g., AdaBoost, XGBoost)
Reduces Variance (prevents
Main Goal Reduces Bias (improves accuracy)
overfitting)
Model Parallel (models are trained Sequential (models are trained one
Training independently) after another)
Each model is trained on the full
Data Each model is trained on a
dataset, but data points are weighted
Sampling random subset (bootstrapping).
based on past errors.
Model Models get a weighted vote based on
All models get an equal vote.
Voting their performance.
Primary Best with complex, high-variance Best with simple, high-bias (weak)
Use models. models.
Typically, model averaging (bagging ensemble) always helps in a machine
learning setting
BUT Training several large neural networks for making an ensemble is
prohibitively expensive and not possible most of the times
• Option 1: Train several neural networks having different
architectures(obviously expensive).
• Option 2: Train multiple instances of the same network using
different training samples (again expensive).
Dropout is a technique which addresses both
these issues.
• Dropout refers to dropping out units.
• Temporarily remove a node and all its
incoming/outgoing connections resulting in a
thinned network.
• A good value for dropout in a hidden layer is
between 0.5 (only for large DNNs)
We initialize all the parameters (weights)
of the network and start training
• For the first training instance (or mini-
batch), we apply dropout resulting in the
thinned network
• We compute the loss and
backpropagate
• Which parameters will we update?
Only those which are active.
For the second training instance (or mini-
batch), we again apply dropout resulting in
a different
thinned network.
• We again compute the loss and
backpropagate to the active weights
• If the weight was active for both the
training instances then it would have
received two updates by now.
• If the weight was active for only one of the
training instances then it would have
received only one updates by now
During test data, we use the full neural Network and scale the output of each
node by the fraction of times it was on during training.
Let's imagine a single forward pass in a small part of our network. We have an input and
one hidden layer with 4 neurons.
• Dropout Rate (p): 0.5 (or 50%). This means we expect to drop half the neurons.
We'll use Inverted Dropout, which is the standard method.
A Standard Forward Pass (No Dropout)
Input (x): [1.0, 2.0]
Weights (W): (A 2x4 matrix for 2 inputs and 4 neurons)
[[0.2, 0.1, 0.4, 0.5],
[0.6, 0.8, 0.1, 0.3]]
Biases (b): [0.1, 0.1, 0.1, 0.1]
Layer's Output (Pre-Activation): [1.5, 1.8, 0.7, 1.2]
The calculation is output = (x * W) + b:
• (1.0 * 0.2) + (2.0 * 0.6) = 1.4
• (1.0 * 0.1) + (2.0 * 0.8) = 1.7
• (1.0 * 0.4) + (2.0 * 0.1) = 0.6
• (1.0 * 0.5) + (2.0 * 0.3) = 1.1
Adding the bias of 0.1 to each:
A Training Pass (With Dropout)
Now, let's do the same pass but during training.
Layer's Output (Pre-Activation): [1.5, 1.8, 0.7, 1.2]
Generate Dropout Mask: We randomly generate a mask of 0s and 1s for our 4 neurons. A 0 means drop.
Let's say our random mask is: [1, 0, 1, 0]
1. Apply Mask: We multiply our output by this mask.
[1.5, 1.8, 0.7, 1.2] * [1, 0, 1, 0]
Dropped Output: [1.5, 0.0, 0.7, 0.0]
Neurons 2 and 4 have been "dropped" (set to zero). They contribute nothing to the next layer in this training step.
Scale the Output (Inverted Dropout): Since we dropped 50% of our neurons,
the total output of this layer is now only half as large as it should be.
To fix this, we scale up the activations of the neurons that survived.
Scaling Factor: 1 / (1 - p) = 1 / (1 - 0.5) = 2.0
We divide the Dropped Output by the keep probability (or multiply by the
scaling factor).
[1.5, 0.0, 0.7, 0.0] / 0.5
Final Layer Output (to next layer): [3.0, 0.0, 1.4, 0.0]
This final vector is what gets passed to the next layer (and then through its
activation function).