MODULE 2

SYLLABUS
Introduction, setup and initialization- Kaiming, Xavier
weight intializations, Vanishing and exploding gradient
problems, Optimization techniques - Gradient Descent (GD),
Stochastic GD, GD with momentum, GD with Nesterov
momentum, AdaGrad, RMSProp, Adam., Regularization
Techniques - L1 and L2 regularization, Early stopping,
Dataset augmentation, Parameter tying and sharing,
Ensemble methods, Dropout, Batch normalization
Training deep models - Introduction
TRAINING PROCESS
Different Weight Initialization Techniques

Zero Initialization (Initialized all weights to 0)

▪If we initialized all the weights with 0, then what happens is that the derivative
with respect to the loss function is the same for every weight in W[l], thus
all weights have the same value in subsequent iterations.
▪This makes hidden layers symmetric and this process continues for all the n
iterations. Thus, initialized weights with zero make your network no better than
a linear model.
Random Initialization (Initialized weights
randomly)

▪This technique tries to address the problems of zero initialization since it

prevents neurons from learning the same features of their inputs.
▪since our goal is to make each neuron learn different functions of its input
and this technique gives much better accuracy than zero initialization.
▪In general, it is used to break the symmetry. It is better to assign random
values except 0 to weights.
▪ Remember, neural networks are very sensitive and prone to overfitting
as it quickly memorize the training data.
Best Practices for Weight Initialization

❑Use RELU or leaky RELU as the activation function, as they both are relatively
robust to the vanishing or exploding gradient problems(especially for networks
that are not too deep). In the case of leaky RELU, they never have zero gradients.
Thus, they never die, and training continues.
❑Use Heuristics for weight initialization: For deep neural networks, we can use
any of the following heuristics to initialize the weights depending on the chosen
non-linear activation function.
❑While these heuristics do not completely solve the exploding or vanishing
gradients problems, they help to reduce it to a great extent.
❑The most common heuristics are as follows:
1. Xavier/Glorot Initialization :

❑Xavier Initialization is a Gaussian initialization heuristic that

keeps the variance of the input to a layer the same as that of the
layer’s [Link] ensures that the variance remains the same
throughout the network.
❑ It works well for sigmoid function
2)He /Kaiming Initialization
Benefits of using these heuristics:

•All these heuristics serve as good starting points for

weight initialization and they reduce the chances of
exploding or vanishing gradients.
• All these heuristics do not vanish or explode too
quickly, as the weights are neither too much bigger
than 1 nor too much less than 1.
• They help to avoid slow convergence and ensure that
we do not keep oscillating off the minima.
OPTIMIZATION TECHNIQUES

• Optimization algorithms are responsible for reducing

losses and providing the most accurate results possible.
• The weight is initialized using some initialization strategies
and is updated with each epoch according to the equation.
• The best results are achieved using some optimization
strategies or algorithms called Optimizer.
❑when we get to realize that our model is performing poor at the current
instance so we need to minimize the loss and maximize the accuracy. That
process is known as optimization.
❑Optimizers are methods or algorithms used to change the attributes of neural
networks, such as weights and learning rate, to reduce the loss.
❑After the calculation of loss, we need to optimize our weights and bias in the
same iteration.
❑Initially we don't know the weights so we start randomly but with some trial
and error based on loss function we can end up getting our loss downwards.
❑Optimization techniques are responsible for reducing the loss and providing
most accurate results possible .
GRADIENT DESCENT
❑Gradient Descent is known as one of the most commonly used optimization algorithms to
minimize a function by optimizing the parameters.
❑The goal of gradient descent is to find the set of weights (or coefficients) that minimize the loss
function. The algorithm works by iteratively adjusting the weights in the direction of the
steepest decrease in the loss function.
❑The basic idea of gradient descent is to start with an initial set of weights and update them in
the direction of the negative gradient of the loss function.
❑The gradient is a vector of partial derivatives that represents the rate of change of the loss
function with respect to the weights. By updating the weights in the direction of the negative
gradient, the algorithm moves towards a minimum of the loss function.
❑The learning rate is a hyperparameter that determines the size of the step taken in the weight
update. A small learning rate results in a slow convergence, while a large learning rate can lead
to overshooting the minimum and oscillating around the minimum.
❑It’s important to choose an appropriate learning rate that balances the speed of convergence
and the stability of the optimization.
Disadvantages of GD
❑A constant learning rate is not desirable because
❑A lower learning rate used early on will cause the algorithm to take too long to
come even close to an optimal solution.
❑On the other hand, a large initial learning rate will allow the algorithm to come
reasonably close to a good solution at first; however, the algorithm will then
oscillate around the point for a very long time, or diverge in an unstable way, if the
high rate of learning is maintained. In either case, maintaining a constant learning
rate is not ideal.
❑it can bounce around the search space based on the gradient. This bouncing
effect can cause the algorithm to converge slowly or to get stuck in a local
minimum, rather than finding the global minimum.
Stochastic gradient descent algorithm
❑In Gradient Descent optimization, we compute the cost gradient based on the complete
training set; hence, it is called batch gradient descent.
❑In case of very large datasets, using Gradient Descent can be quite costly since we are
only taking a single step for one pass over the training set – thus, the larger the training
set, the slower our algorithm updates the weights and the longer it may take until it
converges to the global cost minimum.
❑In Stochastic Gradient Descent ,we don’t accumulate the weight updates as we’ve
seen above for Gradient Descent:
❑Instead of computing the sum of all gradients, stochastic gradient descent selects an
observation uniformly at random
Momentum based gradient descent
❑GD takes a lot of time to converge.
❑Momentum helps the optimization process retain speed in flat regions of the loss surface and
avoid local optima.
❑Momentum involves adding an additional hyperparameter that controls the amount of history
(momentum) to include in the update equation, i.e. the step to a new point in the search space.
❑The value for the hyperparameter is defined in the range 0.0 to 1.0 and often has a value close to
1.0, such as 0.8, 0.9, or 0.99. A momentum of 0.0 is the same as gradient descent without
momentum.
❑With momentum-based descent, the learning is accelerated, because one is
generally moving in a direction that often points closer to the optimal solution and
the useless “sideways” oscillations are muted.
❑The basic idea is to give greater preference to consistent directions over multiple
steps, which have greater importance in the descent.
❑ This allows the use of larger steps in the correct direction without causing
overflows or “explosions” in the sideways direction.
❑As a result, learning is accelerated.
Analogy
❑A marble will overshoot when it is allowed to roll down a bowl.
❑The momentum-based method will generally perform better because the marble
gains speed as it rolls down the bowl; the quicker arrival at the optimal solution
❑The marble’s gathering of speed helps it efficiently navigate flat regions of the
loss surface but it overshoots.
.
Disadvantages
[Link]: Momentum-based gradient descent can overshoot the minimum
of the cost function and lead to oscillations around the minimum. This can
happen if the momentum term is too high or if the learning rate is too
[Link] avoids local optima.
[Link] to the initial conditions: The momentum term can cause the algorithm
to converge to different minima depending on the initial conditions. This can
make the algorithm less reliable, especially if the cost function has several local
minima.
Nesterov Accelerated Gradient Descent
❑The Nesterov momentum algorithm is a modification of the traditional momentum
method used in gradient descent optimization algorithms.
❑In Nesterov momentum, the gradients are computed at a point that would be reached
after executing a discounted version of the previous step again, which is the momentum
portion of the current step.
❑This point is obtained by multiplying the previous update vector with the friction
parameter and then computing the gradient.
❑The idea behind Nesterov momentum is that by using the corrected gradient, which
takes into account the momentum portion of the update, the algorithm can make more
informed updates and move towards the minimum of the cost function more quickly.
❑This is particularly useful when the cost function is curved, as the corrected gradient can
help to avoid overshooting the minimum.
❑In the previous analogy of the rolling marble, such an approach will start applying the
“brakes” on the gradient-descent procedure when the marble starts reaching near the
bottom of the bowl, because the lookahead will “warn” it about the reversal in gradient
direction.
AdaGrad
❑Gradient of f(x) w.r.t to a particular weight is clearly dependent on its corresponding input.
❑ If there are n points, we can just sum the gradients over all the n points to get the total
gradient
❑But what would happen if the feature x2 is very sparse (i.e., if its value is 0 for most inputs)?
❑ It is fair to assume that ∇w2 will be 0 for most inputs and hence w2 will not get enough
updates.
❑To make sure updates happen even when a particular input is sparse, we have a different
learning rate for each parameter.
❑AdaGrad algorithm, which adjusts the learning rate based on the sparsity of the data to ensure
that parameters with low frequency receive higher learning rates.
❑ Additionally, AdaGrad ensures that frequently updated parameters have their learning rates
decreased over time to prevent overshooting the optimal solution.
❑However, it is noted that removing the square root from the denominator of the algorithm may
negatively impact its effectiveness.
❑ Furthermore, it is mentioned that the RMSProp algorithm can help prevent the decay of the
effective learning rate for frequently updated parameters.
AdaGrad got stuck when it was close to convergence, it was no longer able to move in the
vertical (b) direction because of the decayed learning rate.
RMSProp overcomes this problem by being less aggressive on the decay.
RMSProp also tries to dampen the oscillations, but in a different way than
momentum.
RMS prop also takes away the need to adjust learning rate, and does it
automatically.
More so, RMSProp choses a different learning rate for each parameter.
In the first equation, we compute an exponential average of the square of the
gradient.
we multiply the exponential average computed till the last update with a
hyperparameter, represented by the greek symbol nu. We then multiply the
square of the current gradient with (1 - nu). We then add them together to get
the exponential average till the current time step.
Then in the second equation, we decided our step size. We move in the
direction of the gradient, but our step size is affected by the exponential
average. We chose an initial learning rate eta, and then divide it by the
average. In our case, since the average of w1 is much much larger than w2, the
learning step for w1 is much lesser than that of w2. Hence, this will help us
avoid bouncing between the ridges, and move towards the minima.
Adam
Adam or Adaptive Moment Optimization algorithms combines the heuristics
of both Momentum and RMSProp. Here are the update equations.
REGULARISATION
❑Weight regularization is a technique which aims to stabilize an
overfitted network by penalizing the large value of weights in the
network.
❑An overfitted network usually presents with problems with a large value
of weights, as a small change in the input can lead to large changes in the
output.
❑For instance, when the network is given new or test data, it results in
incorrect predictions.
❑Weight regularization penalizes the network’s large weights & forcing
the optimization algorithm to reduce the larger weight values to smaller
weights, and this leads to stability of the network & presents good
performance.
❑In weight regularization, the network configuration remains unchanged
only modifying the value of weights.
L1 REGULARISATION
❑L1 regularization (LASSO regression) (Least Absolute Shrinkage
and Selection Operator) produces sparse matrices.
❑ Sparse matrices are zero-matrices in which some elements are ones
(the sparsity refers to the ones), but in this context a sparse matrix
could be several close-to-zero values and other larger values.
❑If we find a model with neurons whose weights are close to zero it
means we don’t need those neurons because the model deactivates
them with zeros and we might not need a specific feature/input
leading to a simpler model.
❑For instance, if we have 50 coefficients but only 10 are nonzero, the other
40 are irrelevant to make our predictions. This is not only interesting from
the efficiency point of view but also from the economic point of view:
gathering data and extracting its features might be a very expensive task (in
terms of time and money).
❑Reducing this will benefit us.
❑Due to the absolute value, L1 regularization provides with a non-
differentiable term, but despite of that, there are methods to minimize it
 Benefits of L1 Regularization (LASSO)
•Feature Selection: L1 regularization inherently performs
feature selection by driving some coefficients to zero.
•Model Interpretability: The resulting model is often easier to
interpret because it is simpler and focuses on fewer features.
•Cost Efficiency: By identifying and eliminating irrelevant
features, LASSO can reduce the cost of data collection and
processing.
L2 regularization (Ridge regression)
❑L2 regularization (Ridge regression) on the other hand leads to a
balanced minimization of the weights.
❑ Since L2 uses squares, it emphasizes the errors, and it can be a
problem when there are outliers in the data.
❑Unlike L1, L2 has an analytical solution which makes it computationally
efficient.
 The Role of the λ Parameter
•Both L1 and L2 regularizations involve a hyperparameter λ, which
controls the strength of the regularization.
•Large λ: A larger value of λ imposes a stronger penalty on the magnitude
of the coefficients, leading to simpler models with smaller weights. This
can help avoid overfitting by discouraging complex models that might fit
the noise in the training data.
•Small λ: A smaller value of reduces the regularization effect, allowing the
model to fit the data more closely. If λ is set to zero, the regularization is
effectively turned off, and the model reduces to ordinary least squares
(OLS) regression or the original optimization problem.
 Comparison with L1 Regularization:
•Feature Selection: L2 regularization (Ridge) tends to shrink weights but does
not set them to zero, meaning all features are retained. L1 regularization
(LASSO), on the other hand, can result in some weights being exactly zero,
which effectively removes the corresponding features from the model.
•Handling Outliers: L2 regularization is more sensitive to outliers because it
squares the coefficients, making large errors more costly. L1 regularization is
less sensitive to outliers, which can be advantageous in certain situations.
•Computational Efficiency: L2 regularization often has an analytical solution,
making it computationally more efficient for linear models. L1 regularization,
due to its non-differentiable nature at zero, generally requires iterative
methods for optimization.
DATA AUGMENTATION
❑A common trick to reduce overfitting in convolutional neural networks is the idea of
data augmentation.
❑In data augmentation, new training examples are generated by using transformations on
the original examples.
❑Image processing is one domain to which data augmentation is well-suited.
❑This is because many transformations such as translation, rotation, patch extraction,
and reflection, do not fundamentally change the properties of the object in an image.
❑However, they do increase the generalization power of the data set when trained with
the augmented data set. For example, if a data set is trained with mirror images and
reflected versions of all the bananas in it, then the model is able to better recognize
bananas in different orientations.
❑Many of these forms of data augmentation require very little computation.
❑they can be created at training time, when an image is being processed. For
example, while processing an image of a banana, it can be reflected into a
modified banana at training time.
❑Similarly, the same banana might be represented in somewhat different color
intensities in different images, and therefore it might be helpful to create
representations of the same image in different color intensities. In many cases,
creating the training data set using image patches can be helpful.
Common Data Augmentation Techniques
EXAMPLES
[Link] Stopping

Another common form of regularization is early stopping, in which the gradient descent is ended
after only a few iterations.
One way to decide the stopping point is by holding out a part of the training data, and then
testing the error of the model on the held-out set.
The gradient-descent approach is terminated when the error on the held-out set begins to rise.
Early stopping essentially reduces the size of the parameter space to a smaller neighborhood
within the initial values of the parameters.
From this point of view, early stopping acts as a regularizer because it effectively restricts the
parameter space

DEPT OF EI,SJCET,PALAI 87
The model tries to chase the loss function crazily on the training data, by tuning the parameters.
Now, we keep another set of data as the validation set and as we go on training, we keep a record
of the loss function on the validation data, and when we see that there is no improvement on the
validation set, we stop, rather than going all the epochs.
This strategy of stopping early based on the validation set performance is called Early Stopping.

DEPT OF EI,SJCET,PALAI 88
ADDING NOISE TO INPUT
❑The addition of noise to the input has connections with penalty-based regularization.
❑It can be shown that the addition of an equal amount of Gaussian noise to each input is equivalent to
Tikhonov regularization of a single-layer neural network with an identity activation function .
❑Also noise applied to the inputs is a dataset augmentation strategy.
❑Another way that noise has been used in the service of regularizing models is by adding it to the weights.
This technique has been used primarily in the context of recurrent neural networks.
Ensemble methods
BAGGING
❑Bagging (short for bootstrap aggregating) is a technique for reducing generalization error by
combining several models
❑The idea is to train several different models separately, then have all of the models vote on the
output for test examples.
❑This is an example of a general strategy in machine learning called model averaging. Techniques
employing this strategy are known as ensemble methods.
DROPOUT