MODULE 3

SYLLABUS
Introduction to deep learning, Deep feed forward network,
Training deep models, Optimization techniques - Gradient
Descent (GD), GD with momentum, Nesterov accelerated
GD, Stochastic GD, AdaGrad, RMSProp, Adam.
Regularization Techniques - L1 and L2 regularization, Early
stopping, Dataset augmentation, Parameter sharing and
tying, Injecting noise at input, Ensemble methods, Dropout,
Parameter initialization
Introduction to deep learning, Deep feed
forward network,
A deep neural network is a neural network with atleast two hidden layers.
Deep neural networks use sophisticated mathematical modeling to process data in
different [Link] machine learning algorithms are linear, deep learning
algorithms are stacked in a hierarchy.
Deep learning creates many layers of neurons, attempting to learn structured
representation, layer by layer
The goal of a feedforward network is to approximate some function f ∗. For example,for a
classifier, y = f ∗(x) maps an input x to a category y.
A feedforward network defines a mapping y = f (x; θ) and learns the value of the
parameters θ that result in the best function approximation.
These models are called feedforward because information flows through the function
being evaluated from x, through the intermediate computations used to define f, and
finally to the output y. There are no feedback connections in which outputs of the model
are fed back into itself.
Feedforward networks are of extreme importance to machine learning [Link]
form the basis of many important commercial applications. Forexample, the
convolutional networks used for object recognition from photos are aspecialized kind of
feedforward network.
Feedforward neural networks are called networks because they are typically
represented by composing together many different functions.
For example, we might have three functions f (1), f (2), and f (3) connected in a chain, to
form f(x) = f(3)(f (2)(f(1) (x ))). This chain structure is most commonly used structure of
neural networks. In this case, f (1) is called the first layer of the network called input
layer used to feed the input into the network; f (2) is called the second layer called
hidden layer used to train the neural network, and so on. The final layer of a
feedforward network is called the output layer that provides the output of the network.
The overall length of the chain gives the depth of the model and width of the model is
number of neurons in the input layer. It is from this terminology that the name “deep
learning” arises
GRADIENT DESCENT
❑Gradient Descent is known as one of the most commonly used optimization algorithms to
minimize a function by optimizing the parameters.
❑The goal of gradient descent is to find the set of weights (or coefficients) that minimize the loss
function. The algorithm works by iteratively adjusting the weights in the direction of the
steepest decrease in the loss function.
❑The basic idea of gradient descent is to start with an initial set of weights and update them in
the direction of the negative gradient of the loss function.
❑The gradient is a vector of partial derivatives that represents the rate of change of the loss
function with respect to the weights. By updating the weights in the direction of the negative
gradient, the algorithm moves towards a minimum of the loss function.
❑The learning rate is a hyperparameter that determines the size of the step taken in the weight
update. A small learning rate results in a slow convergence, while a large learning rate can lead
to overshooting the minimum and oscillating around the minimum.
❑It’s important to choose an appropriate learning rate that balances the speed of convergence
and the stability of the optimization.
Disadvantages of GD
❑A constant learning rate is not desirable because
❑A lower learning rate used early on will cause the algorithm to take too long to
come even close to an optimal solution.
❑On the other hand, a large initial learning rate will allow the algorithm to come
reasonably close to a good solution at first; however, the algorithm will then
oscillate around the point for a very long time, or diverge in an unstable way, if the
high rate of learning is maintained. In either case, maintaining a constant learning
rate is not ideal.
❑it can bounce around the search space based on the gradient. This bouncing
effect can cause the algorithm to converge slowly or to get stuck in a local
minimum, rather than finding the global minimum.
Momentum based gradient descent
❑GD takes a lot of time to converge.
❑Momentum helps the optimization process retain speed in flat regions of the loss surface and
avoid local optima.
❑Momentum involves adding an additional hyperparameter that controls the amount of history
(momentum) to include in the update equation, i.e. the step to a new point in the search space.
❑The value for the hyperparameter is defined in the range 0.0 to 1.0 and often has a value close to
1.0, such as 0.8, 0.9, or 0.99. A momentum of 0.0 is the same as gradient descent without
momentum.
❑With momentum-based descent, the learning is accelerated, because one is
generally moving in a direction that often points closer to the optimal solution and
the useless “sideways” oscillations are muted.
❑The basic idea is to give greater preference to consistent directions over multiple
steps, which have greater importance in the descent.
❑ This allows the use of larger steps in the correct direction without causing
overflows or “explosions” in the sideways direction.
❑As a result, learning is accelerated.
Analogy
❑A marble will overshoot when it is allowed to roll down a bowl.
❑The momentum-based method will generally perform better because the marble
gains speed as it rolls down the bowl; the quicker arrival at the optimal solution
❑The marble’s gathering of speed helps it efficiently navigate flat regions of the
loss surface but it overshoots.
.
Disadvantages
[Link]: Momentum-based gradient descent can overshoot the minimum
of the cost function and lead to oscillations around the minimum. This can
happen if the momentum term is too high or if the learning rate is too
[Link] avoids local optima.
[Link] to the initial conditions: The momentum term can cause the algorithm
to converge to different minima depending on the initial conditions. This can
make the algorithm less reliable, especially if the cost function has several local
minima.
Nesterov Accelerated Gradient Descent
❑The Nesterov momentum algorithm is a modification of the traditional momentum
method used in gradient descent optimization algorithms.
❑In Nesterov momentum, the gradients are computed at a point that would be reached
after executing a discounted version of the previous step again, which is the momentum
portion of the current step.
❑This point is obtained by multiplying the previous update vector with the friction
parameter and then computing the gradient.
❑The idea behind Nesterov momentum is that by using the corrected gradient, which
takes into account the momentum portion of the update, the algorithm can make more
informed updates and move towards the minimum of the cost function more quickly.
❑This is particularly useful when the cost function is curved, as the corrected gradient can
help to avoid overshooting the minimum.
❑In the previous analogy of the rolling marble, such an approach will start applying the
“brakes” on the gradient-descent procedure when the marble starts reaching near the
bottom of the bowl, because the lookahead will “warn” it about the reversal in gradient
direction.
Stochastic gradient descent algorithm
❑In Gradient Descent optimization, we compute the cost gradient based on the complete
training set; hence, it is called batch gradient descent.
❑In case of very large datasets, using Gradient Descent can be quite costly since we are
only taking a single step for one pass over the training set – thus, the larger the training
set, the slower our algorithm updates the weights and the longer it may take until it
converges to the global cost minimum.
❑In Stochastic Gradient Descent ,we don’t accumulate the weight updates as we’ve
seen above for Gradient Descent:
❑Instead of computing the sum of all gradients, stochastic gradient descent selects an
observation uniformly at random
AdaGrad
❑Gradient of f(x) w.r.t to a particular weight is clearly dependent on its corresponding input.
❑ If there are n points, we can just sum the gradients over all the n points to get the total
gradient
❑But what would happen if the feature x2 is very sparse (i.e., if its value is 0 for most inputs)?
❑ It is fair to assume that ∇w2 will be 0 for most inputs and hence w2 will not get enough
updates.
❑To make sure updates happen even when a particular input is sparse, we have a different
learning rate for each parameter.
❑AdaGrad algorithm, which adjusts the learning rate based on the sparsity of the data to ensure
that parameters with low frequency receive higher learning rates.
❑ Additionally, AdaGrad ensures that frequently updated parameters have their learning rates
decreased over time to prevent overshooting the optimal solution.
❑However, it is noted that removing the square root from the denominator of the algorithm may
negatively impact its effectiveness.
❑ Furthermore, it is mentioned that the RMSProp algorithm can help prevent the decay of the
effective learning rate for frequently updated parameters.
AdaGrad got stuck when it was close to convergence, it was no longer able to move in the
vertical (b) direction because of the decayed learning rate.
RMSProp overcomes this problem by being less aggressive on the decay.
RMSProp also tries to dampen the oscillations, but in a different way than
momentum.
RMS prop also takes away the need to adjust learning rate, and does it
automatically.
More so, RMSProp choses a different learning rate for each parameter.
In the first equation, we compute an exponential average of the square of the
gradient.
we multiply the exponential average computed till the last update with a
hyperparameter, represented by the greek symbol nu. We then multiply the
square of the current gradient with (1 - nu). We then add them together to get
the exponential average till the current time step.
Then in the second equation, we decided our step size. We move in the
direction of the gradient, but our step size is affected by the exponential
average. We chose an initial learning rate eta, and then divide it by the
average. In our case, since the average of w1 is much much larger than w2, the
learning step for w1 is much lesser than that of w2. Hence, this will help us
avoid bouncing between the ridges, and move towards the minima.
Adam
Adam or Adaptive Moment Optimization algorithms combines the heuristics
of both Momentum and RMSProp. Here are the update equations.
Concepts of Regularization, L1 and L2
regularization, Early stopping
Bias / Variance Trade-off
Make sure the distribution of dev/test set is same
as training set
◦ Divide the training, dev and test sets in such a way
that their distribution is similar
◦ Skip the test set and validate the model using the
dev set only

▪ We want our model to be just right, which

means having low bias and low variance.
▪ Overfitting: If the dev set error is much
more than the train set error, the model is
overfitting and has a high variance
▪ Underfitting: When both train and dev set
errors are high, the model is underfitting
and has a high bias

Deep Learning
Image Source: [Link]
Overfitting in Deep Neural Nets
Deep neural networks contain multiple non-linear hidden
layers
◦ This makes them very expressive models that can learn very
complicated relationships between their inputs and outputs.
◦ In other words, model learns even the tiniest details present in
the data.

But with limited training data, many of these complicated

relationships will be the result of sampling noise
◦ So they will exist in the training set but not in real test data even
if it is drawn from the same distribution.
◦ So after learning all the possible patterns it can find, the model
tends to perform extremely well on the training set but fails to
produce good results on the dev and test sets.

Deep Learning
Regularization
Regularization is:
◦ “any modification to a learning algorithm to reduce its
generalization error but not its training error”
◦ Reduce generalization error even at the expense of
increasing training error
◦ E.g., Limiting model capacity is a regularization method

Deep Learning
Source: [Link]
L2 Regularization
❑Increasing or decreasing the value of λ reduces the softness of the penalty.
❑One advantage of this type of parameterized penalty is that one can tune this parameter for optimum
performance on a portion of the training data set that is not used for learning the parameters
DATA AUGMENTATION
❑A common trick to reduce overfitting in convolutional neural networks is the idea of
data augmentation.
❑In data augmentation, new training examples are generated by using transformations on
the original examples.
❑Image processing is one domain to which data augmentation is well-suited.
❑This is because many transformations such as translation, rotation, patch extraction,
and reflection, do not fundamentally change the properties of the object in an image.
❑However, they do increase the generalization power of the data set when trained with
the augmented data set. For example, if a data set is trained with mirror images and
reflected versions of all the bananas in it, then the model is able to better recognize
bananas in different orientations.
❑Many of these forms of data augmentation require very little computation.
❑they can be created at training time, when an image is being processed. For
example, while processing an image of a banana, it can be reflected into a
modified banana at training time.
❑Similarly, the same banana might be represented in somewhat different color
intensities in different images, and therefore it might be helpful to create
representations of the same image in different color intensities. In many cases,
creating the training data set using image patches can be helpful.
❑Although most data augmentation methods are quite efficient, some forms of
transformation that use principal component analysis (PCA) can be more
expensive.
❑ PCA is used in order to change the color intensity of an image. If the
computational costs are high, it becomes important to extract the images up
front and store them.
❑The basic idea here is to use the 3 × 3 covariance matrix of each pixel value and
compute the principal [Link], Gaussian noise is added to each
principal component with zero mean and variance of 0.01.
❑This noise is fixed over all the pixels of a particular image. The approach is
dependent on the fact that object identity is invariant to color intensity and
illumination. It is reported in that data set augmentation reduces error rate by
1%. One must be careful not to apply data augmentation blindly without regard
to the data set and application at hand.
EXAMPLES
[Link] Stopping

Another common form of regularization is early stopping, in which the gradient descent is ended
after only a few iterations.
One way to decide the stopping point is by holding out a part of the training data, and then
testing the error of the model on the held-out set.
The gradient-descent approach is terminated when the error on the held-out set begins to rise.
Early stopping essentially reduces the size of the parameter space to a smaller neighborhood
within the initial values of the parameters.
From this point of view, early stopping acts as a regularizer because it effectively restricts the
parameter space

DEPT OF EI,SJCET,PALAI 48
The model tries to chase the loss function crazily on the training data, by tuning the parameters.
Now, we keep another set of data as the validation set and as we go on training, we keep a record
of the loss function on the validation data, and when we see that there is no improvement on the
validation set, we stop, rather than going all the epochs.
This strategy of stopping early based on the validation set performance is called Early Stopping.

DEPT OF EI,SJCET,PALAI 49
ADDING NOISE TO INPUT
❑The addition of noise to the input has connections with penalty-based regularization.
❑It can be shown that the addition of an equal amount of Gaussian noise to each input is equivalent to
Tikhonov regularization of a single-layer neural network with an identity activation function .
❑Also noise applied to the inputs is a dataset augmentation strategy.
❑Another way that noise has been used in the service of regularizing models is by adding it to the weights.
This technique has been used primarily in the context of recurrent neural networks.
Ensemble methods
BAGGING
❑Bagging (short for bootstrap aggregating) is a technique for reducing generalization error by
combining several models
❑The idea is to train several different models separately, then have all of the models vote on the
output for test examples.
❑This is an example of a general strategy in machine learning called model averaging. Techniques
employing this strategy are known as ensemble methods.
DROPOUT