UNIT–II: TRAINING DEEP NEURAL NETWORKS

Backpropagation with Computational Graph:

Computational graphs are a fundamental tool in deep learning and machine learning, essential
for representing and managing complex computations. They play a crucial role in
understanding and implementing backpropagation, which is vital for training neural networks.
Here’s an overview of computational graphs and their significance in backpropagation:
What is a Computational Graph?
A computational graph is a directed graph where:
• Nodes represent operations (such as addition or multiplication) or variables (such as
inputs or weights).
• Edges represent the flow of data (tensors) between these operations.
This graph captures the sequence of operations and their dependencies, allowing for a clear
visualization and computation of both forward and backward passes in a neural network.
Why Use Computational Graphs?
1. Organize Complex Computations:
o Neural networks involve numerous layers and operations, making them
complex to manage. Computational graphs offer a structured approach to break
down these computations into manageable components.
2. Efficient Gradient Computation:
o Backpropagation depends on the efficient calculation of gradients.
Computational graphs facilitate automatic differentiation by systematically
applying the chain rule across the graph to compute gradients for all parameters.
3. Modularity and Reusability:
o Graphs allow for modular model construction. Different parts of the model can
be represented as subgraphs and reused, making it easier to experiment with and
build models.
4. Optimization:
o Computational graphs can be optimized to enhance performance. For example,
common subexpressions can be shared, and computations can be tailored for
specific hardware (such as GPUs).
5. Debugging and Visualization:

Dr. G. Vennila, Assistant Professor, AIML 25 | P a g e

 o Graphs provide a visual representation of the computation process. This helps
in understanding the model, debugging issues, and ensuring that both the
forward and backward passes are correctly implemented.
How Computational Graphs Work
1. Forward Pass:
o Description: Computes the network's output by propagating inputs through the
graph, performing the operations defined by the nodes, and following the edges.
o Implementation: Evaluates each node according to the operation it represents,
using inputs and intermediate results from connected nodes.
2. Backward Pass (Backpropagation):
o Description: Computes the gradients of the loss with respect to each parameter
by propagating gradients backward through the graph. This involves applying
the chain rule of calculus to the operations represented by the nodes.
o Implementation: Traverses the graph in reverse order, starting from the loss
node and moving toward the input nodes, updating gradients at each node based
on the gradients from subsequent nodes.
Advantages
1. Automatic Differentiation:
o Computational graphs support automatic differentiation, which is crucial for
efficiently computing gradients required for training neural networks.
2. Flexibility:
o They allow for dynamic model definitions and modifications, making them
suitable for complex and variable neural network architectures.
3. Scalability:
o Optimizations and parallelization strategies can be applied to the graph to
enhance computational efficiency, particularly for large models and datasets.
Backpropagation with Post-Activation and Pre-Activation Variables
In neural networks, backpropagation involves computing gradients to update the model's
parameters. The process can vary depending on whether gradients are computed with respect
to post-activation or pre-activation variables. Understanding these concepts is crucial for the
efficient implementation and optimization of neural networks.

Dr. G. Vennila, Assistant Professor, AIML 26 | P a g e

Dr. G. Vennila, Assistant Professor, AIML 27 | P a g e
Gradient based Strategies:
Learning Rate Decay
Learning rate decay is a technique used to adjust the learning rate during neural network
training. It involves gradually reducing the learning rate as training progresses. The strategy is
to start with a relatively high learning rate to accelerate convergence and then decrease it for
more precise adjustments as the model nears the optimal solution.

Why Use Learning Rate Decay?

• Initial Fast Learning: A larger learning rate at the beginning of training allows the
optimizer to make substantial updates quickly, which speeds up learning and helps
avoid shallow local minima.

• Later Precise Adjustments: As the model approaches the minimum, a smaller learning
rate enables finer adjustments, reducing the risk of overshooting and promoting a
smoother convergence.

Without decay, a high learning rate might cause the model to oscillate around the minimum,
while starting with a low learning rate could make training excessively slow.

Dr. G. Vennila, Assistant Professor, AIML 28 | P a g e

This leads to a slower reduction compared to exponential decay.

• The learning rate first increases and then decreases over the course of training. This
strategy can help speed up convergence and improve generalization.

Example

Let's say you're training a neural network for image classification using gradient
descent. You want to use step decay to adjust the learning rate as the training
progresses. You decide to reduce the learning rate by a factor of 0.5 every 10 epochs.

• Initial Learning Rate (LR): 0.1

• Decay Factor: 0.5

• Step Size (epochs): 10

Dr. G. Vennila, Assistant Professor, AIML 29 | P a g e

Momentum-based learning:

Momentum-based learning is an optimization technique used to enhance the performance and

convergence of training deep learning models. It refines the gradient descent algorithm by
incorporating past gradient information into the current update.

Concept of Momentum-Based Learning

Momentum-based learning involves adding a fraction of the previous gradient update to the
current gradient update. This approach smooths the optimization path, reduces oscillations, and
accelerates convergence, particularly in areas with small gradients or narrow valleys.

Dr. G. Vennila, Assistant Professor, AIML 30 | P a g e

Benefits of Momentum-Based Learning

Accelerated Convergence:

Momentum helps accelerate gradient vectors in the correct direction, leading to faster
convergence, especially in regions with small gradients or along long, narrow valleys.
Reduced Oscillations:
By incorporating past gradients, momentum-based learning minimizes oscillations and
smooths the path towards the minimum.

Dr. G. Vennila, Assistant Professor, AIML 31 | P a g e

Improved Performance:
Momentum aids in escaping local minima and navigating complex loss landscapes more
effectively.

Parameter-Specific Learning Rates (PSLR)

The learning rate is a critical hyperparameter that controls the extent of weight adjustments
based on the gradient of the loss function. A smaller learning rate slows learning but can lead
to precise convergence, while a larger learning rate speeds up learning but may overshoot
optimal solutions.

Traditionally, a single global learning rate is used for all model parameters. However, different
parts of a neural network may benefit from distinct learning rates. This is where Parameter-
Specific Learning Rates (PSLR) come into play, involving assigning different learning rates to
various subsets of parameters.

Why Use Parameter-Specific Learning Rates?

Fine-Tuning Pretrained Models:

When fine-tuning a pretrained model, lower layers (with general features) are often frozen,
while higher layers (specific to the new task) are adjusted. Using lower learning rates for
pretrained layers prevents disrupting learned features, while higher learning rates can be
applied to newly added layers.

Different Learning Rates for Different Layers:

In deep networks, lower layers represent general features and may require smaller learning
rates for stability, whereas higher layers, being more task-specific, might benefit from larger
learning rates for quicker adaptation.

Layer Types with Different Dynamics:

Convolutional, recurrent, and fully connected layers have different roles and may require
different learning rates. For example, recurrent layers like LSTMs or GRUs may need smaller
learning rates to handle sensitive updates compared to convolutional or fully connected layers.

Regularization:
Combining parameter-specific learning rates with parameter-specific regularization can

Dr. G. Vennila, Assistant Professor, AIML 32 | P a g e

provide optimized and controlled learning dynamics, particularly when different parts of the
network have varying levels of regularization.

Complex Architectures:
In complex architectures like ResNets, GANs, transformers, or multi-branch networks,
different branches or components may need specific learning rates. PSLR helps manage the
learning dynamics of these complex interactions more effectively.

How Parameter-Specific Learning Rates Work

Parameter-specific learning rates involve grouping parameters and assigning each group its
own learning rate. This approach allows different parameter groups to be updated at different
rates during training.

Example Implementation:

Layer 1: Learning rate of 0.001

Layer 2: Learning rate of 0.01

In this setup, Layer 1 (possibly pretrained) is updated slowly, while Layer 2 adapts more
rapidly, allowing for tailored training dynamics.

Comparison with Global Learning Rate

Dr. G. Vennila, Assistant Professor, AIML 33 | P a g e

Aspect Global Learning Rate Parameter-Specific Learning Rate
Single learning rate for all Different learning rates for different
Learning Rate
parameters parameters
Less flexible; same rate applied More flexible; adapts rates based on
Flexibility
uniformly parameter needs
May be slower if parameters Can converge faster by optimizing
Convergence
have different sensitivities learning rates for each parameter
Complexity Simple to implement More complex to manage and tune
Handling Different Automatically adapts to different
May require manual adjustment
Scales parameter scales

Gradient Clipping

Gradient clipping is a technique used to address the problem of exploding gradients during the
training of deep neural networks. Exploding gradients occur when gradients become
excessively large during backpropagation, leading to instability in the training process. This
can result in excessively large updates to the model parameters, causing divergence and
training failure.

Purpose:
The goal of gradient clipping is to cap the gradients to a predefined range or threshold to ensure
they do not exceed a certain size. This stabilizes and controls the training process, particularly
in deep or recurrent neural networks (RNNs), where exploding gradients are more common.

Types of Gradient Clipping:

1. Gradient Clipping by Value

In this approach, each component of the gradient vector is clipped if it exceeds a specified
range, typically defined by a minimum and maximum value. The idea is to limit the magnitude
of each gradient element to prevent any from becoming too large.

How it Works:

Dr. G. Vennila, Assistant Professor, AIML 34 | P a g e

 o Define a minimum threshold (clip_value_min) and a maximum threshold
(clip_value_max).
o If any gradient component exceeds clip_value_max, it is set to clip_value_max.
o Conversely, if any gradient component is below clip_value_min, it is set to
clip_value_min.

Example:

• Let’s assume a gradient vector g = [0.5, -1.2, 3.0, -4.5, 2.0].

• If we clip the values to lie between -2 and 2, the clipped gradient vector will become g
= [0.5, -1.2, 2.0, -2.0, 2.0].

• Gradient Component (gi): Each individual element of the gradient vector, which
corresponds to the gradient with respect to a specific parameter in the model. This
clipping is applied separately to each component.
• clip_value_max: The upper bound for gradient values. If any gradient component
exceeds this maximum value, it is clipped to clip_value_max.
• clip_value_min: The lower bound for gradient values. If any gradient component is
below
• clip_value_min: This is the lower bound for the gradient value. Any gradient
component smaller than this will be clipped to this minimum value.

Dr. G. Vennila, Assistant Professor, AIML 35 | P a g e

Gradient Clipping by Norm

Gradient Clipping by Norm is a technique used to stabilize training by rescaling the entire
gradient vector if its magnitude exceeds a predefined threshold.

How It Works:

1. Compute the Norm: Calculate the norm (typically the Euclidean or L2 norm) of the
gradient vector.
2. Compare with Threshold: If the norm of the gradient vector exceeds the threshold
(clip_norm), rescale the entire vector.
3. Rescaling: Adjust the gradient vector so that its norm equals clip_norm, while
maintaining its direction.

Advantages:

• Preserves Direction: The gradient's direction is maintained, which helps in preserving

the gradient's relative scales and can lead to more stable convergence.
• Effective for Deep Models: Particularly useful for models where gradients can vary
greatly in magnitude, such as RNNs or very deep networks.

Disadvantages:

Dr. G. Vennila, Assistant Professor, AIML 36 | P a g e

 • Complexity: Slightly more complex to implement than clipping by value, as it involves
computing norms and rescaling vectors.
• Potential for Inefficiency: If norms are frequently close to or exceed the threshold, it
may lead to rescaling more often, which could affect training dynamics.

Dr. G. Vennila, Assistant Professor, AIML 37 | P a g e

gclipped=[3.0,4.0]

Dr. G. Vennila, Assistant Professor, AIML 38 | P a g e

Polyak Averaging

Polyak Averaging is a method employed to enhance the convergence and stability of

optimization algorithms, especially in iterative methods such as gradient descent. This
technique involves computing and using the average of model parameters over multiple
iterations to obtain a more stable and potentially improved solution.

Concept of Polyak Averaging:

1. Definition:

o Polyak Averaging entails calculating an average of the model's parameters

across various iterations. Instead of relying solely on the parameters from the
final iteration, this average is used for making predictions or as the final model.

Dr. G. Vennila, Assistant Professor, AIML 39 | P a g e

Benefits of Polyak Averaging
1. Improves Stability:
o By averaging parameters over multiple iterations, the effects of noisy updates
or fluctuations are mitigated, leading to more stable solutions.
2. Reduces Variance:
o This technique helps smooth out noise in the optimization process, potentially
resulting in a more accurate and reliable final model.
Applications of Polyak Averaging
1. Stochastic Gradient Descent (SGD):
o Frequently employed with SGD to enhance the stability of the final outcome.
2. Neural Networks:
o Utilized to improve the performance and stability of models trained through
iterative optimization techniques.
Local and Spurious Minima
Local Minima
1. Definition:
o A local minimum is a point in the loss landscape where the loss function has a
lower value than at nearby points but is not necessarily the lowest value across
the entire landscape.
2. Characteristics:
o Surrounding Area: The loss value is lower compared to neighboring points,
though it may not be the global minimum.
o Convergence: Optimization algorithms may become trapped in local minima
without mechanisms to escape.

Dr. G. Vennila, Assistant Professor, AIML 40 | P a g e

Spurious Minima
1. Definition:
o A spurious minimum is a local minimum that does not accurately reflect the
global structure of the loss landscape, often arising from noise or irregularities
in the loss surface.
2. Characteristics:
o Non-Optimal: Spurious minima are generally not true solutions and may not
lead to models with good generalization.
o Irregular Landscapes: Common in complex loss landscapes with significant
irregularities.

Dr. G. Vennila, Assistant Professor, AIML 41 | P a g e