06-03-2026
Optimization
Algorithms
Exponentially
weighted averages
1
06-03-2026
2
06-03-2026
Temperature in Nagpur
𝜃1 = 26°C
temperature
𝜃2 = 28°C
𝜃3 = 30°C
.
.
.
𝜃180 = 40°C
𝜃181 = 45°C days
.
.
.
3
06-03-2026
Exponentially weighted averages
temperature
days
Optimization
Algorithms
Understanding
exponentially
weighted averages
4
06-03-2026
Exponentially weighted averages
𝑣𝑡 = 𝛽𝑣𝑡−1 + (1 − 𝛽)𝜃𝑡
temperature
days
5
06-03-2026
Exponentially weighted averages
𝑣𝑡 = 𝛽𝑣𝑡−1 + (1 − 𝛽)𝜃𝑡
𝑣100 = 0.9𝑣99 + 0.1𝜃100
𝑣99 = 0.9𝑣98 + 0.1𝜃99
𝑣98 = 0.9𝑣97 + 0.1𝜃98
…
Implementing exponentially weighted
averages
𝑣0 = 0
𝑣1 = 𝛽𝑣0 + (1 − 𝛽) 𝜃1
𝑣2 = 𝛽𝑣1 + (1 − 𝛽) 𝜃2
𝑣3 = 𝛽𝑣2 + (1 − 𝛽) 𝜃3
…
6
06-03-2026
Optimization
Algorithms
Bias correction
in exponentially
weighted average
Bias correction
temperature
days
𝑣𝑡 = 𝛽𝑣𝑡−1 + (1 − 𝛽)𝜃𝑡
7
06-03-2026
Optimization
Algorithms
Gradient descent
with momentum
Gradient descent example
8
06-03-2026
Implementation details
On iteration 𝑡:
Compute 𝑑𝑊, 𝑑𝑏 on the current mini-batch
𝑣𝑑𝑊 = 𝛽𝑣𝑑𝑊 + 1 − 𝛽 𝑑𝑊
𝑣𝑑𝑏 = 𝛽𝑣𝑑𝑏 + 1 − 𝛽 𝑑𝑏
𝑊 = 𝑊 − 𝛼𝑣𝑑𝑊 , 𝑏 = 𝑏 − 𝛼𝑣𝑑𝑏
Hyperparameters: 𝛼, 𝛽 𝛽 = 0.9
9
06-03-2026
10
06-03-2026
Optimization
Algorithms
RMSprop
11
06-03-2026
RMSprop (Root Mean Square Propagation) Optimizer
• RMSprop is an adaptive learning rate optimization algorithm proposed by Geoffrey
Hinton in his Coursera lecture on Neural Networks. It is designed to solve problems with
vanishing or exploding gradients, particularly in deep neural networks. It helps in
maintaining a balanced and stable learning rate throughout the training process.
• Why Do We Need RMSprop?
Traditional Stochastic Gradient Descent (SGD) updates parameters using a fixed
learning rate. However, in deep networks, this can lead to:
Large oscillations in gradients, causing slow convergence.
Diminishing or exploding gradients, especially in non-convex loss functions.
RMSprop addresses these issues by scaling the learning rate adaptively for each
parameter based on recent gradient magnitudes.
RMSprop
12
06-03-2026
Working Mechanism of RMSprop
• RMSprop modifies the standard gradient descent update rule by introducing a moving
average of squared gradients.
1) Compute the gradient of the loss function w.r.t. weights:
2) Maintain an exponentially decaying average of squared gradients:
3) Update parameters using a normalized learning rate:
RMSprop (Root Mean Square Propagation) Optimizer
• How Does This Help?
• Adaptive Learning Rate: The update step is inversely proportional to
the moving average of squared gradients. This means:
• Smaller learning rates for parameters with large gradients (prevents
divergence).
• Larger learning rates for parameters with small gradients (prevents stagnation).
• Stable Training: RMSprop reduces oscillations and speeds up
convergence.
13
06-03-2026
RMSprop (Root Mean Square Propagation) Optimizer
• Key Features of RMSprop:
Adaptive Learning Rate: Adjusts learning rates dynamically per
parameter.
Reduces Oscillations: Works well with mini-batches, stabilizing
updates.
Good for Non-Stationary Problems: Works well in tasks where the
data distribution changes over time.
Prevents Divergence: Unlike SGD, RMSprop normalizes gradients
for stable learning.
RMSprop (Root Mean Square Propagation) Optimizer
• Advantages of RMSprop:
Performs well in deep networks (e.g., CNNs, RNNs, LSTMs).
Less sensitive to the choice of learning rate compared to SGD.
Prevents vanishing/exploding gradients effectively.
Faster convergence compared to vanilla SGD.
• Disadvantages of RMSprop:
❌ Lack of Momentum: Unlike Adam, RMSprop does not use past
velocity.
❌ Hyperparameter Sensitivity: The decay rate β can affect performance
significantly.
14
06-03-2026
RMSprop
Optimization
Algorithms
Adam optimization
algorithm
15
06-03-2026
Adam (Adaptive Moment Estimation) Optimizer
• Adam (Adaptive Moment Estimation) is one of the most widely used optimization
algorithms in deep learning. It combines the benefits of both RMSprop and
Momentum to provide faster and more stable convergence.
• Adam dynamically adjusts the learning rate for each parameter based on:
Momentum (first moment) – the moving average of past gradients.
RMSprop-style scaling (second moment) – the moving average of
squared gradients.
• It works well across various deep learning architectures, including CNNs, RNNs,
and transformers, making it the go-to optimizer for many tasks.
Adam (Adaptive Moment Estimation) Optimizer
• Why is Adam Needed?
• Standard Stochastic Gradient Descent (SGD) has several issues:
It uses a fixed learning rate, which can be too large (causing divergence) or too
small (leading to slow training).
It does not account for previous gradients (momentum) or gradient variance
(adaptive scaling).
Deep networks can suffer from vanishing or exploding gradients, slowing down
training.
• Adam addresses these problems by adapting learning rates for each
parameter dynamically, leading to faster and more stable training.
16
06-03-2026
Adam (Adaptive Moment Estimation) Optimizer
• How Does Adam Work?
• Adam maintains two moving averages for each parameter:
First moment estimate (Momentum term): Tracks the average of past
gradients.
Second moment estimate (Variance term, like RMSprop): Tracks the
average of squared gradients to normalize updates.
Adam (Adaptive Moment Estimation) Optimizer
• Step-by-Step Working of Adam
• Step 1: Compute the Gradient
• Step 2: Compute the Biased Moment Estimates
1) First Moment Estimate (Momentum Term)
2) Second Moment Estimate (RMSprop-like Variance Term)
• Step 3: Apply Bias Correction
• Step 4: Update the Parameters
17
06-03-2026
Adam (Adaptive Moment Estimation) Optimizer
• Key Features of RMSprop:
Combines Momentum and RMSprop: Uses past gradients and
adaptive learning rates.
Adaptive Learning Rate: Each parameter gets its own learning rate.
Handles Noisy Gradients Well: Works well in mini-batch and online
learning settings.
Fast Convergence: Usually converges faster than SGD and
RMSprop.
Adam (Adaptive Moment Estimation) Optimizer
• Advantages of Adam:
Works well with large datasets and deep networks.
Stable training, even with noisy gradients (mini-batch training).
Less sensitive to initial hyperparameter choices.
Efficient memory usage (only keeps first and second moments per
parameter).
• Disadvantages of Adam:
❌ Can lead to overfitting due to aggressive learning updates.
❌ Sometimes struggles with generalization (SGD often gives better
final performance).
❌ Learning rate scheduling is still needed in some cases.
18
06-03-2026
Adam (Adaptive Moment Estimation) Optimizer
• SGD is simple but requires careful tuning of the learning rate.
• Momentum accelerates training but doesn’t adapt learning rates.
• RMSprop adapts learning rates but lacks momentum.
• Adam combines the best of both worlds for fast, stable optimization.
Adam optimization algorithm
19
06-03-2026
Hyperparameters choice:
20