Adaptive Learning Rates in Deep Learning

Part 2: From Momentum to Adam (and its Variants)

Mohamed Elsyed
AI & Intelligent Systems Specialist
September 2025

Inspired by multiple resources including Deep Learning lectures at Carnegie Mellon University
(CMU), the book “Hands-On Machine Learning”, and online tutorials.
Rewritten and explained in my own words with additional practical insights for better
understanding and application in real-world deep learning problems.
From Second Order to Adaptive Optimization
In the previous section, we saw that second-order methods, despite their theoretical
appeal, are computationally expensive and impractical for deep neural networks. As a
result, we return to conventional first-order Gradient Descent.

𝑊 → 𝑊 − 𝜂 ∇𝑤 𝐸(𝑊)

However, plain Gradient Descent is far from ideal. It suffers mainly from two issues:

1. Vanishing and exploding gradients – when gradients vanish, learning becomes

extremely slow; when they explode, parameters diverge.

2. Global learning rate – using the same step size for all parameters is inefficient, as
some weights may require larger updates while others need smaller ones.

Momentum and Nesterov Accelerated Gradient (NAG) were introduced to partially address
these problems by smoothing updates and reducing oscillations. Yet, they still rely on a
fixed global learning rate and cannot fully solve vanishing or exploding gradients.

These limitations motivate the transition to adaptive optimization methods, which assign
each parameter its own learning rate that adjusts dynamically during training. Algorithms
such as Adagrad, RMSProp, and Adam provide both stability and efficiency, and they have
become the backbone of modern deep learning optimization.

Momentum
Idea
Momentum can be viewed as a bowling ball rolling down a gentle slope: At first, the ball
moves slowly, but as it accumulates momentum it begins to roll faster until it reaches a
terminal velocity (due to friction or resistance).

In contrast, plain Gradient Descent updates the weights using only the current gradient:

𝑊 → 𝑊 − 𝜂 ∇𝑤 𝐸(𝑊)

This approach ignores past gradients. If the local gradient is small, progress becomes
extremely slow.

Momentum, however, considers both the current and previous gradients. The gradient is
treated as an acceleration, not just a speed, which allows the optimizer to move faster and
more smoothly.
Update Rule
The algorithm introduces a new hyperparameter β, the momentum coefficient (0 < β <
1 ), typically set to 0.9:

∆𝑤 𝑘 = 𝛽∆𝑤 𝑘−1 − 𝜂∇𝑤 𝐸(𝑤 𝑘−1 )

𝑤 𝑘 = 𝑤 𝑘−1 + ∆𝑤 𝑘

• 𝜂: Learning rate.
• 𝛽: controls the friction (0 = high friction, 1 = no friction).
• ∆𝑤 𝑘 : update step at iteration k.

Properties
Running average of gradients:

• Steps grow longer in directions where gradients keep the same sign.

• Steps grow shorter where gradients flip frequently.

Terminal velocity (the maximum size of the weight updates): if the gradient is constant, the
maximum update magnitude equals:
𝜂
∇𝑤 𝐸(𝑊)
𝛽−1

For 𝛽 = 0.9, then the terminal velocity is equal to 10 times the gradient times the learning rate.
So, momentum can move up to 10× faster than plain GD. This allows momentum to escape from
plateaus much faster than gradient descent.

At any iteration, to compute the

current step:

1- First take a step against the gradient at

the current location: 𝜂∇𝑤 𝐸(𝑤 𝑘−1 )

2- Then we add a scaled version of

previous step: 𝛽∆𝑤 𝑘−1

Intuition with Example

Initialization: 𝑤 0 , ∆𝑤 0 = 0
Step 1:

∆𝑤 1 = 𝛽∆𝑤 0 − 𝜂∇𝑤 𝐸(𝑤 0 )

𝑤 1 = 𝑤 0 + ∆𝑤 1

Step 2:

∆𝑤 2 = 𝛽∆𝑤 1 − 𝜂∇𝑤 𝐸(𝑤 1 )

𝛽2 < 𝛽 < 1
= 𝛽(𝛽∆𝑤 0 − 𝜂∇𝑤 𝐸(𝑤 0 )) − 𝜂∇𝑤 𝐸(𝑤 1 )
(0.9)2 < (0.9) < 1
2 0 0) 1)
= 𝛽 ∆𝑤 − 𝛽 𝜂∇𝑤 𝐸(𝑤 − 𝜂∇𝑤 𝐸(𝑤

Thus, the update depends mainly on the current gradient, but past gradients also contribute
through the β factor.

Advantages
Momentum helps accelerate convergence in valleys and plateaus, allowing the optimizer to roll
past local optima instead of getting stuck. It also reduces the zigzagging effect that often occurs
in narrow valleys and makes the updates smoother by averaging past gradients rather than
relying only on the current one.

Note on Overshooting
Because of the momentum effect, the optimizer may overshoot the minimum, then come back,
overshoot again, and keep oscillating several times before eventually stabilizing. This is why
adding some friction, that is, choosing a value of β less than one, is important since it dampens
these oscillations and speeds up convergence.

Drawbacks

Despite its benefits, momentum introduces a new hyperparameter β. If β is set too high, it may
lead to excessive overshooting and oscillations, although having moderate friction generally
helps to reduce this issue.
However, the drawback of overshooting and oscillations in Momentum motivated the
development of Nesterov Accelerated Gradient (NAG), which anticipates the next
position before making the update, effectively reducing this issue and improving stability.

Nesterov Accelerated Gradient (NAG)

One small but powerful variant of Momentum optimization. This method, called Nesterov
Accelerated Gradient (NAG), is almost always faster than vanilla Momentum
optimization.

The key idea is simple: instead of calculating the gradient of the cost function at the current
position 𝑤, NAG looks a bit ahead in the direction of the momentum. In other words, the
gradient is measured at 𝑤 𝑘−1 + 𝛽∆𝑤 𝑘−1 rather than at 𝑤 𝑘−1 .

∆𝑤 𝑘 = 𝛽∆𝑤 𝑘−1 − 𝜂∇𝑤 𝐸(𝑤 𝑘−1 + 𝛽∆𝑤 𝑘−1 )

𝑤 𝑘 = 𝑤 𝑘−1 + ∆𝑤 𝑘

At any iteration, to compute the current step:

1- First look ahead by adding a scaled version of

previous step: 𝛽 ∆𝑤 𝑘−1

2- then take a step against the gradient, but

measured at the lookahead position:
𝜂∇𝑤 𝐸(𝑤 𝑘−1 + 𝛽∆𝑤 𝑘−1 )

This small tweak works because in general the momentum vector will be pointing in the
right direction, so it will be slightly more accurate to use the gradient measured a bit farther
in that direction rather than the gradient at the original position.

As you can see (where ∇1 represents the

gradient of the cost function measured at the
starting point 𝑤 𝑘−1 , and ∇2 represents the
gradient of the point located at 𝑤 𝑘−1 + 𝛽∆𝑤 𝑘−1
As you can see, the Nesterov update ends up closer to the optimum. After a while the small
improvements add up and NAG ends up being significantly faster than regular momentum
optimization. Moreover, note that when the momentum pushes the weights across a valley,
∇1 continuous to push farther across the valley, while ∇2 pushes back toward the bottom of
the valley. This helps reduce oscillations and thus NAG converges faster.

𝑊 → 𝑊 − 𝜂 ∇𝑤 𝐸(𝑊)

Momentum methods fix this term to reduce unstable oscillation.

What about 𝜂 ? However, they still suffer from one key limitation: they use a single global
learning rate for all parameters.

This is problematic, because different parameters behave very differently during training:

• Some directions are smooth and stable.

• Others oscillate heavily and become unstable.

Moreover, in many real-world datasets we encounter sparse features:

• Frequent (dense) features appear many times during training. Their gradients are
large and repetitive, which often causes oscillations. These need a smaller learning
rate to avoid overshooting.

• Rare (sparse) features appear infrequently. Their gradients are small or rarely
updated, so they require a larger learning rate to catch up and learn effectively.

To handle this, Adagrad was introduced.

AdaGrad
The AdaGrad algorithm handles this by scaling down the gradient vector along the steepest
dimensions.

Modified update rule: We want to

• scale down learning rates for terms with large mean squared derivatives
• scale up learning rates for terms with small mean squared derivatives
Update Rule
𝑣𝑘 = 𝑣𝑘−1 + ∇𝑤 𝐸(𝑊)2
𝜂
𝑤𝑘 = 𝑤𝑘−1 − ∇𝑤 𝐸(𝑊)
√𝑣𝑘 + 𝜀

• The first step accumulates the square of the gradients into the vector 𝑣𝑘 .
• 𝜀: is smoothing term to avoid division by zero.

In short, this algorithm decays the learning rate, but it does so faster for steep dimensions
than for dimensions with gentler slopes. It requires much less tuning of the learning rate
hyperparameter 𝜂.

Aggressive Decay in AdaGrad

AdaGrad frequently performs well for simple quadratic problems, but it often stops too
early when training neural networks. The reason is that the learning rate gets scaled down
so aggressively that the algorithm ends up stopping entirely before reaching the global
optimum.

This phenomenon is called Aggressive Decay of the Learning Rate. It makes AdaGrad
unsuitable for training deep neural networks, since the steps become too small after a
short period of training. This issue appears especially with dense features, where
gradients exist almost everywhere, leading to a very fast decay of the effective learning
rate. In contrast, AdaGrad remains useful for sparse features, where some dimensions
are rarely updated, so their learning rates stay relatively larger.

However, AdaGrad may still be efficient for simpler tasks such as linear regression.
Understanding AdaGrad and its aggressive decay problem is important, because it
motivated the development of more advanced optimizers such as RMSProp and Adam,
which address this limitation.
RMSProp
The RMSProp algorithm fixes this by accumulating only the gradients from the most recent
iterations, as opposed to all the gradients since the beginning of training. It does so by
using exponential decay in the first step.

Update Rule
𝑣𝑘 = 𝛾 𝑣𝑘−1 + (1 − 𝛾) ∇𝑤 𝐸(𝑊)2
𝜂
𝑤𝑘 = 𝑤𝑘−1 − ∇𝑤 𝐸(𝑊)
√𝑣𝑘 + 𝜀

The decay rate 𝛾 is typically set to 0.9. Yes, it’s once again a new hyperparameter, but this
default value often works well so, you may not need to tune it at all.

Up to this point, we have seen two powerful ideas:

• RMSProp → adapts the learning rate (controls the step size).

• Momentum → smooths the updates (controls the direction).

What if we combine them?

One takes care of the step size, the other of the direction → together they form a stronger
optimizer.

This is exactly the idea behind Adam.

Adam
Just like momentum optimization, it keeps track of an exceptionally decaying average of
past gradients; and just like RMSProp, it keeps track of an exceptionally decaying average of
past squared gradients.

Update Rule
𝑚𝑘 = 𝛽1 𝑚𝑘−1 + (1 − 𝛽1 ) ∇𝑤 𝐸(𝑊)

𝑣𝑘 = 𝛽2 𝑣𝑘−1 + (1 − 𝛽2 ) ∇𝑤 𝐸(𝑊)2
𝑚𝑘 𝑣𝑘
𝑚
̂𝑘 = , 𝑣̂𝑘 =
1 − 𝛽1𝑘 1 − 𝛽2𝑘
𝜂
𝑤𝑘 = 𝑤𝑘−1 − 𝑚
̂𝑘
√𝑣̂𝑘 + 𝜀
If you just look at steps 1, 2 and 4, you will notice Adam’s close similarity
to both momentum optimization and RMSProp: 𝛽1 corresponds to 𝛽 in
momentum optimization, and 𝛽2 corresponds to 𝛾 in RMSProp. The only
difference is that step 1 computes an exponentially decaying average
rather than an exponentially decaying sum, but these are equivalent.

Step 3 is somewhat of a technical detail: since 𝑚 and 𝑣 are initialized at

0, they will be biased toward 0 at the beginning of training, so these two
steps will help boost 𝑚 and 𝑣 at the beginning of training.

Typical Hyperparameters

Generally, these optimizers do not require an explicit learning rate

schedule to be tuned, but they do come with other hyperparameters that
need to be set. The commonly used default values are:

• RMSProp:

o Learning rate 𝜂 = 0.001

o Decay rate 𝛾 = 0.9

• Adam:

o Learning rate 𝜂 = 0.001

o 𝛽1 = 0.9 (controls the momentum term)

o 𝛽2 = 0.999 (controls the RMSProp-like term)

These defaults generally work well in practice and are widely used in
deep learning frameworks.
Figure 1 (Optimizer Trajectories)

The figure shows optimizer trajectories on a contour surface. Adam converges fastest with
a smooth and stable path (15 steps). RMSProp follows closely (17 steps). Momentum
converges with oscillations (31 steps), while AdaGrad moves slowly due to its diminishing
step sizes (31 steps).

Figure 2 (Distance to Optimum vs. Steps)

The figure illustrates the distance to the optimum over training steps. Adam achieves the
fastest and most stable convergence, RMSProp performs similarly but slightly slower,
Momentum reduces the distance with oscillations, and AdaGrad converges slowly as its
learning rate decays.

Finally, three variants of Adam are worth mentioning: AdaMax, Nadam, and AdamW.

AdaMax
The Adam paper also introduced AdaMax. Notice that in step 2 of Adam, the algorithm
accumulates the squares of the gradients in 𝑣𝑘 (with a greater weight for more recent
gradients). In step 4, ignoring 𝜀 and other technical details, Adam scales down the
parameter updates by the square root of 𝑣𝑘 . In short, Adam scales the updates by the 𝐿2
norm of the time-decayed gradients (recall that the 𝐿2 norm is the square root of the sum of
squares).

AdaMax replaces the 𝐿2 norm with the 𝐿∞ norm (the maximum). Specifically, it replaces
step 2 with:

Update Rule
𝑚𝑘 = 𝛽1 𝑚𝑘−1 + (1 − 𝛽1 ) ∇𝑤 𝐸(𝑊)

𝑣𝑘 = max (𝛽2 𝑣𝑘−1 , abs( ∇𝑤 𝐸(𝑊))

𝑚𝑘
𝑚
̂𝑘 =
1 − 𝛽1𝑘
𝜂
𝑤𝑘 = 𝑤𝑘−1 − 𝑚
̂
𝑣̂𝑘 + 𝜀 𝑘

It drops to step 3, and in step 4 the gradient updates are scaled down by,
which represents the maximum of the absolute values of the time-
decayed gradients.
In practice, this modification can make AdaMax more stable than Adam,
but its effectiveness depends on the dataset. Adam performs better, yet
AdaMax can be a useful alternative if problems arise with Adam on
certain tasks.

Nadam
Nadam (Nesterov-accelerated Adam)

Nadam is another variant of Adam that integrates the idea of Nesterov momentum into
the Adam update rule. In standard Adam, the update uses the exponentially moving
average of past gradients (𝑚𝑘 ) and the exponentially moving average of squared gradients
(𝑣𝑘 )to adapt the learning rate for each parameter.

The key difference in Nadam is that instead of applying the momentum term directly, it
applies the Nesterov accelerated gradient (a “lookahead” correction). This means that
rather than updating the parameters with the current momentum, Nadam first takes a
partial step in the direction of the momentum and then calculates the gradient, effectively
providing a more informed adjustment.

Formally, the update rule for Nadam modifies Adam’s step to:

Update Rule
𝑚𝑘 = 𝛽1 𝑚𝑘−1 + (1 − 𝛽1 ) ∇𝑤 𝐸(𝑊)

𝑣𝑘 = 𝛽2 𝑣𝑘−1 + (1 − 𝛽2 ) ∇𝑤 𝐸(𝑊)2
𝑚𝑘 𝑣𝑘
𝑚
̂𝑘 = , 𝑣̂𝑘 =
1 − 𝛽1𝑘 1 − 𝛽2𝑘

Nesterov adjustment:
1 − 𝛽1
𝑚̀ 𝑘 = 𝛽1 𝑚
̂𝑘 + ∇𝑤 𝐸(𝑊)
1 − 𝛽1𝑘
𝑚̀ 𝑘
𝑤𝑘 = 𝑤𝑘−1 − 𝜂
√𝑣̂𝑘 + 𝜀

In practice, Nadam often converges faster than Adam in certain problems, especially when
the optimization landscape has steep ravines or curved surfaces. However, the
improvement is not guaranteed, and in many tasks Adam or AdamW still perform equally
well or better.
When using Nadam: Nadam is preferred over standard Adam when your optimization
landscape has steep ravines or curved surfaces, or when the model exhibits oscillatory
behavior. It often converges slightly faster than Adam in such cases.

AdamW
AdamW (Adam with Weight Decay)

AdamW is a variant of Adam that integrates a regularization technique known as weight

decay. Weight decay works by shrinking the model’s weights at each training step, typically
by multiplying them by a decay factor (e.g., 0.99). At first, this may seem like 𝐿2
regularization, which also keeps weight small, but they are not equivalent when used with
adaptive optimizers like Adam.

𝑚𝑘 = 𝛽1 𝑚𝑘−1 + (1 − 𝛽1 ) ∇𝑤 𝐸(𝑊)

𝑣𝑘 = 𝛽2 𝑣𝑘−1 + (1 − 𝛽2 ) ∇𝑤 𝐸(𝑊)2
𝑚𝑘 𝑣𝑘
𝑚
̂𝑘 = , 𝑣̂𝑘 =
1 − 𝛽1𝑘 1 − 𝛽2𝑘

Update parameters without weight decay:

𝑚
̂𝑘
𝑤̀ 𝑘−1 = 𝑤𝑘−1 − 𝜂
√𝑣̂𝑘 + 𝜀

Apply decoupled weight decay:

𝑤𝑘 = 𝑤̀ 𝑘−1 − 𝜂 𝜆 𝑤𝑘−1

In fact, while 𝐿2 regularization and weight decay are mathematically equivalent in SGD,
they behave differently in Adam. Combining Adam with standard 𝐿2 regularization often
leads to models that converge quickly but fail to generalize well compared to SGD. AdamW
addresses this issue by decoupling weight decay from the adaptive gradient update,
thereby improving generalization.

When to use AdamW: AdamW is recommended over Adam for large models (e.g.,
Transformers, deep CNNs) or when regularization is important. Its decoupled weight decay
improves generalization compared to standard Adam.
The plot shows the paths of Adam, Adamax, Nadam, and AdamW on a stochastic function.
They all start from [2,2] aiming for the minimum at [0,0]. Adam and Nadam converge fastest
with smooth trajectories, Adamax is slightly slower, and AdamW maintains stable updates
due to decoupled weight decay. The distance-to-optimum plot (log-scale) highlights faster
convergence of adaptive momentum-based optimizers compared to standard behavior.

Default Hyperparameters Quick Reference

Optimizer η (Learning β1 β2 (RMS γ (Decay, ε
Rate) (Momentum) decay) RMSProp) (Smoothing)

GD 0.01–0.1 – – – –

Momentum 0.01–0.1 0.9 – – –

NAG 0.01–0.1 0.9 – – –

Adagrad 0.01 – – – 1e-8

RMSProp 0.001 – – 0.9 1e-8

Adam 0.001 0.9 0.999 – 1e-8

Adamax 0.002 0.9 0.999 – 1e-8

Nadam 0.002 0.9 0.999 – 1e-8

AdamW 0.001 0.9 0.999 – 1e-8

(* is bad, ** is average and *** is good)

OptimizerConvergence Convergence Generalization Best Use Cases

speed quality Ability
GD * *** Small models, baseline experiments
**
Momentum ** *** ** or *** Deep nets, vision tasks
NAG ** *** ** or *** Deep networks, convex & non-
convex
AdaGrad *** *(stops early) * NLP with sparse features
RMSProp *** ** or *** ** RNNs, non-stationary problems
Adam *** ** or *** ** Most deep learning tasks
AdaMax *** ** or *** ** NLP, sparse features
Nadam *** ** or *** ** Deep networks, when oscillations
are a problem
AdamW *** ** or *** *** Large models (Transformers, CNNs),
modern DL

• Convergence speed:
• Adagrad appears fast at the beginning (***) but stops early due to learning rate
decay.
• Generalization:
• AdamW performs better than Adam on large models due to decoupled weight
decay.
• Momentum vs NAG:
• NAG is sometimes faster and more stable in cases with oscillations.

Conclusion
Adaptive learning rate optimizers, especially Adam and its variants, combine the benefits
of momentum and per-parameter step size adaptation, allowing faster and more stable
convergence on complex, stochastic loss landscapes. Nadam improves convergence in
oscillatory or curved regions, while AdamW enhances generalization for large models by
decoupling weight decay. Overall, these methods significantly outperform vanilla Gradient
Descent, providing both efficiency and robustness in modern deep learning tasks.