Scribd
Upload
4 views18 pages

Understanding Exponentially Weighted Averages

The document discusses Exponentially Weighted Averages (EWMA), highlighting their application in optimization algorithms and their role in smoothing data such as daily temperatures. It explains the general formula for calculating EWMA and provides insights into bias correction and momentum in stochastic gradient descent. Additionally, it introduces Adam, an adaptive moment estimation method that utilizes first and second moments for weight updates.

Uploaded by

officialtorque05
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
4 views18 pages

Understanding Exponentially Weighted Averages

The document discusses Exponentially Weighted Averages (EWMA), highlighting their application in optimization algorithms and their role in smoothing data such as daily temperatures. It explains the general formula for calculating EWMA and provides insights into bias correction and momentum in stochastic gradient descent. Additionally, it introduces Adam, an adaptive moment estimation method that utilizes first and second moments for weight updates.

Uploaded by

officialtorque05
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Exponentially weighted averages

(also called Exponentially Weighted Moving Average in Statistics)

Daily temperature

An important component of several optimization algorithms


Current days
temp

Prev days temp

General Formula
General formula.

vt
is approximately averaged
over

How?
Reason is
discussed later

Red curve: 10
days’ average
Green Curve: 50
days’ average
Yellow Curve: 2
days’ average
Understanding Exponentially Weighted
Averages

v100 = 0.1θ100 + 0.9(0.1θ 99 + 0.9(0.1θ 98 + 0.9θ 97 ))


= 0.1θ100 + 0.1* 0.9 *θ 99 + 0.1* (0.9) 2 θ 98 + (0.9) 3θ 97
Further expansion gives as follows.
= 0.1θ100 + 0.1* 0.9 *θ 99 + 0.1* (0.9) 2 θ 98 + 0.1* (0.9) 3θ 97
+ 0.1* (0.9) 4 θ 96 + ....
v100 . = 0.1θ100 + 0.1* 0.9 *θ 99 + 0.1* (0.9) 2 θ 98 + 0.1* (0.9) 3θ 97
+ 0.1* (0.9) 4 θ 96 + ....

0.1

t=100
Temperatures Plot of coefficients: an
for 100 days exponential decay function

v100 = element wise dot product of these two


and sum up

Weight for the x-th term is very small, say


1
0.9 x = = 0.35, if it is multipled by 0.1
e
it becomes small
β = 0.9
1
βx =
e
1 1
⇒x=− =
log β log(1 + ( β − 1))
1
≈− [use log(1 + x) series]
( β − 1) 2
( β − 1) −
2
1
≈− [ignoring higher terms]
( β − 1)
1
≈
(1 − β )
Bias Correction

Poor estimate at
the initial few
steps

Bias correction
Contour plot of the
3-D cost function

Stochastic Gradient Descent is noisy. Direction of gradients –(1) towards


the centre (horizontal) (2) vertical .

To quick convergence, the learning in the horizontal direction should be


faster, but it should slower in the vertical direction.

How to achieve it?


Momentum
On iteration t,
Compute dw, db on the current mini batch

Exponentially
weighted
average of
gradients dw =
the derivative of the cost
function w.r.t w
Exponentially
weighted
average of
gradients

Friction
Acceleration of the rolling
Velocity ball on the error bowl

In the literature, beta is


the momentum
parameter. (1 − β ) is
not multiplied with
dW. It is less intuitive.
Bias correction is not used here.

Learning rate Averages last 10 gradients (approx)


RMS: Root Mean Square (see square term and square toot term)

Small because less


oscillation
large because more
oscillation
Large denominator.
Slow update in the
vertical direction
Small denominator. Faster damping oscillation
update
W, b are two parameters . Two axes in two different directions.
In contrast with
momentum,
RMSprop works
well with the
larger learning
rate

Epsilon is added for numerical stability


Like
Momentum

Like
RMSprop

Bias
correction

Weight update
used in
RMSprop
Adam: Adaptive Moment Estimation

dW : first moment, dW 2 : 2nd moment


(variance)

You might also like