Module 3

LSTM Architectures

O. V. Ramana Murthy
 Contents
 Introduction
 LSTM architectures
 Other Gated RNN
 Example Forward pass
 Application

2
 Reference:
 10.10 Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep
Learning, MIT Press, 2016
 Simple Explanation of LSTM | Deep Learning Tutorial 36
(Tensorflow, Keras & Python)
[Link]

3
 Gated Recurrent Network (GRU)
A Gated Recurrent Network (GRN) is a type of recurrent
neural network (RNN) that incorporates gating mechanisms to
manage the flow of information and avoid issues like vanishing
or exploding gradients. A common example of GRNs is the
Gated Recurrent Unit (GRU), which uses update and reset
gates.

Before Hidden state is computed….

4
 Purpose of Each Gate
Reset Gate: Determines how much of the past information
should be forgotten or reset before calculating the new state.
Update Gate: Controls how much of the new information
should replace the past information in the hidden state.
Candidate Activation: Proposes a potential new state based
on the current input and adjusted past state.
Final Hidden State: Combines the past state and the new
candidate state, weighted by the update gate, to form the final
output for the current step.

5
 Long short-term memory
Recall simple chain structure of basic RNNs
The repeating module in a standard RNN contains a
single layer.

6
 Long short-term memory
An example of LSTM: The repeating module in an LSTM
contains four interacting layers.

7
 Core idea behind LSTM
The key to LSTMs is the cell
state, the horizontal line
running through the top of
the diagram.
It runs straight down the
entire chain, with only some
minor linear interactions.

8
 Core idea behind LSTM
Gates are a way to optionally let information
through. They are composed out of a sigmoid
neural net layer and a pointwise multiplication
operation.
The sigmoid layer outputs numbers between zero
and one, describing how much of each component
should be let through.
A value of zero means “let nothing through,” while
a value of one means “let everything through!

9
 Step by Step Walk Through of LSTM

The first step in our LSTM is to decide what information we’re

10
 Step by Step Walk Through of LSTM

The next step is to decide what new information we’re going

11
 Step by Step Walk Through of LSTM

First, a sigmoid layer called the “input gate layer” decides which values
we’ll update.
Next, a tanh layer creates a vector of new candidate values, 𝐶ሚ𝑡 , that
could be added to the state.
12
 Step by Step Walk Through of LSTM

It’s now time to update the old cell state, Ct-1 , into the new cell state
Ct . The previous steps already decided what to do, we just need to
actually do it.

13
 Step by Step Walk Through of LSTM

Finally, we need to decide what we’re going to output.

14
 Summary
1. Input Gate: Decides what information to update
𝑖𝑡 = 𝜎 𝑊𝑖 𝑥𝑡 + 𝑈𝑖 ℎ𝑡−1
2. Forget Gate: Decides what information to discard
𝑓𝑡 = 𝜎 𝑊𝑓 𝑥𝑡 + 𝑈𝑓 ℎ𝑡−1
3. Candidate cell state 𝑐𝑡ǁ = tanh 𝑊𝑐 𝑥𝑡 + 𝑈𝑐 ℎ𝑡−1
4. Cell State Update 𝑐𝑡 = 𝑓𝑡 . 𝑐𝑡−1 + 𝑖𝑡 . 𝑐𝑡ǁ
5. Output Gate: Regulates the hidden state:
𝑜𝑡 = 𝜎 𝑊𝑜 𝑥𝑡 + 𝑈𝑜 ℎ𝑡−1
6. Hidden State: ℎ𝑡 = 𝑜𝑡 . tanh 𝑐𝑡
7. Predicted output 𝑦𝑡 = 𝑊𝑦 . ℎ𝑡

15
 Example (same from RNN)
Inputs: 𝑥=[0.5,0.6,0.7] ; Targets: 𝑦=[0.6,0.7,0.8]
We'll assume a simple LSTM with:
1. One hidden unit.
2. Initialized weights and biases.
3. No bias for simplicity.
4. Sigmoid activation for gates and tanh activation for cell states.
Assumptions
• Input vector size = 1 (since each input xt is scalar).
• Hidden state size = 1 (for simplicity).
• Initial hidden state h0 and cell state c0 are both zero.
Weights
• Input weights:=0.5
• Hidden state weights: =0.3

16
 Time Step t=1(x1=0.5)
Input gate: 𝑖1=𝜎(0.5⋅0.5+0.3⋅0)=𝜎(0.25)≈0.562
Forget gate: 𝑓1=𝜎(0.5⋅0.5+0.3⋅0)=𝜎(0.25)≈0.562
Candidate cell state: 𝑐1ǁ =tanh(0.5⋅0.5+0.3⋅0)=tanh(0.25) ≈ 0.244
Cell state: 𝑐1=𝑓1⋅𝑐0+𝑖1⋅ 𝑐1ǁ =0.562⋅0+0.562⋅0.244≈0.137
Output gate: 𝑜1=𝜎(0.5⋅0.5+0.3⋅0)=𝜎(0.25)≈0.562
Hidden state: ℎ1=𝑜1⋅tanh(𝑐1)=0.562⋅tanh(0.137)≈0.076
Predicted output y1 =Wy ℎ1 = 0.076

17
 Time Step t=2(x2=0.6)
Input gate: 𝑖2=σ(0.5⋅0.6+0.3⋅0.076)=σ(0.318)≈0.578
Forget gate: 𝑓2=σ(0.5⋅0.6+0.3⋅0.076)=σ(0.318)≈0.578
Candidate cell state:
𝑐2ǁ =tanh(0.5⋅0.6+0.3⋅0.076)=tanh(0.318)≈0.308
Cell state: 𝑐2=𝑓2⋅𝑐1+𝑖2⋅ 𝑐2ǁ =0.578⋅0.137+0.578⋅0.308≈0.257
Output gate: 𝑜2=σ(0.5⋅0.6+0.3⋅0.076)=σ(0.318)≈0.578
Hidden state: ℎ2=𝑜2⋅tanh(𝑐2)=0.578⋅tanh(0.257)≈0.144
Predicted output y2 =Wy ℎ2 = 0.144

18
 Time Step t=3(x3=0.7)
Input gate: 𝑖3=σ(0.5⋅0.7+0.3⋅0.144)=σ(0.372)≈0.592
Forget gate: 𝑓3=σ(0.5⋅0.7+0.3⋅0.144)=σ(0.372)≈0.592
Candidate cell state:
𝑐3ǁ =tanh(0.5⋅0.7+0.3⋅0.144)=tanh(0.372)≈0.355
Cell state: 𝑐3=𝑓3⋅𝑐2+𝑖3⋅ 𝑐3ǁ = 0.592⋅0.257+0.592⋅0.355≈0.361
Output gate: 𝑜3= σ(0.5⋅0.7+0.3⋅0.144)=σ(0.372)≈0.592
Hidden state: ℎ3=𝑜3⋅tanh(𝑐3)= 0.592⋅tanh(0.361)≈0.202
Predicted output y3 =Wy ℎ3 = 0.144

19
 RNN vs LSTM
RNN Structure
Single Tanh Layer: RNNs have a simple architecture where
the hidden state at each time step is updated using
ℎ𝑡 = tanh 𝑊𝑥 𝑥𝑡 + 𝑏𝑥 + 𝑊ℎ ℎ𝑡−1
LSTM Structure
Gating Mechanisms: LSTMs introduce additional structures
called gates (input gate, forget gate, and output gate) and a
cell state to better regulate the flow of information:

20
 LSTM Cell

21
 Applications
Speech Recognition: LSTMs model sequential data like audio
waveforms, enabling accurate transcription of spoken language
into text.
Time Series Prediction: Used for forecasting stock prices,
weather patterns, and energy consumption due to their ability
to capture temporal dependencies.
Natural Language Processing (NLP): LSTMs power tasks like
machine translation, text summarization, and sentiment
analysis by understanding word sequences.
Handwriting Recognition: LSTMs decode sequential pen
strokes in handwritten input for digit and text recognition.
Video Analysis: Applied in activity recognition and scene
understanding by processing sequential frames in videos.
22
 Thank You All Very Much

23