Machine Learning and Deep Learning

(ITDLC6031)

Module 5:
Recurrent Neural Networks

Course In-charge:
Dr. Radhika Kotecha
Professor and Head, Department of Information Technology
Contents
Total
COs Hours /
Module Subtopics Hours /
Mapped Subtopic
Module
Introduction to Recurrent Neural
Networks (RNN), Architecture, RNN CO5 02
versus CNN, RNN versus FNN.
Training Recurrent Networks and
Challenges Thereof, Long Short-Term 07
CO5 03
Memory (LSTM), Gated Recurrent
Units (GRUs).
Applications of RNN. CO5 02

2
Recurrent Neural Networks (RNN)
• RNN works on the principle of saving the output of a
particular layer and feeding this back to the input in order to
predict the output of the layer.
• RNN were created because there were a few issues in the
feed-forward neural network:
• Cannot handle sequential data
• Considers only the current input
• Cannot memorize previous inputs
• An RNN can handle sequential data, accepting the current
input data, and previously received inputs. RNNs can
memorize previous inputs due to their internal memory.

3
Recurrent Neural Networks (RNN)

The nodes in different layers of the neural network are compressed to form a single layer of
recurrent neural networks. A, B, and C are the parameters of the network.

4
Recurrent Neural Networks (RNN)

5
Working of Recurrent Neural Networks (RNN):
Step-wise Animated

6
Working of Recurrent Neural Networks (RNN)

7
Math behind RNN

Feed-forward Propagation Recurrent Neural Network

8
Backpropagation of Error in RNN
• On reaching the output, loss is calculated at every timestamp thus
called backpropagation through time which is then propagated to
update the weights to minimize the errors.

9
Types of RNN
• The four types of RNN are:
• One to One RNN
• One to Many RNN
• Many to One RNN
• Many to Many RNN

10
Long Short Term Memory (LSTM)
• Long Short Term Memory networks – usually just called
“LSTMs” – are a special kind of RNN, capable of learning
long-term dependencies. They were introduced by Hochreiter
& Schmidhuber (1997), and were refined and popularized by
many people in following work.
• LSTMs work tremendously well on a large variety of
problems, and are widely used.
• LSTMs are explicitly designed to avoid the long-term
dependency problem. Remembering information for long
periods of time is practically their default behavior.

11
RNN versus LSTM
• RNN is not able to distinguish between important or not so
important information.
• LSTM contains gates that decide which data is important and
can be useful in future and which data has to be erased.
• RNN cell receives the inputs of hidden state(h) and input(x)
while LSTM cell has an additional input called cell state (c).
• Cell state is an internal memory where info is stored and
hidden state is where computations are done.

12
Long Short Term Memory (LSTM) Components
• Forget gate (f): Neural Network with Sigmoid
• Input gate (i): Neural Network with Sigmoid
• Candidate gate (g): Neural Network with tanh
• Output gate (o): Neural Network with Sigmoid
• Cell state (c): Vector
• Hidden state (h): Vector

13
Long Short Term Memory (LSTM)

14
LSTM Process in Detail

15
LSTM Process – Step 1
Forget Gate:
• Irrelevant info coming from
previous timestep (ht-1) is
removed.
• This decision is made by
the forget
gate having Sigmoid function
of range(0,1).
• Then, multiplying its output
with previous cell state gives
the necessary info (cft).
• If the value of forget gate is 0
then info from cell state is
supposed to be removed.

16
LSTM Process – Step 1
Example:
• Shachi is an Indian. She is
multilingual. Shelly is an
international Yoga teacher.

• Here, the change of subject

from Shachi to Shelly should
be identified and hence
“Shachi” subject should be
forgotten.

17
LSTM Process – Step 2
Input Gate (with Candidate
Gate):
• Decides what info should be
stored in cell state.
• The previous cell info is
passed to Sigmoid function of
input gate(it). For value 0
indicates the info need not be
stored. The previous timestep
info passed to tanh layer to
create candidate values to be
added in current cell state.
• The multiplied result of both
above when added to the cell
state, the current cell state is
updated with the new
necessary info (ct).
18
LSTM Process – Step 3
Output Gate:
• The output gate with
Sigmoid function decides
what goes from cell for the
output.
• For value 0, hidden state
(ht-1) is not passed to the
output.
• Updated cell info(ct) is
passed to tanh and
multiplied to the output of
output gate to give current
timestep info (ht).

19
LSTM Process – Step 4
If it’s the last LSTM cell then the hidden state (ht) is passed to
the softmax function to give the output y.

20
LSTM – Numerical Solving
• To predict temperature trend impact using LSTM

21
LSTM – Numerical Solving
• To predict temperature trend impact using LSTM

22
Issues with LSTM
• Issue: Long-term dependencies problem still exists
• Solution: For eradicating this problem, Transformers are
used

• Issue: Computationally expensive

• Solution: For eradicating this problem, GRUs are used

23
Gated Recurrent Unit (GRU)
• The Gated Recurrent Unit (GRU) (Cho et al., 2014) offers a streamlined
version of the LSTM memory cell that often achieves comparable
performance but with the advantage of being faster to compute.
• To solve the vanishing gradient problem of a standard RNN, GRU uses,
so-called, update gate and reset gate.
• Basically, these are two vectors which decide what information should be
passed to the output. They can be trained to keep information from long
ago, without washing it through time or remove information which is
irrelevant to the prediction.

24
Gated Recurrent Unit (GRU) Components
• Update Gate (Z): Neural Network with Sigmoid – helps to
capture short-term dependencies in sequences.
• Reset Gate (R): Neural Network with Sigmoid – helps to
capture long-term dependencies in sequences
• Candidate Hidden State (Ȟ): Vector
• Hidden State (H): Vector

25
Gated Recurrent Unit (GRU)

26
GRU Process – Step 1
Update Gate (Z):
• The update gate helps the
model to determine how
much of the past information
(from previous time steps)
needs to be passed along to
the future.
• With this gate, the model
can decide to copy all the
information from the past
and eliminate the risk of
vanishing gradient problem.
• It is analogous to the Output
Gate in an LSTM recurrent
unit.

27
GRU Process – Step 2
Reset Gate (R):
• The reset gate is used from
the model to decide how
much of the past
information to forget.
• It is analogous to the
combination of the Input
Gate and the Forget Gate
in an LSTM recurrent unit.

28
GRU Process – Step 3
Candidate Hidden State (Ȟ):
• Used to introduce some
non-linearity into the input.
• To reduce the effect that
previous information has on
the current information that
is being passed into the
future.

29
GRU Process – Step 4
Hidden State (H) - Final
memory at current time
step:
• Used to store and filter
the information for the
current unit and passes
it down to the network.
• Whenever the update
gate Zt is close to 1, we
simply retain the old
state.
• In contrast,
whenever Zt is close to
0, the new
state Ht approaches the
candidate state Ȟt.

30
RNNs vs LSTM vs GRU
Recurrent Neural Network
Aspect Long Short-Term Memory (LSTM) Gated Recurrent Unit (GRU)
(RNN)

Simple recurrent structure with Complex architecture with a memory Moderately complex structure
Architecture
feedback connections cell and multiple gates with fewer gates than LSTM

Captures long-term
Memory Captures only short-term Captures both short-term and long-
dependencies with simplified
Capability dependencies term dependencies effectively
mechanism
Gating
Does not use gates Uses input, forget, and output gates Uses update and reset gates
Mechanism

Vanishing Highly prone to vanishing gradient Effectively overcomes vanishing Reduces vanishing gradient
Gradient Issue problem gradient problem problem

Computational High computational cost due to

Low computational cost Moderate computational cost
Complexity multiple gates
Faster training due to simple Faster than LSTM but slower
Training Time Slower training due to complexity
structure than RNN
Performance on Performs poorly on long Performs very well on long Performs well on long
Long Sequences sequences sequences sequences

Suitable for complex tasks such as Suitable for tasks requiring

Suitable for basic sequence
Applications language modeling and speech balance between performance
modeling tasks
recognition and efficiency

31
Deep Recurrent Neural Networks (Deep RNNs)

32
Example Questions
1. Explain RNN.
2. Explain the need of RNN for sequential data.
3. Explain the working of RNN in general.
4. Differentiate CNN and RNN.
5. Differentiate RNN, LSTM, and GRU.
6. List various Gates and States in LSTM.
7. Explain the generic architecture of LSTM.
8. Explain issues with LSTM.
9. For a real-world application of LSTM, given the xt, ht−1, ct−1 and given
weights, calculate values for forget gate ft, input gate it, updated cell
state ct, output gate ot and updated hidden state ht. (Numerical)
[Link] the generic architecture of GRU.
[Link] the role of Update Gate in GRU.
[Link] LSTM / GRU for any application (E.g. time-series based stock
market prediction and explain its architecture)
(Hint: Explain application first, and then the generic architecture).

33
References
1. S. Rose, L. Kumar, D Renuka, Deep Learning using Python, Wiley.
2. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT
Press.
3. T. Mitchell, Machine Learning, McGraw Hill.
4. C. Aggarwal, Neural Networks and Deep Learning: A Textbook,
Springer.
5. J. Han, M. Kamber, Data Mining Concepts and Techniques, Morgan
Kaufmann.
6. M. Nielsen, Neural Networks and Deep Learning, Determination
Press.
7. Online Blogs and Videos

34