Recurrent Neural Networks(RNN)
&
Long-Short Term Memory
A. Prof. Noha El-Attar
Recurrent Neural Networks
• Human brain deals with information streams. Most data is obtained, processed, and
generated sequentially.
– E.g., listening: soundwaves vocabularies/sentences
– E.g., action: brain signals/instructions sequential muscle movements
• Human thoughts have persistence; humans don’t start their thinking from scratch every
second.
– As you read this sentence, you understand each word based on your prior knowledge.
• The applications of standard Artificial Neural Networks (and also Convolutional
Networks) are limited due to:
– They only accepted a fixed-size vector as input (e.g., an image) and produce a fixed-
size vector as output (e.g., probabilities of different classes).
– These models use a fixed amount of computational steps (e.g. the number of layers in
the model).
• Recurrent Neural Networks (RNNs) are a family of neural networks introduced to learn
sequential data.
– Inspired by the temporal-dependent and persistent human thoughts 2
Real-life Sequence Learning Applications
• RNNs can be applied to various type of sequential data to learn
the temporal patterns.
– Time-series data (e.g., stock price) Prediction, regression
– Raw sensor data (e.g., signal, voice, handwriting) Labels or text
sequences
– Text Label (e.g., sentiment) or text sequence (e.g., translation,
summary, answer)
– Image and video Text description (e.g., captions, scene
interpretation)
3
What is the difference between RNN and other
Networks?
• Regular feed-forward networks such as MLP and CNNs only consider the current input.
• They do not have any memory about what happened in the past.
• RNNs differ in that they retain information about the input previously received.
• They are networks with feedback loops that allow information to persist - a trait that is
analogous to short-term memory.
• This sequential memory is preserved in the recurrent network’s hidden state vector and
represents the context based on the prior inputs and outputs.
• Vanilla RNN works on the principle of saving the output of a particular
layer and feeding this back to the input in order to predict the output of
the layer.
The green blocks are called hidden states. The blue circles, defined by the vector a within each block, ar
e called hidden nodes or hidden units.
RNN Calculations
Note: We reuse the Weight matrix at every time step
It’s also important to understand the dimensions of all the variables:
• Where
• k is the dimension of the input vector xᵢ
• d is the number of hidden nodes
Forward Propagation:
Take the word “dogs,” where we want to train an RNN to predict
the letter “s” given the letters “d”-“o”-“g”.
we’ll use 3 hidden nodes in our RNN (d=3). The dimensions for
each of our variables are as follows:
Forward Propagation
• We’ve initialized random weights for the matrices Wx, Wy, and Wh to provide
an example with numbers.
Ht-1 Xi
WH WX
a1
WY h1
Backpropagation through time
• Now that we understand how to make predictions with RNNs,
let’s explore how RNNs learn to make correct predictions.
• RNN backpropagation Algorithm
1. Initialize weight matrices Wx, Wy, Wh randomly
2. Forward propagation to compute predictions
3. Compute the loss
4. Backpropagation to compute gradients
5. Update weights based on gradients
6. Repeat steps 2–5
Error Calculation
Wh Wh Wh
h1 h2 h3
First step- Adjusting WY
h1 h2 h3
Second step- Adjusting Wh
• Formula:
• General Form:
Wh Wh Wh
h1 h2 h3
Third step- Adjusting Wx
h1 h2 h3
Limitations:
• This method of Back Propagation through time (BPTT) can be used up to a
limited number of time steps like 8 or 10.
• Short-term memory: Struggles with long sequences because it mainly retains
recent information.
• Vanishing gradient problem: When training with backpropagation through
time (BPTT), gradients shrink over long sequences, making it hard to learn
long-term dependencies.
RNN Alternatives:
• LSTMs (Long Short-Term Memory): Solve the vanishing gradient
problem by using gates to control information flow.
• An LSTM has a similar control flow as a recurrent neural
network. It processes data passing on information as it
propagates forward. The differences are the operations within
the LSTM's cells.
• GRUs (Gated Recurrent Units): A simpler alternative to LSTM
with fewer parameters.
Understanding LSTM and its diagrams
• Difference between RNN and LSTM:
– RNN: single layer (tanh)
– LSTM: 4 interactive layers
Understanding LSTM and its diagrams
Cell State
Long Memory
Short Memory
Understanding LSTM and its diagrams
• Cell state:
– Like a conveyor belt
– It runs straight down the entire chain
– LSTM can remove or add information to the cell state
• An LSTM has three of these gates, to protect and control the cell state:
Long memory
– Forget gate layer Keep gate
– Input gate layer Write gate
– Output gate layer Read gate
19
Understanding LSTM and its diagrams
• Forget information:
– Decide what information throw away from the cell state
(which part LSTM will keep and which will forget)
– Forget gate layer:
• Output a number between 0 and 1
Understanding LSTM and its diagrams
• Add new information:
– Decide what new information store in the cell state
– Input gate layer:
• Decides which values we’ll update
– Tanh layer:
• creates a vector of new candidate values, 𝐶𝑡
Understanding LSTM and its diagrams
• Update cell state:
– Forgetting the things we decided to forget earlier: f t Ct 1
– Adding information we decide to add: it Ct
* *
Understanding LSTM and its diagrams
• Create output:
– Decide what we’re going to output
– Output gate layer:
• Decides what parts of the cell state we’re going to output
– Tanh layer:
• Push the values between -1 and +1
*
Shadow state/ Short memory
Understanding LSTM and its diagrams
• Conclusion:
– Step 1: Forget gate layer
– Step 2: Input gate layer
– Step 3: Combine step 1 & 2
– Step 4: Output the cell state
Numerical Example on LSTM:
Revise the Example from the PDF file
Exercise
• Implement the numerical example in PDF from scratch by
Python