Module IV

Recurrent Neural Networks: Back propagation, vanishing gradients, exploding

gradients, truncated backpropagation through time, Gated Recurrent Units (GRUs),
Long Short-Term Memory (LSTM) cells, solving the vanishing gradient problem with
LSTMs.

______________________________________________________________________

Recurrent Neural Networks

Explain how recurrent neural networks use sequential data to solve common temporal
problems seen in language translation and speech recognition.

What are recurrent neural networks?

● A recurrent neural network (RNN) is a type of artificial neural network which uses
sequential data or time series data.
● These deep learning algorithms are commonly used for ordinal or temporal
problems, such as language translation, natural language processing (nlp),
speech recognition, and image captioning; they are incorporated into popular
applications such as Siri, voice search, and Google Translate.
● Like feedforward and convolutional neural networks (CNNs), recurrent neural
networks utilize training data to learn.
● They are distinguished by their “memory” as they take information from prior
inputs to influence the current input and output.
● While traditional deep neural networks assume that inputs and outputs are
independent of each other, the output of recurrent neural networks depend on
the prior elements within the sequence.
● While future events would also be helpful in determining the output of a given
sequence, unidirectional recurrent neural networks cannot account for these
events in their predictions.
Recurrent Neural Network vs. Feedforward Neural Network

Comparison of Recurrent Neural Networks (on the left) and Feedforward Neural
Networks (on the right)

Let’s take an idiom, such as “feeling under the weather”, which is commonly used when
someone is ill, to aid us in the explanation of RNNs. In order for the idiom to make
sense, it needs to be expressed in that specific order. As a result, recurrent networks
need to account for the position of each word in the idiom and they use that information
to predict the next word in the sequence.

Looking at the visual below, the “rolled” visual of the RNN represents the whole neural
network, or rather the entire predicted phrase, like “feeling under the weather.” The
“unrolled” visual represents the individual layers, or time steps, of the neural network.
Each layer maps to a single word in that phrase, such as “weather”. Prior inputs, such
as “feeling” and “under”, would be represented as a hidden state in the third timestep to
predict the output in the sequence, “the”.
Another distinguishing characteristic of recurrent networks is that they share parameters
across each layer of the network.
While feedforward networks have different weights across each node, recurrent neural
networks share the same weight parameter within each layer of the network. That said,
these weights are still adjusted through the processes of backpropagation and gradient
descent to facilitate reinforcement learning.

Recurrent neural networks leverage backpropagation through time (BPTT) algorithm to

determine the gradients, which is slightly different from traditional backpropagation as it
is specific to sequence data. The principles of BPTT are the same as traditional
backpropagation, where the model trains itself by calculating errors from its output layer
to its input layer. These calculations allow us to adjust and fit the parameters of the
model appropriately. BPTT differs from the traditional approach in that BPTT sums
errors at each time step whereas feedforward networks do not need to sum errors as
they do not share parameters across each layer.

Through this process, RNNs tend to run into two problems, known as exploding
gradients and vanishing gradients. These issues are defined by the size of the gradient,
which is the slope of the loss function along the error curve.

When the gradient is too small, it continues to become smaller, updating the weight
parameters until they become insignificant—i.e. 0. When that occurs, the algorithm is no
longer learning. Exploding gradients occur when the gradient is too large, creating an
unstable model. In this case, the model weights will grow too large, and they will
eventually be represented as NaN. One solution to these issues is to reduce the
number of hidden layers within the neural network, eliminating some of the complexity in
the RNN model.

Types of recurrent neural networks

Feedforward networks map one input to one output, and while we’ve visualized
recurrent neural networks in this way in the above diagrams, they do not actually have
this constraint. Instead, their inputs and outputs can vary in length, and different types of

RNNs are used for different use cases, such as music generation, sentiment
classification, and machine translation.

Different types of RNNs are usually expressed using the following diagrams:

One-to-one:

One-to-many:

Many-to-one:
Many-to-many:

Many-to-many:
Common activation functions

An activation function determines whether a neuron should be activated. The nonlinear

functions typically convert the output of a given neuron to a value between 0 and 1 or -1
and 1. Some of the most commonly used functions are defined as follows:
Sigmoid: This is represented with the formula g(x) = 1/(1 + e^-x).

Tanh: This is represented with the formula g(x) = (e^-x - e^-x)/(e^-x + e^-x).
Relu: This is represented with the formula g(x) = max(0 , x)

Variant RNN architectures

Bidirectional recurrent neural networks (BRNN): These are a variant network

architecture of RNNs.

While unidirectional RNNs can only be drawn from previous inputs to make predictions
about the current state, bidirectional RNNs pull in future data to improve the accuracy of
it.

If we return to the example of “feeling under the weather” earlier in this article, the
model can better predict that the second word in that phrase is “under” if it knew that the
last word in the sequence is “weather.”

Long short-term memory (LSTM): This is a popular RNN architecture, which was
introduced by Sepp Hochreiter and Juergen Schmidhuber as a solution to the vanishing
gradient problem.

In their paper (PDF, 388 KB) (link resides outside IBM), they work to address the
problem of long-term dependencies. That is, if the previous state that is influencing the
current prediction is not in the recent past, the RNN model may not be able to
accurately predict the current state. As an example, let’s say we wanted to predict the
italicized words in the following, “Alice is allergic to nuts. She can’t eat peanut butter.”
The context of a nut allergy can help us anticipate that the food that cannot be eaten
contains nuts. However, if that context was a few sentences prior, then it would make it
difficult, or even impossible, for the RNN to connect the information. To remedy this,
LSTMs have “cells” in the hidden layers of the neural network, which have three gates–
an input gate, an output gate, and a forget gate. These gates control the flow of
information which is needed to predict the output in the network. For example, if gender
pronouns, such as “she”, was repeated multiple times in prior sentences, you may
exclude that from the cell state.

Gated recurrent units (GRUs): This RNN variant is similar to the LSTMs as it also works
to address the short-term memory problem of RNN models. Instead of using a “cell
state” to regulate information, it uses hidden states, and instead of three gates, it has
two—a reset gate and an update gate. Similar to the gates within LSTMs, the reset and
update gates control how much and which information to retain.

Introduction to Recurrent Neural Network

● Difficulty Level : Medium
● Last Updated : 23 Aug, 2022
Read
Discuss
Recurrent Neural Network(RNN) are a type of Neural Network where the output from
previous step are fed as input to the current step. In traditional neural networks, all
the inputs and outputs are independent of each other, but in cases like when it is
required to predict the next word of a sentence, the previous words are required and
hence there is a need to remember the previous words. Thus RNN came into existence,
which solved this issue with the help of a Hidden Layer. The main and most important
feature of RNN is Hidden state, which remembers some information about a sequence.
RNN have a “memory” which remembers all information about what has been
calculated. It uses the same parameters for each input as it performs the same task on
all the inputs or hidden layers to produce the output. This reduces the complexity of
parameters, unlike other neural networks.
How RNN works
The working of a RNN can be understood with the help of below example:
Example:
Suppose there is a deeper network with one input layer, three hidden layers and one
output layer. Then like other neural networks, each hidden layer will have its own set of
weights and biases, let’s say, for hidden layer 1 the weights and biases are (w1, b1),
(w2, b2) for second hidden layer and (w3, b3) for third hidden layer. This means that
each of these layers are independent of each other, i.e. they do not memorize the
previous outputs.
Now the RNN will do the following:
● RNN converts the independent activations into dependent activations by
providing the same weights and biases to all the layers, thus reducing the
complexity of giving each output as input to the next hidden layer.
● Hence these three layers can be joined together such that the weights and
 bias of all the hidden layers is the same, into a single recurrent layer.
increasing parameters and memorizing each previous outputs by

Formula for calculating current state:

where:
ht -> current state
ht-1 -> previous state
xt -> input state
●
Formula for applying Activation function(tanh):

where:
whh -> weight at recurrent neuron
wxh -> weight at input neuron
●
Formula for calculating output:
Yt -> output
Why -> weight at output layer
● 1
Training through RNN
1. A single time step of the input is provided to the network.
2. Then calculate its current state using set of current input and the previous
state.
3. The current ht becomes ht-1 for the next time step.
4. One can go as many time steps according to the problem and join the
information from all the previous states.
5. Once all the time steps are completed the final current state is used to
calculate the output.
6. The output is then compared to the actual output i.e the target output and the
error is generated.
7. The error is then back-propagated to the network to update the weights and
hence the network (RNN) is trained.
Advantages of Recurrent Neural Network
1. An RNN remembers each and every information through time. It is useful in
time series prediction only because of the feature to remember previous
inputs as well. This is called Long Short Term Memory.
2. Recurrent neural network are even used with convolutional layers to extend
the effective pixel neighborhood.
Disadvantages of Recurrent Neural Network
1. Gradient vanishing and exploding problems.
2. Training an RNN is a very difficult task.
3. It cannot process very long sequences if using tanh or relu as an activation
function.

Recurrent Neural Networks Explanation

● Difficulty Level : Hard
● Last Updated : 06 Oct, 2021
Read
Discuss
Today, different Machine Learning techniques are used to handle different types of
data. One of the most difficult types of data to handle and the forecast is sequential
data. Sequential data is different from other types of data in the sense that while all the
features of a typical dataset can be assumed to be order-independent, this cannot be
assumed for a sequential dataset. To handle such type of data, the concept of
Recurrent Neural Networks was conceived. It is different from other Artificial Neural
Networks in its structure. While other networks “travel” in a linear direction during the
feed-forward process or the back-propagation process, the Recurrent Network follows a
recurrence relation instead of a feed-forward pass and uses Back-Propagation
through time to learn.
The Recurrent Neural Network consists of multiple fixed activation function units, one
for each time step. Each unit has an internal state which is called the hidden state of the
unit. This hidden state signifies the past knowledge that the network currently holds at a
given time step. This hidden state is updated at every time step to signify the change in
the knowledge of the network about the past. The hidden state is updated using the
following recurrence relation:-

[Tex]- The new hidden state[/Tex]

[Tex]- The old hidden state[/Tex]
[Tex]- The current input[/Tex]

[Tex]- The fixed function with trainable weights[/Tex]

Note: Typically, to understand the concepts of a Recurrent Neural Network, it is often
illustrated in its unrolled form and this norm will be followed in this post.
At each time step, the new hidden state is calculated using the recurrence relation as
given above. This new generated hidden state is used to generate indeed a new hidden
state and so on.
The basic work-flow of a Recurrent Neural Network is as follows:-

Note that is the initial hidden state of the network. Typically, it is a vector of zeros,
but it can have other values also. One method is to encode the presumptions about the
data into the initial hidden state of the network. For example, for a problem to determine
the tone of a speech given by a renowned person, the person’s past speeches’ tones
may be encoded into the initial hidden state. Another technique is to make the initial
hidden state a trainable parameter. Although these techniques add little nuances to the
network, initializing the hidden state vector to zeros is typically an effective choice.
Working of each Recurrent Unit:

1. Take input the previously hidden state vector and the current input vector.
Note that since the hidden state and current input are treated as vectors,
each element in the vector is placed in a different dimension which is
orthogonal to the other dimensions. Thus each element when multiplied by
another element only gives a non-zero value when the elements involved are
non-zero and the elements are in the same dimension.
2. Element-wise multiplies the hidden state vector by the hidden state weights
and similarly performs the element-wise multiplication of the current input
vector and the current input weights. This generates the parameterized
hidden state vector and the current input vector.
Note that weights for different vectors are stored in the trainable weight
matrix.
3. Perform the vector addition of the two parameterized vectors and then
calculate the element-wise hyperbolic tangent to generate the new hidden
state vector.

During the training of the recurrent network, the network also generates an output at
each time step. This output is used to train the network using gradient descent.
The Back-Propagation involved is similar to the one used in a typical Artificial Neural
Network with some minor changes. These changes are noted as:-
Let the predicted output of the network at any time step be and the actual output be
. Then the error at each time step is given by:-

The total error is given by the summation of the errors at all the time steps.

Similarly, the value can be calculated as the summation of gradients at each time
step.

Using the chain rule of calculus and using the fact that the output at a time step t is a
function of the current hidden state of the recurrent unit, the following expression
arises:-

Note that the weight matrix W used in the above expression is different for the input
vector and hidden state vector and is only used in this manner for notational
convenience.
Thus the following expression arises:-

Thus, Back-Propagation Through Time only differs from a typical Back-Propagation in

the fact the errors at each time step are summed up to calculate the total error.
Although the basic Recurrent Neural Network is fairly effective, it can suffer from a
significant problem. For deep networks, The Back-Propagation process can lead to the
following issues:-

● Vanishing Gradients: This occurs when the gradients become very small
and tend towards zero.
● Exploding Gradients: This occurs when the gradients become too large due
to back-propagation.
The problem of Exploding Gradients may be solved by using a hack – By putting a
threshold on the gradients being passed back in time. But this solution is not seen as a
solution to the problem and may also reduce the efficiency of the network. To deal with
such problems, two main variants of Recurrent Neural Networks were developed –
Long Short Term Memory Networks and Gated Recurrent Unit Networks.

Long Short Term Memory Networks Explanation

● Difficulty Level : Medium
● Last Updated : 29 Sep, 2021
Read
Discuss
Prerequisites: Recurrent Neural Networks
To solve the problem of Vanishing and Exploding Gradients in a Deep Recurrent Neural
Network, many variations were developed. One of the most famous of them is the Long
Short Term Memory Network(LSTM). In concept, an LSTM recurrent unit tries to
“remember” all the past knowledge that the network is seen so far and to “forget”
irrelevant data. This is done by introducing different activation function layers called
“gates” for different purposes. Each LSTM recurrent unit also maintains a vector called
the Internal Cell State which conceptually describes the information that was chosen to
be retained by the previous LSTM recurrent unit. A Long Short Term Memory Network
consists of four different gates for different purposes as described below:-
1. Forget Gate(f): It determines to what extent to forget the previous data.
2. Input Gate(i): It determines the extent of information be written onto the
Internal Cell State.
3. Input Modulation Gate(g): It is often considered as a sub-part of the input
gate and much literature on LSTM’s does not even mention it and assume it
is inside the Input gate. It is used to modulate the information that the Input
gate will write onto the Internal State Cell by adding non-linearity to the
information and making the information Zero-mean. This is done to reduce
the learning time as Zero-mean input has faster convergence. Although this
gate’s actions are less important than the others and are often treated as a
finesse-providing concept, it is good practice to include this gate in the
structure of the LSTM unit.
4. Output Gate(o): It determines what output(next Hidden State) to generate
from the current Internal Cell State.
The basic workflow of a Long Short Term Memory Network is similar to the workflow of
a Recurrent Neural Network with the only difference being that the Internal Cell State is
also passed forward along with the Hidden State.

Working of an LSTM recurrent unit:

1. Take input the current input, the previous hidden state, and the previous
internal cell state.
2. Calculate the values of the four different gates by following the below steps:-
○ For each gate, calculate the parameterized vectors for the current
input and the previous hidden state by element-wise multiplication
with the concerned vector with the respective weights for each
gate.
○ Apply the respective activation function for each gate element-wise
on the parameterized vectors. Below given is the list of the gates
with the activation function to be applied for the gate.
3. Calculate the current internal cell state by first calculating the element-wise
 multiplication vector of the input gate and the input modulation gate, then
calculate the element-wise multiplication vector of the forget gate and the
previous internal cell state and then adding the two vectors.

4. Calculate the current hidden state by first taking the element-wise hyperbolic
tangent of the current internal cell state vector and then performing element-
wise multiplication with the output gate.
The above-stated working is illustrated as below:-

Note that the blue circles denote element-wise multiplication. The weight matrix W
contains different weights for the current input vector and the previous hidden state for
each gate.
Just like Recurrent Neural Networks, an LSTM network also generates an output at
each time step and this output is used to train the network using gradient descent.

The only main difference between the Back-Propagation algorithms of Recurrent Neural
Networks and Long Short Term Memory Networks is related to the mathematics of the
algorithm.
Let be the predicted output at each time step and be the actual output at each
time step. Then the error at each time step is given by:-
The total error is thus given by the summation of errors at all time steps.

Similarly, the value can be calculated as the summation of the gradients at each
time step.

Using the chain rule and using the fact that is a function of and which indeed is a
function of , the following expression arises:-

Thus the total error gradient is given by the following:-

Note that the gradient equation involves a chain of for an LSTM Back-Propagation
while the gradient equation involves a chain of for a basic Recurrent Neural
Network.
How does LSTM solve the problem of vanishing and exploding gradients?
Recall the expression for .

The value of the gradients is controlled by the chain of derivatives starting from .
Expanding this value using the expression for :-

For a basic RNN, the term after a certain time starts to take values either greater
than 1 or less than 1 but always in the same range. This is the root cause of the

vanishing and exploding gradients problem. In an LSTM, the term does not have
a fixed pattern and can take any positive value at any time step. Thus, it is not
guaranteed that for an infinite number of time steps, the term will converge to 0 or
diverge completely. If the gradient starts converging towards zero, then the weights of
the gates can be adjusted accordingly to bring it closer to 1. Since during the training
phase, the network adjusts these weights only, it thus learns when to let the gradient
converge to zero and when to preserve it.
Deep Learning | Introduction to Long Short Term Memory
● Difficulty Level : Easy
● Last Updated : 29 Sep, 2021
Read
Discuss
Long Short Term Memory is a kind of recurrent neural network. In RNN output from the
last step is fed as input in the current step. LSTM was designed by Hochreiter &
Schmidhuber. It tackled the problem of long-term dependencies of RNN in which the
RNN cannot predict the word stored in the long-term memory but can give more
accurate predictions from the recent information. As the gap length increases RNN does
not give an efficient performance. LSTM can by default retain the information for a long
period of time. It is used for processing, predicting, and classifying on the basis of time-
series data.
Structure Of LSTM:
LSTM has a chain structure that contains four neural networks and different memory
blocks called cells.

Information is retained by the cells and the memory manipulations are done by the
gates. There are three gates –
1. Forget Gate: The information that is no longer useful in the cell state is removed with
the forget gate. Two inputs x_t (input at the particular time) and h_t-1 (previous cell
output) are fed to the gate and multiplied with weight matrices followed by the addition
of bias. The resultant is passed through an activation function which gives a binary
output. If for a particular cell state the output is 0, the piece of information is forgotten
and for output 1, the information is retained for future use.
2. Input gate: The addition of useful information to the cell state is done by the input
gate. First, the information is regulated using the sigmoid function and filter the values to
be remembered similar to the forget gate using inputs h_t-1 and x_t. Then, a vector is
created using tanh function that gives an output from -1 to +1, which contains all the
possible values from h_t-1 and x_t. At last, the values of the vector and the regulated
values are multiplied to obtain the useful information

3. Output gate: The task of extracting useful information from the current cell state to
be presented as output is done by the output gate. First, a vector is generated by
applying tanh function on the cell. Then, the information is regulated using the sigmoid
function and filter by the values to be remembered using inputs h_t-1 and x_t. At last,
the values of the vector and the regulated values are multiplied to be sent as an output
and input to the next cell.
Some of the famous applications of LSTM includes:
1. Language Modelling
2. Machine Translation
3. Image Captioning
 4. Handwriting generation
5. Question Answering Chatbots
Understanding of LSTM Networks
● Difficulty Level : Expert
● Last Updated : 25 Jun, 2021
Read
Discuss
This article talks about the problems of conventional RNNs, namely, the vanishing and
exploding gradients and provides a convenient solution to these problems in the form of
Long Short Term Memory (LSTM). Long Short-Term Memory is an advanced version of
recurrent neural network (RNN) architecture that was designed to model chronological
sequences and their long-range dependencies more precisely than conventional RNNs.
The major highlights include the interior design of a basic LSTM cell, the variations
brought into the LSTM architecture, and few applications of LSTMs that are highly in
demand. It also makes a comparison between LSTMs and GRUs. The article concludes
with a list of disadvantages of the LSTM network and a brief introduction of the
upcoming attention-based models that are swiftly replacing LSTMs in the real world.
Introduction:
LSTM networks are an extension of recurrent neural networks (RNNs) mainly
introduced to handle situations where RNNs fail. Talking about RNN, it is a network that
works on the present input by taking into consideration the previous output (feedback)
and storing in its memory for a short period of time (short-term memory). Out of its
various applications, the most popular ones are in the fields of speech processing, non-
Markovian control, and music composition. Nevertheless, there are drawbacks to RNNs.
First, it fails to store information for a longer period of time. At times, a reference to
certain information stored quite a long time ago is required to predict the current output.
But RNNs are absolutely incapable of handling such “long-term dependencies”. Second,
there is no finer control over which part of the context needs to be carried forward and
how much of the past needs to be ‘forgotten’. Other issues with RNNs are exploding
and vanishing gradients (explained later) which occur during the training process of a
network through backtracking. Thus, Long Short-Term Memory (LSTM) was brought
into the picture. It has been so designed that the vanishing gradient problem is almost
completely removed, while the training model is left unaltered. Long time lags in certain
problems are bridged using LSTMs where they also handle noise, distributed
representations, and continuous values. With LSTMs, there is no need to keep a finite
number of states from beforehand as required in the hidden Markov model (HMM).
LSTMs provide us with a large range of parameters such as learning rates, and input
and output biases. Hence, no need for fine adjustments. The complexity to update each
weight is reduced to O(1) with LSTMs, similar to that of Back Propagation Through Time
(BPTT), which is an advantage.
Exploding and Vanishing Gradients:
During the training process of a network, the main goal is to minimize loss (in terms of
error or cost) observed in the output when training data is sent through it. We calculate
the gradient, that is, loss with respect to a particular set of weights, adjust the weights
accordingly and repeat this process until we get an optimal set of weights for which loss
is minimum. This is the concept of backtracking. Sometimes, it so happens that the
gradient is almost negligible. It must be noted that the gradient of a layer depends on
certain components in the successive layers. If some of these components are small
(less than 1), the result obtained, which is the gradient, will be even smaller. This is
known as the scaling effect. When this gradient is multiplied with the learning rate which
is in itself a small value ranging between 0.1-0.001, it results in a smaller value. As a
consequence, the alteration in weights is quite small, producing almost the same output
as before. Similarly, if the gradients are quite large in value due to the large values of
components, the weights get updated to a value beyond the optimal value. This is
known as the problem of exploding gradients. To avoid this scaling effect, the neural
network unit was re-built in such a way that the scaling factor was fixed to one. The cell
was then enriched by several gating units and was called LSTM.
Architecture:
The basic difference between the architectures of RNNs and LSTMs is that the hidden
layer of LSTM is a gated unit or gated cell. It consists of four layers that interact with
one another in a way to produce the output of that cell along with the cell state. These
two things are then passed onto the next hidden layer. Unlike RNNs which have got the
only single neural net layer of tanh, LSTMs comprises of three logistic sigmoid gates
and one tanh layer. Gates have been introduced in order to limit the information that is
passed through the cell. They determine which part of the information will be needed by
the next cell and which part is to be discarded. The output is usually in the range of 0-1
where ‘0’ means ‘reject all’ and ‘1’ means ‘include all’.
Hidden layers of LSTM :
Each LSTM cell has three inputs , and and two outputs and . For
a given time t, is the hidden state, is the cell state or memory, is the current
data point or input. The first sigmoid layer has two inputs– and where is
the hidden state of the previous cell. It is known as the forget gate as its output selects
the amount of information of the previous cell to be included. The output is a number in
[0,1] which is multiplied (point-wise) with the previous cell state .
Conventional LSTM:

The second sigmoid layer is the input gate that decides what new information is to be
added to the cell. It takes two inputs and . The tanh layer creates a vector
of the new candidate values. Together, these two layers determine the information to be

stored in the cell state. Their point-wise multiplication tells us the amount of
information to be added to the cell state. The result is then added with the result of the

forget gate multiplied with previous cell state to produce the current cell
state . Next, the output of the cell is calculated using a sigmoid and a tanh layer. The
sigmoid layer decides which part of the cell state will be present in the output whereas
tanh layer shifts the output in the range of [-1,1]. The results of the two layers undergo
point-wise multiplication to produce the output ht of the cell.
Variations:
With the increasing popularity of LSTMs, various alterations have been tried on the
conventional LSTM architecture to simplify the internal design of cells to make them
work in a more efficient way and to reduce the computational complexity. Gers and
Schmidhuber introduced peephole connections which allowed gate layers to have
knowledge about the cell state at every instant. Some LSTMs also made use of a
coupled input and forget gate instead of two separate gates that helped in making both
the decisions simultaneously. Another variation was the use of the Gated Recurrent
Unit(GRU) which improved the design complexity by reducing the number of gates. It
uses a combination of the cell state and hidden state and also an update gate which
has forgotten and input gates merged into it.
LSTM(Figure-A), DLSTM(Figure-B), LSTMP(Figure-C) and DLSTMP(Figure-D)

1. Figure-A represents what a basic LSTM network looks like. Only one layer of
LSTM between an input and output layer has been shown here.
2. Figure-B represents Deep LSTM which includes a number of LSTM layers in
between the input and output. The advantage is that the input values fed to
the network not only go through several LSTM layers but also propagate
through time within one LSTM cell. Hence, parameters are well distributed
within multiple layers. This results in a thorough process of inputs in each
time step.
3. Figure-C represents LSTM with the Recurrent Projection layer where the
recurrent connections are taken from the projection layer to the LSTM layer
input. This architecture was designed to reduce the high learning
computational complexity (O(N)) for each time step) of the standard LSTM
RNN.
4. Figure-D represents Deep LSTM with a Recurrent Projection Layer consisting
of multiple LSTM layers where each layer has its own projection layer. The
 increased depth is quite useful in the case where the memory size is too
large. Having increased depth prevents overfitting in models as the inputs to
the network need to go through many nonlinear functions.
GRUs Vs LSTMs
In spite of being quite similar to LSTMs, GRUs have never been so popular. But what
are GRUs? GRU stands for Gated Recurrent Units. As the name suggests, these
recurrent units, proposed by Cho, are also provided with a gated mechanism to
effectively and adaptively capture dependencies of different time scales. They have an
update gate and a reset gate. The former is responsible for selecting what piece of
knowledge is to be carried forward, whereas the latter lies in between two successive
recurrent units and decides how much information needs to be forgotten.
Activation at time t:

Update gate:

Candidate activation:

Reset gate:

Another striking aspect of GRUs is that they do not store cell state in any way, hence,
they are unable to regulate the amount of memory content to which the next unit is
exposed. Instead, LSTMs regulate the amount of new information being included in the
cell. On the other hand, the GRU controls the information flow from the previous
activation when computing the new, candidate activation, but does not independently
control the amount of the candidate activation being added (the control is tied via the
update gate).
Applications:
LSTM models need to be trained with a training dataset prior to its employment in real-
world applications. Some of the most demanding applications are discussed below:

1. Language modelling or text generation, that involves the computation of

words when a sequence of words is fed as input. Language models can be
operated at the character level, n-gram level, sentence level or even
paragraph level.
2. Image processing, that involves performing analysis of a picture and
concluding its result into a sentence. For this, it’s required to have a dataset
comprising of a good amount of pictures with their corresponding descriptive
captions. A model that has already been trained is used to predict features of
 images present in the dataset. This is photo data. The dataset is then
processed in such a way that only the words that are most suggestive are
present in it. This is text data. Using these two types of data, we try to fit the
model. The work of the model is to generate a descriptive sentence for the
picture one word at a time by taking input words that were predicted
previously by the model and also the image.
3. Speech and Handwriting Recognition
4. Music generation which is quite similar to that of text generation where
LSTMs predict musical notes instead of text by analyzing a combination of
given notes fed as input.
5. Language Translation involves mapping a sequence in one language to a
sequence in another language. Similar to image processing, a dataset,
containing phrases and their translations, is first cleaned and only a part of it
is used to train the model. An encoder-decoder LSTM model is used which
first converts input sequence to its vector representation (encoding) and then
outputs it to its translated version.
Drawbacks:
As it is said, everything in this world comes with its own advantages and disadvantages,
LSTMs too, have a few drawbacks which are discussed as below:

1. LSTMs became popular because they could solve the problem of vanishing
gradients. But it turns out, they fail to remove it completely. The problem lies
in the fact that the data still has to move from cell to cell for its evaluation.
Moreover, the cell has become quite complex now with the additional features
(such as forget gates) being brought into the picture.
2. They require a lot of resources and time to get trained and become ready for
real-world applications. In technical terms, they need high memory-bandwidth
because of linear layers present in each cell which the system usually fails to
provide for. Thus, hardware-wise, LSTMs become quite inefficient.
3. With the rise of data mining, developers are looking for a model that can
remember past information for a longer time than LSTMs. The source of
inspiration for such kind of model is the human habit of dividing a given piece
of information into small parts for easy remembrance.
4. LSTMs get affected by different random weight initialization and hence
behave quite similar to that of a feed-forward neural net. They prefer small
weight initialization instead.
5. LSTMs are prone to overfitting and it is difficult to apply the dropout algorithm
to curb this issue. Dropout is a regularization method where input and
recurrent connections to LSTM units are probabilistically excluded from
activation and weight updates while training a network.
[Link]

LSTM – Derivation of Back propagation through time

● Difficulty Level : Hard
● Last Updated : 27 Dec, 2021
Read
Discuss
LSTM (Long short term Memory ) is a type of RNN(Recurrent neural network), which
is a famous deep learning algorithm that is well suited for making predictions and
classification with a flavour of the time. In this article, we will derive the algorithm
backpropagation through time and find the gradient value for all the weights at a
particular timestamp.
As the name suggests backpropagation through time is similar to backpropagation in
DNN(deep neural network) but due to the dependency of time in RNN and LSTM, we
will have to apply the chain rule with time dependency.

Let the input at time t in the LSTM cell be xt, the cell state from time t-1 and t be ct-1
and ct and the output for time t-1 and t be ht-1 and ht . The initial value of ct and ht at t =
0 will be zero.
Step 1 : Initialization of the weights .
Weights for different gates are :
Input gate : wxi, wxg, bi, whj, wg , bg

Forget gate : wxf, bf, whf

Output gate : wxo, bo, who

Step 2 : Passing through different gates .

Inputs: xt and ht-i , ct-1 are given to the LSTM cell

Passing through input gate:

Zg = wxg *x + whg * ht-1 + bg

g = tanh(Zg)
Zj = wxi * x + whi * ht-1 + bi
i = sigmoid(Zi)
 Input_gate_out = g*i

Passing through forget gate:

Zf = wxf * x + whf *ht-1 + bf

f = sigmoid(Zf)

Forget_gate_out = f

Passing through the output gate:

Zo = wxo*x + who * ht-1 + bo

o = sigmoid(zO)

Out_gate_out = o
Step 3 : Calculating the output ht and current cell state ct.
Calculating the current cell state ct :
ct = (ct-1 * forget_gate_out) + input_gate_out

Calculating the output gate ht:

ht=out_gate_out * tanh(ct)
Step 4 : Calculating the gradient through back propagation through time at time stamp t
using the chain rule.

Let the gradient pass down by the above cell be:

 E_delta = dE/dht

If we are using MSE (mean square error)for error then,

E_delta=(y-h(x))
Here y is the original value and h(x) is the predicted
value.

Gradient with respect to output gate

dE/do = (dE/dht ) * (dht /do) = E_delta * ( dht /

do)
dE/do = E_delta * tanh(ct)

Gradient with respect to ct

dE/dct = (dE / dht )*(dht /dct)= E_delta *(dht /dct)
dE/dct = E_delta * o * (1-tanh2 (ct))

Gradient with respect to input gate dE/di, dE/dg

dE/di = (dE/di ) * (dct / di)

dE/di = E_delta * o * (1-tanh2 (ct)) * g
Similarly,
dE/dg = E_delta * o * (1-tanh2 (ct)) * i

Gradient with respect to forget gate

dE/df = E_delta * (dE/dct ) * (dct / dt) t

dE/df = E_delta * o * (1-tanh2 (ct)) * ct-1

Gradient with respect to ct-1

dE/dct = E_delta * (dE/dct ) * (dct / dct-1)

dE/dct = E_delta * o * (1-tanh2 (ct)) * f

Gradient with respect to output gate weights:

dE/dwxo = dE/do *(do/dwxo) = E_delta * tanh(ct) *

sigmoid(zo) * (1-sigmoid(zo) * xt
dE/dwho = dE/do *(do/dwho) = E_delta * tanh(ct) *
sigmoid(zo) * (1-sigmoid(zo) * ht-1
dE/dbo = dE/do *(do/dbo) = E_delta * tanh(ct) *
sigmoid(zo) * (1-sigmoid(zo)
 Gradient with respect to forget gate weights:

dE/dwxf = dE/df *(df/dwxf) = E_delta * o * (1-tanh2 (ct)) *

ct-1 * sigmoid(zf) * (1-sigmoid(zf) * xt
dE/dwhf = dE/df *(df/dwhf) = E_delta * o * (1-tanh2 (ct)) *
ct-1 * sigmoid(zf) * (1-sigmoid(zf) * ht-1
dE/dbo = dE/df *(df/dbo) = E_delta * o * (1-tanh2 (ct)) *
ct-1 * sigmoid(zf) * (1-sigmoid(zf)

Gradient with respect to input gate weights:

dE/dwxi = dE/di *(di/dwxi) = E_delta * o * (1-tanh2 (ct)) *

g * sigmoid(zi) * (1-sigmoid(zi) * xt
dE/dwhi = dE/di *(di/dwhi) = E_delta * o * (1-tanh2 (ct)) *
g * sigmoid(zi) * (1-sigmoid(zi) * ht-1
dE/dbi = dE/di *(di/dbi) = E_delta * o * (1-tanh2 (ct)) * g
* sigmoid(zi) * (1-sigmoid(zi)

dE/dwxg = dE/dg *(dg/dwxg) = E_delta * o * (1-tanh2 (ct)) *

i * (1?tanh2(zg))*xt
dE/dwhg = dE/dg *(dg/dwhg) = E_delta * o * (1-tanh2 (ct)) *
i * (1?tanh2(zg))*ht-1
dE/dbg = dE/dg *(dg/dbg) = E_delta * o * (1-tanh2 (ct)) *
i * (1?tanh2(zg))
Finally the gradients associated with the weights are,

Using all gradient, we can easily update the weights associated with input gate, output
gate, and forget gate