MODULE 5
SYLLABUS
Recurrent neural networks – Computational graphs, RNN
design, encoder – decoder sequence to sequence
architectures, deep recurrent networks, recursive neural
networks, modern RNNs LSTM and GRU, Practical use
cases for RNNs. Case study – Build simple RNN for tasks in
speech and audio processing and image processing domains
Recurrent neural networks
Sentiment analysis using CNN
❑Consider a situation in which one tried to use a conventional neural network in
order to perform sentiment analysis.
❑The sentiment can be a binary label depending on whether it is positive or
negative.
❑The first problem that one would face is that the length of different sentences
is different.
❑ Therefore, if we used a neural network with 5 sets of one-hot encoded word
inputs it would be impossible to enter a sentence with more than five words.
❑Furthermore, any sentence with less than five words would have missing
inputs.
Sequence Learning Problems
Computational graphs
RNN design
❑We often call the input to the RNN the “context.” We want to produce a
representation of this context, C . The context C might be a vector or sequence of
vectors that summarize the input sequence X = (x(1), . . . , x(nx )).
✓Steps
(1) an encoder or reader or input RNN processes the input sequence. The
encoder emits the context C, usually as a simple function of its final hidden state.
(2) a decoder or writer or output RNN is conditioned on that fixed-length vector
to generate the output sequence Y = (y(1), . . . , y(ny )).
❑One clear limitation of this architecture is when the context C output by the
encoder RNN has a dimension that is too small to properly summarize a long
sequence.
❑In order to avoid this They proposed to make C a variable-length sequence
rather than a fixed-size vector. Additionally, they introduced an attention
mechanism that learns to associate elements of the sequence C to elements of the
output sequence.
Deep Recurrent Networks
❑each of these three blocks is associated with a single weight matrix.
❑In other words, when the network is unfolded, each of these
corresponds to a shallow transformation.
❑ By a shallow transformation, we mean a transformation that
would be represented by a single layer within a deep MLP.
❑Decomposing the state of an RNN into multiple layers as in Fig. 10.13 (left). We
can think of the lower layers in the hierarchy depicted in Fig. 10.13a as playing a
role in transforming the raw input into a representation that is more
appropriate, a the higher levels of the hidden state.
❑ Pascanu et al. (2014a) go a step further and propose to have a separate MLP
(possibly deep) for each of the three blocks enumerated above, as illustrated in
Fig. [Link] so by adding depth may hurt learning by making optimization
difficult.
❑In general, it is easier to optimize shallower architectures
❑For example, if an MLP with a single hidden layer is used for the state-to-state
transition, we have doubled the length of the shortest path between variables in
any two different time steps, compared with the ordinary RNN
❑This can be mitigated by introducing skip connections in the hidden-to-hidden
path, as illustrated in Fig. 10.13c.
Vanishing Gradient problem in RNN
❑ RNNs suffer from the problem of vanishing gradients, which hampers the learning of long data
sequences.
❑ The gradients carry information used in the RNN parameter update and when the gradient
becomes smaller and smaller, the parameter updates become insignificant which means no real
learning is done.
❑The network has an input sequence of vectors [x(1), x(2),…,
x(k)], at time step t the network has an input vector x(t).
❑Past information and learned knowledge is encoded in the
network state vectors [c(1), c(2),…, c(k-1)], at time step t the
network has an input state vector c(t-1).
❑The network has two weight matrices: Wrec and Win
connecting c(t-1) and x(t), the two parts of the input vector [c(t-
1), x(t)], to the hidden layer.
❑ For simplicity, W = [Wrec, Win].
❑The sigmoid function is used as the activation function in the
hidden layer.
❑The network outputs a single vector at the last time step
❑The last expression tends to vanish when k is large, this is due to the derivative of the tanh
activation function which is smaller than 1.
LSTM
❑Introducing self-loops to produce paths where the gradient
can flow for long durations is a core contribution of the initial
long short-term memory (LSTM) model.
❑The LSTM has been found extremely successful in many
applications,
❑such as unconstrained handwriting recognition
❑ speech recognition
❑handwriting generation
❑machine translation
❑image captioning.
❑LSTM network has three gates that update and control the cell states,
❑ these are the forget gate,
❑input gate and
❑ output gate.
❑The gates use hyperbolic tangent and sigmoid activation functions.
❑The forget gate controls what information in the cell state to forget, given new
information than entered the network.
❑The output gate controls what information encoded in the cell state is sent to the network as
input in the following time step, this is done via the output vector h(t).
hidden state output – 2 stage process