M06 TAN9224 02 GE C06 page 464

464 Chapter 6 Classification: Alternative Techniques

6. Training an ANN is a time consuming process, especially when the

number of hidden nodes is large. Nevertheless, test examples can be
classified rapidly.

7. Just like logistic regression, ANN can learn in the presence of interacting
variables, since the model parameters are jointly learned over all vari-
ables together. In addition, ANN cannot handle instances with missing
values in the training or testing phase.

6.8 Deep Learning

As described above, the use of hidden layers in ANN is based on the general
belief that complex high-level features can be constructed by combining sim-
pler lower-level features. Typically, the greater the number of hidden layers,
the deeper the hierarchy of features learned by the network. This motivates
the learning of ANN models with long chains of hidden layers, known as deep
neural networks. In contrast to “shallow” neural networks that involve only
a small number of hidden layers, deep neural networks are able to represent
features at multiple levels of abstraction and often require far fewer nodes per
❦ layer to achieve generalization performance similar to shallow networks.
Despite the huge potential in learning deep neural networks, it has re-
mained challenging to learn ANN models with a large number of hidden layers
using classical approaches. Apart from reasons related to limited computa-
tional resources and hardware architectures, there have been a number of
algorithmic challenges in learning deep neural networks. First, learning a deep
neural network with low training error has been a daunting task because of
the saturation of sigmoid activation functions, resulting in slow convergence
of gradient descent. This problem becomes even more serious as we move
away from the output node to the hidden layers, because of the compounded
effects of saturation at multiple layers, known as the vanishing gradient
problem. Because of this reason, classical ANN models have suffered from
slow and ineffective learning, leading to poor training and test performance.
Second, the learning of deep neural networks is quite sensitive to the initial
values of model parameters, chiefly because of the non-convex nature of the
optimization function and the slow convergence of gradient descent. Third,
deep neural networks with a large number of hidden layers have high model
complexity, making them susceptible to overfitting. Hence, even if a deep
neural network has been trained to show low training error, it can still suffer
from poor generalization performance.

❦
 M06 TAN9224 02 GE C06 page 465

6.8 Deep Learning 465

These challenges have deterred progress in building deep neural networks

for several decades and it is only recently that we have started to unlock their
immense potential with the help of a number of advances being made in the
area of deep learning. Although some of these advances have been around
for some time, they have only gained mainstream attention in the last decade,
with deep neural networks continually beating records in various competitions
and solving problems that were too difficult for other classification approaches.
There are two factors that have played a major role in the emergence of
deep learning techniques. First, the availability of larger labeled data sets, e.g.,
the ImageNet data set contains more than 10 million labeled images, has made
it possible to learn more complex ANN models than ever before, without falling
easily into the traps of model overfitting. Second, advances in computational
abilities and hardware infrastructures, such as the use of graphical processing
units (GPU) for distributed computing, have greatly helped in experimenting
with deep neural networks with larger architectures that would not have been
feasible with traditional resources.
In addition to the previous two factors, there have been a number of algo-
rithmic advancements to overcome the challenges faced by classical methods
in learning deep neural networks. Some examples include the use of more
❦ responsive combinations of loss functions and activation functions, better
initialization of model parameters, novel regularization techniques, more agile
architecture designs, and better techniques for model learning and hyper-
parameter selection. In the following, we describe some of the deep learning
advances made to address the challenges in learning deep neural networks.
Further details on recent developments in deep learning can be obtained from
the Bibliographic Notes.

6.8.1 Using Synergistic Loss Functions

One of the major realizations leading to deep learning has been the importance
of choosing appropriate combinations of activation and loss functions. Classical
ANN models commonly made use of the sigmoid activation function at the
output layer, because of its ability to produce real-valued outputs between 0
and 1, which was combined with a squared loss objective to perform gradient
descent. It was soon noticed that this particular combination of activation and
loss function resulted in the saturation of output activation values, which can
be described as follows.

❦
 M06 TAN9224 02 GE C06 page 466

466 Chapter 6 Classification: Alternative Techniques

1 0.25

0.8 0.2

0.6 0.15

0.4 0.1

0.2
0.05

0
0
−15 −10 −5 0 5 10 15 −15 −10 −5 0 5 10 15

(a) σ(z). (b) ∂σ(z)/∂z.

Figure 6.27. Plots of sigmoid function and its derivative.

Saturation of Outputs
Although the sigmoid has been widely-used as an activation function, it easily
saturates at high and low values of inputs that are far away from 0. Observe
❦ from Figure 6.27(a) that σ(z) shows variance in its values only when z is close
to 0. For this reason, ∂σ(z)/∂z is non-zero for only a small range of z around
0, as shown in Figure 6.27(b). Since ∂σ(z)/∂z is one of the components in the
gradient of loss (see Equation 6.55), we get a diminishing gradient value when
the activation values are far from 0.
To illustrate the effect of saturation on the learning of model parameters
at the output node, consider the partial derivative of loss with respect to the
weight wjL at the output node. Using the squared loss function, we can write
this as
∂ Loss
= 2(aL − y) × σ(z L )(1 − σ(z L )) × aL−1
j . (6.60)
∂wjL
In the previous equation, notice that when z L is highly negative, σ(z L ) (and
hence the gradient) is close to 0. On the other hand, when z L is highly positive,
(1 − σ(z L )) becomes close to 0, nullifying the value of the gradient. Hence,
irrespective of whether the prediction aL matches the true label y or not, the
gradient of the loss with respect to the weights is close to 0 whenever z L is
highly positive or negative. This causes an unnecessarily slow convergence of
the model parameters of the ANN model, often resulting in poor learning.
Note that it is the combination of the squared loss function and the sigmoid
activation function at the output node that together results in diminishing
gradients (and thus poor learning) upon saturation of outputs. It is thus

❦
 M06 TAN9224 02 GE C06 page 467

6.8 Deep Learning 467

important to choose a synergistic combination of loss function and activation

function that does not suffer from the saturation of outputs.

Cross entropy loss function

The cross entropy loss function, which was described in the context of logistic
regression in Section 6.6.2, can significantly avoid the problem of saturating
outputs when used in combination with the sigmoid activation function. The
cross entropy loss function of a real-valued prediction ŷ ∈ (0, 1) on a data
instance with binary label y ∈ {0, 1} can be defined as

Loss (y, ŷ) = −y log(ŷ) − (1 − y) log(1 − ŷ), (6.61)

where log represents the natural logarithm (to base e) and 0 log(0) = 0 for
convenience. The cross entropy function has foundations in information theory
and measures the amount of disagreement between y and ŷ. The partial
derivative of this loss function with respect to ŷ = aL can be given as

∂ Loss −y (1 − y)
❦
δL = = + .
∂aL aL (1 − aL )
(aL − y)
= . (6.62)
aL (1 − aL )

Using this value of δ L in Equation 6.56, we can obtain the partial derivative
of the loss with respect to the weight wjl at the output node as

∂ Loss (aL − y)
= × aL (1 − aL ) × aL−1 .
∂wjL aL (1 − aL ) j

= (aL − y) × aL−1
j . (6.63)

Notice the simplicity of the previous formula using the cross entropy loss
function. The partial derivatives of the loss with respect to the weights at
the output node depend only on the difference between the prediction aL and
the true label y. In contrast to Equation 6.60, it does not involve terms such
as σ(z L )(1 − σ(z L )) that can be impacted by saturation of z L . Hence, the
gradients are high whenever (aL − y) is large, promoting effective learning of
the model parameters at the output node. This has been a major breakthrough
in the learning of modern ANN models and it is now a common practice to use
the cross entropy loss function with sigmoid activations at the output node.

❦
 M06 TAN9224 02 GE C06 page 468

468 Chapter 6 Classification: Alternative Techniques

6.8.2 Using Responsive Activation Functions

Even though the cross entropy loss function helps in overcoming the problem
of saturating outputs, it still does not solve the problem of saturation at hidden
layers, arising due to the use of sigmoid activation functions at hidden nodes.
In fact, the effect of saturation on the learning of model parameters is even
more aggravated at hidden layers, a problem known as the vanishing gradient
problem. In the following, we describe the vanishing gradient problem and
the use of a more responsive activation function, called the rectified linear
output unit (ReLU), to overcome this problem.

Vanishing Gradient Problem

The impact of saturating activation values on the learning of model parameters
increases at deeper hidden layers that are farther away from the output node.
Even if the activation in the output layer does not saturate, the repeated
multiplications performed as we backpropagate the gradients from the output
layer to the hidden layers may lead to decreasing gradients in the hidden
layers. This is called the vanishing gradient problem, which has been one of
the major hindrances in learning deep neural networks.
❦ To illustrate the vanishing gradient problem, consider an ANN model that
consists of a single node at every hidden layer of the network, as shown in
Figure 6.28. This simplified architecture involves a single chain of hidden nodes
where a single weighted link wl connects the node at layer l − 1 to the node at
layer l. Using Equations 6.56 and 6.59, we can represent the partial derivative
of the loss with respect to wl as

∂ Loss
= δ l × al (1 − al ) × al−1 ,
∂wl
L−1
!
where δ l = 2(aL − y) × (ar+1 (1 − ar+1 ) × wr+1 ). (6.64)
r=l

Notice that if any of the linear predictors z r+1 saturates at subsequent lay-
ers, then the term ar+1 (1 − ar+1 ) becomes close to 0, thus diminishing the
overall gradient. The saturation of activations thus gets compounded and has

Figure 6.28. An example of an ANN model with only one node at every hidden layer.

❦
 M06 TAN9224 02 GE C06 page 469

6.8 Deep Learning 469

15

10

0
−15 −10 −5 0 5 10 15

Figure 6.29. Plot of the rectified linear unit (ReLU) activation function.

multiplicative effects on the gradients at hidden layers, making them highly

unstable and thus, unsuitable for use with gradient descent. Even though
the previous discussion only pertains to the simplified architecture involving
a single chain of hidden nodes, a similar argument can be made for any
❦ generic ANN architecture involving multiple chains of hidden nodes. Note
that the vanishing gradient problem primarily arises because of the use of
sigmoid activation function at hidden nodes, which is known to easily saturate
especially after repeated multiplications.

Rectified Linear Units (ReLU)

To overcome the vanishing gradient problem, it is important to use an activa-
tion function f (z) at the hidden nodes that provides a stable and significant
value of the gradient whenever a hidden node is active, i.e., z > 0. This
is achieved by using rectified linear units (ReLU) as activation functions at
hidden nodes, which can be defined as
"
z, if z > 0.
a = f (z) = (6.65)
0, otherwise.

The idea of ReLU has been inspired from biological neurons, which are either
in an inactive state (f (z) = 0) or show an activation value proportional to
the input. Figure 6.29 shows a plot of the ReLU function. We can see that it
is linear with respect to z when z > 0. Hence, the gradient of the activation

❦
 M06 TAN9224 02 GE C06 page 470

470 Chapter 6 Classification: Alternative Techniques

value with respect to z can be written as

"
∂a 1, if z > 0.
= (6.66)
∂z 0, if z < 0.

Although f (z) is not differentiable at 0, it is common practice to use ∂a/∂z = 0

when z = 0. Since the gradient of the ReLU activation function is equal to
1 whenever z > 0, it avoids the problem of saturation at hidden nodes, even
after repeated multiplications. Using ReLU, the partial derivatives of the loss
with respect to the weight and bias parameters can be given by

∂ Loss
l
= δil × I(zil ) × al−1
j , (6.67)
∂wij
∂ Loss
= δil × I(zil ), (6.68)
∂bli
n
#
where δil = (δil+1 × I(zil+1 ) × wij
l+1
),
❦
i=1
"
1, if z > 0.
and I(z) =
0, otherwise.

Notice that ReLU shows a linear behavior in the activation values whenever
a node is active, as compared to the nonlinear properties of the sigmoid func-
tion. This linearity promotes better flows of gradients during backpropagation,
and thus simplifies the learning of ANN model parameters. The ReLU is
also highly responsive at large values of z away from 0, as opposed to the
sigmoid activation function, making it more suitable for gradient descent.
These differences give ReLU a major advantage over the sigmoid function.
Indeed, ReLU is used as the preferred choice of activation function at hidden
layers in most modern ANN models.

6.8.3 Regularization
A major challenge in learning deep neural networks is the high model com-
plexity of ANN models, which grows with the addition of hidden layers in
the network. This can become a serious concern, especially when the train-
ing set is small, due to the phenomena of model overfitting. To overcome
this challenge, it is important to use techniques that can help in reducing

❦
 M06 TAN9224 02 GE C06 page 471

6.8 Deep Learning 471

the complexity of the learned model, known as regularization techniques.

Classical approaches for learning ANN models did not have an effective way
to promote regularization of the learned model parameters. Hence, they had
often been sidelined by other classification methods, such as support vector
machines (SVM), which have in-built regularization mechanisms. (SVMs will
be discussed in more detail in Section 6.9).
One of the major advancements in deep learning has been the development
of novel regularization techniques for ANN models that are able to offer
significant improvements in generalization performance. In the following, we
discuss one of the regularization techniques for ANN, known as the dropout
method, that have gained a lot of attention in several applications.

Dropout
The main objective of dropout is to avoid the learning of spurious features at
hidden nodes, occurring due to model overfitting. It uses the basic intuition
that spurious features often “co-adapt” themselves such that they show good
training performance only when used in highly selective combinations. On the
other hand, relevant features can be used in a diversity of feature combinations
❦ and hence are quite resilient to the removal or modification of other features.
The dropout method uses this intuition to break complex “co-adaptations”
in the learned features by randomly dropping input and hidden nodes in the
network during training.
Dropout belongs to a family of regularization techniques that uses the
criteria of resilience to random perturbations as a measure of the robustness
(and hence, simplicity) of a model. For example, one approach to regularization
is to inject noise in the input attributes of the training set and learn a model
with the noisy training instances. If a feature learned from the training data
is indeed generalizable, it should not be affected by the addition of noise.
Dropout can be viewed as a similar regularization approach that perturbs the
information content of the training set not only at the level of attributes but
also at multiple levels of abstractions, by dropping input and hidden nodes.
The dropout method draws inspiration from the biological process of gene
swapping in sexual reproduction, where half of the genes from both parents
are combined together to create the genes of the offspring. This favors the
selection of parent genes that are not only useful but can also inter-mingle
with diverse combinations of genes coming from the other parent. On the other
hand, co-adapted genes that function only in highly selective combinations are
soon eliminated in the process of evolution. This idea is used in the dropout

❦
 M06 TAN9224 02 GE C06 page 472

472 Chapter 6 Classification: Alternative Techniques

(a) Original network.

❦
(b) Sub-networks.

Figure 6.30. Examples of sub-networks generated in the dropout method using γ = 0.5.

method for eliminating spurious co-adapted features. A simplified description

of the dropout method is provided in the rest of this section.
Let (wk , bk ) represent the model parameters of the ANN model at the
k iteration of the gradient descent method. At every iteration, we randomly
th

select a fraction γ of input and hidden nodes to be dropped from the network,
where γ ∈ (0, 1) is a hyper-parameter that is typically chosen to be 0.5. The
weighted links and bias terms involving the dropped nodes are then eliminated,
resulting in a “thinned” sub-network of smaller size. The model parameters
of the sub-network (wsk , bks ) are then updated by computing activation values
and performing backpropagation on this smaller sub-network. These updated
values are then added back in the original network to obtain the updated
model parameters, (wk+1 , bk+1 ), to be used in the next iteration.
Figure 6.30 shows some examples of sub-networks that can be generated
at different iterations of the dropout method, by randomly dropping input
and hidden nodes. Since every sub-network has a different architecture, it is
difficult to learn complex co-adaptations in the features that can result in
overfitting. Instead, the features at the hidden nodes are learned to be more

❦
 M06 TAN9224 02 GE C06 page 473

6.8 Deep Learning 473

agile to random modifications in the network structure, thus improving their

generalization ability. The model parameters are updated using a different
random sub-network at every iteration, till the gradient descent method con-
verges.
Let (wkmax , bkmax ) denote the model parameters at the last iteration kmax
of the gradient descent method. These parameters are finally scaled down by
a factor of (1 − γ), to produce the weights and bias terms of the final ANN
model, as follows:

(w∗ , b∗ ) = ((1 − γ) × wkmax , (1 − γ) × bkmax )

We can now use the complete neural network with model parameters
(w∗ , b∗ )for testing. The dropout method has been shown to provide significant
improvements in the generalization performance of ANN models in a number
of applications. It is computationally cheap and can be applied in combination
with any of the other deep learning techniques. It also has a number of
similarities with a widely-used ensemble learning method known as bagging,
which learns multiple models using random subsets of the training set, and
then uses the average output of all the models to make predictions. (Bagging
❦
will be presented in more detail later in Section 6.10.4). In a similar vein, it
can be shown that the predictions of the final network learned using dropout
approximates the average output of all possible 2n sub-networks that can
be formed using n nodes. This is one of the reasons behind the superior
regularization abilities of dropout.

6.8.4 Initialization of Model Parameters

Because of the non-convex nature of the loss function used by ANN models,
it is possible to get stuck in locally optimal but globally inferior solutions.
Hence, the initial choice of model parameter values plays a significant role in
the learning of ANN by gradient descent. The impact of poor initialization is
even more aggravated when the model is complex, the network architecture is
deep, or the classification task is difficult. In such cases, it is often advisable
to first learn a simpler model for the problem, e.g., using a single hidden layer,
and then incrementally increase the complexity of the model, e.g., by adding
more hidden layers. An alternate approach is to train the model for a simpler
task and then use the learned model parameters as initial parameter choices
in the learning of the original task. The process of initializing ANN model
parameters before the actual training process is known as pretraining.
Pretraining helps in initializing the model to a suitable region in the
parameter space that would otherwise be inaccessible by random initialization.

❦
 M06 TAN9224 02 GE C06 page 474

474 Chapter 6 Classification: Alternative Techniques

Pretraining also reduces the variance in the model parameters by fixing the
starting point of gradient descent, thus reducing the chances of overfitting due
to multiple comparisons. The models learned by pretraining are thus more
consistent and provide better generalization performance.

Supervised Pretraining
A common approach for pretraining is to incrementally train the ANN model
in a layer-wise manner, by adding one hidden layer at a time. This approach,
known as supervised pretraining, ensures that the parameters learned at
every layer are obtained by solving a simpler problem, rather than learning
all model parameters together. These parameter values thus provide a good
choice for initializing the ANN model. The approach for supervised pretraining
can be briefly described as follows.
We start the supervised pretraining process by considering a reduced ANN
model with only a single hidden layer. By applying gradient descent on this
simple model, we are able to learn the model parameters of the first hidden
layer. At the next run, we add another hidden layer to the model and apply
gradient descent to learn the parameters of the newly added hidden layer, while
❦ keeping the parameters of the first layer fixed. This procedure is recursively
applied such that while learning the parameters of the lth hidden layer, we
consider a reduced model with only l hidden layers, whose first (l − 1) hidden
layers are not updated on the lth run but are instead fixed using pretrained
values from previous runs. In this way, we are able to learn the model param-
eters of all (L − 1) hidden layers. These pretrained values are used to initialize
the hidden layers of the final ANN model, which is fine-tuned by applying a
final round of gradient descent over all the layers.

Unsupervised Pretraining
Supervised pretraining provides a powerful way to initialize model parameters,
by gradually growing the model complexity from shallower to deeper net-
works. However, supervised pretraining requires a sufficient number of labeled
training instances for effective initialization of the ANN model. An alternate
pretraining approach is unsupervised pretraining, which initializes model
parameters by using unlabeled instances that are often abundantly available.
The basic idea of unsupervised pretraining is to initialize the ANN model
in such a way that the learned features capture the latent structure in the
unlabeled data.

❦
 M06 TAN9224 02 GE C06 page 475

6.8 Deep Learning 475

Figure 6.31. The basic architecture of a single-layer autoencoder.

Unsupervised pretraining relies on the assumption that learning the dis-

tribution of the input data can indirectly help in learning the classification
model. It is most helpful when the number of labeled examples is small and the
features for the supervised problem bear resemblance to the factors generating
❦ the input data. Unsupervised pretraining can be viewed as a different form
of regularization, where the focus is not explicitly toward finding simpler
features but instead toward finding features that can best explain the input
data. Historically, unsupervised pretraining has played an important role in
reviving the area of deep learning, by making it possible to train any generic
deep neural network without requiring specialized architectures.

Use of Autoencoders
One simple and commonly used approach for unsupervised pretraining is to
use an unsupervised ANN model known as an autoencoder. The basic archi-
tecture of an autoencoder is shown in Figure 6.31. An autoencoder attempts to
learn a reconstruction of the input data by mapping the attributes x to latent
features c, and then re-projecting c back to the original attribute space to
create the reconstruction x̂. The latent features are represented using a hidden
layer of nodes, while the input and output layers represent the attributes and
contain the same number of nodes. During training, the goal is to learn an
autoencoder model that provides the lowest reconstruction error, RE(x, x̂),
on all input data instances. A typical choice of the reconstruction error is the
squared loss function:
RE(x, x̂) = ||x − x̂||2 .

❦
 M06 TAN9224 02 GE C06 page 476

476 Chapter 6 Classification: Alternative Techniques

The model parameters of the autoencoder can be learned by using a similar

gradient descent method as the one used for learning supervised ANN models
for classification. The key difference is the use of the reconstruction error on
all training instances as the training loss. Autoencoders that have multiple
layers of hidden layers are known as stacked autoencoders.
Autoencoders are able to capture complex representations of the input
data by the use of hidden nodes. However, if the number of hidden nodes
is large, it is possible for an autoencoder to learn the identity relationship,
where the input x is just copied and returned as the output x̂, resulting in a
trivial solution. For example, if we use as many hidden nodes as the number
of attributes, then it is possible for every hidden node to copy an attribute
and simply pass it along to an output node, without extracting any useful
information. To avoid this problem, it is common practice to keep the number
of hidden nodes smaller than the number of input attributes. This forces the
autoencoder to learn a compact and useful encoding of the input data, similar
to a dimensionality reduction technique. An alternate approach is to corrupt
the input instances by adding random noise, and then learn the autoencoder
to reconstruct the original instance from the noisy input. This approach is
known as the denoising autoencoder, which offers strong regularization
❦ capabilities and is often used to learn complex features even in the presence
of a large number of hidden nodes.
To use an autoencoder for unsupervised pretraining, we can follow a similar
layer-wise approach like supervised pretraining. In particular, to pretrain the
model parameters of the lth hidden layer, we can construct a reduced ANN
model with only l hidden layers and an output layer containing the same num-
ber of nodes as the attributes and is used for reconstruction. The parameters
of the lth hidden layer of this network are then learned using a gradient descent
method to minimize the reconstruction error. The use of unlabeled data can
be viewed as providing hints to the learning of parameters at every layer that
aid in generalization. The final model parameters of the ANN model are then
learned by applying gradient descent over all the layers, using the initial values
of parameters obtained from pretraining.

Hybrid Pretraining
Unsupervised pretraining can also be combined with supervised pretraining by
using two output layers at every run of pretraining, one for reconstruction and
the other for supervised classification. The parameters of the lth hidden layer
are then learned by jointly minimizing the losses on both output layers, usually
weighted by a trade-off hyper-parameter α. Such a combined approach often

❦
 M06 TAN9224 02 GE C06 page 477

6.8 Deep Learning 477

shows better generalization performance than either of the approaches, since

it provides a way to balance between the competing objectives of representing
the input data and improving classification performance.

6.8.5 Characteristics of Deep Learning

Apart from the basic characteristics of ANN discussed in Section 6.7.3, the use
of deep learning techniques provides the following additional characteristics:

1. An ANN model trained for some task can be easily re-used for a different
task that involves the same attributes, by using pretraining strategies.
For example, we can use the learned parameters of the original task
as initial parameter choices for the target task. In this way, ANN pro-
motes re-usability of learning, which can be quite useful when the target
application has a smaller number of labeled training instances.

2. Deep learning techniques for regularization, such as the dropout method,

help in reducing the model complexity of ANN and thus promoting good
generalization performance. The use of regularization techniques is espe-
cially useful in high-dimensional settings, where the number of training
❦ labels is small but the classification problem is inherently difficult.

3. The use of an autoencoder for pretraining can help eliminate irrelevant

attributes that are not related to other attributes. Further, it can help
reduce the impact of redundant attributes by representing them as copies
of the same attribute.

4. Although the learning of an ANN model can succumb to finding infe-

rior and locally optimal solutions, there are a number of deep learning
techniques that have been proposed to ensure adequate learning of an
ANN. Apart from the methods discussed in this section, some other
techniques involve novel architecture designs such as skip connections
between the output layer and lower layers, which aids the easy flow of
gradients during backpropagation.

5. A number of specialized ANN architectures have been designed to handle

a variety of input data sets. Some examples include convolutional
neural networks (CNN) for two-dimensional gridded objects such as
images, and recurrent neural network (RNN) for sequences. While
CNNs have been extensively used in the area of computer vision, RNNs
have found applications in processing speech and language.