Unit-3

Deep Neural Networks

Trainer: Ms. Nidhi Grover Raheja
Deep Neural Network (DNN)
• A Deep Neural Network (DNN) is a type of Artificial Neural Network
(ANN) that has multiple hidden layers between the input and output
layers.
• Each layer in a DNN extracts higher-level features from the output of
the previous layer, enabling it to learn complex patterns in the data.

In simple terms:
• A DNN is just a neural network with more than one hidden layer —
allowing it to “go deeper” into understanding data.
Input Layer → Hidden Layer 1 → Hidden Layer 2 → Hidden Layer 3 → ... → Output Layer
Structure of Deep Neural Networks
• Deep learning automatically finds the important features for
classification because of deep neural networks, whereas in Machine
Learning, we had to define these features manually.
• As shown in the image above, Deep Learning works as follows:
1. At the lowest level, the network fixates on patterns of local contrast as
important.
2. The following layer is then able to use those patterns of local contrast to
fixate on things that resemble eyes, noses, and mouths
3. Finally, the top layer can apply those facial features to face templates.
4. A deep neural network is capable of composing more and more complex
features in each of its successive layers.
 Example: Image Classification (Cat vs Dog)
• Let’s say you are training a DNN to identify whether an image is of a cat or a dog.

Layer Role What it Learns

Input Layer Receives raw pixel values (e.g., 128×128 image → 16,384 inputs) Raw image data
Hidden Layer 1 Applies filters to detect edges Basic features (edges, corners)
Hidden Layer 2 Combines edges to detect shapes Ears, nose, eyes
Hidden Layer 3 Combines shapes to form higher-level features Cat or dog face
Output Layer Gives probability scores (e.g., [Cat=0.9, Dog=0.1]) Final classification
 Difference Between ANN and DNN
Basis ANN (Artificial Neural Network) DNN (Deep Neural Network)
A neural network with one or few hidden A neural network with many hidden layers (two
Definition
layers. or more).
Depth Shallow architecture (1 hidden layer). Deep architecture (multiple hidden layers).

Feature Learning Features are often manually extracted. Features are automatically learned from data.

Complexity Suitable for simple problems. Suitable for complex and large-scale problems.

Computation Time Requires less computation. Requires high computation power (GPUs).

Training Data Works with small datasets. Needs large datasets for good performance.

Example Predicting house price using few input variables. Image recognition, NLP, speech recognition.

Interpretability Easier to interpret. Harder to interpret (acts as a black box).

Higher accuracy for complex, high-dimensional
Performance Limited accuracy on complex data.
data.
Learning Capability Learns basic patterns. Learns hierarchical (low to high-level) patterns.
Applications of Deep Learning
• Speech Recognition: We all use Siri, Alexa, Google Assistant, Voice Search,
etc., which are voice-controlled intelligent assistants.
• Automatic Machine Translation: We all know that Google can instantly
translate between 100 different human languages, that too very quickly, as
if by magic. The technology behind Google Translate is called Machine
Translation and has been a savior for people who can’t communicate with
each other because of the difference in language.
• Instant Visual Translation: As you know, deep learning is used to identify
images that have letters and where the letters are in the scene. Once
identified, they can be turned into text, translated and the image recreated
with the translated text. This is often called instant visual translation.
• Now, imagine a situation where you have visited any other country whose
native language is not known to you. Well, no need to worry, using various
apps like Google Translate you can go ahead and perform instant visual
translations to read signs or shop boards written in another language. This
has been possible only because of Deep Learning.
• Automated Self Driven Cars: Google is trying to take their self-driving car initiative,
known as WAYMO, to a whole new level of perfection using Deep Learning.
• Therefore, rather than using old hand-coded algorithms, they can now program a
system that can learn by itself using data provided by different sensors.
• The name Waymo is not a traditional acronym with a full, formal expansion of each
letter. Instead, it is a combination of words meant to signify "a new way forward in
mobility".
Difficulty of training deep neural networks
1. Overfitting and Underfitting
• Balancing model complexity to ensure it generalizes well to new data is
challenging. Overfitting occurs when a model is too complex and captures noise
in the training data. Underfitting happens when a model is too simple and fails to
capture the underlying patterns.

2. Data Quality and Quantity

• Deep learning models require large, high-quality datasets for training. Insufficient
or poor-quality data can lead to inaccurate predictions and model failures.
Acquiring and annotating large datasets is often time-consuming and expensive.

3. Computational Resources
• Training deep learning models demands significant computational power and
resources. This can be expensive and inaccessible for many organizations. High-
performance hardware like GPUs and TPUs is often necessary to handle the
intensive computations.
4. Interpretability
• Deep learning models often function as "black boxes," making it difficult to
understand how they make decisions. This lack of transparency can be
problematic, especially in critical applications. Understanding the decision-making
process is crucial for trust and accountability.

5. Hyperparameter Tuning
• Finding the optimal settings for a model’s hyperparameters requires expertise.
This process can be time-consuming and computationally intensive.
Hyperparameters significantly impact the model’s performance, and tuning them
effectively is essential for achieving high accuracy.

6. Scalability
• Scaling deep learning models to handle large datasets and complex tasks
efficiently is a major challenge. Ensuring models perform well in real-world
applications often requires significant adjustments. This involves optimizing both
algorithms and infrastructure to manage increased loads.
7. Ethical and Bias Issues
• Deep learning models can inadvertently learn and perpetuate biases
present in the training data. This can lead to unfair outcomes and ethical
concerns. Addressing bias and ensuring fairness in models is critical for
their acceptance and trustworthiness.

8. Hardware Limitations
• Training deep learning models requires substantial computational
resources, including high-performance GPUs or TPUs. Access to such
hardware can be a bottleneck for researchers and practitioners.

9. Adversarial Attacks
• Deep learning models are susceptible to adversarial attacks, where subtle
perturbations to input data can cause misclassification. Robustness against
such attacks remains a significant concern in safety-critical applications.
Strategies to Overcome Deep Learning Challenges
• Addressing the challenges in deep learning is crucial for developing
effective and reliable models. By implementing the right strategies, we
can mitigate these issues and enhance the performance of our deep
learning systems.
• Here are the key strategies:

1.
2.

3.
4.

5.
6.

7.
Model Parameters Vs Hyperparameters
• The two most confusing terms in Machine Learning are Model
Parameters and Hyperparameters.
• In deep neural networks, the distinction between parameters and
hyperparameters is fundamental to understanding how these models
learn and are constructed.
1. Model Parameters
• Parameters are the internal variables of the model that are learned directly
from the training data during the training process as the algorithm used
tries to remember the mapping between the input features and the labels
or targets..
• They represent the model's internal state and are adjusted to minimize the
error between the model's predictions and the actual target values.
Parameter Learning and Training Process:
1. Model training typically starts with parameters being initialized to some
values (random values or set to zeros).
2. As training/learning progresses, the initial values are updated using an
optimization algorithm (e.g., gradient descent).
3. The learning algorithm is continuously updating the parameter values as
learning progresses but hyperparameter values set by the model
designer remain unchanged.
4. At the end of the learning process, model parameters are what
constitute the model itself.
Learning Outcome of Model Parameters:
• Parameters are updated iteratively during training using optimization
algorithms like Gradient Descent and its variants (e.g., Adam,
RMSprop), often guided by backpropagation.

Examples of parameters
1. The coefficients (or weights) of linear and logistic regression
models.
2. Weights and biases of an NN
3. The cluster centroids in clustering
2. Hyperparameters
• Hyperparameters are external configuration variables that the
practitioner sets before the training process begins.
• They control the overall structure of the model and the learning
process itself, influencing how the parameters are learned.
• The prefix ‘hyper_’ suggests that they are ‘top-level’ parameters that
control the learning process and the model parameters that result
from it.
• As a machine learning engineer designing a model, you choose and
set hyperparameter values that your learning algorithm will use
before the training of the model even begins.
• In this light, hyperparameters are said to be external to the model
because the model cannot change its values during learning/training.
Learning Hyperparameters in a Model
• Hyperparameters are not learned from the data but are typically
chosen through experimentation, domain knowledge, or systematic
search techniques like grid search, random search, or Bayesian
optimization.
• Hyperparameters are used by the learning algorithm when it is
learning but they are not part of the resulting model.
• At the end of the learning process, we have the trained model
parameters which effectively is what we refer to as the model.
• The hyperparameters that were used during training are not part of
this model.
Examples of Hyperparameters in Deep Neural Networks:
1. Weights and Biases
2. Train-test split ratio
3. Learning rate in optimization algorithms (e.g., gradient descent)
4. Choice of optimization algorithm (e.g., gradient descent, stochastic gradient descent,
or Adam optimizer)
5. Choice of activation function in a neural network (NN) layer (e.g., Sigmoid, ReLU, Tanh)
6. The choice of the cost or loss function that the model will use
7. Number of hidden layers in a NN
8. Number of activation units in each layer
9. The drop-out rate in NN (dropout probability)
10. Number of iterations (epochs) in training a nn
11. Number of clusters in a clustering task
12. Kernel or filter size in convolutional layers
13. Pooling size
14. Batch size
Summary
• Basically, anything in machine learning and deep learning that you
decide their values or choose their configuration before training
begins, and whose values or configuration will remain the same when
training ends, is a hyperparameter.
Greedy layer-wise training (GLWT)
• Before the development of modern GPUs and techniques like dropout
and ReLU activations, training deep networks (with many layers) was
extremely difficult due to:
Poor weight initialization
Limited labeled data
Training deep networks from scratch using backpropagation often
failed.
Problems like vanishing gradients/exploding gradients, slow
convergence, and poor generalization occurred.
• To solve this, researchers, notably Geoffrey Hinton around 2006)
proposed Greedy Layer-Wise Training — a way to train deep neural
networks one layer at a time in an unsupervised or semi-supervised
fashion.
Greedy layer-wise training (GLWT)
• Instead of training all layers of a deep network together,
we train one layer at a time — greedily — meaning each layer tries
to do its best without waiting for the others.
• Each layer learns to represent the data slightly better (more
abstractly) than the previous one.
• Assume you have a deep neural network with multiple hidden layers:
Input → Layer 1 → Layer 2 → Layer 3 → Output

Instead of training this whole network at once, we do it layer by layer:

Working of GLWT
1. Train the first layer as an unsupervised autoencoder (or Restricted
Boltzmann Machine, RBM).
Unsupervised Neural Network Models in Step 1

• In unsupervised learning, we don’t have labeled data (no

target/output). The system learns patterns or structures hidden in
the input data itself.
• Example: If we give a neural network many images of animals (but
without telling which animal), it tries to discover features that group
similar images— e.g., “has fur,” “has four legs,” “has wings,” etc.
• These are neural network architectures that learn useful
representations or features from raw input data without labels.
• They form the core of Greedy Layer-Wise Training because each layer
is trained to reconstruct or model its input before passing
information upward.
 Types of Unsupervised Neural Network Models
Model Key Idea Example Usage

Learns to reconstruct its input through a hidden compressed

representation. It has two parts:
Used for Dimensionality reduction,
1. Encoder: Compresses input into a smaller representation
feature learning.
(latent space)
Example:
2. Decoder: Reconstructs the original input from the encoded
If we input a 28×28 MNIST digit,
representation
the autoencoder learns to
Input → [Encoder] → Hidden/Latent → [Decoder] →
compress it into a small code (say,
1. Autoencoder Reconstructed Output. Let us see it’s working:
32 values)
(AE)
and then reconstruct the digit
back.
After training, the hidden layer
contains useful features that
represent shapes or edges.
Model Key Idea Example Usage
Used in :
• A probabilistic neural network that learns to model data distribution.
 Pretraining deep neural networks
• It is used to pretrain layers in Deep Belief Networks (DBNs) — the
(as in DBNs)
original way to do Greedy Layer-Wise Training. RBMs have 2 layers :
 Collaborative filtering (movie
Layer Role Example recommendation)
Takes the input data Example: image pixels, movie  Feature extraction
Visible layer (v)
(what you show it). preferences, or movie ratings.
Example1. : Suppose we give binary input
Tries to find patterns or Example: edges in an image, or
patterns like
2. Hidden layer (h) learn features inside that “action-movie lovers” pattern in
[1,0,1,0,1,0,0,1] representing image
data. movie data.
Restricted pixels (on/off).
Boltzmann It’s called “Restricted” because: RBM learns to represent statistical
Machine There are no connections between nodes dependencies — e.g., “when pixel 1 is
(RBM) within the same layer on, pixel 4 is likely off”. After training, the
hidden units represent abstract patterns
(only between layers).
or features in the data.
Example 2: Imagine your data is whether
[Visible Layer] [Hidden Layer] a person likes movies. The RBM learns
(no visible-visible or hidden-hidden hidden patterns like:
connections) “People who like Action and Comedy
often dislike Romance” or “People who
Each visible node connects to all hidden like Romance often like Comedy”
nodes. After training, hidden neurons capture
these correlations.
How RBM Works (Step-by-Step)
Model Key Idea Example Usage

A stack of multiple RBMs trained layer- Feature learning, initialization for

3. Deep Belief Network (DBN)
by-layer. deep networks

Projects high-dimensional data to a

4. Self-Organizing Map (SOM) Data visualization, clustering
low-dimensional map (grid).

5. Generative Adversarial
Learns to generate new data similar to
Network (GAN) (partially Image generation, data synthesis
input examples.
unsupervised)
Working of GLWT (cont.)
2. Freeze the first layer and use its output as input to train the second
layer.
Layer 2 learns higher-level features from the representation of Layer 1.
Working of GLWT
3. Continue this process for each layer.
Each layer is trained independently.
Repeat this process for all hidden layers: each time using the output of the
previous layer as the input for the next.

4. Fine-tuning
After all layers are pretrained:
 Stack them together to form the full network.
 Now fine-tune the entire model using supervised learning (backpropagation)
on labeled data.
 This adjusts all weights slightly to improve final performance.
Final Verdict
• Greedy Layer-Wise Training is a bottom-up unsupervised pretraining
method where each layer learns independently to extract meaningful
features, and the full network is later fine-tuned using supervised
learning.
Example: Greedy Layer-Wise Training using
Autoencoders
• Let’s take a simple example: Suppose we want to build a Deep
Autoencoder for MNIST handwritten digits (28×28 images).

Layer 1: First Autoencoder

• Input: Raw pixels (784)
• Hidden layer: 500 neurons
• Train Autoencoder 1 to reconstruct input. It learns feature
representation .
• After training, discard the decoder and keep only encoder weights.
Layer 2: Second Autoencoder
• Input: (500)
• Hidden layer: 250 neurons
• Train Autoencoder 2 to reconstruct .
It learns higher-level features .

Layer 3: Third Autoencoder

• Input: (250)
• Hidden layer: 100 neurons
• Train Autoencoder 3 to reconstruct .
It learns even more abstract features .

Fine-tune
• Stack encoders: Input(784) → 500 → 250 → 100 → Output(10)
• Add a softmax layer for digit classification.
• Train the whole network using backpropagation on labeled data (digits 0–
9).
Advantages of GLWT
1. Solves vanishing gradients: It provides a better initialization for
deep networks, which helps prevent the vanishing gradient problem
during the subsequent supervised training phase.
2. Improved accuracy: It can lead to better model accuracy compared
to training a deep network from random initialization.
3. Data efficiency: It is particularly useful when you have a lot of
unlabeled data and a smaller amount of labeled data.
4. Faster training: Pre-training with good initial weights can speed up
the overall training process.
Limitations of GLWT
1. Computational cost: Pre-training each layer can be computationally
intensive.
2. Requires large datasets: While it can be used with small labeled
datasets, the unsupervised pre-training stage still requires a large
amount of data to be effective.
3. Not always necessary: With the advent of other techniques like
improved activation functions and architectures, the need for this
specific pre-training method has decreased for many modern deep
learning tasks.
Recurrent Neural Networks
(RNNs)
Types of Neural Networks
• There are three types of widely used Neural Networks:

1. Artificial Neural Network (ANN)

2. Convolutional Neural Network (CNN)
3. Recurrent Neural Network (RNN)
 Artificial Neural Network Convolutional Neural Network Recurrent Neural Network
Feature
(ANN) (CNN) (RNN)

 Basic form of neural  Specialized for image or

 Designed for sequential
network. spatial data.
or time-dependent data.
 Consists of input layer,  Uses convolutional layers with
 Has loops so it can
hidden layers, and filters that automatically
remember information
output layer. detect patterns like edges,
Structure / Working from previous steps
 Each neuron passes shapes, and textures.
(context memory).
information forward (no  Followed by pooling and fully
 Data flows in a
loops). connected layers.
sequence rather than
 Works on tabular or
just forward.
structured data.

 Text and speech

 Image classification (cats vs.
 Classification (spam or processing (chatbots,
dogs)
not spam) translation)
 Object detection, facial
Use Cases  Regression (predicting  Time series forecasting
recognition
prices, sales, etc.) (stock, weather)
 Medical image analysis,
 Basic prediction models  Sentiment analysis,
autonomous vehicles
handwriting recognition
 Artificial Neural Network Convolutional Neural Network Recurrent Neural Network
Feature
(ANN) (CNN) (RNN)

Data Type Structured / tabular Image / spatial Sequential / time series

Memory of Past Data No No Yes

Main Layers Used Dense (fully connected) Convolution + Pooling Recurrent (with feedback)

Detects visual features Handles sequences and

Key Advantage Simple and versatile
automatically temporal patterns

Predict next word in a

Example Application Predict house prices Identify handwritten digits
sentence
What is a Recurrent Neural Network (RNN)?
• A Recurrent Neural Network (RNN) is a type of neural network that is
specially designed to handle sequential data — data that comes in a
sequence where order matters.
Examples of sequential data:
1. Sentences in a paragraph (sequence of words)
2. Stock prices over days (sequence of numbers)
3. Speech or audio signals (sequence of sound frames)
4. Sensor readings over time
• Unlike normal neural networks (which take a fixed-size input and
produce a fixed-size output), RNNs can process variable-length
sequences.
How RNNs Are Different from Normal Neural
Networks
• In a traditional feedforward neural network, all inputs and outputs
are independent of each other.
• But in an RNN, the output at one time step depends on the previous
inputs — because it has a memory.
This “memory” helps it remember what it saw before.
• So, RNNs are like humans who read a sentence word by word and
remember the context.
Key Idea: Hidden State (Memory)
• An RNN maintains a hidden state, which acts as a kind of memory of
what the network has seen so far.
This hidden state is updated at each time step as new input comes in.
So, at every step:
A. RNN takes the current input (say the current word)
B. Takes the previous hidden state (memory from the last word)
C. Produces a new hidden state (updated memory)
D. Produces an output (like the next word prediction)
 The structure of Recurrent Neural Networks (RNNs)

Simple Feed Forward Neural Network architecture. Simple Recurrent Neural Network architecture.
 First, let’s remind ourselves what a typical Feed Forward Neural Network looks like. Note
that it can contain any number of input nodes, hidden nodes, and output nodes. The below
2–3–2 structure is purely for illustration.

Simple Feed Forward Neural Network architecture. Simple Recurrent Neural Network architecture.

Next, if we look at RNN, we notice a slight difference. The hidden units inside RNN have a
built-in feedback loop, enabling the information to be passed back to the same node multiple
times. These hidden units are commonly called RECURRENT UNITS.
Recurrent unit operation
• A recurrent unit processes information for a predefined number of timesteps,
each time passing a hidden state and an input for that specific timestep through
an activation function.
• Timestep – single processing of the inputs through the recurrent unit. E.g., if you
have only one timestep, your inputs will be processed only once (equivalent to a
regular hidden node). If you have 7 timesteps, your inputs will be processed 7
times. See the illustration below showing the feedback loop inside the recurrent
unit:

Recurrent unit operation at timestep t.

• Note that at the initial timestep, the hidden state h0 is initialized to 0.
• Next, the output (a hidden state h at t+1) is passed back to a recurrent unit and
processed again together with the following input:

The process repeats until the specified number of timesteps is reached.

Let’s tie all of it together and see what a simple RNN with one input,
one hidden node (containing three timesteps), and one output would
look like.
Assume you want to predict tomorrow’s air temperature based on the
sequence of air temperatures from the last three days. Then, an RNN typically
has three main layers with following actions:

Layer Purpose Description

Takes one element of the sequence
1. Input Layer Feeds the input sequence
at each time step
Stores information about past
2. Hidden (Recurrent) Layer Main processing + memory
inputs (acts like memory)
Converts hidden state to
3. Output Layer Produces output at each step
meaningful output

Training: A typical feed-forward neural network is trained using a

backpropagation algorithm. Meanwhile, training an RNN uses a slightly
modified version of backpropagation, which includes the unfolding in time to
train the weights of the network. The algorithm is based on computing the
gradient vector and is called BACKPROPAGATION IN TIME or BPTT for short.
Types of RNN architectures
1. One to One: This type of RNN behaves the same as any simple
Neural network; it is also known as a Vanilla Neural Network. In
this Neural network, there is only one input and one output.
2. One To Many: In this type of RNN, there is one input and many
outputs associated with it. One of the most used examples of this
network is Image captioning, where given an image, we predict a
sentence having Multiple words.
3. Many to One: In this type of network, Many inputs are fed to the network at
several stages of the network, generating only one output. This type of network is
used in problems like sentiment analysis. We give multiple words as input and
predict only the sentiment of the sentence as output.
4. Many-to-Many: In this type of neural network, multiple inputs and multiple
outputs correspond to a problem. One Example of this Problem will be language
translation. In language translation, we provide multiple words from one language
as input and predict multiple words from the second language as output.
Backpropagation through Time (BPTT)
• In RNNs, the data is processed step-by-step (one at a time).
• Backpropagation through time (BPTT) is a method used in recurrent
neural networks (RNNs) to train the network by backpropagating
errors through time.
• In a traditional feedforward neural network, the data flows through
the network in one direction, from the input layer through the hidden
layers to the output layer.
• However, in RNNs, there are connections between nodes in different
time steps, which means that the output of the network at one time
step depends on the input at that time step as well as the previous
time steps.
What is a Time Step?
• A time step in a Recurrent Neural Network (RNN) refers to a specific
moment or instance in a sequence of data being processed by the
network.
• In the context of sequential data, such as text, speech, or time series,
the RNN processes one element of the sequence at each time step.
• This element could be a word in a sentence, a single data point in a
time series, or a phoneme in speech.
Forward and Backward Pass in BPTT
• In RNN, Backpropagation is applied to each time step, starting from the
last one and moving backward to the first. This process is known as
Backpropagation Through Time.
• Forward Pass:
• Here is the diagram that shows the inputs, weights, outputs, and hidden
states, of an RNN,
• The Xt, Xt−1, and Xt−2 are the inputs to each corresponding time step,
and yt, yt−1, and yt−2 are the outputs from each time step. However, we need
to represent the information passing from one time step to another. Let's
call ht the hidden state at a particular time step. So automatically, the
hidden states in the previous layers become ht−1, ht−2, and so on.

• The basic idea of the forward pass in an RNN is to compute the hidden state
values at each time step, i.e., ht. So, considering this idea, the general
formula for performing forward propagation through the hidden states will
be

• In RNNs and their variations, we typically use a Hyperbolic tangent (Tanh)

activation function, so when applying the tanh activation the formula can be
changed to,
• We now have the formula for passing inputs forward through hidden
states. However, to achieve the intended outcome, we also need to obtain
outputs at each time step. Here is the formula to do so.

• Simple, just computing the dot product between Woh and ht plus a bias bo.
The ϕ is basically the activation function, which can be any function
according to the problem you are trying to solve. For instance, you can use
sigmoid or SoftMax if you are dealing with probability problems.
Backward pass
• Let's figure out how to derive BPTT in RNN. The first thing we need to do
is to choose a Loss function. Here, we choose the Mean Squared Error
(MSE) Loss.

• Below is the formula for calculating the MSE loss function,

• For propagating the loss backward, we need to find the derivative of loss;
the derivative of MSE is simply,
Here is a way to understand the flow of gradients in a single time step,
When dealing with multiple time steps, the gradients of each time step should
be summed up.
The generalized formula is:
Updating Weights and Bias Using Gradient Descent

The update rule is quite similar across different types of neural networks, including
recurrent neural networks (RNNs), with some variations depending on the
optimization algorithm used. Since we are using Gradient Descent, the update rule
will be,
 Output Output Output
We can visualize
each time step as a
collection of Feed
Forward Neural
Networks where the
hidden layers of
each network are
connected together.
Now if you flatten it
you'll get something
like this, on the right.

Timestamp 1 Timestamp 2 Timestamp 3

• On the left side is a recurrent neuron, and on the right-hand side is the recurrent
neuron unrolled through time.
• This adds some inherent “memory” in the system that aids the model in picking up
historical patterns that happened previously in time.
• When predicting Y1, the recurrent neuron uses the inputs of X1 and the output from
the previous time step, Y0. This means that Y0′s influence on Y1 is direct, and it also
indirectly influences Y2.
Backpropagation is applied
to each time step, starting
from the last one and
moving backward to the
first. This process is known
as Backpropagation
Through Time.
Why tanh (Hyperbolic Tangent) used as
Activation Function?
• The hyperbolic tangent (tanh) activation function is frequently employed in
Recurrent Neural Networks (RNNs) due to several key advantages:
1. Zero-Centered Output: Tanh outputs values ranging from -1 to 1. This
zero-centered property aids in the optimization process during training, as it
can lead to more stable and efficient weight updates compared to activation
functions like sigmoid, which output values between 0 and 1. The gradients
tend to oscillate less, potentially leading to faster convergence.
2. Stronger Gradients: Compared to the sigmoid function, tanh exhibits
steeper gradients across a larger portion of its domain. This characteristic
can result in faster learning and convergence during the backpropagation
through time (BPTT) process in RNNs.
3. Better Representation of State Changes: In RNNs, the hidden state acts as
a form of memory, carrying information from previous time steps. The ability
of tanh to output both positive and negative values (between -1 and 1)
allows for a more nuanced and precise representation of changes in the
hidden state. This can be particularly beneficial for modeling complex
temporal dependencies and sequences.
4. Differentiability for Gradient-Based Optimization: Tanh is a differentiable
function, which is a prerequisite for using gradient-based optimization
algorithms like stochastic gradient descent (SGD) to train neural networks,
including RNNs.
While tanh offers advantages, it is important to note that it can still suffer
from the vanishing gradient problem, especially in very deep RNNs or when
dealing with extremely long sequences. This occurs when the gradients
become very small, hindering effective learning in earlier layers. However,
the zero-centered output and stronger gradients of tanh often make it a
more effective choice than sigmoid for hidden layers in many RNN
architectures.
Problems in BPTT
BPTT faces two common issues when applied to long sequences. When we
train it using Backpropagation Through Time (BPTT), gradients either cause:
1. Vanishing Gradient Problem: When gradients become very small as they’re
multiplied through time, the network “forgets” early information.
2. Exploding Gradient Problem: When gradients grow too large, and cause
unstable training.
As a result, simple RNNs can’t learn long-term relationships in sequences (like
meaning in long sentences).
Solution:

Use advanced RNN architectures like:

A. LSTM (Long Short-Term Memory)
B. GRU (Gated Recurrent Unit)
These use gates to control the flow of information and preserve long-term
dependencies.
Applications of RNNs in NLP (Natural Language
Processing)
• Natural Language Processing deals with sequences of words, where
the meaning of a word depends on previous words (context).
• For example:
“He went to the bank to deposit money.”
vs.
“He sat by the bank of the river.”
• A simple neural network (like a feedforward NN) cannot handle this
sequence dependency because it processes all inputs independently.
• RNNs solve this by maintaining a hidden state — a kind of memory
that carries information from previous time steps (words).
How RNN Works in NLP?
Major Applications of RNNs in NLP
1. Text Classification: Used to classify text sequences — for example,
sentiment analysis, spam detection, or topic categorization.
Example:
Input: “The movie was absolutely wonderful.”
Output: Positive sentiment

How RNN helps:

• It reads words one by one.
• Maintains a hidden state capturing emotional tone.
• At the end of the sequence, it outputs the overall sentiment.
2. Language Modeling & Text Generation: Used to predict the next
word given previous ones.
Example:
Input: “I love to play”
Output: “football”
• RNNs are trained to predict the next word in the sequence.
Once trained, you can use it to generate text word by word, as in:
“Once upon a time … there was a little girl … who loved …”
Applications:
Predictive keyboards
Autocomplete systems
Chatbots
Story or code generation
3. Machine Translation (Seq2Seq Models)
• Used for translating text from one language to another — e.g.,
English → Hindi.
Example:
Input: “How are you?”
Output: “आप कैसे ह?”
Working:
An Encoder RNN reads the English sentence and summarizes it into a
context vector.
A Decoder RNN generates the Hindi sentence from that context.
Foundation for:
Google Translate
Neural Machine Translation (NMT)
Sequence-to-sequence models
4. Speech Recognition
• Although this involves audio, it’s still an NLP-related sequence task.
Example:
Audio Input: “Hello world”
Output Text: “hello world”
• RNNs (especially LSTMs & BiLSTMs) help model the temporal nature
of sound sequences, converting them into text.
Used in:
• Siri, Alexa, Google Assistant
• Speech-to-text engines
5. Named Entity Recognition (NER)
• NER identifies important entities (names, locations, organizations, dates,
etc.) in text and classifies them into predefined categories.
Example:
“Barack Obama was born in Hawaii.”
→ Barack Obama → Person
→ Hawaii → Loca on
Why RNNs are useful:
They capture context (e.g., knowing “Barack” before “Obama” signals it’s a
person’s name). NER helps computers understand who/what/where is
mentioned in text — essential for chatbots, search engines, and
information extraction systems.
Often implemented using:
• BiLSTM + CRF models (Bidirectional LSTM with Conditional Random
Fields)
Workflow of RNN for NER

Let’s break down the RNN-based NER pipeline:

Step Description

1. Input Text Sentence is tokenized into words.

Converts each word into a numeric vector (e.g., using Word2Vec, GloVe, or
2. Word Embedding Layer
learned embeddings).

3. RNN Layer (LSTM/GRU) Processes words one by one, maintaining context through hidden states.

For each word, outputs a probability distribution over entity labels (e.g.,
4. Output Layer
PERSON, ORG, LOC, O).

5. Prediction The label with the highest probability is chosen for each word.
Example Sentence and Flow
Let’s take:
“Barack Obama visited Delhi.”

Step 1: Input Sequence

Tokens:
["Barack", "Obama", "visited", "Delhi"]

Step 2: Embedding

Each word is converted into a dense vector (say, 50 dimensions):

x₁ = [0.3, 0.6, 0.9, ...]

x₂ = [0.2, 0.7, 0.4, ...]
...
Time Step (t) Word Hidden State ( ht ) (Memory)
t=1 Barack Context about “Barack”

t=2 Obama Context about “Barack Obama” (likely a PERSON)

t=3 visited Context now includes subject + verb

t=4 Delhi Context indicates a LOCATION

Word PERSON ORG LOC O Predicted Label

Barack 0.95 0.02 0.01 0.02 PERSON

Obama 0.98 0.01 0.00 0.01 PERSON

visited 0.01 0.00 0.01 0.98 O

Delhi 0.03 0.02 0.90 0.05 LOC

Final Output:

Barack (PERSON)
Obama (PERSON)
visited (O)
Delhi (LOCATION)
6. Part-of-Speech (POS) Tagging
• Assigns grammatical tags to words.
Example:
“She eats apples.”
→ She (PRONOUN), eats (VERB), apples (NOUN)
Why RNNs work well:
They learn that:
“She” likely precedes a verb.
“Eats” is followed by a noun.
Common architectures:
RNN
BiLSTM
GRU
Long Short-Term Memory (LSTM)
 Problems of RNN
Problem Explanation Example
During backpropagation, gradients (used
In a long sentence, RNN forgets the
to update weights) become extremely
1. Vanishing Gradient Problem subject when reaching the verb at the
small → the network “forgets” early
end.
information.

Some mes gradients become very large → The model gives NaN (not-a-number)
2. Exploding Gradient Problem
unstable training and weight overflow. or diverges.

RNNs remember only recent inputs — In “The man who went to the park is
3. Short-Term Memory they can’t capture long-term tall,” the word “man” should link to “is
dependencies. tall,” but RNN forgets it.
Simple RNNs treat all previous data
Doesn’t know that “not” changes
4. Difficulty in Learning Context equally, can’t decide what to remember or
meaning of “good” in “not good.”
forget.
Gradients must pass through every time
5. Slow Training for Long Large text data takes a long time to
step → slow and inefficient for long text or
Sequences converge.
time-series.
Long Short-Term Memory (LSTM)
• LSTM stands for Long Short-Term Memory.
• It is a special kind of Recurrent Neural Network (RNN) that is very good at
remembering information for a long time.
• LSTMs introduce memory cells that can store information over long periods,
thereby addressing issues like vanishing gradients during backpropagation.
• It is mainly used when the data comes in a sequence — for example:
A sentence (sequence of words)
A sound clip (sequence of audio samples)
A video (sequence of frames)
A stock price over time
• LSTM fixes the “forgetfulness” of RNNs —
it helps the model remember important things for long and ignore the rest,
making it perfect for language, time-series, and sequential data.
Why do we need LSTM?
• A normal RNN can remember only short-term information.
When the sequence becomes long, the RNN forgets the earlier
information — this is called the vanishing gradient problem.
Example:
In the sentence — “The movie that I watched yesterday was amazing”
The word “amazing” refers to “movie,” which came a long time before.
A simple RNN often forgets “movie” by the time it reads “amazing.”
• LSTM solves this problem by adding a memory cell — a special
structure that can decide what to remember and what to forget.
How LSTM Works??
• The LSTM architectures involve the memory cell which is controlled by three gates, as
shown in the image below, and each part performs an individual function.
 Previous Cell State Updated Cell State

Previous Hidden State Updated Hidden State

1. Input gate: Controls what information is added to the memory cell.

2. Forget gate: Determines what information is removed from the memory cell.
3. Output gate: Controls what information is output from the memory cell.
This allows LSTM networks to selectively retain or discard information as it flows through
the network which allows them to learn long-term dependencies. The network has a hidden
state which is like its short-term memory. This memory is updated using the current input, the
previous hidden state and the current state of the memory cell.
Detailed LSTM Architecture
• LSTM architecture has a chain structure that contains four neural networks
and different memory blocks called cells.
1. Forget Gate
• The information that is no longer useful in the cell state is removed with the
forget gate.
• Two inputs ) input at the particular time) and ) previous cell output)
are fed to the gate and multiplied with weight matrices followed by the
addition of bias.
• The resultant is passed through sigmoid activation function which gives
output in range of [0,1].
• If for a particular cell state the output is 0 or near to 0, the piece of
information is forgotten and for output of 1 or near to 1, the information is
retained for future use. The equation for the forget gate is:

Where:
 𝑊 represents the weight matrix associated with the forget gate.

 ℎ −1, 𝑥 denotes the concatenation of the current input and the previous hidden state.

 𝑏 is the bias with the forget gate and 𝜎 is the sigmoid activation function.
2. Input gate
• The addition of useful information to the cell state is
done by the input gate.
• First the information (it) is regulated using the sigmoid
function and filter the values to be remembered similar
to the forget gate using inputs and .
• Then, a vector (Ct) is created using tanh function that
gives an output from -1 to +1 which contains all the
possible values from and .
• At last the values of the vector and the regulated
values are multiplied to obtain the useful information.
• The equation for the input gate is:
3. Output gate
• The output gate is responsible for deciding what part of the current
cell state should be sent as the hidden state (output) for this time
step.
• First, the gate uses a sigmoid function to determine which information
from the current cell state will be output.
• This is done using the previous hidden state ℎ and the current
input 𝑥 :

• Next, the current cell state 𝐶 is passed through a tanh activation to scale
its values between −1 and +1.
• Finally, this transformed cell state is multiplied element-wise with 𝑜 to
produce the hidden state ℎ :

 𝑜 is the output gate activation and 𝐶 is the current cell state.

 ⊙ represents element-wise multiplication and 𝜎 is the sigmoid activation function.

The cell state carries long-
term memory through all
time steps. The forget and
input gates decide how
much of the old memory to
keep and how much new
information to add.

This hidden state 𝒕 is

then passed to the
next time step and can
also be used for
generating the output
of the network.
Simple LSTM Example
Think of LSTM like a student preparing for an exam :

1. The forget gate decides which old notes are no longer needed.
(“Forget unimportant topics, not in the syllabus!”)
2. The input gate decides which new notes to keep.
(“This new chapter on probability looks important.”)
3. The cell state is like the notebook where the student writes important
information.
4. The output gate decides what to recall in the exam.
(“From all I know, this formula seems useful now.”)

This is how LSTM “manages memory” — it doesn’t remember everything,

only what’s useful.
Applications of LSTM

Application Area Example

Natural Language Processing Sentiment analysis, text generation, machine
(NLP) translation, name entity recognition
Speech Processing Speech recognition (like Alexa, Siri)
Time-Series Prediction Stock price forecasting, weather prediction
Video Analysis Action recognition, video captioning
Chatbots & Conversational AI Remembering conversation context
Bidirectional Long Short-Term Memory (BiLSTM)
• A Bidirectional Long Short-Term Memory (BiLSTM) is an extension of the
traditional LSTM network that processes sequential data in both
a forward (past to future) and a backward (future to past) direction.
• Unlike conventional Long Short-Term Memory (LSTM) that process
sequences in only one direction, BiLSTMs allow information to flow from
both forward and backward, enabling them to capture more contextual
information.
• This makes BiLSTMs particularly effective for tasks where understanding
both past and future context is crucial.
• In simple words:
A normal LSTM reads data only from the start to the end, but a
Bidirectional LSTM reads it both ways.
How BiLSTM works?
• In many problems — like text or speech — the meaning of a word or
event depends on both previous and next words.
• For example:
Sentence: “He said it was not good.”
• If you read left to right, when you see the word “good”, you already
know “not” came before it → meaning it is negative.
But if you also read right to left, you can understand relationships even
more accurately.
• So, BiLSTM captures complete context — past and future — improving
understanding.
 Bidirectional LSTM (BiLSTM)

Here:
• is the input token and is the output token
• and are Forward and backward LSTM units
•The final output of is the combination of and LSTM nodes.
A Bidirectional LSTM (BiLSTM) consists of two separate LSTM layers:
1. Forward LSTM: Processes the sequence from start to end
2. Backward LSTM: Processes the sequence from end to start
• The outputs of both LSTMs are then combined to form the final output.
Mathematically, the final output at time t is computed as:

Where:
• :Final probability vector of the network.
• :Probability vector from the forward LSTM network.

• :Probability vector from the backward LSTM network.

The combined output gives context from both past and future, leading
to better predictions or understanding.
LSTM Vs BiLSTM

Feature Normal LSTM Bidirectional LSTM

Direction One-way (past → future) Two-way (past future)
Context Partial (only past) Full (past + future)
Accuracy Good Better (in many NLP tasks)
Complexity Lower Higher (2× parameters)
Use case Time-series prediction Text/speech understanding
Applications of BiLSTM Limitations
1. Speech Recognition – understand sound 1. Slower (processes input twice).
patterns from both sides. 2. More memory usage (2× parameters).
2. Sentiment Analysis – context-aware 3. Not ideal for real-time tasks (can’t wait
emotion detection. for the future input in streaming data).
3. Named Entity Recognition (NER) –
identify names, places, or dates using full
sentence context.
4. Machine Translation – understand both
start and end of a sentence before
translating.
5. Text Summarization / Generation – better
context grasp.
Gated Recurrent Unit (GRU)
• GRU (Gated Recurrent Unit) is a type of Recurrent Neural Network (RNN) — just like LSTM —
designed to solve the same problems that normal RNNs have, like vanishing gradient and short-
term memory.
• We can think of GRU as a simpler version of LSTM — it works almost as well, but has fewer
gates and parameters, making it faster and easier to train.
• LSTMs are powerful but complex — they have three gates (input, forget, output) and two states
(hidden + cell). GRU simplifies that idea by combining some gates and removing the separate cell
state.
LSTM = Complex but powerful
GRU = Simple and fast, but still very effective
Architecture of GRU
GRU has two gates only:
1. Update Gate (z )
2. Reset Gate (r )
and one hidden state (h ) (no separate cell state like in LSTM).
1. Update Gate (z ) – “How much old information to keep”
 Decides how much of the previous memory (h ₋₁) should be carried
forward.

 If z is close to 1 → keep most of the old informa on.

 If z is close to 0 → replace it with new informa on.
2. Reset Gate (r ) – “How much past information to forget”

• Decides how much of the past to forget.

• Helps the model reset the memory when older information is no longer useful.

Formula:

3. Candidate Hidden State (h ) – “Proposed new memory”

• Creates a new candidate memory using the current input and the previous
memory (controlled by the reset gate).

4. Final Hidden State (h ) – “Actual memory for this step”

• Combines old memory and new memory using the update gate.
Applications of GRU Limitations of GRU
• Text and Speech Processing • Sometimes less accurate for
• Sentiment analysis very long sequences
• Machine translation • Slightly less flexible than LSTM
• Chatbots and text generation (no separate long-term
• Time-Series Forecasting memory)
• Stock prices, weather prediction
• Sequential Data Tasks
• Speech recognition
• Handwriting recognition
Key Differences Between GRU and LSTM

Feature LSTM GRU

Gates 3 (Input, Forget, Output) 2 (Update, Reset)

States 2 (Hidden + Cell) 1 (Hidden only)

Complexity Higher (more parameters) Lower (faster training)

Performance Slightly better for complex tasks Almost equal in many tasks

Training Speed Slower Faster

Best For Longer sequences Moderate-length sequences