DR D PAUL JOSEPH,

SCOPE, VIT-AP

MODULE 4
NLP USING DEEP LEARNING

Natural Language Processing

Course code: CSE3015
DR D PAUL JOSEPH 1
 ▪Types of learning techniques,

▪Chunking,

▪Information extraction & Relation Extraction,

Syllabus ▪Recurrent neural networks,

▪LSTMs/GRUs,

▪Transformers,

▪Self-attention Mechanism

▪Sub-word tokenization

▪Positional encoding
DR D PAUL JOSEPH
Dr D PAUL JOSEPH 2
Chunking in NLP
▪ Chunking in NLP refers to the process of breaking down a text into meaningful
phrases or segments called "chunks.“
▪ Chunking, also known as shallow parsing, is a technique in NLP used to extract
meaningful phrases (chunks) from a sentence.
▪ It groups words into phrases(chunks) based on their Part-of-Speech (POS) tags.
▪ These chunks are usually bigger than individual words but smaller than full
sentences.
▪ Chunking helps in better understanding the structure and meaning of a sentence.
▪ chunking focuses on smaller, useful phrases, such as noun phrases (NPs) and verb
phrases (VPs).

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 3
 Chunking in NLP

How Chunking Works

1. POS Tagging: The sentence is first tokenized and assigned POS tags.
2. Pattern Rules: Regular expressions or machine learning models are used
to define patterns for grouping words.
3. Chunking Process: Using these patterns, groups of words are extracted.

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 4
 Chunking in NLP
import nltk
sentence = "The quick brown fox jumps over the lazy dog"
tokens = nltk.word_tokenize(sentence)
pos_tags = nltk.pos_tag(tokens)

# Define the chunk grammar for NP (Noun Phrase), VP (Verb Phrase), and PP (Prepositional Phrase)
chunk_grammar = r"""
NP: {<DT>?<JJ>*<NN.*>} # Determiner (optional) + Adjective (0 or more) + Noun
VP: {<VB.*><NP|PP>*} # Verb + (Optional NP or PP)
PP: {<IN><NP>} # Preposition + NP
"""
# Create chunk parser
chunk_parser = [Link](chunk_grammar)
chunks = chunk_parser.parse(pos_tags)
print(chunks)

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 5
 Types of learning techniques

Machine learning (ML) is a

subset of artificial
intelligence (AI) that enables
computers to learn patterns
from data and make
decisions or predictions
based on learned patterns.

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 6
 Types of learning techniques
▪Supervised Learning

▪In this approach, the model is trained on labeled data.

▪Supervised learning is a type of machine learning where a model learns from labeled
data (i.e., input-output pairs).

▪The goal is to find a mapping function(weights) from inputs to outputs so that the
model can make accurate predictions on unseen data.

▪labeled data refers to a dataset that includes input data paired with the correct output,
or labels.

▪Example models include linear regression, logistic regression, support vector machines,
and neural networks

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 7
 Types of learning techniques
▪ Unsupervised Learning
▪ Here, the model is trained on unlabeled data and find patterns and relationships within the data.
▪ The model learns patterns from unlabelled data without explicit outputs.
▪ Common techniques :
▪ Clustering (e.g., K-Means, DBSCAN, Hierarchical Clustering)
▪ Dimensionality Reduction (e.g., PCA, t-SNE, Autoencoders)
▪ Semi-Supervised Learning
▪ This is a mix of supervised and unsupervised learning.
▪ The model is trained on a small amount of labeled data and a large amount of unlabeled data.
▪ This is useful when labeling data is expensive or time-consuming
▪ Examples: Medical diagnosis with a few labeled cases and many unlabeled images

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 8
 Types of learning techniques
▪ Reinforcement Learning:
▪ In this type, an agent learns by interacting with its environment and receiving feedback in the form of
rewards or penalties.
▪ It's commonly used in robotics, gaming, and autonomous systems.
▪ Examples:
▪ Self-driving cars optimizing driving strategies
▪ Self-Supervised Learning:
▪ A subset of supervised learning where the model generates its own labels.
▪ For example, predicting the next word in a sentence.

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 9
 • Information Extraction (IE) is task of finding structured information
INFORMATION from unstructured or semi structured text.
EXTRACTION • The input to IE system is a collection of documents (email, web pages,
news groups, news articles, business reports, research papers, blogs,
resumes, proposals, and soon) and output is a representation of the
relevant information.
• This process typically involves identifying and extracting specific
types of information such as
Entities(NER)
• People, organizations, locations, times, dates, prices, ...
• Or sometimes: genes, proteins, diseases, medicines, ...
Relations between entities(Relation extraction)
• Located in, employed by, part of, married to, ...
larger events that are taking place(Event extraction)
DR D PAUL JOSEPH 10
 Named Entity Recognition
SUB TASKS
IN Relation extraction
INFORMATION Event extraction
EXTRACTION
Coreference Resolution

DR D PAUL JOSEPH 11
 Named Entity
Recognition (NER)
▪ The first step in information extraction is to detect the entities in the text.

▪ A named entity is, anything that can be referred to with a proper name: a
person, a location, an organization.

▪ The term is commonly extended to include things that aren’t entities,

including dates, times, and prices.

▪ Named Entity Recognition (NER) is used to identify and classify named

entities in text into predefined categories such as Person, Organization,
Location, Date, Time, Money, Percentages, Products, etc.

▪ The text contains 13 mentions of named entities including 5

organizations, 4 locations, 2 times, 1 person, and 1 mention of money.

DR D PAUL JOSEPH 12
 Named Entity Recognition (NER)

• A list of generic named entity types with the kinds of Type ambiguities in the use of the name
entities they refer to. Washington
Type ambiguity in NER • [PER Washington] was born into slavery on
• Type ambiguity in NER occurs when a word belongs the farm of James Burroughs.
• [ORG Washington] went up 2 games to 1 in
to multiple named entity types depending on the
the four-game series.
context. • Blair arrived in [LOC Washington] for what
• This can lead to incorrect classification by NER may well be his last state visit.
models.
DR D PAUL JOSEPH
Dr D PAUL JOSEPH 13
Rule-Based Approaches:
• These rely on predefined sets of rules and patterns to identify named Approaches to NER
entities.
• They are simple to implement but can be limited in their ability to
Deep Learning Approaches:
generalize to new data.
Dictionary-Based Approaches:
• These use neural networks, such as Recurrent

• These use dictionaries or gazetteers of known named entities to Neural Networks (RNNs) and Transformers, to
match and identify entities in text. automatically learn features from raw text.
• They are effective for well-defined domains but may struggle with new • They have shown state-of-the-art performance
entities. in NER tasks but require large amounts of
Machine Learning Approaches:
labelled data and computational resources.
• These involve training models on labelled datasets to learn patterns
that distinguish named entities.
• Common algorithms include Conditional Random Fields (CRFs) and
Hidden Markov Models (HMMs).

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 14
import re Using Rule & Dictionary-Based Approaches
import nltk

text = "Elon Musk is the CEO of Tesla Inc. He was born on 1971-06-28 in South
Africa."

# Predefined lists (gazetteers/Dictionary)

companies = {"Tesla", "Google", "Microsoft", "Amazon"}
persons = {"Elon Musk", "Bill Gates", "Jeff Bezos"}
locations = {"South Africa", "United States", "India"}

# Rule-based entity extraction

entities = {}

# Recognizing dates (YYYY-MM-DD format)

date_pattern = r'\b\d{4}-\d{2}-\d{2}\b'
entities["DATE"] = [Link](date_pattern, text)

entities["PERSON"] = [name for name in persons if name in text]

entities["ORG"] = [name for name in companies if name in text]

entities["GPE"] = [name for name in locations if name in text]

Dr D PAUL JOSEPH
print(entities) {'DATE': ['1971-06-28'], 'PERSON': ['Elon Musk'], 'ORG': ['Tesla'], 'GPE': ['South Africa']}
DR D PAUL JOSEPH 15
 Relation extraction
• Relation extraction is the process of finding and extracting semantic relations among
the text entities from text.
• Once entities are recognized, identify specific relations between entities
• These are often binary relations like child-of, employment, part-whole, and geospatial
relations.

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 16
 Rule-based Relation Extraction
• In Rule-based relation extraction, predefined linguistic rules or patterns are used to identify and classify
relationships between entities in text.
• Hearst (1992a, 1998) proposed five patterns for identifying is-a relationships.
• These patterns help extract structured knowledge from unstructured text by recognizing the "is-a"
relationship.
Pattern Example
"X such as Y" "Vehicles such as cars and bikes are common."
"Programming languages including Python and Java are
"X including Y"
popular."
"X is a type of Y" "A tulip is a type of flower."
"X, especially Y" "Fruits, especially apples and oranges, are healthy."
"Y and other X" "Eagles and other birds can fly.
DR D PAUL JOSEPH
Dr D PAUL JOSEPH 17
 Rule-based Relation Extraction
• Many instances of relations can be identified through hand-crafted patterns, looking for
triples (X, α, Y) where X & Y are entities and α are words in between.
• For the “Paris is in France” example, α = ”is in”. This could be extracted with a regular
expression.

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 18
 Rule-based Relation Extraction
import re
def extract_custom_relations(text):

pattern = r"(\w+) (was born in|is located in|works at|is a type of|and other) (\w+)"
matches = [Link](pattern, text)

return [(match[0], match[1], match[2]) for match in matches]

text = "Musk was born in South Africa. Microsoft is located in the USA.\
Sundar Pichai works at [Link] is a type of [Link] and other birds
can fly"

# Extract relations
print(extract_custom_relations(text))

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 19
 Extracting Richer Relations Using Rules and Named Entities

([A-Z][a-z]+(?:\s[A-Z][a-z]+)*)\s*,\s*([A-Z][a-z]+(?:\s[A-Z][a-z]+)*)\s+of\s+([A-Z][a-z]+(?:\s[A-Z][a-z]+)*)

•([A-Z][a-z]+(?:\s[A-Z][a-z]+)*) → Captures the Person (PER)

•,\s* → Matches the comma and optional spaces
•([A-Z][a-z]+(?:\s[A-Z][a-z]+)*) → Captures the Position (POSITION)
•\s+of\s+ → Matches the phrase " of "
•([A-Z][a-z]+(?:\s[A-Z][a-z]+)*) → Captures the Organization (ORG)

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 20
import re
pattern = r"(\b[A-Z][a-z]+(?>\s[A-Z][a-z]+)*>\s*,\s*([A-Z][a-z]+(?>\s[A-Z][a-z]+)*>\s+of\s+([A-
Z][a-z]+(?>\s[A-Z][a-z]+)*>"
def extract_person_position_org(text):
match = [Link](pattern, text)
if match:
return [Link]() # Returns (Person, Position, Organization)
return None
sentences = [
"George Marshall, Secretary of State of the United States.",
"John Doe, Chief Executive Officer of TechCorp.",
"Alice Brown, Vice President of Marketing of Global Enterprises."
]

for sentence in sentences:

result = extract_person_position_org(sentence)
if result:
print(f"\nSentence: {sentence}")
print(f"Extracted: Person = {result[0]}, Relation = {result[1]}, Organization =
{result[2]}")

Dr D PAUL JOSEPH DR D PAUL JOSEPH 21

 Relation Extraction via Supervised Learning
▪ Supervised Learning: A method where a model is trained on a dataset containing labeled entity pairs and
their relationships.
▪ A fixed set of relations and entities is chosen.
▪ a training corpus is hand-annotated with the relations and entities, and the annotated text.

Data Collection & Annotation

▪ Gather a dataset containing text with entity data = [
pairs. ("John works at Google.", "John", "Google", "works_at"),
▪ Manually annotate the relationships ("Elon founded Tesla.", "Elon", "Tesla", "founded"),
between entities with predefined relation ("Apple acquired Beats.", "Apple", "Beats", "acquired"),
]
labels (or use existing labeled datasets like
SemEval, TAC KBP).

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 22
 Relation Extraction via Supervised Learning
▪ Preprocessing
▪ Tokenization: Split text into words or sub-words.
▪ Named Entity Recognition (NER): Identify and classify named entities (e.g., persons, organizations,
locations).
▪ Part-of-Speech (POS) Tagging: Label words with their grammatical roles.
▪ Dependency Parsing: Extract syntactic relations between words.

▪ Feature Extraction
▪ Some common feature types for relation extraction include:
▪ Lexical Features: Words between and around the entity pair.
▪ Syntactic Features: POS tags, dependency relations.
▪ Entity-Based Features: Named entity types, entity distance.
▪ Positional Features: Distance of words from entities.

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 23
 Relation Extraction via Supervised Learning
Model Training
•Choose a supervised learning model:
• Traditional ML models: SVM, Decision Trees, Random Forest,
Logistic Regression.
• Deep Learning models: CNN, BiLSTM, Transformer-based models
(BERT, RoBERTa).
•Train the model on labelled data to learn patterns.

Prediction
•Apply the trained model to new text data.
•Extract relationships between entities.

Pros & Cons:

•Can get high accuracies if enough training data
• Labeling a large training set is expensive
DR D PAUL JOSEPH
Dr D PAUL JOSEPH 24
 Event extraction
▪ Event extraction is the process of identifying and classifying events mentioned in text.
▪ It involves extracting structured information about events, such as the participants,
locations, dates, and other relevant details.

▪ Key Components of Event Extraction

▪ Event Trigger – The main verb or noun indicating an event (e.g., "launched," "announced,"
"elected").
▪ Event Type – The category of the event (e.g., "Business," "Disaster," "Political," "Sports").
▪ Event Arguments – The entities participating in the event (e.g., Person, Organization,
Location, Date, Time).

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 25
 Event extraction
▪ Applications:
Financial News Monitoring – Detecting stock-related events (e.g., mergers, acquisitions).
Disaster Alert Systems – Extracting crisis events from social media (e.g., earthquakes, floods).
Legal Document Analysis – Extracting case details (e.g., court cases, judgments).

▪ Example:
text = "Elon Musk announced the launch of the new Tesla model at the event in California on March 3, 2025, at 10:00
AM."
▪ Output:

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 26
import spacy

# Load spaCy model

nlp = [Link]("en_core_web_sm")

text = "Elon Musk announced the launch of the new Tesla model at the event in California on March 3, 2025,
at 10:00 AM."

# Process the text

doc = nlp(text)

# Extract named entities (including TIME)

entities = {"PERSON": [], "ORG": [], "GPE": [], "DATE": [], "TIME": []}
for ent in [Link]:
if ent.label_ in entities:
entities[ent.label_].append([Link])

# Extract event triggers (filtering meaningful verbs)

event_triggers = [token.lemma_ for token in doc if token.pos_ => "VERB"]

# Print extracted details

print("Event Details:")
print(f"- Event Triggers (Actions): {event_triggers}")
print(f"- Participants: {entities['PERSON']}")
print(f"- Organizations: {entities['ORG']}")
print(f"- Location: {entities['GPE']}")
print(f"- Date: {entities['DATE']}")
print(f"- Time: {entities['TIME']}") DR D PAUL JOSEPH
Dr D PAUL JOSEPH 27
 Coreference Resolution in NLP
▪ Coreference resolution (CR) is the task of finding all words (called mentions) in a given text that refer to the
same entity.
▪ After finding and grouping these words we can resolve them by replacing, pronouns with noun phrases.
▪ It helps in understanding context, improving text coherence, and enabling more accurate question answering,
summarization, and dialogue systems.

Input:
"Elon Musk founded SpaceX in 2002. He is also the CEO of Tesla."
Output (Resolved Coreference):
"Elon Musk founded SpaceX in 2002. Elon Musk is also the CEO of Tesla."
DR D PAUL JOSEPH
Dr D PAUL JOSEPH 28
 Coreference Resolution in NLP
1️⃣ Pronominal Coreference
•Resolving pronouns (he, she, it, they, etc.) to actual entities.
•Example:
• "Obama was the U.S. President. He served for two terms."
• "He" → "Obama“
2️⃣ Named Entity Coreference
•Resolving different mentions of the same entity (e.g., “Tesla” and “the company”).
•Example:
• "Apple launched a new iPhone. The company expects high sales."
• "The company" → "Apple"
DR D PAUL JOSEPH
Dr D PAUL JOSEPH 29
import nltk
import spacy X_train = []
import numpy as np y_train = []
from [Link] import SVC
from sklearn.feature_extraction import DictVectorizer for sentence, e1, e2, relation in train_data:
from [Link] import make_pipeline features = extract_features(sentence, e1, e2)
X_train.append(features)
# Load Spacy NLP model y_train.append(relation)
nlp = [Link]("en_core_web_sm")
# Sample training data: (sentence, entity1, entity2, relation) # Convert features to numerical form
train_data = [ vectorizer = DictVectorizer(sparse=False)
("John works at Google.", "John", "Google", "works_at"),
X_train_vectorized = vectorizer.fit_transform(X_train)
("Elon founded Tesla.", "Elon", "Tesla", "founded"),
("Apple acquired Beats.", "Apple", "Beats", "acquired"),
]
# Train SVM model
# Extract features from sentences svm_clf = SVC(kernel="linear", probability=True)
def extract_features(sentence, entity1, entity2): svm_clf.fit(X_train_vectorized, y_train)
doc = nlp(sentence) # Function to predict relation in a new sentence
features = {} def predict_relation(sentence, entity1, entity2):
# POS tagging of words features = extract_features(sentence, entity1, entity2)
for token in doc: features_vectorized = [Link]([features])
features[f"word_{[Link]}"] = token.pos_ prediction = svm_clf.predict(features_vectorized)
# Distance between entities return prediction[0]
e1_idx = [Link](entity1)
e2_idx = [Link](entity2) # Test the model with a new sentence
features["entity_distance"] = abs(e1_idx - e2_idx) test_sentence = "Mark acquired Facebook."
# Dependency parsing
print(predict_relation(test_sentence, "Mark", "Facebook"))
for token in doc:
features[f"dep_{[Link]}"] = token.dep_
return features
DR D PAUL JOSEPH
Dr D PAUL JOSEPH acquired 30
Neural
Networks

DR D PAUL JOSEPH 31
 Feedforward Neural
Network (FNN)
▪ A Feedforward Neural Network (FNN) is
a type of artificial neural network where
information moves in one direction—
from the input nodes, through the
hidden nodes and to the output nodes—
without cycles or loops.

▪ It consists of an input layer, hidden

layers, and an output layer.

▪ The network learns by adjusting weights

through backpropagation.

DR D PAUL JOSEPH 32
 In this phase, the input At each hidden layer, the
This process continues
data is fed into the weighted sum of the inputs
until the output layer is
Feedforward Phase: network, and it propagates is calculated and passed
reached, and a prediction
forward through the through an activation
is made.
network. function.

Once a prediction is made, This error is then

The process of adjusting
the error (difference propagated back through
weights is typically done
Backpropagation Phase: between the predicted the network, and the
using a gradient descent
output and the actual weights are adjusted to
optimization algorithm.
output) is calculated. minimize this error.

Working of Feedforward Neural Network

DR D PAUL JOSEPH 33
01 02 03
FNNs process inputs FNNs do not have any In FNNs, each input
independently memory of past has its own set of
without considering inputs; parameters, making
past information. each input is learning inefficient for
processed in isolation. sequences.

FNN’s Contd..
DR D PAUL JOSEPH 34
 Sequence Models
▪ Sequence Models are neural network architectures that process sequence data.

▪ Sequential data is data—such as words, sentences, or time-series data

▪ Applications of Sequence Models are in Speech Recognition, Machine Translation,

Music Generation, Sentiment classification, Image captioning etc.

1. Recurrent Neural Networks(RNN),

Types 2.
3.
Gated Recurrent Units(GRU),
Long-short-term Memory(LSTM),
4. Transformers

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 35
 Recurrent neural networks

Recurrent Neural RNNs are designed

RNN has a concept
Networks (RNNs) to process
of “memory” which RNNs are
are a type of neural sequential data by
remembers all called recurrent bec The main difference The output of This way, the model
network maintaining a
information about ause they perform between RNN and hidden state at time can learn the
architecture hidden state that
what has been the same task for conventional ANN is step t-1 goes as information from
designed to handle captures
calculated till time every element of a that RNN has a input into the next previous time steps.
sequential data, information about
step -t sequence, with the feedback loop that time step t.
where the order of previous inputs
output being goes as input into
the data matters. depended on the the next time step.
previous
computations.

DR D PAUL JOSEPH 36
 Recurrent neural networks

Key Features of RNNs: Sequential Processing – Unlike Weight Sharing – The same set Backpropagation Through Temporal Dependencies –
traditional neural networks, of weights is applied at each Time (BPTT) – A variant of RNNs capture relationships
RNNs maintain a hidden time step, reducing the backpropagation used to over time, enabling them to
state(memory) that captures number of parameters. update weights in RNNs. recognize patterns in time-
information from previous series data and text.
time steps.

DR D PAUL JOSEPH 37
 The Structure
of RNN
A basic RNN consists of three layers:

1. Input Layer – Takes in the current input

(e.g., a word, a time-series value).

2. Hidden Layer (Memory Unit) – Stores

past information and updates itself at
each time step.

3. Output Layer – Produces the final

prediction based on the hidden state.

DR D PAUL JOSEPH 38
 Working of RNN

▪ RNNs work the same way as conventional ANNs.

▪ It has weights, bias, activation, nodes, and layers.

▪ We train the RNN model with multiple sequences of data, and each sequence has time steps.

▪ A RNN processes input data in a sequence, maintaining a hidden state that gets updated at each step based on
the current input and the previous hidden state.

▪ The output from the hidden state goes to the output layer and to the next hidden state.

▪ While working with sequential data, the output at any time step(t) should depend on the input at that time
step as well as previous time steps.

Dr D PAULDRJOSEPH
D PAUL JOSEPH 39
RNN - Computation at the hidden state

Dr D PAULDRJOSEPH
D PAUL JOSEPH 40
 Working of RNN Contd..
Compute Loss

Dr D PAULDRJOSEPH
D PAUL JOSEPH 41
 Working of RNN
Backward Propagation Through Time
(BPTT):

• Backward Propagation Through Time

(BPTT) is an extension of standard
backpropagation used to train
Recurrent Neural Networks (RNNs).

DR D PAUL JOSEPH 42
 Working of RNN

Backward Propagation Through

Time (BPTT):
• Since the hidden states are
dependent on previous time
steps, we compute gradients
recursively using the chain
rule.

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 43
 Working of RNN

Weight Updates Using Gradient Descent

▪ Using the computed gradients, update all weights with Stochastic Gradient Descent (SGD)

▪ Repeat for Multiple Epochs:

▪ Forward and backward propagation are repeated for multiple epochs.
▪ The model learns patterns and improves accuracy over time.
Dr D PAULDRJOSEPH
D PAUL JOSEPH 44
 Types of RNN
Recurrent Neural Networks (RNNs) have different architectures
based on how inputs and outputs are structured.

One-to-One (Vanilla Neural Network)

1. Input: Single input

2. Output: Single output

3. Example: Image classification (e.g., CNNs on MNIST)

One-to-Many

1. Input: Single input

2. Output: Sequence of outputs

3. Use Case: Sequence generation

Image captioning

DR D PAUL JOSEPH 45
 Types of RNN
Many-to-One
1. Input: Sequence of inputs
2. Output: Single output
3. Use Case: Sentiment analysis, text classification
Many-to-Many (Same Length)
1. Input: Sequence of inputs
2. Output: Sequence of outputs (same length)
3. Use Case: POS tagging, Named Entity Recognition (NER)
Many-to-Many (Different Lengths, Encoder-Decoder)
1. Input: Sequence of inputs
2. Output: Sequence of outputs (different length)
3. Use Case: Machine Translation, Speech-to-Text

Dr D PAULDRJOSEPH
D PAUL JOSEPH 46
 RNN for Text classification tasks
Negative

Movie is very Bad 47

Dr D PAULDRJOSEPH
D PAUL JOSEPH
 Vanishing and Exploding Gradients problem in RNNs
• The vanishing and exploding gradient problem occurs during backpropagation in
RNNs due to repeated multiplication of gradients through multiple time steps. This
makes it difficult to train deep networks effectively.

Vanishing Gradient Problem

Why It Happens
•In RNNs, gradients are propagated backward through many time steps.
•If the weight matrices have small values, repeated multiplication causes the gradients to
shrink exponentially.
•Eventually, gradients become so small (close to zero) that earlier layers in the network
stop updating weights.
Effects
•The network struggles to learn long-range dependencies.
•The model relies mostly on recent inputs and ignores older ones.
•Training becomes slow, and weights stop updating.
Dr D PAULDRJOSEPH
D PAUL JOSEPH 48
 Vanishing and Exploding Gradients problem in RNNs
Exploding Gradient Problem
Why It Happens
•If the weight matrices have large values, repeated multiplication causes gradients to grow
exponentially.
•The model weights may become NaN during training.
Effects
•Model parameters update too aggressively.
•Loss does not converge properly.
•The model may produce meaningless or repetitive outputs.

DR D PAUL JOSEPH
Dr D PAUL JOSEPH 49
 Bidirectional Recurrent Neural Networks
• A Bidirectional Recurrent Neural Network (BiRNN) is an extension of a standard RNN
that processes data in both forward and backward directions.
• This allows the network to have context from both past and future time steps, making it
especially useful for NLP tasks like Named Entity Recognition (NER), POS tagging, and
Machine Translation.

How BiRNN Works

A BiRNN consists of two RNN layers:
•Forward RNN → Processes the sequence from start to end.
•Backward RNN → Processes the sequence from end to start.
•The outputs from both RNNs are concatenated or combined

Dr D PAULDRJOSEPH
D PAUL JOSEPH 50
 Bidirectional Recurrent Neural Networks
Mathematical Representation
For a given input sequence X=(x1,x2,…,xT):

Forward RNN: Backward RNN:

Where:
•X<t>: is the input at time step t Where:
•W_x: is the weight matrix for the input •X<t>: is the input at time step t (same as
•W_h: is the weight matrix for the hidden state from the in the forward RNN)
previous time step •W_x, W_h, and b_h are the same
•W_y: is the weight matrix from the hidden state to the matrices and bias terms used in the
output forward RNN but applied in the reverse
•b_h, b_y: are the bias terms order
•g: activation function, typically a non-linear function like •h←<t+1>: is the hidden state from the
tanh or ReLU next time step (as we are processing
•y→<t>: is the output at time step t Dr D PAULDRJOSEPH
D PAUL JOSEPH
backward) 51
 Bidirectional Recurrent Neural Networks
BPTT
• In the case of a bidirectional RNN, BPTT involves two separate Backpropagation passes:
one for the forward RNN and one for the backward RNN.
• During the forward pass, the forward RNN processes the input sequence in the usual way
and makes predictions for the output sequence.
• These predictions are then compared to the target output sequence, and the error is
backpropagated through the network to update the weights of the forward RNN.

• The backward RNN processes the input sequence in reverse order during the backward
pass and predicts the output sequence.
• These predictions are then compared to the target output sequence in reverse order, and
the error is backpropagated through the network to update the weights of the backward
RNN.
• Once both passes are complete, the weights of the forward and backward RNNs are
updated based on the errors computed during the forward and backward passes,
DR D PAUL JOSEPH 52
respectively. Dr D PAUL JOSEPH
 Bidirectional Recurrent Neural Networks
Combined Output:
There are several ways in which the outputs of the forward and backward RNNs can be merged, depending on the
specific needs of the model and the task it is being used for.
Some common merge modes include:
Concatenation:
• In this mode, the outputs of the forward and backward RNNs are concatenated together, resulting in a single
output tensor that is twice as long as the original input.
Sum:
• In this mode, the outputs of the forward and backward RNNs are added together element-wise, resulting in a
single output tensor that has the same shape as the original input.
Average:
• In this mode, the outputs of the forward and backward RNNs are averaged element-wise, resulting in a single
output tensor that has the same shape as the original input.
Maximum:
• In this mode, the maximum value of the forward and backward outputs is taken at each time step, resulting in
a single output tensor with the same shape as the original input.

Dr D PAULDRJOSEPH
D PAUL JOSEPH 53