UNIT - II SEQUENCE PROCESSING 9

Text Classification – Sentiment Analysis – Recurrent neural networks – Context in RNNs –

Transformer networks – Machine translation – Encoder-decoder RNNs – Attention – Beam search
– Evaluation of translation

Process of Text Classification

1. Text Preprocessing

Before classification, text must be cleaned.

a. Tokenization

Breaking sentence → individual words

Example: “I love NLP.” → [I, love, NLP]

b. Lowercasing

“I” → “i”; “NLP” → “nlp”

c. Stopword Removal

Words like “the”, “is”, “and” removed.

d. Stemming

Reduces words to root form:

 playing, played → play

e. Lemmatization

Maps words to dictionary form:

 running → run

2. Feature Extraction Techniques

a. Bag of Words (BoW)

Counts frequency of words.

Pros: Simple
Cons: Ignores order & meaning.

b. TF-IDF

Improves BoW by giving high weight to important words.

c. Word Embeddings

Dense vector representation capturing meaning

(Word2Vec, GloVe)

d. Contextual Embeddings

Models like BERT generate meaning based on surrounding context.

3. Machine Learning Algorithms

 Naive Bayes → Probabilistic

 SVM → Maximizes class margin
 Logistic Regression

4. Deep Learning Models

 CNN → Works well on reviews

 RNN, LSTM → Sequence models
 Transformers → Best for classification tasks

5. Evaluation Methods

 Accuracy
 Precision, Recall
 F1 Score
 Confusion Matrix
2. SENTIMENT ANALYSIS (Highly
Elaborate Notes)
Definition

Sentiment Analysis identifies the emotional tone (positive, negative, neutral) in text.

Types of Sentiment Analysis

1. Binary Classification

Positive vs. Negative

Example: “I love this movie.” → Positive

2. Multi-Class Classification

Positive / Neutral / Negative

3. Fine-Grained

Star ratings: 1, 2, 3, 4, 5

4. Aspect-Based Sentiment Analysis

Separates sentiment based on aspect:

Example:
“The camera is good but the battery is poor.”

 Camera → positive
 Battery → negative

Approaches of Sentiment Analysis

1. Rule-Based / Lexicon-Based

Uses dictionaries of sentiment words (SentiWordNet).

Words → polarity score
Sentence sentiment = sum of scores.

2. Machine Learning–Based

Uses labeled datasets and classifiers:

 Naive Bayes
 SVM
 Random Forest

Workflow:

 Preprocess text
 Extract features (BoW, TF-IDF)
 Train classifier
 Predict sentiment

3. Deep Learning–Based

 RNN
 LSTM
 GRU
 CNN
 Transformers (BERT, RoBERTa)

Advantages:

 Understands context
 Detects sarcasm better
 Higher accuracy

3. RECURRENT NEURAL NETWORKS

(RNN) – Fully Elaborated
Why RNN?
Traditional neural networks treat inputs independently.
But language requires memory of previous words.

Example:
"I want to eat an apple."
The word eat influences the meaning of apple.

Architecture of RNN
At time step t:

 Input xₜ
 Hidden state hₜ (memory)
 Previous state hₜ₋₁
 Output yₜ

Hidden state equation

ht=tanh⁡(Wxxt+Whht−1+b)h_t = \tanh(W_x x_t + W_h h_{t-1} + b)ht=tanh(Wxxt+Whht−1+b)

Output

yt=softmax(Wyht)y_t = \text{softmax}(W_y h_t)yt=softmax(Wyht)

Unfolding in Time
RNN is drawn like a chain of repeating modules:

 h₁ → h₂ → h₃ → … → hₙ

Each step passes memory forward.

Limitations
1. Vanishing Gradient Problem
o Gradients shrink → cannot learn long-term dependencies.
2. Exploding Gradient
o Gradients become too large → unstable.
 3. Cannot learn relationships far apart
Example:
“The book that I bought yesterday … is good.”

Solutions
 LSTM (Long Short-Term Memory)
 GRU (Gated Recurrent Unit)
 Attention mechanism
 Transformers (replacement for RNN)

4. CONTEXT IN RNNs (Highly Elaborate)

What is Context?

Context refers to the influence of surrounding words that determine a word’s meaning.

Example:
“She visited the bank.”
Meaning depends on the context:

 River bank
 Financial bank

How RNN Maintains Context?

RNN uses hidden state hₜ to store everything it has seen so far.

Short-Term Context

Stored by simple RNN.

Long-Term Context

Stored by LSTM/GRU through:

 Forget gate
 Input gate
  Output gate

Importance of Context
 Helps understand ambiguous words
 Captures dependencies across sentences
 Improves NLP tasks like translation and summarization
 Helps recognize sentiment and sarcasm

5. TRANSFORMER NETWORKS –
(FULLY ELABORATED)
Why Transformers?

RNNs are slow and struggle with long sequences.

Transformers replaced RNN/LSTM in almost all NLP tasks.

Key Concept: Attention Mechanism

Instead of processing sequentially, each word attends to all other words.

Transforms input into:

 Query
 Key
 Value

Transformer Architecture
1. Encoder

 N identical layers
 Each layer has:
o Multi-head self-attention
o Feed-forward neural network
 o Layer norm + residual connections

2. Decoder

 Masked self-attention
 Encoder–decoder attention
 Feed-forward network

Advantages
 Parallel training → extremely fast
 Captures long-range dependencies
 State-of-the-art for:
o Translation
o Summarization
o Question Answering
o Chatbots

6. MACHINE TRANSLATION (MT) – Full

Elaborate Notes
Definition

Machine Translation automatically converts text from one human language to another (e.g.,
English → Tamil).

Types of Translation Systems

1. Rule-Based MT

Uses linguistic rules.

Pros: interpretable
Cons: requires expert effort, low accuracy
2. Statistical Machine Translation (SMT)

Uses probabilities learned from bilingual corpora.

Key techniques:

 Word alignment
 Phrase-based translation

Cons:

 Ignores global context

 Limited fluency

3. Neural Machine Translation (NMT)

Uses deep learning (encoder–decoder + attention).

Advantages:

 Fluent translations
 Captures long-term dependencies
 End-to-end system

7. ENCODER–DECODER RNNs – Full

Notes
What is Encoder–Decoder?

A sequence-to-sequence architecture.

Used for:

 Machine translation
 Summarization
 Chatbots
 Text generation
Encoder
 Reads input sequence word by word
 Produces context vector (summary of input)

Decoder
 Produces output sequence word by word
 Uses context vector as input

Limitation
A single context vector cannot hold long sentence information → performance drops.

Solution
Attention mechanism
→ Gives decoder access to all encoder hidden states.

8. ATTENTION MECHANISM – Fully

Elaborated
Why Attention?

Some words are more important than others.

Example (translation):
“I am very happy today.”
The word very increases intensity.

Working of Attention
1. Compute alignment score between encoder outputs and decoder state.
2. Apply softmax to get attention weights.
 3. Weighted sum of encoder states → context vector for each output word.

Types of Attention
 Bahdanau (Additive)
 Luong (Multiplicative)
 Self-Attention (used in Transformers)

Benefits
 Removes context-vector bottleneck
 Improves translation quality
 Enables long sentences understanding
 Foundation for Transformers

9. BEAM SEARCH – Very Elaborate Notes

What is Beam Search?

A search algorithm used during text generation to choose the best possible sentence.

Why Not Greedy Search?

Greedy selects best word at each step, which may lead to poor results.

How Beam Search Works

Keeps top k sequences at each decoding step.

If beam width = 3:

 At each step, keep 3 best sentence candidates.

Advantages
 Produces better, coherent translations
 Balanced between accuracy and speed

10. EVALUATION OF TRANSLATION –

Fully Elaborated
Evaluating translation quality is essential.

1. BLEU Score
Most widely used metric.

How BLEU Works

 Compares system output with reference translation.

 Uses n-gram precision (1-gram, 2-gram, 3-gram…).
 Applies brevity penalty if translation is too short.

Score range:

 0 = worst
 1 = best

A score of 0.6–0.7 is very good for MT.

2. METEOR
Considers:

 Stemming
 Synonym matching
 Word ordering
Produces more human-like evaluation.
3. ROUGE
Used in summarization evaluation (recall-based).

4. TER (Translation Edit Rate)

Measures edits needed to convert output to reference.

Lower TER = better translation.