Overview of Text
Classification
DEEP LEARNING FOR TEXT WITH PYTORCH
Shubham Jain
Instructor
Text classification defined
Assigning labels to text Organizes and gives structure to
Giving meaning to words and sentences unstructured data
Applications:
Analyzing customer sentiment in reviews
Detecting spam in emails
Tagging news articles with relevant
topics
Types: binary, multi-class, multi-label
DEEP LEARNING FOR TEXT WITH PYTORCH
Binary classification
Sorting into two categories
Example: email spam detection
Emails can be classified as 'spam' or 'not
spam'
1 [Link]
DEEP LEARNING FOR TEXT WITH PYTORCH
Multi-class classification
Sorting into multiple categories
Example: News articles can be sorted into
various categories like
1. Politics
2. Sports
3. Technology
DEEP LEARNING FOR TEXT WITH PYTORCH
Multi-label classification
Each text can be assigned multiple labels
Example: Books can be multiple genres
Action
Adventure
Fantasy
DEEP LEARNING FOR TEXT WITH PYTORCH
What are word embeddings
Previous encoding techniques are a good
first step
Often create too many features and
can't identify similar words
Word embeddings map words to numerical
vectors
Example of semantic relationship:
King and queen
Man and woman
DEEP LEARNING FOR TEXT WITH PYTORCH
Word to index mapping
Example:
"King" -> 1
"Queen" -> 2
Compact and computationally efficient
Follows tokenization in the pipeline
DEEP LEARNING FOR TEXT WITH PYTORCH
Word embeddings in PyTorch
[Link] :
Creates word vectors from indexes
Input: Indexes for ['The', 'cat', 'sat', 'on', 'the', 'mat']
Embedding for 'the': tensor([-0.4689, 0.3164, -0.2971, -0.1291, 0.4064])
Embedding for 'cat': tensor([-0.0978, -0.4764, 0.0476, 0.1044, -0.3976])
Embedding for 'sat': tensor([ 0.2731, 0.4431, 0.1275, 0.1434, -0.4721])
DEEP LEARNING FOR TEXT WITH PYTORCH
Using [Link]
import torch
from torch import nn
words = ["The", "cat", "sat", "on", "the", "mat"]
word_to_idx = {word: i for i, word in enumerate(words)}
inputs = [Link]([word_to_idx[w] for w in words])
embedding = [Link](num_embeddings=len(words), embedding_dim=10)
output = embedding(inputs)
print(output)
tensor([[ 1.0624, 0.6792, 0.0459, ... -1.0828, -0.4475, 0.4868],
...
[1.5766, 0.0106, 0.1161, ...,, -0.0859, 1.3160, 1.3621])
DEEP LEARNING FOR TEXT WITH PYTORCH
Using embeddings in the pipeline
def preprocess_sentences(text): def text_processing_pipeline(text):
# Tokenization tokens = preprocess_sentences(text)
# Stemming dataset = TextDataset(tokens)
... dataloader = DataLoader(dataset, batch_size=2,
# Word to index mapping shuffle=True)
class TextDataset(Dataset): return dataloader, vectorizer
def __init__(self, encoded_sentences):
[Link] = encoded_sentences text = "Your sample text here."
dataloader, vectorizer = text_processing_pipeline(text)
def __len__(self): embedding = [Link](num_embeddings=10,
return len([Link]) embedding_dim=50)
def __getitem__(self, index): for batch in dataloader:
return [Link][index] output = embedding(batch)
print(output)
DEEP LEARNING FOR TEXT WITH PYTORCH
Let's practice!
DEEP LEARNING FOR TEXT WITH PYTORCH
Convolutional neural
networks for text
classification
DEEP LEARNING FOR TEXT WITH PYTORCH
Shubham Jain
Instructor
CNNs for text classification
Classifying tweets as
Positive
Negative
Neutral
DEEP LEARNING FOR TEXT WITH PYTORCH
The convolution operation
Convolution operation
Sliding a filter (kernel) over the input
data
For each position of the filter, perform
element-wise calculations
For text: learns structure and meaning of
words
1 Animation from Vincent Dumoulin, Francesco Visin
DEEP LEARNING FOR TEXT WITH PYTORCH
Filter and stride in CNNs
Filter:
Small matrix that we slide over the input
Stride:
Number of positions the filter moves
1 Animation from Vincent Dumoulin, Francesco Visin
DEEP LEARNING FOR TEXT WITH PYTORCH
CNN architecture for text
Convolutional layer: applies filters to input data
Pooling layer: reduces data size while preserving important information
Fully connected layer: makes final predictions based on previous layer output
DEEP LEARNING FOR TEXT WITH PYTORCH
Implementing a text classification model using CNN
class SentimentAnalysisCNN([Link]): __init__ method configures the
def __init__(self, vocab_size, embed_dim): architecture
super().__init__()
[Link] = [Link](vocab_size, super() initializes the base class
embed_dim)
[Link] = nn.Conv1d(embed_dim, embed_dim,
[Link]
kernel_size=3, stride=1,
padding=1)
[Link] creates dense word vectors
[Link] = [Link](embed_dim, 2)
...
nn.Conv1d for one dimensional data
DEEP LEARNING FOR TEXT WITH PYTORCH
Implementing a text classification model using CNN
... Embedding layer converts text to
def forward(self, text):
embedded = [Link](text).permute(0, 2, 1)
embedding
conved = [Link]([Link](embedded))
Match tensors to convolution layer's
conved = [Link](dim=2)
return [Link](conved)
expected input
Extract important features with ReLU
Eliminate extra layers and dimensions
DEEP LEARNING FOR TEXT WITH PYTORCH
Preparing data for the sentiment analysis model
vocab = ["i", "love", "this", "book", "do", "not", "like"]
word_to_idx = {word: i for i, word in enumerate(vocab)}
vocab_size = len(word_to_ix)
embed_dim = 10
book_samples = [
("The story was captivating and kept me hooked until the end.".split(),1),
("I found the characters shallow and the plot predictable.".split(),0)
]
model = SentimentAnalysisCNN(vocab_size, embed_dim)
criterion = [Link]()
optimizer = [Link]([Link](), lr=0.1)
DEEP LEARNING FOR TEXT WITH PYTORCH
Training the model
for epoch in range(10):
for sentence, label in data:
model.zero_grad()
sentence = [Link]([word_to_idx.get(w, 0) for w in sentence]).unsqueeze(0)
outputs = model(sentence)
label = [Link]([int(label)])
loss = criterion(outputs, label)
[Link]()
[Link]()
DEEP LEARNING FOR TEXT WITH PYTORCH
Running the Sentiment Analysis Model
for sample in book_samples:
input_tensor = [Link]([word_to_idx[w] for w in sample], dtype=[Link]).unsqueeze(0)
outputs = model(input_tensor)
_, predicted_label = [Link]([Link], 1)
sentiment = "Positive" if predicted_label.item() == 1 else "Negative"
print(f"Book Review: {' '.join(sample)}")
print(f"Sentiment: {sentiment}\n")
Book Review: The story was captivating and kept me hooked until the end
Sentiment: Positive
Book Review: I found the characters shallow and the plot predictable
Sentiment: Negative
DEEP LEARNING FOR TEXT WITH PYTORCH
Let's practice!
DEEP LEARNING FOR TEXT WITH PYTORCH
Recurrent neural
networks for text
classification
DEEP LEARNING FOR TEXT WITH PYTORCH
Shubham Jain
Data Scientist
RNNs for text
Handle sequences of varying lengths
Maintain an internal short-term memory
CNNs spot patterns in chunks
RNNs remember past words for greater meaning
DEEP LEARNING FOR TEXT WITH PYTORCH
RNNs for text classification
Why?
RNNs can read sentences like humans, one
word at a time
Understand context and order
Example: Detecting sarcasm in a tweet
"I just love getting stuck in traffic."
Sarcastic
DEEP LEARNING FOR TEXT WITH PYTORCH
Recap: Implementing Dataset and DataLoader
# Import libraries
from [Link] import Dataset, DataLoader
# Create a class
class TextDataset(Dataset):
def __init__(self, text):
[Link] = text
def __len__(self):
return len([Link])
def __getitem__(self, idx):
return [Link][idx]
DEEP LEARNING FOR TEXT WITH PYTORCH
RNN implementation
sample_tweet = "This movie had a great plot and amazing acting."
# Preprocess the review and convert it to a tensor (not shown for brevity)
# ...
sentiment_prediction = model(sample_tweet_tensor)
Train an RNN model to classify tweet as positive or negative
Output: "Positive"
DEEP LEARNING FOR TEXT WITH PYTORCH
RNN variation: LSTM
Tweet:
"Loved the cinematography,
hated the dialogue.
The acting was exceptional,
but the plot fell flat."
Long Short Term Memory (LSTM) can
capture complexities where RNNs may
struggle
DEEP LEARNING FOR TEXT WITH PYTORCH
LSTM
LSTM architecture: Input gate, forget gate, and output gate
class LSTMModel([Link]):
def __init__(self, input_size, hidden_size, output_size):
super(LSTMModel, self).__init__()
[Link] = [Link](input_size, hidden_size, batch_first=True)
[Link] = [Link](hidden_size, output_size)
def forward(self, x):
_, (hidden, _) = [Link](x)
output = [Link]([Link](0))
return output
DEEP LEARNING FOR TEXT WITH PYTORCH
RNN variation: GRU
Email subject:
"Congratulations!
You've won a free trip
to Hawaii!"
Gated Recurrent Unit (GRU) can quickly
recognize spammy patterns without
needing the full context
DEEP LEARNING FOR TEXT WITH PYTORCH
GRU
class GRUModel([Link]):
def __init__(self, input_size, hidden_size, output_size):
super(GRUModel, self).__init__()
[Link] = [Link](input_size, hidden_size, batch_first=True)
[Link] = [Link](hidden_size, output_size)
def forward(self, x):
_, hidden = [Link](x)
output = [Link]([Link](0))
return output
DEEP LEARNING FOR TEXT WITH PYTORCH
Let's practice!
DEEP LEARNING FOR TEXT WITH PYTORCH
Evaluation metrics
for text classification
DEEP LEARNING FOR TEXT WITH PYTORCH
Shubham Jain
Instructor
Why evaluation metrics matter
Spotlight on Book Reviews:
Imagine a model that assesses the sentiment of book reviews
The model claims a best-selling novel is poorly reviewed. Do we accept this?
Use evaluation metrics
DEEP LEARNING FOR TEXT WITH PYTORCH
Evaluation RNN Models
# Initialize model, criterion, and optimizer
rnn_model = RNNModel(input_size, hidden_size, num_layers, num_classes)
...
# Model training
for epoch in range(10):
outputs = rnn_model(X_train)
...
print(f'Epoch: {epoch+1}, Loss: {[Link]()}')
outputs = rnn_model(X_test)
_, predicted = [Link](outputs, 1)
DEEP LEARNING FOR TEXT WITH PYTORCH
Accuracy
The ratio of correct predictions to the total predictions
from torchmetrics import Accuracy
actual = [Link]([0, 1, 1, 0, 1, 0])
predicted = [Link]([0, 0, 1, 0, 1, 1])
accuracy = Accuracy(task="binary", num_classes=2)
acc = accuracy(predicted, actual)
print(f"Accuracy: {acc}")
Accuracy: 0.6666666666666666
DEEP LEARNING FOR TEXT WITH PYTORCH
Beyond accuracy
10,000 reviews: 9,800 are positive
A model that always predicts positive: 98% accuracy
The model failed to classify negative reviews
Precision: confidence in labeling a review as negative
Recall: how well the model spots negative reviews
F1 Score: balance between precision and recall
DEEP LEARNING FOR TEXT WITH PYTORCH
Precision and Recall
Precision: correctly predicted positive observations / total predicted positives
Recall: correctly predicted positive observations / all observations in the positive class
from torchmetrics import Precision, Recall
precision = Precision(task="binary", num_classes=2)
recall = Recall(task="binary", num_classes=2)
prec = precision(predicted, actual)
rec = recall(predicted, actual)
print(f"Precision: {prec}")
print(f"Recall: {rec}")
Precision: 0.6666666666666666
Recall: 0.5
DEEP LEARNING FOR TEXT WITH PYTORCH
Precision and Recall
Precision: 0.6666666666666666
Recall: 0.5
Precision: 66.66% accurately predicted as positive
Recall: captured 50% of positives
DEEP LEARNING FOR TEXT WITH PYTORCH
F1 score
Harmonizes precision and recall
Better measure for imbalanced classes
from torchmetrics import F1Score
f1 = F1Score(task="binary", num_classes=2)
f1_score = f1(predicted, actual)
print(f"F1 Score: {f1_score}")
F1 Score: 0.5714285714285715
F1 Score of 1 = perfect precision and recall
F1 Score of 0 = worst performance
DEEP LEARNING FOR TEXT WITH PYTORCH
Considerations
Multiclass cores may be identical
Can indicate good model performance
Always consider the problem when interpreting results!
DEEP LEARNING FOR TEXT WITH PYTORCH
Let's practice!
DEEP LEARNING FOR TEXT WITH PYTORCH