M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Module 3: Classifying Topics of Newsgroup Posts

Dr. Nikos Tsourakis
 Course outline

• Module 0: Python Crash Course • Module 6: Teaching Machines to

Translate
• Module 1: Intro to Machine Learning
• Module 7: Summarizing Wikipedia
• Module 2: Detecting Spam Emails Articles
• Module 3: Classifying Topics of • Module 8: Detecting Hateful and
Newsgroup Posts Offensive Language
• Module 4: Extracting Sentiments from • Module 9: Generating Text in Chatbots
Product Reviews
• Module 10: Clustering Speech-to-Text
• Module 5: Recommending Music Titles Transcriptions

Classifying Topics of Newsgroup Posts 2

Overview

• The large volumes of unstructured text that large corporations and organizations
need to sort daily necessitate automatizing tedious and time-consuming manual
tasks
• We deal with how to tag a text document using a list of predefined topics. The
aim is to assign each sample to one and only one label
• We attack the problem by utilizing supervised and unsupervised ML techniques
• We expand on the basic exploratory data analysis presented in the previous module and
create richer visualizations with extra meaning and depth
• The transformation of data from a high-dimensional space into a low-dimensional one
• Then, we implement two classifiers and compare the different models
• Finally, we introduce state-of-the-art word representation techniques with unique properties
Classifying Topics of Newsgroup Posts 3
Module objectives

After completing this module, you should be able to:

• Creating comprehensive plots
• Reducing the complexity of data either for visualization or classification

• Setting up a baseline model

• Training the classification models

• Fine-tuning the hyperparameters

• Understanding state-of-the-art word representation techniques

Recommending Music Titles 4

M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Section 1: Understanding topic classification

Topic classification

• Businesses deal with many other unstructured texts, such as news

posts, support tickets, or customer reviews
• Failing to glean this data efficiently can lead to missed opportunities or,
even worse, angry customers
• We focus on the problem of topic classification, with the aim to assign
a topic to some piece of text
• We focus on the problem of topic classification (multiclass
classification), intending to assign a label (or topic) to a piece of text
• For this task, we use the 20 newsgroups dataset available in the scikit-
learn module, which comprises around 18,000 news posts on 20 topics

Classifying Topics of Newsgroup Posts 6

M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Section 2: Performing exploratory data analysis

Exploratory data analysis

• A primary concern during exploratory data analysis (EDA) is to verify

that the dataset is appropriately formatted
• For instance, it is not uncommon to encounter missing or out-of-the
range values
• Plotting the data or extracting various statistics can reveal this
unpleasant situation
• We also might need to transform or exclude part of the data
• Having an imbalanced dataset where one class monopolizes the whole
corpus is also a source of concern
• The ML algorithm is overexposed and subsequently learns data of one class type
well while having difficulty with samples from the less frequent classes
Classifying Topics of Newsgroup Posts 8
Dimensionality reduction

• Selecting the appropriate features for a given problem is not easy

• We can end up with redundant or highly correlated features that unnecessarily
tangle the ML algorithm
• For example, consider the task of classifying planets based on two attributes,
radius (r) and circumference (2πr)
• We are using two highly correlated quantities, and there is no extra benefit to
including both in the feature space
• The solution is to either keep one of them or introduce a new feature that is a
linear combination of radius and circumference
• This process is called dimensionality reduction and proves to be very
helpful for speeding up the training of ML algorithms, filtering noise out
of the data, performing feature extraction, and data visualization
Classifying Topics of Newsgroup Posts 9
Principal Component Analysis

• As part of the EDA, it can be helpful to visualize high-dimensional

spaces in a way that our limited human brains can comprehend
• Principal component analysis (PCA) is dimensionality reduction
technique that deals with unlabeled data, and for this reason, it is an
unsupervised learning method
• The method creates a new coordinate system with a new set of
orthogonal axes (principal components)
• the first axis goes toward the highest variance in the data
• the second one goes toward the second-highest variance

Classifying Topics of Newsgroup Posts 10

Principal Component Analysis

• As part of the EDA, it can be helpful to visualize high-dimensional

spaces in a way that our limited human brains can comprehend
• Principal component analysis (PCA) is dimensionality reduction
technique that deals with unlabeled data, and for this reason, it is an
unsupervised learning method
• The method creates a new coordinate system with a new set of
orthogonal axes (principal components)
• the first axis goes toward the highest variance in the data
• the second one goes toward the second-highest variance

Variance is a statistical measure of dispersion that shows

how far data points are spread out from their mean value
Classifying Topics of Newsgroup Posts 10
Principal Component Analysis

• Plot of 20 random points in a 3-D space

Classifying Topics of Newsgroup Posts 11

Principal Component Analysis

• Plot of 20 random points in a 3-D space

Classifying Topics of Newsgroup Posts 12

Principal Component Analysis

• A plot of the data points clusters in the new space

Classifying Topics of Newsgroup Posts 13

 Demos
Let’s try! • [Link]
component-analysis/
• [Link]

14 Classifying Topics of Newsgroup Posts

Linear Discriminant Analysis

• Linear discriminant analysis (LDA) is also a dimensionality reduction

technique
• While PCA aims to identify the combination of principal components
that maximize the variance in a dataset, LDA maximizes the separability
between different classes by projecting the points onto a lower-
dimensional space
• It aims to find the linear projection of the data in this subspace that
optimizes some measure of class separation
• In contrast to the PCA algorithm, LDA is a supervised method

Classifying Topics of Newsgroup Posts 15

Linear Discriminant Analysis

• The aim of both is to find the right components PCA: highest variance,
LDA: highest separability

Classifying Topics of Newsgroup Posts 16

Occam’s razor

• A much smaller representation with less features can provide the same
performance as a model with many features
• How can we choose between several possible and more complex
alternatives for solving a particular problem?

Classifying Topics of Newsgroup Posts 17

Occam’s razor

• A much smaller representation with less features can provide the same
performance as a model with many features
• How can we choose between several possible and more complex
alternatives for solving a particular problem?

Classifying Topics of Newsgroup Posts 17

Occam’s razor

• A much smaller representation with less features can provide the same
performance as a model with many features
• How can we choose between several possible and more complex
alternatives for solving a particular problem?

Classifying Topics of Newsgroup Posts 17

Occam’s razor

• A much smaller representation with less features can provide the same
performance as a model with many features
• How can we choose between several possible and more complex
alternatives for solving a particular problem?

Classifying Topics of Newsgroup Posts 17

Occam’s razor

• A much smaller representation with less features can provide the same
performance as a model with many features
• How can we choose between several possible and more complex
alternatives for solving a particular problem?

Classifying Topics of Newsgroup Posts 17

Occam’s razor

• A much smaller representation with less features can provide the same
performance as a model with many features
• How can we choose between several possible and more complex
alternatives for solving a particular problem?
Precedence should be given
to simplicity; the simpler
explanation of the problem
must be preferred

Classifying Topics of Newsgroup Posts 17

 Tasks
Let’s practice! • Exploratory data analysis
• Dimensionality reduction

[Link]
hub/PacktPublishing/Machine-
Learning-Techniques-for-
Text/blob/main/chapter-03/topic-
[Link]

18 Classifying Topics of Newsgroup Posts

M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Section 3: Performing classification

K-Nearest Neighbors

• Consider the cloud that contains three types of smiley faces – happy,
sad, and neutral
• There is also a hidden face depicted by a question mark. If you had to
guess what its actual type was, what would that be?

Classifying Topics of Newsgroup Posts 20

K-Nearest Neighbors

• K-Nearest Neighbors (KNN) is a non-parametric and lazy learning

method that stores the position of all data samples and classifies new
cases based on some similarity measure
• Lazy learning means that the algorithm takes almost zero time to learn
in this case
• The training samples are stored and used to classify new observations
based on a majority vote
• K is the only hyperparameter of KNN and specifies the number of
closest neighbors to be considered
• when K = 1, the nearest neighbor class is assigned to the new sample
• when K = 3, the three closest neighbors are examined
Classifying Topics of Newsgroup Posts 21
K-Nearest Neighbors

• We choose different values for K and examine the data points in each
neighborhood

Classifying Topics of Newsgroup Posts 22

Cross-validation

• What should the value of K be?

• Fine-tuning it using cross-validation
• Three basic steps
• Partitioning the data into several subsets (folds)
• Holding out one of the subsets each time and training the model with the rest
• Evaluating the model with the holdout test
• 5-fold cross-validation:

Classifying Topics of Newsgroup Posts 23

Confusion matrix

• The confusion matrix provides a better analysis of the strengths and weaknesses of
the model
• Each row (or column) represents the instances in the actual class, while each
column (or row) represents the instances in the predicted one

Classifying Topics of Newsgroup Posts 24

Decision trees

• Decision trees are one of the most popular supervised ML algorithms because their
models are intuitive and easy to explain
• The data is represented in a tree hierarchy where:
• each internal (non-leaf) node is labeled with an input feature
• the arcs in the internal nodes signify possible values for a specific feature
• each leaf represents a class

Classifying Topics of Newsgroup Posts 25

Random forest

• In ensemble learning, multiple classifiers are generated and combined

to solve a particular problem

or

• The random forest method exploits the benefits of ensemble learning

by constructing a multitude of decision trees on randomly selected data
samples
• Each decision tree produces its own prediction and the method is
responsible for choosing the best result by voting
Classifying Topics of Newsgroup Posts 26
Singular Value Decomposition

• PCA and LDA help to visualize high-dimensional data

• Techniques of this kind can also be applied during classification
to reduce the feature space of the problem
• Too many features can degrade the performance of ML algorithms
while increasing computation and memory requirements
• A suitable method for dimensionality reduction is the Singular
Value Decomposition (SVD)
• expresses the feature space in a new components system
• works well with sparse matrices frequently encountered in text
classification
Classifying Topics of Newsgroup Posts 27
Singular Value Decomposition

• PCA and LDA help to visualize high-dimensional data

• Techniques of this kind can also be applied during classification
to reduce the feature space of the problem
• Too many features can degrade the performance of ML algorithms
while increasing computation and memory requirements
• A suitable method for dimensionality reduction is the Singular
Value Decomposition (SVD)
• expresses the feature space in a new components system
• works well with sparse matrices frequently encountered in text
classification
Classifying Topics of Newsgroup Posts 27
 Tasks
Let’s practice! • Exploratory data analysis
• Dimensionality reduction
• Classification

[Link]
hub/PacktPublishing/Machine-
Learning-Techniques-for-
Text/blob/main/chapter-03/topic-
[Link]
28 Classifying Topics of Newsgroup Posts
M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Section 4: Extracting word embedding representation

Match profiles

• You are assigned to create the matching algorithm for a new dating
service
• This algorithm must identify people with similar characteristics (Big
Five) and propose candidate profiles

Classifying Topics of Newsgroup Posts 30

Word embedding

• Just as the five traits

represent each person as a
unique point in a five-
dimensional space, word
embedding represent words
in a multidimensional space,
typically in the order of
hundreds
• Following the same approach
as before, we show the
embedding vector of different
English words
Classifying Topics of Newsgroup Posts 31
Word embedding

Embed the points of a set of English words into a three-dimensional space

Classifying Topics of Newsgroup Posts 32

Vector arithmetic

• We can build word analogies using statements “a is to b as c is to d”. For

example:
• “Paris is to France as Berlin is to Germany”
• “King is to man as queen is to woman”
• etc.

• Essentially, we subtract
embedding vectors in all these
equations, a process called
vector arithmetic. For example:
• man – psychiatrist =
woman - psychologist

Classifying Topics of Newsgroup Posts 33

 Tasks
Let’s practice! • Exploratory data analysis
• Dimensionality reduction
• Classification
• Word embedding

[Link]
hub/PacktPublishing/Machine-
Learning-Techniques-for-
Text/blob/main/chapter-03/topic-
[Link]
34 Classifying Topics of Newsgroup Posts
 Key takeaways

Visualizations Dimensionality reduction ML algorithms & models

• N-gram frequencies • Principal Component Analysis • ZeroR
• Pie charts • Linear Discriminant Analysis • K-Nearest Neighbor
• Scatter plots • Singular Value Decomposition • Random Forest
• Heatmaps • Decision Trees

Text representations ML concepts Tools

• Word2Vec • Unsupervised learning • fastText
• Cross-Validation

Classifying Topics of Newsgroup Posts 35

M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Questions?