M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Module 2: Detecting Spam Emails

Dr. Nikos Tsourakis
 Course outline

• Module 0: Python Crash Course • Module 6: Teaching Machines to

Translate
• Module 1: Intro to Machine Learning
• Module 7: Summarizing Wikipedia
• Module 2: Detecting Spam Emails Articles
• Module 3: Classifying Topics of • Module 8: Detecting Hateful and
Newsgroup Posts Offensive Language
• Module 4: Extracting Sentiments from • Module 9: Generating Text in Chatbots
Product Reviews
• Module 10: Clustering Speech-to-Text
• Module 5: Recommending Music Titles Transcriptions

Detecting Spam Emails 2

Overview

• Electronic mail is one of the most ubiquitous Internet services to exchange

messages asynchronously
• The problem in this communication scheme is identifying and blocking
unsolicited and unwanted messages
• We deal with this problem from a machine learning perspective and unfolds as a
series of steps for developing and evaluating a typical spam detector
• We elaborate on the limitations of performing spam detection using traditional
programming
• Next, we introduce the basic techniques for text representation and preprocessing
• Finally, we implement two classifiers using an open-source dataset and evaluate their
performance on standard metrics

Detecting Spam Emails 3

Module objectives

After completing this module, you should be able to:

• Obtaining the data

• Understanding its content

• Preparing the datasets for the analysis

• Training the classification models

• Realizing the tradeoffs of the algorithms

• Assessing the performance of the models

Recommending Music Titles 4

M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Section 1: Understanding spam detection

Spam detection

• Software inside the mail server that decides whether an incoming email is spam
or not
• Without a spam detector the inbox will be flooded with irrelevant and possibly
dangerous correspondence
• A serious technical, economic and even social threat!
• If you were to implement a detector program from scratch how would you do it?

Courtesy : Google images (Medium post)

Detecting Spam Emails 6
Spam detection

• How can we build a spam detector for the following email?

Detecting Spam Emails 7

Spam detection

• Let’s try to think of possible strategies for the detector

Detecting Spam Emails 8

Spam detection

• Let’s try to think of possible strategies for the detector

➢ T1 – The text in the subject field is

typical for spam. It is characterized by
a manipulative style that creates
unnecessary urgency and pressure

Detecting Spam Emails 9

Spam detection

• Let’s try to think of possible strategies for the detector

➢ T2 – The message begins with the

phrase Dear MR tjones. The last
word was probably extracted
automatically from the recipient's
email address

Detecting Spam Emails 10

Spam detection

• Let’s try to think of possible strategies for the detector

➢ T3 – Bad spelling and the incorrect

use of grammar are potential spam
indicators

Detecting Spam Emails 11

Spam detection

• Let’s try to think of possible strategies for the detector

➢ T4 – The text in the body of the

message contains sequences with
multiple punctuation marks or capital
letters

Detecting Spam Emails 12

Write the code

• We can now implement a program taking into account the

four triggers
• When a new email arrives, the detector runs the following
code and decides whether it is a spam

• Obviously, this is not the best detector ever built!

Detecting Spam Emails 13
Add more triggers

• Words can also serve as part of our separation criteria

• Visualize the text data using word clouds (also known as tag clouds)

• The image suggests that the most common word in our spam
message is “virus”
Detecting Spam Emails 14
Add more triggers

• Next, we adapt the pseudocode

Detecting Spam Emails 15

Add more triggers

• Next, we adapt the pseudocode

• Is this new version of the program better?

Detecting Spam Emails 15

Add more triggers

• Next, we adapt the pseudocode

• Is this new version of the program better?

• Slightly
• We can engineer even more criteria, but the problem becomes
insurmountable at some point

Detecting Spam Emails 15

Add more triggers

• Next, we adapt the pseudocode

• Is this new version of the program better?

• Slightly
• We can engineer even more criteria, but the problem becomes
insurmountable at some point
• This is where machine learning algorithms become handy and can
help us to solve this problem

Detecting Spam Emails 15

Learn by example

• As humans learn by example; touching something very hot (input)

causes pain (output), we can train a machine to read emails (input)
and classify them automatically into spam/non-spam (output)

[Link]

Detecting Spam Emails 16

Feature engineering

• The input to the learning algorithm was not the actual emails text but
some kind of information extracted from those (T1-T4)
• These are called features and the process of creating them is called
feature engineering
• Identifying a suitable list of features for any ML task requires domain
knowledge – comprehending the problem you want to solve in-
depth.
• How you choose them directly impacts the algorithm’s performance
and determines its success to a significant degree:
• certain features can overlap with others
• specific features might be less relevant
Detecting Spam Emails 17
Feature engineering

• Let’s see a few examples and propose good features

Detecting Spam Emails 18

Feature engineering

• Let’s see a few examples and propose good features

Detecting Spam Emails 18

Feature engineering

• Let’s see a few examples and propose good features

Detecting Spam Emails 18

Feature engineering

• Let’s see a few examples and propose good features

Detecting Spam Emails 18

Feature engineering

• Let’s see a few examples and propose good features

Detecting Spam Emails 18

Feature engineering

• Let’s see a few examples and propose good features

Detecting Spam Emails 18

Feature engineering

• Let’s see a few examples and propose good features

• But as we are dealing with text, and we need pertinent features

Detecting Spam Emails 18
M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Section 2: Extracting word representations

Data representation

• What does a word mean to a computer? What about an

image or an audio file?
• To put it simply, nothing!
• A computer circuit can only process signals that contain two
voltage levels or states, similar to an on-off switch
• This representation is the well-known binary system where
every quantity is expressed as a sequence of 1s (high
voltage) and 0s (low voltage)
• All data should be represented numerically

Detecting Spam Emails 20

Word representations

• Numbers can codify a word, the pixels of an image, the audio

samples of an audio file, etc.

Detecting Spam Emails 21

Word representations

• Numbers can codify a word, the pixels of an image, the audio

samples of an audio file, etc.
I want to become
a numbers
vector!!!

• There are various ways to represent words in machine learning

problems
Detecting Spam Emails 21
Using label encoding

• Consider this quote from Aristotle “a friend to all is a friend to none”

• Using label encoding we can produce the following mapping:

• This representation, however, implies some short of order (because

0 < 2 < 5), which is not true “a friend to all
is a friend to
none”

Detecting Spam Emails 22

Using one-hot encoding

• One-hot encoding codifies every word as a vector with zeros and a

single one
• no two words exist with the same one-hot vector

Detecting Spam Emails 23

Using one-hot encoding

• One-hot encoding codifies every word as a vector with zeros and a

single one
• no two words exist with the same one-hot vector

• The majority of the elements in the array are zeros and can pose
challenges due to the memory required to store them
Detecting Spam Emails 23
Using one-hot encoding

• One-hot encoding codifies every word as a vector with zeros and a

single one
• no two words exist with the same one-hot vector

Matrixes of this
kind are called
sparse

• The majority of the elements in the array are zeros and can pose
challenges due to the memory required to store them
Detecting Spam Emails 23
Using token count encoding

• Also known as bag-of-words (BoW) counts the absolute frequency of

each word within a sentence or a document
• The input is represented as a bag of words without taking into
account grammar or word order
• Create a table (Term Document Matrix) where each cell represents
the number of times a word from the vocabulary appears the quote
“a friend to all
is a friend to
none”

Detecting Spam Emails 24

N-grams as tokens

• Certain word combinations are more frequent than others in any

human language
• Consist of a single word (unigrams), two words (bigrams), three
words (trigrams), etc.

Detecting Spam Emails 25

Using tf-idf encoding

• One limitation of BoW representations is that they do not consider

the value of words inside the corpus
• If solely frequency were of prime importance, articles such as a or
the would provide the most information for a document
• The term frequency-inverse document frequency (tf-idf) encoding
scheme:
• penalizes these frequent words
• allows us to weigh each word in the text
• Heuristic where more common words tend to be less relevant for
most semantic classification tasks, and the weighting reflects this
approach
Detecting Spam Emails 26
Using tf-idf encoding

• From a virtual dataset with 10 million emails, we randomly pick one

containing 100 words
• Suppose that the word virus appears three times in this email, so its
3
term frequency (tf) is: = 0.03
100
• Moreover, the same word appears in 1000 emails in the corpus, so
10000000
the inverse document frequency (idf) is: log( )=4
1000
• The tf-idf weight is simply the product of these two statistics:
0.03 * 4 = 0.12

Detecting Spam Emails 27

Calculating vector similarity

• Cosine similarity is the degree to which two vectors point in the

same direction
• It targets orientation, rather than magnitude
• When the two vectors are aligned to the same direction cosine
similarity is 1, when they are perpendicular it is 0, and when they
point in opposite directions it is -1

Detecting Spam Emails 28

Cosine similarity

• Which vectors are more similar (A, B or C)?

A=(4, 4, 4),
B=(1, 7, 5)
C=(-5, 5, 1)

Detecting Spam Emails 29

Cosine similarity
𝑉•U
• Given that: cos(𝑉, 𝑈) =
𝑉 × 𝑈
• A=(4, 4, 4), B=(1, 7, 5) and C=(-5, 5, 1)
• We have:
𝐴 • 𝐵 = 4 × 1 + 4 × 7 + 4 × 5 = 52
𝐴 • 𝐶 = 4 × −5 + 4 × 5 + 4 × 1 = 4
𝐵 • 𝐶 = 1 × (−5) + 7 × 5 + 5 × 1 = 35

𝐴 = 42 + 42 + 42 = 48, 𝐵 = 75 and 𝐶 = 51
• We obtain:
52 4 35
cos(𝐴, 𝐵) = ≅ 0.87 , cos(𝐴, 𝐶) = ≅ 0.08 and cos(𝐵, 𝐶) = ≅ 0.57
48× 75 48× 51 75× 51

Detecting Spam Emails 30

M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Section 3: Executing data preprocessing

Tokenizing the input

• During tokenization we split textual data into smaller components

called tokens
• These can be words, phrases, symbols, or other meaningful elements
• We can use regular expressions (regexp) that can assist with the creation
of a tokenizer

“The best course ever taught about Text Mining”

“The”, “best”, “course”, “ever”, “taught”, “about”, “Text”, “Mining”

Detecting Spam Emails 32

Understanding regular expressions

• Regular expressions (regexp) are used to:

• find a string in a document
• replace part of the text with something else
• examine the conformity of some textual input

Detecting Spam Emails 33

Understanding regular expressions

• Regular expressions (regexp) are used to:

• find a string in a document
• replace part of the text with something else
• examine the conformity of some textual input

Detecting Spam Emails 33

Understanding regular expressions

• Check the validity of an email address

Detecting Spam Emails 34

Removing stop words

• A typical task during the preprocessing phase is removing all the

words that presumably help us focus on the most important
information in the text
• These are called stop words and there is no universal list in English or
any other language
• Examples of stop words include determiners (such as another and
the), conjunctions (such as but and or), and prepositions (such as
before and in)
tokenization “The”, “best”, “course”, “ever”, “taught”, “about”, “Text”, “Mining”

stop word removal “best”, “course”, “ever”, “taught”, “Text”, “Mining”

Detecting Spam Emails 35

Stemming the words

• Map words with the same core meaning to a central word or

symbol
• During stemming we cut off the end (suffix) or the beginning
(prefix) of an inflected word and ending up with its stem (the
root word)
• For example, the stem of the word plays is play
stop word removal “best”, “course”, “ever”, “taught”, “Text”, “Mining”

stemming or/and lemmatization “best”, “course”, “ever”, “teach”, “text”, “mine”

Detecting Spam Emails 36

Lemmatizing the words

• Another approach for reducing the inflectional forms of a word to a

base root is called lemmatization
• The method performs morphological analysis of the word and
obtains its proper lemma (the base form under which it appears in a
dictionary)
• For example, the lemma of led is lead
• Lemmatization differs from stemming, as it requires detailed
dictionaries to look up a word
stop word removal “best”, “course”, “ever”, “taught”, “Text”, “Mining”

stemming or/and lemmatization “best”, “course”, “ever”, “teach”, “text”, “mine”

Detecting Spam Emails 37
 Tasks
Let’s practice! • Word representations
• Data preprocessing

[Link]
hub/PacktPublishing/Machine-
Learning-Techniques-for-
Text/blob/main/chapter-02/spam-
[Link]

38 Detecting Spam Emails

M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Section 4: Performing classification

Support Vector Machines

• Support Vector Machines is a supervised machine learning algorithm

Detecting Spam Emails 40

Support Vector Machines

• The data below is separated with two different lines. Which

separation is better?

Detecting Spam Emails 41

Support Vector Machines

• The data below is separated with two different lines. Which

separation is better?

• The line in the left plot has a higher classification error

• The margin in the second is small and therefore the model lacks in
generalization
Detecting Spam Emails 41
Hyperparameter C

• The SVM algorithm permits the adjustment of the training accuracy

versus generalization tradeoff using the hyperparameter C
• A frequent point of confusion is that hyperparameters and model
parameters are the same things, but this is not true
• Hyperparameters are parameters whose values are used to control the
learning process
• On the other hand, model parameters update their value in very training
iteration until we obtain a good classification model
• We can direct the SVM to create the most efficient model for each
problem by adjusting the hyperparameter C

Detecting Spam Emails 42

Support Vector Machines

• Which one seems to work better this time?

• At first glance, the curved line in the plot on the right perfectly
separates the data into two classes
• But getting too specific boundaries entails the risk of overfitting
• The model learns the training data perfectly but fails to classify a slightly
different example correctly
Detecting Spam Emails 43
Overfitting

• Most of us have grown in a certain

cultural context, trained (overfit) to
interpret social signals like body
posture, facial expressions, voice tone,
etc. in a certain way
• When socializing with people of
diverse social cultures we might fail to
interpret similar social signals correctly
(generalization)

Detecting Spam Emails 44

Overfitting

• Most of us have grown in a certain

cultural context, trained (overfit) to
interpret social signals like body
posture, facial expressions, voice tone,
etc. in a certain way
• When socializing with people of
diverse social cultures we might fail to
interpret similar social signals correctly
(generalization)

There is always a tradeoff between accuracy during training and

generalization during inference
Detecting Spam Emails 44
Hyperparameter gamma

• We can also prevent overfitting by adjusting the gamma

hyperparameter
• Defines how far the influence of a single training example reaches
• The curvature of the decision boundary can also be affected by
points that are pretty far from it.
• The takeaway here is that both C and gamma hyperparameters help
us create more efficient models but identifying their best values
demands experimentation

Detecting Spam Emails 45

Data pipeline

• Getting the data

• Use the SpamAssassin public mail corpus is a selection of mail messages,
suitable for use in testing spam filtering systems
• [Link]
• Creating the train and test sets
• We choose a 75:25 split between the training and test sets, attributing the
larger proposition to the training data
• Preprocessing the data
• tokenization, stop word removal, and lemmatization
• Extracting the features
• tf-idf vectorization
Detecting Spam Emails 46
 Tasks
Let’s practice! • Word representations
• Data preprocessing
• Classification

[Link]
hub/PacktPublishing/Machine-
Learning-Techniques-for-
Text/blob/main/chapter-02/spam-
[Link]
47 Detecting Spam Emails
M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Section 5: Measuring classification performance

Accuracy

• Accuracy is the percentage of correctly classified examples by an

algorithm divided by the total number of examples, defined as:

• If we had to choose the best algorithm this should probably be the

one with the highest accuracy
• The argument is that the algorithm with the highest number of
correct classifications should be the right choice
• Although this is not far from the truth, it is not always the case

Detecting Spam Emails 49

Accuracy

• A model classifies 1000 emails that we already know whether they

spam or ham
• A possible result of classifying these emails is shown in the table
below, known as confusion matrix

Detecting Spam Emails 50

Accuracy

• Rewrite the f𝑜𝑟𝑚𝑢𝑙𝑎

𝑇𝑃 + 𝑇𝑁 15 + 880
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = = = 0.895
𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁 15 + 880 + 20 + 85

Detecting Spam Emails 51

Accuracy

• Rewrite the f𝑜𝑟𝑚𝑢𝑙𝑎

𝑇𝑃 + 𝑇𝑁 15 + 880
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = = = 0.895
𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁 15 + 880 + 20 + 85

• Out of the 1000 spam emails (TP + FN) only 15 were correctly
identified and the other 85 were considered as ham emails
Detecting Spam Emails 51
Accuracy

• To assess the performance of a model correctly, we need to make this

analysis and consider the type of errors that are most important
within the task
• Is it better to have a strict model that can
block a legitimate email for the sake of
fewer spam ones (increased FPs)?
• Or is it preferable to have a lenient model
that doesn’t block most ham emails but
allows more undetected spam in your
mailbox (increased FNs)?

Detecting Spam Emails 52

Accuracy

• Similar questions arise in all ML problems and generally in many real-

world situations
• For example, wrong affirmative decisions (FPs) in a fire alarm system
are preferable to wrong negative ones (FNs)
• In the first case, we get a false alert of a fire that didn’t occur
• Conversely, declaring innocent a guilty prisoner implies higher FNs,
which is preferable to finding guilty an innocent one (higher FPs)
• Accuracy is a good metric when the test data is balanced and the
classes are equally important

Detecting Spam Emails 53

Precision and Recall

• Precision us the proportion of positive identifications that are, in

reality, correct, given by:
𝑇𝑃 15
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = = = 0.43
𝑇𝑃 + 𝐹𝑃 15 + 20

• Recall tells us the proportion of the actual positives that are

identified correctly, given by:
𝑇𝑃 15
𝑅𝑒𝑐𝑎𝑙𝑙 = = = 0.15
𝑇𝑃 + 𝐹𝑁 15 + 85

• Improving precision often deteriorates recall and vice versa

Detecting Spam Emails 54

F-score

• We can combine precision and recall (harmonic mean) in one more

reliable F-score metric

𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ∗ 𝑟𝑒𝑐𝑎𝑙𝑙 0.43 ∗ 0.15

F−score = 2 ∗ =2∗ = 0.22
𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑟𝑒𝑐𝑎𝑙𝑙 0.43 + 0.15

• When precision and recall reach their perfect score (equal to 1), the
F-score becomes 1
• Based on this metric we can evaluate more reliably different ML
models

Detecting Spam Emails 55

ROC and AUC

• When the classifier returns some kind of confidence score for each
prediction, we can use another technique for evaluating performance
called the Receiver Operator Characteristic (ROC) curve
• A ROC curve is a graphical plot that shows the model’s performance
at all classification thresholds
• It utilizes two rates, namely the True Positive Rate (TPR), the same as
recall, and the False Positive Rate (FPR)

Detecting Spam Emails 56

ROC and AUC

• True Positive Rate (TPR)

𝑇𝑃
𝑇𝑃𝑅 =
𝑇𝑃 + 𝐹𝑁
• False Positive Rate (FPR)
𝐹𝑃
𝐹𝑃𝑅 =
𝐹𝑃 + 𝑇𝑁

• The benefit of ROC curves is that they help us visually identify the
trade-offs between the TPR and FPR
• In this way, we can find which classification threshold better suits the
problem under study
Detecting Spam Emails 57
ROC and AUC

• A simplified example
with 10 emails
• 7 thresholds are used
• For each prediction a
probability score is
generated by the model

Detecting Spam Emails 58

ROC and AUC

• The grayed area in these plots, called the Area Under the ROC Curve
(AUC), is related to the quality of our model; the higher its surface,
the better it is

Detecting Spam Emails 59

AUC for two models

• The model based the support vector machines provides better results

Detecting Spam Emails 60

Precision-recall curves

• ROC curves can sometimes perform too optimistically with

imbalanced datasets
• For example, using the TN factor during the FPR calculation can skew
the results
• The solution, in this case, is to generate another visualization called
the Precision-Recall curve
• Fortunately, this factor is not part of the precision or recall formulas
• We must scrutinize both ROC and precision-recall curves to
understand the models’ performance and the differences between
the classifiers

Detecting Spam Emails 61

 Tasks
Let’s practice! • Word representations
• Data preprocessing
• Classification
• Performance

[Link]
hub/PacktPublishing/Machine-
Learning-Techniques-for-
Text/blob/main/chapter-02/spam-
[Link]
62 Detecting Spam Emails
M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Section 6: Typical pipeline

Detecting Spam Emails 64
 input “The best course ever taught about Text Mining”

Detecting Spam Emails 64

 input “The best course ever taught about Text Mining”

tokenization “The”, “best”, “course”, “ever”, “taught”, “about”, “Text”, “Mining”

Detecting Spam Emails 64

 input “The best course ever taught about Text Mining”

tokenization “The”, “best”, “course”, “ever”, “taught”, “about”, “Text”, “Mining”

stop word removal “best”, “course”, “ever”, “taught”, “Text”, “Mining”

Detecting Spam Emails 64

 input “The best course ever taught about Text Mining”

tokenization “The”, “best”, “course”, “ever”, “taught”, “about”, “Text”, “Mining”

stop word removal “best”, “course”, “ever”, “taught”, “Text”, “Mining”

stemming or/and lemmatization “best”, “course”, “ever”, “teach”, “text”, “mine”

Detecting Spam Emails 64

 input “The best course ever taught about Text Mining”

tokenization “The”, “best”, “course”, “ever”, “taught”, “about”, “Text”, “Mining”

stop word removal “best”, “course”, “ever”, “taught”, “Text”, “Mining”

stemming or/and lemmatization “best”, “course”, “ever”, “teach”, “text”, “mine”

word representation 2456 3456 3176 896 1267 697 …

Detecting Spam Emails 64

 input “The best course ever taught about Text Mining”

tokenization “The”, “best”, “course”, “ever”, “taught”, “about”, “Text”, “Mining”

stop word removal “best”, “course”, “ever”, “taught”, “Text”, “Mining”

stemming or/and lemmatization “best”, “course”, “ever”, “teach”, “text”, “mine”

word representation 2456 3456 3176 896 1267 697 …

dimensionality reduction 4567 367 8721 …

Detecting Spam Emails 64

 input “The best course ever taught about Text Mining”

tokenization “The”, “best”, “course”, “ever”, “taught”, “about”, “Text”, “Mining”

stop word removal “best”, “course”, “ever”, “taught”, “Text”, “Mining”

stemming or/and lemmatization “best”, “course”, “ever”, “teach”, “text”, “mine”

word representation 2456 3456 3176 896 1267 697 …

dimensionality reduction 4567 367 8721 …

split in training/test sets training (70%) test (30%)

Detecting Spam Emails 64

 input “The best course ever taught about Text Mining”

tokenization “The”, “best”, “course”, “ever”, “taught”, “about”, “Text”, “Mining”

stop word removal “best”, “course”, “ever”, “taught”, “Text”, “Mining”

stemming or/and lemmatization “best”, “course”, “ever”, “teach”, “text”, “mine”

word representation 2456 3456 3176 896 1267 697 …

dimensionality reduction 4567 367 8721 …

split in training/test sets training (70%) test (30%)

classification algorithm=KNN, k=9

Detecting Spam Emails 64

 input “The best course ever taught about Text Mining”

tokenization “The”, “best”, “course”, “ever”, “taught”, “about”, “Text”, “Mining”

stop word removal “best”, “course”, “ever”, “taught”, “Text”, “Mining”

stemming or/and lemmatization “best”, “course”, “ever”, “teach”, “text”, “mine”

word representation 2456 3456 3176 896 1267 697 …

dimensionality reduction 4567 367 8721 …

split in training/test sets training (70%) test (30%)

classification algorithm=KNN, k=9

performance assessment accuracy=90%

Detecting Spam Emails 64

 input “The best course ever taught about Text Mining”

tokenization “The”, “best”, “course”, “ever”, “taught”, “about”, “Text”, “Mining”

stop word removal “best”, “course”, “ever”, “taught”, “Text”, “Mining”

stemming or/and lemmatization “best”, “course”, “ever”, “teach”, “text”, “mine”

word representation 2456 3456 3176 896 1267 697 …

dimensionality reduction 4567 367 8721 …

split in training/test sets training (70%) test (30%)

classification algorithm=KNN, k=9

performance assessment accuracy=90%

hyperparameter tuning k=1-100

Detecting Spam Emails 64
Detecting Spam Emails 65
 input

Detecting Spam Emails 65

 input

tokenization

Detecting Spam Emails 65

 input

tokenization

stop word removal

Detecting Spam Emails 65

 input

tokenization

stop word removal

stemming or/and lemmatization Porter Stemmer, Wordnet

Detecting Spam Emails 65

 input

tokenization

stop word removal

stemming or/and lemmatization Porter Stemmer, Wordnet

word representation Label, One-hot, Token count, tf-idf,

Word Embedding

Detecting Spam Emails 65

 input

tokenization

stop word removal

stemming or/and lemmatization Porter Stemmer, Wordnet

word representation Label, One-hot, Token count, tf-idf,

Word Embedding

dimensionality reduction PCA, LDA, SVD, t-SNE

Detecting Spam Emails 65

 input

tokenization

stop word removal

stemming or/and lemmatization Porter Stemmer, Wordnet

word representation Label, One-hot, Token count, tf-idf,

Word Embedding

dimensionality reduction PCA, LDA, SVD, t-SNE

split in training/test sets

Detecting Spam Emails 65

 input

tokenization

stop word removal

stemming or/and lemmatization Porter Stemmer, Wordnet

word representation Label, One-hot, Token count, tf-idf,

Word Embedding

dimensionality reduction PCA, LDA, SVD, t-SNE

split in training/test sets

classification SVM, Naïve Bayes, K-Nearest Neighbor, Random Forest,

Logistic Regression, Neural Networks

Detecting Spam Emails 65

 input

tokenization

stop word removal

stemming or/and lemmatization Porter Stemmer, Wordnet

word representation Label, One-hot, Token count, tf-idf,

Word Embedding

dimensionality reduction PCA, LDA, SVD, t-SNE

split in training/test sets

classification SVM, Naïve Bayes, K-Nearest Neighbor, Random Forest,

Logistic Regression, Neural Networks
performance assessment Accuracy, Precision, Recall, F-score, Confusion matrix

Detecting Spam Emails 65

 input

tokenization

stop word removal

stemming or/and lemmatization Porter Stemmer, Wordnet

word representation Label, One-hot, Token count, tf-idf,

Word Embedding

dimensionality reduction PCA, LDA, SVD, t-SNE

split in training/test sets

classification SVM, Naïve Bayes, K-Nearest Neighbor, Random Forest,

Logistic Regression, Neural Networks
performance assessment Accuracy, Precision, Recall, F-score, Confusion matrix

hyperparameter tuning Grid search

Detecting Spam Emails 65
 Key takeaways

Visualizations Text representations ML algorithms & models

• Word clouds • Label encoding • Support Vector Machines
• One-hot encoding • Naïve Bayes
• Token count encoding
• Tf-idf encoding

Text preprocessing ML concepts Performance metrics

• Tokenization • Supervised learning • Accuracy
• Stop words removal • Creating train and test sets • Precision
• Stemming • Feature engineering • Recall
• Lemmatization • Overfitting • F-score
• Regular expressions • ROC and AUC
• True Positive Rate

Detecting Spam Emails 66

M a c h i n e L e a r n i n g Te c h n i q u e s f o r Te x t

Questions?