Tokenization
MODULE OBJECTIVES
○ Define and explain the concept of
tokenization and its importance in
NLP.
o Apply tokenization techniques using
Python libraries (e.g., NLTK, spaCy)
o Understand stemming and
lemmatization
Tokenizati
o on
is the process of breaking up the original
text into component pieces (tokens).
o by converting text into smaller
components, machines can more
effectively analyze and understand
human language
Tokens
o are the basic building blocks of a
document object
o everything that helps us understand the
meaning of the text is derived from
tokens and their relationship to one
another.
Tokenizati
on
tokens are pieces of the original text
Tokenizati
on
Tokenizati
on
Stemming
o is a somewhat crude method for cataloging
related words.
o it essentially chops off letters from the end until
the stem is reached.
o It simplifies words to their most basic form,
making it easier to analyse and process text.
Examples of
Stemming
Types of Stemmer in
NLTK
o Porter’s Stemmer
o Snowball Stemmer
o Lancaster Stemmer
o Regexp Stemmer
Porter’s
o Stemmer
one of the most common and effective stemming
tools
o developed by Martin Porter in 1980
o the algorithm employs five phases of word
reduction, each with its own set of mapping rules.
Porter’s
o Stemmer
it is well-known for its simplicity, speed and
reliability.
o it does not guarantee to have meaningful words
and limited to English language.
Porter’s
o
Stemmer
In the first phase, simple suffix mapping rules are
defined, such as:
Porter’s
o
Stemmer
From a given set of stemming rules only one rule is
applied, based on the longest suffix S1. Thus, caresses
reduces to caress but not cares.
Porter’s
o
Stemmer
More sophisticated phases consider the
length/complexity of the word before applying a rule.
For example:
Porter’s
Stemmer
Stemming words
Porter’s
Stemmer
Stemming words from sentences
Snowball
o
o Stemmer
is an enhanced version of the Porter Stemmer
the algorithm used here is more accurately called
the "English Stemmer" or "Porter2 Stemmer"
o it offers a slight improvement over the original
Porter stemmer, both in logic and speed
o it supports multiple languages
o its limitation is more aggressive which might lead
to over-stemming.
Snowball
Stemmer
Lancaster
o
Stemmer
is known for being more aggressive and faster
than other stemmers
o it is somehow destructive and leads to excessively
shortened stems
o uses a set of external rules that applies an
iterative manner.
o Less efficient than Snowball in larger datasets
Lancaster
Stemmer
Regexp
is a flexible stemming algorithm that allows users
Stemmer
o
to define custom rules using regular expressions.
o this can be very helpful for very specific tasks
where predefined rules are necessary for
stemming.
o requires manual rule definition
o can be computationally expensive for large
datasets
Regexp
Stemmer
References
o
:
[Link]
cking%20Text%20Data%20with%20Machine%20Learning%20and%20Deep%20Learning%20using%20Python%20(%20PDFDr
ive%20).pdf
o [Link]
o [Link]
230119_1-ps-other~dsa~tofu_2-b2c_3-apac_4-prc_5-na_6-na_7-le_8-pdsh-go_9-nb-e_10-na_11-na&utm_loc=9067282-&u
tm_mtd=-c&utm_kw=&utm_source=google&utm_medium=paid_search&utm_content=ps-other~apac-en~dsa~tofu~blog-a
rtificial-intelligence&gad_source=1&gad_campaignid=19589720824&gbraid=0AAAAADQ9WsGQRivbblCZNhL4qzs3Je5uN&g
clid=Cj0KCQjw5onGBhDeARIsAFK6QJYV01iom8NgVgJFRLvbkyr6nEu8olK7iJ5uNektqxQ3-czrJi6yW30aAo1eEALw_wcB
o [Link]
Thank you for
listening ☺