Text Preprocessing
21/11/2025
1
Tokenization
• Given a character sequence and a defined
document unit, tokenization is the task of
chopping it up into pieces, called tokens,
21/11/2025
perhaps at the same time throwing away certain
characters, such as punctuation.
• Term – an element of the vocabulary
• Token – an instance of that type in running text
• Input - “Welcome to this session on text
analytics”
• Output – Welcome, to, this, session, on, text,
2
analytics
Tokenization (cont…)
• nltk Tokenizers
• Sentence tokenizer
21/11/2025
• Word tokenizer
3
from [Link] import word_tokenize, sent_tokenize
line = "Hello! Welcome to text pre-processing session. Do you
21/11/2025
know python? Mr. XXX is the tutor for tis session"
sentences = sent_tokenize(line)
print(sentences)
tokens = word_tokenize(line)
print(tokens)
4
import nltk
from nltk import word_tokenize
raw = “Type any text that you wish to type for checking the
21/11/2025
tokenization examples of nltk tokenizer”
print (type(raw))
tokens = word_tokenize(raw)
print(tokens)
print (type(tokens))
print(len(tokens))
5
>>> import nltk
>>> french_tokenize = [Link]('tokenizers/punkt/[Link]")
21/11/2025
>>> french_tokenize.tokenize(“French Text”)
6
Tokenization - Issues
• Finland’s capital
Finland AND s? Finlands? Finland’s?
• Hewlett-Packard Hewlett and Packard as two
21/11/2025
tokens?
• state-of-the-art: break up hyphenated
sequence.
• co-education
• lowercase, lower-case, lower case ?
• It can be effective to get the user to put in
possible hyphens
7
• San Francisco: one token or two?
• How do you decide it is one token?
Tokenization: language issues
• Arabic (or Hebrew) is basically written right to
left, but with certain items like numbers written
21/11/2025
left to right
• Words are separated, but letter forms within a
word form complex ligatures
•
• ← → ←→ ← start
8
Tokenization: language issues
• Chinese and Japanese have no spaces between words:
• 莎拉波娃现在居住在美国东南部的佛罗里达。
• Not always guaranteed a unique tokenization
21/11/2025
• Use Maximum matcher segmentation algorithm
• Further complicated in Japanese, with multiple
alphabets intermingled
フォーチュン 500 社は情報不足のため時間あた $500K( 約 6,000 万円 )
Katakana Hiragana Kanji Romaji
End-user can express query entirely in hiragana!
9
• Dates/amounts in multiple formats
Normalization
• Process of transforming text into a
single canonical form that it might not have had
21/11/2025
before.
• Text normalization requires being aware of what
type of text is to be normalized and how it is to
be processed afterwards; there is no all-purpose
normalization procedure.
10
Normalization (cont…)
• Removing the punctuation
• Converting the entire text into lower case or
21/11/2025
upper case
• Converting numbers into words
• Stop word removal
• Expanding the abbreviations etc.
11
Punctuation removal
import string
stringIn = "[Link]!"
21/11/2025
out = [Link]([Link](",.!"," " ,",.!"))
print(out)
###### OR #################
out = "".join(c for c in stringIn if c not in ('!','.',':'))
print(out)
12
Case folding
stringIn = "THIS IS IN CAPITAL CASE"
stringOut = [Link]()
21/11/2025
print(stringOut)
13
Stop words
• Stop words are the words that need to filtered
before the actual task (in information retrieval,
21/11/2025
NLP, text mining etc.)
• Stop words do not contribute much to the
overall meaning of a sentence.
• In the case of search engines search space is
reduces by deleting the stop words.
14
import nltk
from [Link] import stopwords
Stops = set([Link]('english'))
text = "When we use a web search engine, we usually don't mind if the
words in the document differ from our search terms in having different
endings.
21/11/2025
my_tokens =[Link]()
print(my_tokens)
myNewList =[]
for word in my_tokens:
if word not in stops:
[Link](word)
print("**********After Removal of Stop words***********")
print(myNewList)
15
print([Link]())
print([Link]('english'))
Stemming
• Reduce terms to their “roots” before indexing
• “Stemming” suggests crude affix chopping
• language dependent
21/11/2025
• e.g., automate(s), automatic, automation all
reduced to automat.
for exampl compress and
for example compressed compress ar both accept
and compression are both as equival to compress
accepted as equivalent to
compress.
16
Porter’s algorithm
• Commonest algorithm for stemming English
• Results suggest it’s at least as good as other
21/11/2025
stemming options
• Conventions + 5 phases of reductions
• phases applied sequentially
• each phase consists of a set of commands
• sample convention: Of the rules in a compound
command, select the one that applies to the
longest suffix. 17
Typical rules in Porter
Step 1a Example
sses ss caresses caress
21/11/2025
ies i ponies poni
ss -> ss caress -> caress
s -> Φ cars -> car
Step 1b
Examaple
(*v*)ing Φ
walking walk
sing ->sing
(*v*)ed Φ
plastered -> plaster
18
Typical rules in Porter
Step 2 (for long stems) Example:
ational ate Relational -> relate
21/11/2025
izer ize Digitizer -> digitize
ator -> ate operator ->operate
Example
Step 3 (for longer stems)
al Φ
revival reviv
able Φ
adjustable adjust
ate -> Φ
activate -> activ
s -> Φ
s -> Φ
19
import nltk
porter = [Link]()
myList =
21/11/2025
["cry","crying","cried","look","make","making","into","intimate"]
print(myList)
print([[Link](t) for t in myList])
lancaster = [Link]()
print([[Link](t) for t in myList])
20
Lemmatization
• Reduce inflectional/variant forms to base form
• E.g.,
21/11/2025
• am, are, is be
• car, cars, car's, cars' car
• the boy's cars are different colors the boy car
be different color
• Lemmatization implies doing “proper” reduction
to dictionary headword form
21
from [Link] import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
21/11/2025
my_tokens =
["leaves","calves","books","surprising","datamining"]
for word in my_tokens:
print([Link](word))
22
Parts-of-speech tagging
• Process of assigning a category ( for example,
noun, verb, adjective etc.) tag to individual
21/11/2025
tokens in a sentence.
23
import nltk
text1 = nltk.word_tokenize("It is a pleasant day")
21/11/2025
print(nltk.pos_tag(text1))
24