Natural Language Processing

Chapter 03
Basic text processing

Abdelkrime Aries
Laboratoire de la Communication dans les Systèmes Informatiques (LCSI)
École nationale Supérieure d’Informatique (ESI, ex. INI), Algiers, Algeria

Academic year: 2025-2026

‫ﻞ‬
り‫ﻫ‬
な o ‫ض‬の
‫ع‬l

‫ب‬
‫ط‬
本
πⵛ 勉 ‫ل‬
f‫ ك‬tl‫م‬Rt ⵓ F
た
り

Lⴿ語 ‫أ‬

F‫ر‬
Ω ⵣ
j m
φふ‫فإ‬
ⴼ ‫ ش‬e
ⴵ
License: CC-BY 4.0

Attribution 4.0 International (CC BY 4.0)

[Link]

You are free to:

Share — copy and redistribute the material in any medium or format
Adapt — remix, transform, and build upon the material for any purpose, even
commercially.

The licensor cannot revoke these freedoms as long as you follow the license terms.

Under the following terms:

Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were
made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses
you or your use.
No additional restrictions — You may not apply legal terms or technological measures that legally restrict
others from doing anything the license permits.
 Characters
Text Segmentation
Text normalization and filtering
Morphology

Natural Language Processing

03- Basic text processing: Introduction

A text is structured in paragraphs, sentences, words and characters.

Some tasks use this structure.
Many tasks rely on words as their processing unit.
Preprocessing phase:
Segmenting text;
Filtering out unnecessary data;
Normalizing words to limit their variations.

3/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation
Text normalization and filtering
Morphology

Natural Language Processing

03- Basic text processing: Plan

3 Text normalization and

1 Characters
filtering
Regular Expressions
Text normalization
Edit distance
Text filtering
2 Text Segmentation
Sentence boundary 4 Morphology
disambiguation Word formation
Word Tokenization Form reducing

4/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

Section 1

Characters

5/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing

Characters

Regular expressions
used to search for strings in text.
recognizes languages of type 3 (regular languages) in Chomsky hierarchy.
useful for lexical analysis (word tokenization).
Edit distance
used to measure the difference between two strings.
useful for fuzzy search.

6/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Regular Expressions: Syntax (1)

RE Meaning Example
. any character beg.n: I begun at the beginning.
[aeuio] specific characters [Cc][au]t: Cats are cute.
[a-e] character range [A-Z]..: I saw Karim.
[ˆaeuio] exclude characters [ˆA-Z]a.: I saw Karim.
c? one or zero colou?r: It is colour or color.
c* zero or more Ye*s : Ys! Yes! Yeeees!
c+ one or more Ye+s : Ys! Yes! Yeeees!
c{n} n occurrences Ye{3}s : Ys! Yes! Yees! Yeees!
c{n,m} between n and m occurrences Ye{1,2}s : Ys! Yes! Yees! Yeees!
c{n,} at least n occurrences Ye{2,}s : Ys! Yes! Yees! Yeees!
c{,m} at most m occurrences Ye{,2}s : Ys! Yes! Yees! Yeees!

7/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Regular Expressions: Syntax (2)

Grouping: ( )
E.g., /([Pp]retty, )+/: Pretty, pretty, pretty
good. RE Equivalence
Disjunction: | \d [0-9]
E.g., /continu(e|ation|al(ly)?)/ \D [ˆ0-9]
Text beginning: ˆ
\w [a-zA-Z0-9 ]
E.g., /ˆK/: Key of Karim.
Text ending: $ \W [ˆ\w]
E.g., /\.[ˆ.]+$/ : [Link] \s [ \r\t\n\f]
To capture a group: $n or \n where n is \S [ˆ\s]
the group’s position.
E.g., /(.*)(ual|uation)$/\1ue/.

8/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Regular Expressions: Use case

Text editors use REs for search and replacement.

Most programming languages introduce a mechanism to use REs.
Data Extraction: e.g., extract emails and phone numbers from blogs and
social media.
[Link]
One of my crazy projects.
Arabic poems meter detection.
Pros: readable rules.
Cons: many passes to process one verse.

9/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Regular Expressions: Exercise

Find the regular expression which searches all conjugations of the verb
“go” (whithout auxilary verb).
Given a log, we want to list all lines starting with “Error”; containing a
number (which starts with a digit other than zero, has 3 to 5 consecutive
zeros and ends with a digit other than zero); and ending with “...”
Find a regular expression which searches for words containing letters
“l”, “i” and “n” in this order. The beginning of the word can be in
uppercase, the rest in lowercase. For example, lion, Linux, violin,
absolution, Aladdin, ...

10/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Regular Expressions: Some humor

11/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Edit distance: Edit operations

Insertion: inserting a character into a string.

uv → uxv / u, v ∈ X ∗ ; uv ∈ X + ; x ∈ X
case → casse, entraı̂nement → entraı̂nnement
Deletion: deleting a character from a string.
uxv → uv / u, v ∈ X ∗ ; uv ∈ X + ; x ∈ X
future → futur, héros → héro
Substitution: substituting a character by another.
uxv → uyv / u, v ∈ X ∗ ; x, y ∈ X; x , y
design → dezign, croient → croyent
Transposition: changing two characters’ positions.
uxwyv → uywxv / u, v, w ∈ X ∗ ; x, y ∈ X; x , y
play → paly, cueillir → ceuillir

12/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Edit distance: Algorithms

Hamming distance: allows only substitution. Strings must have the

same size.
The longest common subsequence (LCS): allows only deletion and
insertion.
Levenshtein distance: allows insertion, deletion and substitution.
Jaro distance: allows only transposition.
Damerau–Levenshtein distance: allows insertion, deletion,
substitution end transposition between two adjacent characters.

13/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Edit distance: Hamming distance

X: source string of length n.

Y: destination string of length n.

0 if Xi = Yi

Di = Di−1 + 

1 otherwise


Examples of Hamming distance

D(100100, 101001) = 3
D(abdelkrim, abderahim) = 3

14/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Edit distance: Levenshtein distance

X: source string of length n.

Y: destination string of length m.
D: a matrix; where D[i, j] is the edit distance between the substrings
X[1..i] and Y[1..j].
To calculate D[n, m], dynamic programming is used.
D[0, 0] = 0.

D[i − 1, j] + 1 // Deletion





D[i, j − 1] + 1 // Insertion



D[i, j] = min 



 2 if X[i] , Y[ j] // Substitution

+

D[i − 1, j − 1]

 
 

 0 otherwise


15/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Edit distance: Levenshtein distance (Example)

Example of Levenshtein distance calculation

# i m m a t u r e

# 0 ←1 ←2 ←3 ←4 ←5 ←6 ←7 ←8

a ↑1 ↖←↑ 2 ↖←↑ 3 ↖←↑ 4 ↖3 ←4 ←5 ←6 ←7

m ↑2 ↖←↑ 3 ↖2 ↖← 3 ←↑ 4 ↖←↑ 5 ↖←↑ 6 ↖←↑ 7 ↖←↑ 8

i ↑3 ↖2 ←↑ 3 ↖←↑ 4 ↖←↑ 5 ↖←↑ 6 ↖←↑ 7 ↖←↑ 8 ↖←↑ 9

n ↑4 ↑3 ↖←↑ 4 ↖←↑ 5 ↖←↑ 6 ↖←↑ 7 ↖←↑ 8 ↖←↑ 9 ↖←↑ 10

e ↑5 ↑4 ↖←↑ 5 ↖←↑ 6 ↖←↑ 7 ↖←↑ 8 ↖←↑ 9 ↖←↑ 10 ↖9

16/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Edit distance: Some applications

file revision: e.g., the UNIX diff command which compares two files.
spell checking: suggesting possible corrections for a spell error (e.g.
Hunspell).
plagiarism detection: Here, words are used instead of characters.
spam filtering: Sometimes spammers intentionally use spell errors to
deceive spam detection tools.
bioinformatics: similarity between two DNA sequences.

17/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Edit distance: Exercise

In order to write the word “magnificent”, the wrong word: “magnecifint”

was written instead. Indicate each edit operation position(s) according
to the correct word (If the operation does not exist, write 0. Transposition
and substitution are prioritized, i.e. they must not e considered as a
combination of insertion/deletion operations):
Insertion : ............ Substitution : ............
Deletion : ............ Transposition : ............
Calculate Hamming and Levenstein distances of the two words: “tray”
and “tary”. Indicate different edit operations for each distance. Redo the
same thing with Lavenstein distance (substitution weight = 1).

18/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Regular Expressions
Text normalization and filtering Edit distance
Morphology

03- Basic text processing: Characters

Edit distance: Some humor

19/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Sentence boundary disambiguation
Text normalization and filtering Word Tokenization
Morphology

Section 2

Text Segmentation

20/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Sentence boundary disambiguation
Text normalization and filtering Word Tokenization
Morphology

03- Basic text processing

Text Segmentation

New paragraphs are marked by a line break, or using HTML tag <p>.
Using tools to extract text from PDFs, many line breaks may occur.
New sentences are marked by a period (or another marker).
Periods are not always used to mark sentence ending.
Words are separated by blanks.
Not always!

21/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Sentence boundary disambiguation
Text normalization and filtering Word Tokenization
Morphology

03- Basic text processing: Text Segmentation

Sentence boundary disambiguation: Problematic

/[.?!]/ is a simple RE to delimit sentences (French, English, etc.).

Periods are used in numbers: 123,456.78 (American style) 123.456,78
(European style)
Abbreviations contain periods. e.g. exempli gratia (e.g.), Doctor of
Philosophy (Ph.D.), page (p.).
Long sentences are hard to process; it will be better to split them. e.g.
After he woke up, Karim showered, brewed a cup of coffee, and settled
down to start his workday..
Quotations. e.g. After receiving the news, Karim exclaimed, ”I can’t
believe it! This is incredible!”
Some languages, such as Thai, do not use markers to delimit
sentences.
22/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing
 Characters
Text Segmentation Sentence boundary disambiguation
Text normalization and filtering Word Tokenization
Morphology

03- Basic text processing: Text Segmentation

Sentence boundary disambiguation: Features

Letter case: sentences start with uppercase.

Parts of speech (PoS): PoS can be used to decide a sentence
boundary.
Word length: abbreviations are usually short.
Prefixes and suffixes: words having infixes are less probable to be
abbreviations.
Abbreviation’s class: some abbreviations can be in the end of a
sentence (e.g. Inc.).
Proper nouns: proper nouns always start with an uppercase; they can
occur in the middle of a sentence.

23/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Sentence boundary disambiguation
Text normalization and filtering Word Tokenization
Morphology

03- Basic text processing: Text Segmentation

Sentence boundary disambiguation: Detection (rule based)

Example of a simple rule-based SBD algorithm

Data: text
Result: list of sentences’ start positions
positions = [0];
foreach delimiter d∼!?. ∈ text do
if d∼!? then
positions ← positions ∪ {position(d)+1}
else
if word after d starts with uppercase then
if word before d has infixes or has more than 5 letters or not a noun then
positions ← positions ∪ {position(d)+1}
else
if word before d not in list of abbreviations then
positions ← positions ∪ {position(d)+1}
else
if word before d ∈ list of ending abbr. then
positions ← positions ∪ {position(d)+1}
end
end
end
end
end
end
return positions;

24/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Sentence boundary disambiguation
Text normalization and filtering Word Tokenization
Morphology

03- Basic text processing: Text Segmentation

Sentence boundary disambiguation: Detection (ML-based)

Mr. Karim teaches NLP. He is very boring.

PoS Case Size HasInfix? EndClass? PoS Case ProperN?

OneHot Binary Num. Binary Binary OneHot Binary Binary

Past word Next word

Preprocessing phase

0 1 0 0 0 0 0 1 2 0 0 0 1 0 0 0 0 0 1 1

MLP
Σ
σ

Delimiter?

Figure: Example of ML-based SBD model with predifined features

25/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing
 Characters
Text Segmentation Sentence boundary disambiguation
Text normalization and filtering Word Tokenization
Morphology

03- Basic text processing: Text Segmentation

Sentence boundary disambiguation: Some humor

26/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Sentence boundary disambiguation
Text normalization and filtering Word Tokenization
Morphology

03- Basic text processing: Text Segmentation

Word Tokenization: Problematic

/[ ]+/ is a RE used to separate words (Arabic, French, English, etc.)

Some languages, such as Japanese, do not use markers to separate
words (今年は本当に忙しかったです。).
Compound words: open (“wake up”), closed (“makeup”; German:
“Lebensversicherung: life insurance”; Arabic: “ ÑîEð Q.Jm': they test them”)
or hyphenated (“well-prepared”; French: “va-t-il: will he”; “c-à-dire: which
means”).
Confusion between characters: apostrophe is used for quotation or
contraction (She’s, J’ai).
Multiple words expressions: numerical expressions such as dates.

27/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Sentence boundary disambiguation
Text normalization and filtering Word Tokenization
Morphology

03- Basic text processing: Text Segmentation

Word Tokenization: Approaches

Rules: using regular expressions (REs).

[Link]
[Link]
[Link]
[Link]
[Link]
Statistics: using language models to calculate the probability of a
character being the marker of word boundary.
[Link]
[Link]

28/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Sentence boundary disambiguation
Text normalization and filtering Word Tokenization
Morphology

03- Basic text processing: Text Segmentation

Word Tokenization: Some humor

29/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Text normalization
Text normalization and filtering Text filtering
Morphology

Section 3

Text normalization and filtering

30/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Text normalization
Text normalization and filtering Text filtering
Morphology

03- Basic text processing

Text normalization and filtering

Text normalization
PROBLEMATIC: Texts can contain different variations of the same term.
Also, some tasks, such as information retrieval, do not need the exact text
content.
SOLUTION: transform a text to its canonical form.
[Link]
english-language
Text filtering
PROBLEMATIC: Texts can contain some non useful characters, words or
expressions.
SOLUTION: deletion.

31/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Text normalization
Text normalization and filtering Text filtering
Morphology

03- Basic text processing: Text normalization and filtering

Text normalization(1)

Acronyms and abbreviations;

standard form: US → USA, U.S.A. → USA
long version: Mr. → Mister
Using the same formatting for values such as dates and numbers;
conversion to textual form: 1205 DZD → One thousand, two hundreds,
five Algerian dinars.
specific format: 12 janvier 1986, 12.01.86, January 12th, 1986 →
1986-01-12.
replacement by type: January 12th, 1986 → DATE,
kariminfo0@[Link] → EMAIL.
Transform uppercase to lowercase;
Text → text
Sometimes, letters case must not transformed, such as in proper nouns
(Will)
32/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing
 Characters
Text Segmentation Text normalization
Text normalization and filtering Text filtering
Morphology

03- Basic text processing: Text normalization and filtering

Text normalization(2)

Contractions;
y’ll → you all, s’il → si il
Diacritics;
unaccentuation: système → systeme
unvocalization: €PYK → €PYK. Unless diacritics must be used (poems)
Encoding.
The same encoding must be used for processing

33/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Text normalization
Text normalization and filtering Text filtering
Morphology

03- Basic text processing: Text normalization and filtering

Text filtering

Special characters such as non printable characters;

they can lead to erroneous processing.
Keywords in textual formats;
Tags: HTML, XML, etc.
stop words: insignificant words such as prepositions, articles and
pronouns.
In information retrieval, there is no need to index them.
In automatic text summarization, these words can alter sentence scores.

34/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Text normalization
Text normalization and filtering Text filtering
Morphology

03- Basic text processing: Text normalization and filtering

Text filtering: Some humor

35/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Word formation
Text normalization and filtering Form reducing
Morphology

Section 4

Morphology

36/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Word formation
Text normalization and filtering Form reducing
Morphology

03- Basic text processing

Morphology

Some languages admit words forming using flexion (e.g. conjugation)

and derivation (e.g. nominalization).
The most used method is affixation.
Automatizing this task can help other tasks, such as natural language
generation (NLG).
Also, the inverse task can help other tasks, such as natural language
comprehension (NLU).

37/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Word formation
Text normalization and filtering Form reducing
Morphology

03- Basic text processing: Morphology

Word formation

Flexion : morphological variation of a word according to grammatical

features (number, gender, etc.).
Verbs conjugation.
Nouns, pronouns, adjectives, adverbs, and articles declension. E.g. ¡ ¯ →
¡¢¯.
Derivation: morphological variation of a word to create a new lexeme or
to change category.
Create a new lexeme: couper → découper, ÉÔ« → ÉÒªJƒ@.
Change category: nominalization (classer → classement, classeur ; €P X.
 @ X , €PYÓ , éƒPYÓ , €P X), adjective (fatiguer → fatigant).
→ €P

38/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Word formation
Text normalization and filtering Form reducing
Morphology

03- Basic text processing: Morphology

Word formation: Example (Automatic conjugation)

Databases:
storing verbs conjugations in a database.
Template:
storing conjugations of certain verbs as models and use a list to index
similar verbs.
E.g., French verbs conjugation.
Rules:
using IF-ELSE rules and regular expression.
E.g., Arabic, English, French and Japanese verbs conjugation:
[Link]
E.g., Arabic verbs conjugation:
[Link]

39/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Word formation
Text normalization and filtering Form reducing
Morphology

03- Basic text processing: Morphology

Word formation: Some humor

40/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Word formation
Text normalization and filtering Form reducing
Morphology

03- Basic text processing: Morphology

Form reducing: Stemming

Affixes deletion.
Result: stem.
E.g., chercher → cherch, arguing → argu.
Database: storing all terms and their stems in a table.
Statistics: using a language model (N-Gram) to estimate truncation
position.
Rules:
Porter algorithm [Porter et al., 1980].
[Link]
[Link]

41/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Word formation
Text normalization and filtering Form reducing
Morphology

03- Basic text processing: Morphology

Form reducing: Stemming (e.g. Porter algorithm)

A set of rules: condition/action.

[Link] a framework to create stemmers
Condition on the stem:
length, ending, if it contains vowels, etc.
e.g., (*v*) Y → I: happy → happi, sky → sky
Condition on the affix:
For Porter stemmer, there is only suffixes
e.g., SSES → SS, ATIONAL → ATE
Condition on the rules:
if a rule was executed, others must not.

42/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Word formation
Text normalization and filtering Form reducing
Morphology

03- Basic text processing: Morphology

Form reducing: Lemmatization

Search canonical form of a word.

Result: lemma.
e.g., comprennent →comprendre, better →good.
Word’s context is used. e.g. saw → (V) see ou (N) saw.
Lexical bases:
[Link] (NLTK
WordNetLemmatizer)
[Link]
[Link]
Machine learning:
[Link]

43/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Word formation
Text normalization and filtering Form reducing
Morphology

03- Basic text processing: Morphology

Form reducing: Lemmatization (E.g. Morphy)

Wordnet’s “morphy” Lemmatization

Data: word, PoS
Result: list of possible lemmas
if word ∈ exceptions list[PoS] then
return search in dictionary({word} ∪ exceptions list[PoS]);
end
forms = {word};
while forms , ∅ do
forms = delete affixes(forms, PoS);
results = search in dictionary({word} ∪ forms);
if results , ∅ then return results ;
end
return ∅;

44/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Characters
Text Segmentation Word formation
Text normalization and filtering Form reducing
Morphology

03- Basic text processing: Morphology

Form reducing: Some humor

45/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Bibliography

Section 5

Bibliography

46/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing

 Bibliography

Bibliography

Indurkhya, N. and Damerau, F. J. (2010).

Handbook of Natural Language Processing.
Chapman & Hall/CRC, 2nd edition.

Jurafsky, D. and Martin, J. H. (2019).

Speech and Language Processing.
[Link]

Porter, M. F. et al. (1980).

An algorithm for suffix stripping.
Program, 14(3):130–137.

46/46 Abdelkrime Aries (2025-2026) NLP: 03- Basic text processing