ABDERRAHIM Med Alaeddine

[Link]@[Link]
Text preprocessing
1. Text cleaning
2. Text tokenization
3. Text normalization

2
Introduction to text Preprocessing

3
What is NLP?
Text preprocessing is the set of techniques used to transform raw text
into a clean and structured format suitable for computational processing
and machine learning models. Natural language in its raw form
contains:
▪ Noise (URLs, emojis, HTML tags)
▪ Inconsistent casing
▪ Spelling variations
▪ Morphological variations
▪ Ambiguity

4
Text Cleaning
Text cleaning removes non-linguistic noise and standardizes formatting.
Formally, it applies filtering functions:
𝑑′ = 𝑓𝑐𝑙𝑒𝑎𝑛 𝑑
Common operations include:
➢Lowercasing
➢Removing punctuation
➢Removing non-alphanumeric characters
➢Removing HTML tags
➢Normalizing whitespace
➢Handling contractions
Cleaning reduces irrelevant token variance and prevents vocabulary
inflation.
5
Lowercasing
sentences = [ "HELLO World!",
"Handling Contractions Isn't Hard.",
"Version2 of the MODEL Achieved 95% Accuracy."]
lowercased_sentences = [[Link]() for sentence in sentences]
print(lowercased_sentences)
[
'hello world!',
"handling contractions isn't hard.",
'version2 of the model achieved 95% accuracy.'
]

6
Lowercasing
import re
def clean_text(text):
text = [Link]()
text = [Link](r"[^\w\s]", "", text) # remove punctuation
return text
example = "Hello World! NLP is AMAZING."
print(clean_text(example))
--------------------------------------------------------
hello world nlp is amazing

7
Removing punctuation
import nltk
from [Link] import word_tokenize
import string
text = "Hello, world! NLP is amazing :)"
tokens = word_tokenize(text)
tokens_clean = [w for w in tokens if w not in [Link]]
print(tokens_clean)
-----------------------------
['Hello', 'world', 'NLP', 'is', 'amazing', ':', ')']
8
Removing punctuation
import spacy
nlp = [Link]("en_core_web_sm")
text = "Hello, world! NLP is amazing :)"
doc = nlp(text)
tokens_clean = [[Link] for token in doc if not token.is_punct]
print(tokens_clean)
-------------------------------------
['Hello', 'world', 'NLP', 'is', 'amazing', ':)']

9
Removing punctuation
from [Link] import preprocess_string
text = "Hello, world! NLP is amazing."
clean_tokens = preprocess_string(text)
print(clean_tokens)
-----------------------------------
Gensim automatically:
▪ lowercases
▪ removes punctuation
▪ removes stopwords
▪ stems words

10
When you should remove punctuation
1) Classic Machine Learning models:
• Naive Bayes
• Logistic Regression
• SVM
• Random Forest
• Bag-of-Words / TF-IDF
Because punctuation usually doesn’t add much meaning and just adds
noise.
• Example:
"good!!!" and "good" should be treated similarly.

11
When you should remove punctuation
2) Topic modeling
▪ LDA
▪ NMF
▪ Punctuation can create useless tokens.

3) Search engines / keyword matching

▪ If you’re building:
▪ a basic search tool
▪ text similarity with TF-IDF

12
 When you should NOT remove punctuation
1) Transformers / BERT / GPT models
• Models like:
• BERT
• RoBERTa
• GPT
• T5
Punctuation helps the tokenizer + meaning.
• Example:
• "Let's eat, grandma."
• "Let's eat grandma." (cannibal vibes)

13
 When you should NOT remove punctuation
2) Sentiment analysis
• Punctuation can carry emotion:
• "Great." (neutral / cold)
• "Great!!!" (very positive / excited)
• "Great??" (sarcastic / confused)

3) Chatbots / conversational text

• Punctuation helps detect:
• questions (?)
• excitement (!)
• tone (...)

14
 When you should NOT remove punctuation
4) Named Entities and abbreviations
Example:
• U.S.A.
• Mr.
• Ph.D.
Removing punctuation may destroy meaning.

15
Removing non-alphanumeric characters
That means removing things like:
@#$%&*!
emojis
punctuation , . ? !
but keeping letters + digits: a-z A-Z 0-9

16
Removing non-alphanumeric characters
import re
text = "Hello!!! NLP@2026 #AI "
clean_text = [Link](r"[^a-zA-Z0-9\s]", "", text)
print(clean_text)
------------------
Hello NLP 2026 AI

17
Removing non-alphanumeric characters
import spacy
nlp = [Link]("en_core_web_sm")
text = "Hello!!! NLP@2026 #AI "
doc = nlp(text)
tokens_clean = [[Link] for token in doc if token.is_alpha or
token.is_digit]
print(tokens_clean)

18
Removing non-alphanumeric characters
• Removing non-alphanumeric characters can be dangerous if your text
includes:
• emails: abderrahim@[Link]
• hashtags: #AI
• currencies: $100
• programming text: C++, C#

19
Removing HTML tags
import re
text = "<p>Hello <b>World</b>!</p>"
clean_text = [Link](r"<.*?>", "", text)
print(clean_text)
------------------
from bs4 import BeautifulSoup
text = "<p>Hello <b>World</b>!</p>"
soup = BeautifulSoup(text, "[Link]")
clean_text = soup.get_text()
print(clean_text)

20
Removing HTML tags
from bs4 import BeautifulSoup
html = ""“ <html>
<head><style>body{color:red;}</style></head>
<body>
<p>Hello <b>World</b>!</p>
<script>alert("Hi");</script>
</body>
</html> """
soup = BeautifulSoup(html, "[Link]")
for script in soup(["script", "style"]):
[Link]()
clean_text = soup.get_text(separator=" ").strip() 21
Removing HTML tags
import re
from bs4 import BeautifulSoup
def remove_html(text):
soup = BeautifulSoup(text, "[Link]")
return soup.get_text()
def clean_text(text):
text = remove_html(text)
text = [Link](r"\s+", " ", text)
return [Link]()
example = "<div>Hello <b>NLP</b> learners!</div>"
print(clean_text(example))
22
Normalizing Whitespace
What is Normalizing Whitespace?
It means:
• Removing extra spaces
• Replacing multiple spaces with one space
• Removing tabs (\t)
• Removing newlines (\n)
• Trimming leading and trailing spaces
-------------------------
text = [Link](r"\s+", " ", text).strip()

23
Handling contractions
Awesome Handling contractions is a key NLP cleaning step.
Contractions are words like:
▪ don't → do not
▪ I'm → I am
▪ can't → cannot
▪ you're → you are
Why Handle Contractions?
Because models may treat:
"do not like" and "don't like" as different tokens
and that hurts consistency in classic NLP pipelines.
24
Handling contractions
import contractions
text = "I'm happy because you're here, but I can't stay."
expanded_text = [Link](text)
print(expanded_text)
---------------
I am happy because you are here, but I cannot stay.

25
Handling contractions
import contractions
import nltk
from [Link] import word_tokenize
[Link]("punkt")
text = "Don't worry, I'm fine!"
expanded = [Link](text)
tokens = word_tokenize([Link]())
print(tokens)
----------
['do', 'not', 'worry', ',', 'i', 'am', 'fine', '!']

26
Handling contractions
import spacy
import contractions

nlp = [Link]("en_core_web_sm")

text = "Don't worry, I'm fine!"

expanded = [Link](text)

doc = nlp(expanded)
print([[Link] for token in doc])

27
TO DO

28
Text tokenization
What is Text Tokenization?
Tokenization is the process of splitting text into smaller units called
tokens. Tokens can be:
• Words
• Subwords
• Characters
The children are playing.→ ["The", "children", "are", "playing", "."]
.‫"]→يجلس األطفال في المدرسة‬." ,"‫ "المدرسة‬,"‫ "في‬,"‫ "األطفال‬,"‫] "يجلس‬
‫ "المدرسة"] → وبالمدرسة‬,"‫ "ب‬,"‫] "و‬

29
Subword Token (Subword Tokenization)
Definition
• A subword is a part of a word that is automatically learned by models
such as BERT or GPT.
• Modern models use subword tokenization to better handle:
• rare or unseen words
• morphologically rich languages (such as Arabic)
• Examples
• "incroyablement" → ["incroy", "##able", "##ment"]
• "]"‫ "ة‬,"‫ "مدرس‬,"‫المدرسة" → ["ال‬

30
Sentence Token (Sentence Tokenization)
Definition
• A sentence is a complete unit of speech, usually ending with:
• a period .
• an exclamation mark !
• a question mark ?
• Examples
• "Bonjour ! J’adore l’IA." → ]"Bonjour !", "J’adore l’IA."[
• "]".‫ "أتعلمها كل يوم‬,".‫" → ["أحب البرمجة‬.‫ أتعلمها كل يوم‬.‫أحب البرمجة‬

31
NLP Tools for Tokenization
• spaCy: robust segmentation (sentences + words), full French support
• Stanza: segmentation + MWT (multi-word tokens), excellent Arabic
support
• Flair: SegTok segmentation, simple and fast
• Transformers: subword tokenization (BPE / WordPiece)
• OpenAI (tiktoken): token counting for GPT models
• LangChain: token-based splitting for RAG (chunking)

32
Examples
import spacy
# English
nlp_en = [Link]("en_core_web_sm")
text_en = "Hello! The children are playing in the garden."
doc_en = nlp_en(text_en)
print("Tokens (EN):", [[Link] for t in doc_en])
print("Sentences (EN):", [[Link] for s in doc_en.sents])
# Arabic (no official POS model in spaCy, basic tokenization only)
nlp_ar = [Link]("ar")
text_ar = "".‫وبالمدرسة الكبيرة يجلس األطفال‬
doc_ar = nlp_ar(text_ar)
print("Tokens (AR):", [[Link] for t in doc_ar])
33
Examples
import stanza
# Download models (run once)
[Link]('en')
[Link]('ar')
# Load English and Arabic pipelines
nlp_en = [Link]('en', processors='tokenize', use_gpu=False)
nlp_ar = [Link]('ar', processors='tokenize,mwt', use_gpu=False)
text_en = "Hello! I love NLP."
text_ar = "".‫وبالمدرسة الكبيرة يجلس األطفال‬
#English
doc_en = nlp_en(text_en)
print("Stanza EN tokens:", [[Link] for s in doc_en.sentences for w in [Link]])
# Arabic
doc_ar = nlp_ar(text_ar)
print("Stanza AR tokens:", [[Link] for s in doc_ar.sentences for w in [Link]])
34
Examples
• from transformers import AutoTokenizer

• # English (BERT)
• tok_en = AutoTokenizer.from_pretrained("bert-base-uncased")
• text_en = "Hello! I love NLP."
• print("BERT English subwords:", tok_en.tokenize(text_en))

• # Arabic (AraBERT)
• tok_ar = AutoTokenizer.from_pretrained("aubmindlab/bert-base-
arabertv02")
• text_ar = "".‫وبالمدرسة الكبيرة يجلس األطفال‬
• print("AraBERT Arabic subwords:", tok_ar.tokenize(text_ar))
35
Examples
• import tiktoken

• text_en = "Hello! I love NLP."

• text_ar = "".‫وبالمدرسة الكبيرة يجلس األطفال‬

• #GPT tokenizer (used by GPT-4 / GPT-3.5 family)

• enc = tiktoken.get_encoding("cl100k_base")

• print("EN tokens:", [Link](text_en))

• print("AR tokens:", [Link](text_ar))
• print("Number of Arabic tokens:", len([Link](text_ar)))

36
Examples
• from langchain_text_splitters import TokenTextSplitter,
HuggingFaceTokenTextSplitter
• # Arabic text
• text_ar = ""!‫ إنهم يدرسون بسرع ٍة اليوم‬.‫وبالمدرسة الكبيرة يجلس األطفال‬
• #English text
• text_en = "The children are sitting in the big school. They are studying quickly
today!"
• # Using tiktoken (OpenAI / GPT tokenizer)
• splitter = TokenTextSplitter(
• encoding_name="cl100k_base",
• chunk_size=20,
• chunk_overlap=5 37
Examples
• print("OpenAI Chunks (Arabic):", splitter.split_text(text_ar))
• print("OpenAI Chunks (English):", splitter.split_text(text_en))
• # Using Hugging Face tokenizer (AraBERT for Arabic)
• hf_splitter = HuggingFaceTokenTextSplitter(
• model_name="aubmindlab/bert-base-arabertv02",
• chunk_size=15,
• chunk_overlap=3
• print("AraBERT Chunks (Arabic):", hf_splitter.split_text(text_ar))

38
Text normalization
▪ Text normalization is the process of transforming raw textual data
into a standardized, consistent, and canonical form in order to reduce
variability and improve computational processing.
▪ It aims to minimize superficial differences in text that do not change
meaning, so that different surface forms of the same concept are
treated uniformly by NLP systems.

39
Text normalization
Text normalization ensures that a model sees fewer variations of the
same semantic content, which:
▪ Reduces vocabulary size
▪ Decreases noise
▪ Improves model generalization
▪ Enhances training efficiency

40
What Text Normalization Typically Includes
Text normalization may involve:
▪ Lowercasing (e.g., Apple → apple)
▪ Removing punctuation
▪ Removing or standardizing special characters
▪ Expanding contractions (don’t → do not)
▪ Normalizing whitespace
▪ Standardizing numbers (twenty → 20)
▪ Lemmatization or stemming
▪ Correcting spelling variations
▪ Normalizing accented characters (café → cafe)
▪ Language-specific normalization (e.g., Arabic diacritics removal)
41
Examples
import spacy
import re
nlp_en = [Link]("en_core_web_sm")
text_en = "Hello!!! I'm LOVING NLP in 2026 :)"
text_en = [Link](r"\s+", " ", text_en).strip() # whitespace normalize
doc_en = nlp_en(text_en)
tokens_en = [
t.lemma_.lower()
for t in doc_en
if not t.is_punct and not t.is_space
]
print("spaCy EN normalized tokens:", tokens_en)
print("spaCy EN normalized text:", " ".join(tokens_en)) 42
Examples
import spacy
import re

nlp_ar = [Link]("ar")

ْ َ ‫س األ‬
text_ar = ""!!!‫طفَا ُل‬ َ ‫َو ِب ْال َم ْد َر‬
َ ‫س ِة ال َك ِب‬
ُ ‫ير ِة يَ ْج ِل‬
text_ar = [Link](r"\s+", " ", text_ar).strip()

doc_ar = nlp_ar(text_ar)

tokens_ar = [[Link] for t in doc_ar if not t.is_space]

print("spaCy AR tokens (basic):", tokens_ar)

43
Examples
• import stanza

• [Link]("en")
• nlp_en = [Link]("en", processors="tokenize,pos,lemma", use_gpu=False)

• text_en = "Hello!!! I'm loving NLP in 2026."

• doc = nlp_en(text_en)

• en_norm = [[Link]() for s in [Link] for w in [Link]]

• print("Stanza EN lemmas:", en_norm)
• print("Stanza EN normalized text:", " ".join(en_norm))

44
Examples
• import stanza

• [Link]("ar")
• nlp_ar = [Link]("ar", processors="tokenize,mwt,pos,lemma", use_gpu=False)

• text_ar = "".‫وبالمدرسة الكبيرة يجلس األطفال‬

• doc = nlp_ar(text_ar)

• ar_tokens = [[Link] for s in [Link] for w in [Link]]

• ar_lemmas = [[Link] for s in [Link] for w in [Link]]

• print("Stanza AR tokens (with MWT):", ar_tokens) # e.g., ["]...,"‫"المدرسة‬,"‫"ب‬,"‫و‬

• print("Stanza AR lemmas:", ar_lemmas)

45
Examples
• from transformers import AutoTokenizer

• tok_en = AutoTokenizer.from_pretrained("bert-base-uncased")

• text_en = "Hello!!! I'm LOVING NLP."

• print("BERT EN tokens:", tok_en.tokenize(text_en))
• print("BERT EN ids:", tok_en(text_en)["input_ids"])

46
Examples
• from transformers import AutoTokenizer

• tok_ar = AutoTokenizer.from_pretrained("aubmindlab/bert-base-
arabertv02")

• text_ar = "".‫وبالمدرسة الكبيرة يجلس األطفال‬

• print("AraBERT tokens:", tok_ar.tokenize(text_ar))
• print("AraBERT ids:", tok_ar(text_ar)["input_ids"])

47
Examples
• from camel_tools.[Link] import (
• normalize_alef_maksura_ar,
• normalize_alef_ar,
• normalize_teh_marbuta_ar
• )
• from camel_tools.[Link] import dediac_ar

• text_ar = ""‫س األطفا ُل‬

ُ ‫إلى المدرس ِة الكبرى َي ْج ِل‬

• #Remove diacritics + normalize common letter variants

• t = dediac_ar(text_ar)
• t = normalize_alef_ar(t)
• t = normalize_alef_maksura_ar(t)
• t = normalize_teh_marbuta_ar(t)

• print("CAMeL normalized Arabic:", t)

48