STEMMING AND LEMMATIZATION:

In every language, words take on different surface forms depending on

the roles they play in sentences. However, since all of these forms are
derived from a common root (stem), they share a core meaning that can
greatly assist us in later stages of analysis and interpretation.

For this reason, in many NLP-based approaches, it is necessary to first

identify the root of words. Two common techniques are used for this
purpose: Stemming and Lemmatization. Both methods aim to extract the
base form of a word, although they do so in different ways.

Stemming:

There are various algorithms designed to perform stemming, and the

Porter algorithm is one of the most well-known algorithms for the
English language. This algorithm applies a set of systematic rules (for
example, removing the letter “s” at the end of plural words) to extract the
stem of words with reasonable accuracy. Stemming helps us standardize
words by reducing them to their root form or base form. The final output
of a stemming process— which typically follows predefined rules and a
fixed set of patterns is not necessarily a meaningful word found in a
dictionary. However, prefixes and suffixes are removed, and the simplest
form of the word, or its computational stem, is produced as the output.
Lemmatization

In the process of lemmatization, the goal is to obtain a meaningful word

one that actually exists in a dictionary representing the base or canonical
form (lemma) of a word. In other words, unlike stemming,
lemmatization aims to return a linguistically valid root form.

As illustrated in the example presented in Week 2 of the Stanford NLP

course, the sentence:

“the boy’s cars are different colors”

after lemmatization becomes:

“the boy car be different color”

In this transformation, plural forms, possessives, and verb inflections are

reduced to their standard dictionary forms while preserving meaningful
base words.
In the following section, we will examine the Porter algorithm and explore
how it identifies and extracts the stems of words.

Porter Stemmer Algorithm

The Porter Stemmer is one of the most well-known stemming methods

in Natural Language Processing (NLP). It was introduced in 1980 by
Martin Porter.

🔎 What Is Its Goal?

Its main objective is to reduce English words to their base stem by
removing suffixes.

Examples:

 playing → play
 studies → studi
 connected → connect
 relational → relat

Note: The output is not always a fully correct dictionary word (for
example, studi). Instead, it produces a computational stem, which is
sufficient for many NLP tasks such as information retrieval and text
processing.

How Does the Porter Algorithm Work?

The Porter Stemmer is a rule-based algorithm. This means it applies a

predefined set of linguistic rules to examine and modify word suffixes.
If certain conditions are satisfied, the suffix is either removed or
replaced.

The original algorithm operates in five main steps (Step 1 to Step 5).
Each step contains several rules.

The general structure of each rule is as follows:

1. The algorithm checks whether the word matches a specific suffix

pattern.
2. It verifies whether the stem before the suffix is long enough, using
a measure called m (measure).
3. If the required conditions are met, the suffix is removed or
replaced.
4. If not, the algorithm proceeds to the next rule.

The Important Concept of Measure (m)

To avoid excessive reduction of words, Porter introduced a concept
called measure (m).

The value m represents the number of consonant–vowel (VC) sequences

in a word.

For example:

 tree → m = 0
 trouble → m = 1
 controlling → m = 2

Many rules are applied only if:

 m>0
 or m > 1

This prevents the stem from becoming too short or meaningless.

Step 1

Step 1 is the most important stage and is divided into three sub-steps.

Step 1a – Handling Plurals

This sub-step deals with plural forms.

Examples of rules:

 sses → ss
dresses → dress
 ies → i
studies → studi
 ss → ss (no change)
class → class
 s → (remove)
cats → cat
Step 1b – Removing -ed and -ing

This step removes verb endings such as:

 eed
 ed
 ing

However, the removal only happens if a valid stem exists before the
suffix.

Examples:

 agreed → agree
 played → play
 playing → play
 controlling → control

But:

 sing → remains unchanged (because the stem would be too short)

After removing the suffix, small adjustments may be made:

 hopping → hop (removal of double consonant)

 making → make (restoring the final “e” in certain cases)

Step 1c – Replacing “y” with “i”

If a word ends in y and there is a vowel before it:

 happy → happi

But:

 sky → remains unchanged

Step 2

In this step, longer suffixes are replaced with simpler forms.

Examples:

 relational → relate
 conditional → condition
 digitizer → digitize
 nationalization → nationalize

This step is usually applied only if m > 0.

Step 3

This step removes or simplifies more abstract suffixes.

Examples:

 icate → ic
communicate → communic
 ative → (remove)
formative → form
 alize → al
formalize → formal

Step 4

In Step 4, more general suffixes are removed, but only if m > 1.

Common suffixes removed include:

 al
 ance
 ence
 er
 ic
  able
 ment
 ion

Examples:

 rational → ration
 adjustment → adjust

Step 5

This is the final refinement stage.

Removing final “e”

 make → mak (in certain conditions)

 rate → may remain unchanged if removal would over-shorten the
word

Removing double “l”

 controll → control

Summary

The Porter algorithm:

 Is entirely rule-based
 Does not rely on a dictionary
 Is computationally efficient
 May not always produce a meaningful dictionary word
 Is primarily designed to reduce word variation for statistical tasks
such as search engines, information retrieval, and text
classification
Exercise: Porter Stemmer Algorithm

Using the Python programming language and the NLTK library, write a
program that performs the following tasks:

1. Define a list containing at least 10 English words (including verbs,

plural nouns, and words with suffixes).
Example:
playing, studies, connected, relational, happiness, running,
nationalization, cats, agreed, different
2. Use the Porter Stemmer algorithm to extract the stem of each
word.
3. Display the output in the following format:

Original Word → Stemmed Word

At the end of your program, briefly answer the following questions in a

few lines:

 Are all the resulting outputs meaningful dictionary words?

 Why are some words transformed into unusual forms (such as studi
or relat)?
 What is the difference between a computational stem and a true
linguistic root?

Bonus Question (Optional – Extra Credit)

Receive a complete sentence from the user, tokenize the sentence, and
then apply the stemming process to all the words using the Porter
Stemmer algorithm.

Display the output both before and after stemming.

Exercise Solution: Porter Stemmer Algorithm

In this exercise, we implement the Porter Stemming algorithm using Python and the NLTK
library.
Step 1: Install the NLTK Library (If Not Installed): pip install nltk
Step 2: Import Required Libraries:
import nltk
from [Link] import PorterStemmer
from [Link] import word_tokenize
# Required only the first time:
# [Link]('punkt')
Step 3: Create a Porter Stemmer Object:

ps = PorterStemmer()
Step 4: Define a List of Words:

words = [
"playing",
"studies",
"connected",
"relational",
"happiness",
"running",
"nationalization",
"cats",
"agreed",
"different"
]
Step 5: Apply Stemming to the Word List:

print("=== Stemming Word List ===")

for word in words:
stem = [Link](word)
print("Original Word:", word, " --> Stemmed Word:", stem)
Expected Output:

Original Word: playing --> Stemmed Word: play

Original Word: studies --> Stemmed Word: studi
Original Word: connected --> Stemmed Word: connect
Original Word: relational --> Stemmed Word: relat
Original Word: happiness --> Stemmed Word: happi
Original Word: running --> Stemmed Word: run
Original Word: nationalization --> Stemmed Word: nation
Original Word: cats --> Stemmed Word: cat
Original Word: agreed --> Stemmed Word: agre
Original Word: different --> Stemmed Word: differ
Step 6: Bonus Question – Sentence Input, Tokenization, and Stemming:
Now we receive a full sentence from the user, tokenize it, and apply stemming to each word.

print("\n=== Sentence Stemming ===")

sentence = input("Enter a sentence: ")
# Tokenize the sentence
tokens = word_tokenize(sentence)
print("Original Sentence Tokens:", tokens)
# Apply stemming
stemmed_tokens = []
for word in tokens:
stemmed_tokens.append([Link](word))
print("Stemmed Tokens:", stemmed_tokens)

User Input:
The boys were playing different games in the parks
Output:

Original Sentence Tokens:

['The', 'boys', 'were', 'playing', 'different', 'games', 'in', 'the', 'parks']
Stemmed Tokens:
['the', 'boy', 'were', 'play', 'differ', 'game', 'in', 'the', 'park']

Analytical Explanation
1. Not all outputs are valid dictionary words (e.g., studi, relat).
2. The Porter algorithm is rule-based and does not rely on a dictionary.
3. The goal is to produce a computational stem, not a linguistically perfect root.
4. This method is especially useful for search engines, information retrieval, and statistical
text analysis, where reducing word variation improves efficiency.

Lemmatization

Scientific Definition
Lemmatization is a process in Natural Language Processing (NLP) in which words are
converted into their base or dictionary form, known as the lemma.

Unlike stemming, which simply removes suffixes based on predefined rules, lemmatization:

 Takes the word’s grammatical role (Part of Speech) into account.

 Uses a lexical database (dictionary).
 Produces a meaningful word that exists in the dictionary.

In other words, lemmatization focuses on returning the correct linguistic base form of a word
rather than just reducing it mechanically.

What Is the Goal of Lemmatization?

The goal of lemmatization is to reduce different forms of a word to its true, meaningful base
form (lemma).
As shown in the table, lemmatization returns the correct dictionary form of each word based on its
grammatical role.

How Does Lemmatization Work?

The lemmatization process usually consists of three main steps:

1️.Part-of-Speech Tagging (POS Tagging)

First, the system determines the grammatical role of the word. It identifies whether the word is:

 A noun
 A verb
 An adjective
 An adverb

For example:

The word “running” can function either as a noun or as a verb.

 If it is used as a noun → running (no change)

 If it is used as a verb → run

Therefore, identifying the correct part of speech is extremely important in the lemmatization
process.

2. Dictionary Lookup
After determining the part of speech (POS), the system searches a lexical database to find the
correct base form of the word.

In Python, Word Net is commonly used for this purpose.

3. Returning the Base Form (Lemma)
Finally, the system returns the standard dictionary form of the word.

The output is a meaningful word that exists in the lexicon.

Conceptual Difference between Lemmatization and Stemming

Although both Stemming and Lemmatization aim to reduce words to a base form, they differ
significantly in methodology and output quality.

Stemming is a rule-based process that mechanically removes prefixes or suffixes without

considering the word’s grammatical role or meaning. As a result, the output may not always be a
valid dictionary word.

Lemmatization, on the other hand, is linguistically informed. It considers the word’s part of
speech (POS) and uses a lexical database to return the correct and meaningful base form found in
a dictionary.

In summary, stemming is computationally efficient and suitable for tasks like search engines,
while lemmatization is more accurate and appropriate for deeper linguistic analysis.

Exercise: Lemmatization in Python

Using the Python programming language and the NLTK library:

1. Receive an English sentence from the user.

2. Tokenize the sentence.
 3. Determine the Part of Speech (POS) of each word.
4. Apply Lemmatization using WordNetLemmatizer.
5. Display the output in three stages:
o Tokens
o POS tags
o Lemmatization results

Finally, explain why some words changed and why others did not.

Step 1: Install the Library (If Needed)

pip install nltk

Step 2: Import Required Libraries

import nltk

from [Link] import word_tokenize

from [Link] import WordNetLemmatizer

from [Link] import wordnet

from nltk import pos_tag

# Run only once:

# [Link]('punkt')

# [Link]('averaged_perceptron_tagger')

# [Link]('wordnet')

Step 3: Convert POS Tags to WordNet Format

Since NLTK POS tags differ from WordNet tags, we convert them:

def get_wordnet_pos(treebank_tag):

if treebank_tag.startswith('J'):

return [Link]

elif treebank_tag.startswith('V'):
 return [Link]

elif treebank_tag.startswith('N'):

return [Link]

elif treebank_tag.startswith('R'):

return [Link]

else:

return [Link]

Step 4: Main Program

lemmatizer = WordNetLemmatizer()

sentence = input("Enter a sentence: ")

# Tokenize

tokens = word_tokenize(sentence)

# POS Tagging

pos_tags = pos_tag(tokens)

print("Tokens:")

print(tokens)

print("\nPOS Tags:")

print(pos_tags)

print("\nLemmatization Result:")

lemmas = []

for word, tag in pos_tags:

wordnet_pos = get_wordnet_pos(tag)

lemma = [Link](word, wordnet_pos)

 [Link](lemma)

print(word, " --> ", lemma)

Input:
The boys were running faster than the girls.
Output:
Tokens:
['The', 'boys', 'were', 'running', 'faster', 'than', 'the', 'girls']
POS Tags:
[('The', 'DT'), ('boys', 'NNS'), ('were', 'VBD'),
('running', 'VBG'), ('faster', 'RBR'),
('than', 'IN'), ('the', 'DT'), ('girls', 'NNS')]
Lemmatization Result:
The --> The
boys --> boy
were --> be
running --> run
faster --> fast
than --> than
the --> the
girls --> girl

Why did these changes occur?

Why did some words remain unchanged?

 Articles such as “the”

 Prepositions such as “than”
 Words that are already in their base form

Important Note:
If we do not specify the POS:

[Link]("running")
The output will be:

Running
Why?
Because by default, it is treated as a noun.
However:

[Link]("running", [Link])
Output:

Run
Therefore, POS tagging is the core of accurate lemmatization.