STEMMING & LEMMATIZATION
Group members:
AP24122040001
AP24122040007
AP24122040016
AP24122040018
AP24122040019
Table of
Contents
1 Stemming
2 Lemmatization 6 Regex - Based stemmer
3 Types of stemming 7 Context-aware processing
4 Types of lemmatization
8 Handling inflected forms
5 Lovins stemmer
Stemming
Stemming is a technique used in Natural Language Processing (NLP) to reduce words to their root
form. It's like finding the core of a word, stripping away grammatical endings like "ing," "ed," or "s."
● Intelligence, intelligent, and intelligently, all these words are originated with a single root word
"intelligen." In English, the word "intelligen" do not have any meaning.
● "running," "ran," and "runs" all stem from the root word "run." Stemming helps simplify text data by
bringing together different word variations under a single root.
Limitations :
1. Overstemming (Loss of Precision) : Stemming may reduce words to an overly generalized form that
doesn't retain the original meaning.
Example: "university" and "universe" both stemmed to "univers".
2. Understemming (Failure to Normalize): The stemmer may fail to recognize that two words should have the
same root.
Example: "analysis" and "analyze" are not reduced to the same stem by many stemming algorithms.
3. Lack of Context Awareness: Stemming operates purely on the word's form, ignoring its context or part of
speech.
Example: "book" (noun) and "book" (verb) will be treated the same.
4. Not Producing Valid Words: Stemming often produces non-dictionary words, which can be confusing in
downstream tasks.
Example: "running" → "run" (valid word), but "fishing" → "fish" might become "fish" (not always valid in all
contexts).
5. Inability to Handle Irregular Forms:Stemming struggles with irregular word forms, such as plural forms or
irregular past tense [Link]: "went" and "go" are not stemmed to the same root by most stemmers.
Examples:
Original: running → Stemmed: run
Original: jumps → Stemmed: jump
Original: happily → Stemmed: happili
Original: computing → Stemmed: comput
Original: fishing → Stemmed: fish
Original: studies → Stemmed: studi
Original: argued → Stemmed: argu
Original: organization → Stemmed: organ
Original: categories → Stemmed: categori
Original: analysis → Stemmed: analysi
Lemmatization
Lemmatization is quite similar to the Stemming. It is used to group different inflected forms of the word,
called Lemma. The main difference between Stemming and lemmatization is that it produces the root
word, which has a meaning.
For example: In lemmatization, the words intelligence, intelligent, and intelligently has a root word
intelligent, which has a meaning.
Limitations of lemmatization:
1. Dependency on Language Resources : Lemmatization requires detailed linguistic resources, such
as lexicons and morphological rules, which may not be available or complete for all languages.
Example: For less-resourced languages, creating a reliable lemmatizer can be challenging.
2. Computational Cost : Lemmatization is more computationally expensive than stemming because it
considers part of speech, linguistic rules, and word context.
Example: Large-scale text processing tasks can become slower due to the overhead of lemmatization.
3. Part-of-Speech Ambiguity: Lemmatization depends heavily on the correct identification of a word's part of
speech (POS). Errors in POS tagging can lead to incorrect lemmatization.
Example: "playing" could be lemmatized to "play" (verb) or remain as "playing" (noun) depending on the POS
tag.
4. Domain-Specific Vocabulary: Lemmatization may not handle domain-specific terms, jargon, or
neologisms effectively because such words may not exist in its dictionary.
Example: "microservices" or "cloudification" might not be lemmatized appropriately in a technical text.
5. Inconsistencies Across Lemmatizers: Different lemmatization tools may use different underlying
dictionaries or rules, leading to inconsistent results.
Example: One lemmatizer may reduce "better" to "good", while another may leave it unchanged.
Examples:
Original: running → Lemmatized: run
Original: jumps → Lemmatized: jump
Original: better → Lemmatized: good
Original: studies → Lemmatized: study
Original: was → Lemmatized: be
Original: children → Lemmatized: child
Original: mice → Lemmatized: mouse
Original: geese → Lemmatized: goose
Original: better → Lemmatized: good
Original: cars → Lemmatized: car
Types of stemming Types of lemmatization
● Lovins stemmer ● Context-Aware processing
● Regex - based stemmer ● Handling inflected forms
Lovins stemmer
The Lovins Stemmer is one of the earliest stemming algorithms, developed by Julie Beth Lovins in
1968. It is used in natural language processing to reduce words to their base or root form by
removing suffixes. Unlike other stemmers like Porter or Snowball, the Lovins Stemmer uses a single-
pass approach with a large suffix list, making it relatively fast but less precise compared to newer
algorithms.
Key Features:
● Removes the longest suffix from a predefined list.
● Applies transformations to handle spelling adjustments.
● Used to reduce inflected or derived words to their base forms.
Rules of Lovins Stemmer
The rules can be classified into two main parts: suffix removal rules* and *heuristic adjustments..
Here are the main rules:
Single Step Suffix Removal: It removes specific suffixes if they match certain patterns. For
example: - -ing → -e
- -ed → -e
- -er → ``
- -ly → ``
- -es → ``
Multi-Step Suffix Removal: Some words may require multiple suffix removal steps based on
specific conditions, such as length and preceding characters.
Special Rules:Certain words have exceptions or special conditions for how they can be stemmed.
Replacement Rules: After suffix removal, certain letter combinations may be replaced to ensure
valid stem formation.
Examples
Here are some examples illustrating how the Lovins Stemmer transforms words:
→Word: "running"
Stem: "run"
Rule Applied: The suffix -ing is removed.
→Word: "happiness"
Stem: "hap"
Rule Applied: The suffix -ness is removed.
→Word: "fishing"
Stem: "fish"
Rule Applied: The suffix -ing is removed.
→ Word: "talked"
Stem: "talk"
Rule Applied: The suffix -ed is removed.
Regex - Based stemmer
A regex-based stemmer in NLP is a simple, rule-based approach to reducing words to their base
or stem forms by applying predefined patterns using regular expressions (regex).
Example Rules:
For English words, you might define rules like:
Remove -ing, -ed, or -ly at the end of words (e.g., "running" → "run", "happily" → "happy").
Replace -ies with -y (e.g., "stories" → "story").
Remove plural suffix -s (e.g., "cats" → "cat").
Predefined patterns in regular expressions (regex) refer to specific sequences or symbols
used to match certain types of text patterns.
1) Character Classes: Match specific types of charactyers in text
Pattern Description Example Match
\d Matches any digit (0-9). 123 in file 123
2) Anchors: Match positions in text rather than characters
Pattern Description Example Match
^ Matches the start of a string ^hello matches “hello
world”
3) Quantifier: Specify how many times does a pattern should appear
Pattern Description Example Match
* Matches 0 or more occurrences ab* matches a,ab,abb
4) Groups and alternatives: Group patterns or provide alternatives
Pattern Description Example Match
(pattern) captures a group for later use or reference (ab)+matches abab
5) Special characters: Escape or match the specific symbols
Character Description Example Match
● Matches any character except newline a.c matches abc,a1c
Context - Aware processing
Context-aware processing in lemmatization techniques involves understanding the meaning and grammatical
role of words in sentences to accurately reduce them to their base forms (lemmas).
Rules of Context-Aware Lemmatization:
Morphological Analysis: Identify the root form of words by analyzing their morphological structures, including
prefixes and suffixes.
Part-of-Speech Tagging: Determine the part of speech (noun, verb, adjective, etc.) of a word to correctly
extract its lemma.
Contextual Meaning: Utilize the surrounding words to infer the correct meaning and base form, as the same
word may have different lemmas based on context.
Word Sense Disambiguation: Differentiate between different meanings of the same word based on its usage
in sentences.
Grammatical Gender and Number: Incorporate rules for grammatical gender (masculine, feminine)
and number (singular, plural) to select the correct lemma.
Examples:-
1. Verb Conjugation:
Context: I am being born Lemmatization: to be born
[Link] Variations:
Context: schools Lemmatization: school
3. Adjective Agreement:
Context: The doubtful books Lemmatization: doubt
4. Gender and Number Agreement:
Context: employees Lemmatization: employee
Handling Inflected forms
Handling inflected forms and rules in Natural Language Processing (NLP) refers to the ability of an NLP system
to correctly interpret and process different forms of a word that share the same root or lemma. This is crucial
because words in many languages (like English, Spanish, German etc.) change their form depending on their
grammatical role (e.g., tense, number, gender, case).
This is essential for consistent analysis and processing across various grammatical contexts.
Examples:
1. Verbs
Inflected Forms:
Normal Word: play, played, playing Lemmatized Word: play
Normal Word: is walking Lemmatized Word: walk
2. Nouns
Inflected Forms:
Normal Word: cat, cats Lemmatized Word: cat
Normal Word: child, children Lemmatized Word: child
3. Adjectives and Adverbs
Inflected Forms:
Normal Word: tall, taller, tallest Lemmatized Word: tall
Normal Word: quickly, more quickly, most quickly Lemmatized Word: quickly
4. Languages with Complex Inflection Rules
Spanish Verbs
● Normal Word: hablar (to speak), hablo (I speak), habló (he/she spoke), hablando (speaking)
● Lemmatized Word: hablar
German Nouns: Normal Word: Haus (house), Häuser (houses)
● Lemmatized Word: Haus
Thank you
Questions?