RegEx, Lematization,

Tokenization
Dr. Humaira Waqas
Today’s Topics
• Stemming
• Tokeninzation
• RegExpresiion
• Logistic Regression
 Stemming
Stemming is the process of reducing words to their root form, or stem, by removing
suffixes and prefixes.
• Stemming does not consider the context of the word; it simply truncates the word to
its base form.
Why stemming?
• It is purely rule-based and doesn't require a dictionary lookup.
• Search engines like Elasticsearch and Google still use stemming for indexing because
absolute linguistic correctness is not necessary.
• If a user searches "organizations", it’s fine to return documents with "organiz", even if
"organization" is the proper lemma
• It reduces vocabulary size more aggressively than lemmatization, leading to smaller
models
Stemming
• A stemmer works by removing or modifying affixes (prefixes, suffixes,
infixes) to reduce a word to its base stem.
• The process follows rule-based heuristics rather than linguistic analysis,
which is why the output may not always be a valid word.
• It removes suffixes (e.g., -ing, -ly, -es, -ment).
• Does not check meaning, so the stem may not be a real word.
• Faster than lemmatization but less accurate.
Ignores Linguistics analysis means:
• It only follows pattern-based transformations.
• Does not care if the result is a real word or makes sense in a sentence.
• No understanding of word function (noun vs. verb vs. adjective).
 Stemming
import nltk "cats"]
from [Link] import PorterStemmer
# Stemming the words
# Download the necessary NLTK stemmed_words = [[Link](word)
resources (if not already downloaded) for word in words]
[Link]('punkt')
# Print the original and stemmed words
# Create an instance of the Porter for original, stemmed in zip(words,
Stemmer stemmed_words):
stemmer = PorterStemmer() print(f"Original: {original} ->
Stemmed: {stemmed}")
# Example words to be stemmed
words = ["running", "runner", "ran",
"easily", "fairly", "happily", "happiness",
 Lemmatization
Lemmatization is the process of reducing a word to its base or root form, known as its
lemma.

Words and Their Lemmas:

"running" → "run"
"children" → "child"

• Lemmatization requires knowing a word’s Part-of-Speech (POS) to determine the

correct lemma
• Lemmatization is slower because it needs to check each word in a dictionary to
determine its correct root form.
 Lemmatization
• Lemmatization requires the Part-of-Speech (POS) tag before converting a word to its
lemma.
• If no POS is provided, it assumes the word is a noun by default.
• If a POS tag is given, it finds the correct lemma based on the word’s role in the
sentence.
• It returns the correct lemma based on meaning and grammar.
 Lemmatization
How lemmatize?
• Using NLTK (Natural Language Toolkit)
• Using SpaCy
• Using TextBlob
Stemming vs Lemmatization
• If speed is more important than accuracy, use stemming.
• If accuracy matters more, use lemmatization.
• For production-grade NLP models → Lemmatization is better.
• For quick preprocessing in text mining/search → Stemming is sufficient.
Lemmatization
 Lemmatization
import nltk ("flying", "v"), # Verb
from [Link] importrunning --> run
WordNetLemmatizer ("was", "v"), # Verb
ran --> run
from nltk import pos_tag ("men", "n") # Noun
better --> good ]
children --> child
# Initialize the lemmatizer
geese --> goose
lemmatizer = WordNetLemmatizer() # Lemmatize each word with its POS
flying --> fly lemmas_with_pos = {word:
# Example words with was -->intended
their be POS [Link](word, pos=pos) for word, pos
in words_with_pos}
words_with_pos = [ men --> man
("running", "v"), # Verb # Print the results
("ran", "v"), # Verb for word, lemma in lemmas_with_pos.items():
("better", "a"), # Adjective print(f"{word} --> {lemma}")
("children", "n"), # Noun
("geese", "n"), # Noun
 Regular Expressions
• Regular expressions are patterns used to match character combinations in strings.
• Assertions: Assertions are used to define the conditions that the characters before or
after a specific point in the string must meet for a successful match.
• Boundaries which indicate the beginnings and endings of lines and words, and
other patterns indicating in some way that a match is possible.
• ^ matches the beginning of input /^A/ “an A”, “An A”
• $ matches the end of input /t$/ “eater”, “eat”
• \b matches a word boundary. This is the position where a word character is not
followed or preceded by another word character, such as between a letter and a
space.
• /\bm/ matches the “m” in “moon”
• /oo\b/ does not match the “oo” in “moon”, because “oo” is followed by “n”
which is a word character
• /oon\b/ matches the “oon” in “moon” because “oon” is the end of the string, not
followed by a word character.
 Regular Expressions
Quantifiers: Quantifiers indicate numbers of characters or expressions to match
* matches 0 or more times
/bo*/ matches “boooo” in “A ghost booooed” and “b” in “A bird warbled”
+ matches 1 or more times equivalent to {1,}
/a+/ matches the “a” in “candy” and all the “a” in “”caaaaaaandy”
? Matches the preceding item 0 or 1 times.
/e?le?/ matches “el” in “angel” and “le” in “angle”
X{n,} and X{n,m} where n is a positive integer, matches atleast “n” occurrces of the preceding item.
/a{2, }/ “candy”, “caandy”, “caaaaaandy”
/a{1, 4}/
/<.*>/ “<foo><bar></bar></foo>”
By default, quantifiers like * and + are greedy, they try to match as much of the string as possible.
The ? character after the quantifier makes the quantifier non-greedy, meaning that it will stop as
soon as it find a match. /<.*?>/ “<foo>”
Regular Expressions
Groups and ranges
• They indicate groups and ranges of expression characters
• () used to define groups
• [] used to specify ranges
• (x|y) matches either x or y
• [xyz], [a-c], [a-d], [abcd-], [-abcd]
• [^xyz] matches anything except what is enclosed in the brackets
• [^a-c]
• [0-9a-zA-Z]
• [a]
Regular Expressions
Other Assertions
• X(?=y) lookahead assertion
• Matches “x” only if “x” is followed by “y”
• /Jack(?=Spart)/
• /Jack(?=Spart|Frost)/
• X(?!y) negative lookahead assertion
• Matches “x” only if “x” is not followed by “y”
• /\d+(?!\.)/ matches a number only if it is not followed by a decimal
point
• (?<=y)x lookbehind assertion
• Matches “x” only if “x” is preceded by “y”
• /(?<=Jack)Spart/ matches “Spart” only if it is preceded by “Jack”
• /(?<=Jack|Tom)Spart/ matches only if it is preceded by “Jack” or “Tom”
• (?<!y)x negative lookbehind assertion
• Matches “x” only if “x” is not preceded by “y”
• /(?<!-)\d+/ matches a number only if it is not preceded by a minus sign
 Regular Expressions
Character Classes: These classes distinguish kinds of characters.
. Matches any single character except the terminators e.g. \n, \r etc.
/.y/ matches “my” and “ay” in “yes make my day”
\d matches any digit. Equivalent to [0-9]
\D matches any character that is not a digit
\w matches any alphanumeric character i.e. [A-Za-z0-9]
\W matches any character that is not a word character i.e. [^A-Za-z0-9]
\s matches a single white space character, including space, tab, form feed, line feed and
other Unicode spaces i.e. [ \f\n\r\t\v\u00a0\u1680\u2000-
\u200a\u2028\u2029\u202f\u205f\u3000\ufeff]
\S matches a single character other than white space
\t matches a horizontal tab
\r matches a carriage return
 Regular Expression Patterns

Brackets ([]) are used to find a range of characters:

Example:

[^a-zA-Z]
• It matches any string not containing any of the characters ranging from a
through z and A through Z.
Regular Expression Quantifiers
• Quantifiers define quantities. It defines
frequency or position of bracketed
character sequences and single
characters can be denoted by a special
character.
• Each special character has a specific
connotation. The +, *, ?, and $ flags all
follow a character sequence.

Example:
• p(hp)*
• It matches any string containing
a p followed by zero or more instances of
the sequence hp.
 Regular Expression Examples
1. Telephone must be a valid 11 digits number.
• /^[0-9]{11}$/
2. Username must contain 5 - 12 characters
• /^[0-9a-z]{5,12}$/i
3. Password must be alphanumeric (@, _ and - are also allowed) and be 8 - 20
characters
• /^[0-9a-z@-_]{8,20}$/
4. Email must be a valid address, e.g. me@[Link] {.uk}
• /^([a-z0-9\.-]+)@([a-z0-9-]+)\.([a-z]{2,8})(\.[a-z]{2,8})?$/
• me@[Link] {.uk}
 Regular Expression
• Capture group
words that contain an apostrophe ('), commonly seen in contractions or possessives.

Tokenizer using Regular Expression

(REGEX)
1. [\w'|$.]+ Will match charaters a-z A-Z 0-9 _ ‘ | $ .
2. [0-4]+(\.[0-9]+)? Decimal number where the whole part is 0-4 followed by . and the
decimal part can be 0-9 and it is optional
3. \. Any character
4. \$[0-9]+\.[0-9]+ Price beginning with $
5. [A-Z][A-Z]+ matches uppercase words that are at least two letters long
6. [\0-9 ^.]+ string that contains digits \ ^ .
7. [0-9]+-[0-9]+ numeric ranges or hyphen-separated numbers
8. \w+\'\w+ words that contain an apostrophe (‘), like contractions
9. \w+\'\w+|[A-Z][a-z]+|[a-z][a-z]+|[A-Z][A-Z]+
Tokenizer using RegEx
Write a Python function that takes a text string as input and
tokenizes it into words using regular expressions (Regex). Your
function should:
• Remove punctuation.
• Split the text into individual words.
• Convert all words to lowercase.
 Tokenizer using RegEx
import re return tokens
def regex_tokenizer(text):
# Define a regex pattern to match words # Example Usage
pattern = r"\b\w+\b" text = "Hello, world! This is a simple
# Matches words (letters, numbers, and tokenizer using Regex."
underscores) tokens = regex_tokenizer(text)
# Find all words using regex print(tokens)
tokens = [Link](pattern, text)
# Convert to lowercase #output
tokens = [[Link]() for word in ['hello', 'world', 'this', 'is', 'a', 'simple',
tokens] 'tokenizer', 'using', 'regex']
Python Codes
• [Link]
syMwVIK-IqcwbhS?usp=sharing