Natural Language Processing(NLP)

Lecture 2 : Basic Text Processing

 Basic Text Processing in NLP
Basic Text Processing in Natural Language Processing (NLP) refers to a set of fundamental techniques used to clean,
standardize, and prepare textual data for analysis, modeling, and machine learning applications. These techniques help
convert raw text into a structured format that computers can efficiently process for various NLP tasks such as sentiment
analysis, machine translation, text classification, and speech recognition.
Since natural language data often contains punctuation, stopwords, inconsistent cases, special characters, and
variations in word forms, basic text processing ensures that the text is clean and suitable for computational models.
• Tokenization
• Stopword Removal
• Stemming
• Lemmatization
• Punctuation Removal
• Lowercasing
• Handling Numbers
• Removing Extra Spaces
 TOKENIZATION
Tokenization is the process of breaking text into smaller meaningful units called tokens (words, subwords, or sentences). It helps in
analyzing text more effectively.
Example:
• Sentence: "Natural Language Processing is amazing!"
• Word Tokens: ["Natural", "Language", "Processing", "is", "amazing", "!"]
• Sentence Tokens: ["Natural Language Processing is amazing!"]

…
…

…
TOKENIZATION

…
.

…
…
TOKENIZATION
TOKENIZATION
TOKENIZATION
TOKENIZATION
TOKENIZATION
TOKENIZATION
TOKENIZATION
TOKENIZATION
 Stemming
Stemming
Stemming is the process of reducing a word to its word stem that affixes to suffixes and prefixes or to the roots of words
known as a lemma. Stemming is important in natural language understanding (NLU) and natural language processing
(NLP).
Stemming
Stemming
Stemming
Stemming
Stemming
 Lemmatization
Wordnet Lemmatizer
Lemmatization technique is like stemming. The output we will get after lemmatization is
called ‘lemma’, which is a root word rather than root stem, the output of stemming. After
lemmatization, we will be getting a valid word that means the same thing.
Lemmatization
Lemmatization
Stopwords
Stopwords
Stopwords
Stopwords
Stopwords
Stopwords
Stopwords
 Regular Expression

A Regular Expression (Regex) is a powerful tool used

in text processing to define search patterns. These
patterns can be used for:
•Matching: Identifying specific text patterns in
strings.
•Searching: Finding and extracting specific
sequences of characters.
•Replacing: Modifying parts of text based on defined
patterns.
Regular expressions are widely used in tasks like:
•Validating inputs (e.g., email addresses).
•Extracting data (e.g., dates or phone numbers).
•Tokenizing text for Natural Language Processing.
 Regular Expressions Play a Surprisingly Large Role

Widely used in both academics and industry:

[Link] for text cleaning and preprocessing tasks:
[Link] unwanted characters (e.g., punctuation).
[Link] specific patterns like dates or emails.
2.A cornerstone for information extraction:
[Link] names, addresses, or phone numbers in documents.
[Link] in creating pipelines for search engines or chatbots.
 Regular Expressions: Matching Patterns (Disjunction)
Letters inside square brackets []
Square brackets define a set of characters, matching any single character
within the brackets .

Ranges using the dash [A-Z]

Dashes specify a range of characters inside the square
brackets..
 Regular Expressions: Negation in Disjunction

Carat (^) as the First Character in [] Negates the List

•Note: Carat (^) negates the list only when it is the first character inside
square brackets.
•Special characters (e.g., ., *, +, ?) lose their special meaning inside [].
Regular Expressions: Tasks
 Regular Expressions: Task 1

Find words without vowels (a, e, i, o, u).

Pattern: \b[^aeiouAEIOU\s]+\b
Hint: Match entire words that exclude vowels.
 Regular Expressions: Task 2

Identify all characters that are not lowercase letters.

Pattern: [^a-z]
Hint: Use negation to exclude lowercase characters.
 Regular Expressions: Task 3

Find all non-digit characters in the sample text.

Pattern: [^0-9]
Hint: Use negation to exclude digits.
 Regular Expressions: Task 4

Match sentences that do not end with a period (.).

Pattern: [^.]\s*$
Hint: Look for sentence endings that do not include a period.
 Regular Expressions: Task 5

Creative Challenge:
Find all special characters (e.g., !, @, #, $) but exclude letters and digits.
Pattern: [^a-zA-Z0-9\s]
Regular Expressions: Convenient Aliases
Regular Expressions: Convenient Aliases
Regular Expressions: Convenient Aliases
 Regular Expressions: More Disjunction

Soccer is also called football in many countries!

The pipe symbol | for disjunction.

Pattern: soccer|football
Matches: Either "soccer" or "football," commonly used interchangeably in
different countries.
Pattern: [sS]occer|[fF]ootball
Matches: Adds case insensitivity to the matching (e.g., Soccer, FOOTBALL).
 Wildcards, Optionality, Repetition: . ? * +

Explanation:
•.: Matches any single character (e.g., col.r matches "color" or "colar").
•?: Matches zero or one occurrence of the preceding character (e.g., favo[u]?r matches "favor"
or "favour").
•*: Matches zero or more occurrences of the preceding character (e.g., ha*ppy matches "hppy"
or "happy").
•+: Matches one or more occurrences of the preceding character (e.g., go+od matches "good"
or "goood").
 Regular Expressions: Anchors ^ and $

•Pattern: ^[0-9]
Matches: "123 Main Street", "42 is the answer."
•Pattern: ^[^a-zA-Z]
Matches: "123 ABC", "!Hello".
•Pattern: \.$
Matches: "End of the line." (lines ending with a period).
•Pattern: [!?]$
Matches: "What is this?", "Surprise!".
 Python Regular Expressions
Regex and Python both use backslash \ for special characters. You need to
escape the backslash with another backslash!
•Example: "\\s+" matches one or more spaces.
•"\\w+" matches one or more alphanumeric characters.
•"\\n" in Python is the newline character, not a regex pattern. Use "\\\n" if
you want to match a literal backslash followed by n.
Use Python's Raw String Notation
•Raw strings make it easier to write regex patterns without escaping the
backslash multiple times.
•Example:
•r"\d+" matches one or more digits (instead of "\\d+").
•r"\bword\b" matches "word" as a whole word.
 Iterative Process of Writing Regex

•cat: Matches only lowercase "cat".

•[cC]at: Matches case-insensitive variations but includes substrings like
"concatenate".
•\b[cC]at\b: Uses word boundaries (\b) to match "cat" or "Cat" as
standalone words only.
 False Positives and False Negatives
The process we just went through was based on fixing two kinds of errors:
[Link] matching things that we should have matched
•Example: Missing "Apple" when we were searching for fruit names.
False negatives

[Link] strings that we should not have matched

•Example: Matching "applepie" or "pineapple" when we wanted only "apple".
False positives
 Characterizing Work on NLP
In NLP, we are always dealing with two key types of errors:
Reducing the error rate for an application often involves two complementary
efforts:
[Link] coverage (or recall):
[Link]: Minimize false negatives by ensuring that as many relevant items
as possible are captured.
[Link]: Capturing every occurrence of a product review, even if
phrased unconventionally.
[Link] accuracy (or precision):
[Link]: Minimize false positives by ensuring only the correct items are
captured.
[Link]: Identifying only the names of people, avoiding matches with
company names.
More Regular Expressions: Substitutions
and ELIZA
Substitutions in Text Processing
[Link] text patterns:
•Example: Replacing all occurrences of "don't" with "do not".
•Regex: [Link](r"\bdon't\b", "do not", text).
[Link] data:
•Example: Masking email addresses in a document.
•Regex: [Link](r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b", "[email
removed]", text).
ELIZA Chatbot
•ELIZA was an early chatbot that used pattern substitution to simulate conversation.
•Example: Converting statements into questions.
•Input: "I feel sad."
•Output: "Why do you feel sad?"
•Regex: [Link](r"I feel (.*)", r"Why do you feel \1?", text).
 Capture Groups: Multiple Registers

(.*): Captures text before "er" and after "we."

\1 and \2: Reuse the captured text in the second segment for validation.
 Non Capture Group
Regex Pattern | /(?:some|a few) (people|cats) like some \1/

•Non-Capturing Group (?:...):

Groups "some" or "a few" without capturing them.
•Capturing Group (people|cats):
Captures "people" or "cats" to allow backreference.
•Backreference \1:
Reuses the captured group to ensure consistency in the match.
 Lookahead Assertions
(?= pattern) is true if the pattern matches, but is zero-width; doesn't advance
the character pointer.
(?! pattern) is true if a pattern does not match.
How to match, at the beginning of a line, any single word that doesn’t start
with “Apple”:
/^(?!Apple)[A-Za-z]+/
 ELIZA: An Early NLP Application
Overview
• ELIZA was an early Natural Language Processing (NLP) system.
• Simulated a Rogerian psychotherapist.
• Created by Joseph Weizenbaum in 1966.
How It Works
• Pattern Matching: Identifies user input patterns, e.g.,
◦ Input: "I need X"
• Response Generation: Reframes input into questions, e.g.,
◦ Response: "Why do you need X?"
Significance
• First conversational AI demonstrating human-like dialogue.
• Highlighted the potential of AI to mimic human interaction.
• Inspired advancements in conversational AI, like chatbots and virtual assistants.
 How ELIZA Works
Pattern Matching Rules
•Identifies key emotions or patterns in input and generates responses.
Examples of rules:
•s/.* I’M (depressed|sad) .*/I AM SORRY TO HEAR YOU ARE \1/
•s/.* I AM (depressed|sad) .*/WHY DO YOU THINK YOU ARE \1?/
•s/.* all .*/IN WHAT WAY?/
•s/.* always .*/CAN YOU THINK OF A SPECIFIC EXAMPLE?/
Explanation
• Regex-Based: Matches input patterns with regular expressions.
• Dynamic Responses: Maps identified patterns to empathetic replies.
• Focus on Reflection: Encourages users to elaborate on their statements.
Word and Corpus
 How Many Words in a Sentence?
Example Sentence:
"The fox jumped over the lazy dog, and the fox ran away."
•Type:
•A unique element of the vocabulary (distinct word).
•Example: "fox" and "ran" are separate types.
•Repeated words are counted only once as types.
•Token:
•Each instance of a word in running text.
•Example: The word "the" appears 3 times as tokens.
Analysis
•Tokens: 12 (every word, including repeats, is counted).
•Types: 9 (unique words: the, fox, jumped, over, lazy, dog, and, ran, away).
 How Many Words in a Corpus?
Definitions:
• N = Number of Tokens:
Total instances of words in a corpus.
• V = Vocabulary (Set of Types):
Unique words in the corpus.
|V| = Size of Vocabulary.
Words & Corpus Example
 Corpora
Words don’t appear out of nowhere!
A text is produced by:
• A specific writer(s),
• At a specific time,
• In a specific variety (e.g., dialect or register),
• Of a specific language,
• For a specific function (e.g., inform, persuade, entertain).
"The weather today is sunny with a chance of rain in the evening."
Corpora vary along dimension like
Word tokenization
Space Based tokenization
 Simple Tokenization in UNIX
This approach uses basic Unix tools to tokenize a text file, inspired by Ken
Church's "UNIX for Poets." The goal is to extract individual word tokens and
count their frequencies.
Steps:
[Link] Non-Alphabetic Characters to Newlines:
•Command: tr -sc 'A-Za-z' '\n' < [Link]
•Explanation: The tr command replaces all characters that are not letters (non-alphanumeric) with newline
characters (\n). This effectively breaks the text into one word per line for easier processing.
[Link] Tokens Alphabetically:
•Command: | sort
•Explanation: The sort command organizes all tokens (words) in alphabetical order. This step is crucial to group
identical tokens together for counting.
[Link] Identical Tokens and Count:
•Command: | uniq -c
•Explanation: The uniq -c command combines identical tokens and outputs each unique word along with its
frequency in the text.
Simple Tokenization in UNIX
 Issues in Tokenization
Tokenization, or breaking text into smaller units (tokens), is not always
straightforward. Special cases make the process more complex.
Why We Can't Blindly Remove Punctuation:
•Abbreviations: Words like m.p.h., Ph.D., or AT&T rely on punctuation to retain meaning.
•Contractions: Examples like cap’n (for captain) use apostrophes that can’t be removed without losing the
word's form.
•Prices: Tokens such as $45.55 mix symbols and numbers.
•Dates: Formats like 01/02/06 need careful handling to avoid splitting into meaningless tokens.
•URLs: Examples like [Link] require tokenization as a whole unit.
•Email Addresses: Tokens such as someone@[Link] must remain intact.
•Hashtags: Tokens like #nlproc carry specific social media and content metadata.
Clitics & Multiword Expressions (MWEs)
Clitics:
Clitics are words that don’t stand alone and are often attached to others:
Examples:
•English: "are" in we're.
•French: "je" in j'ai, or "le" in l'honneur.
Multiword Expressions (MWEs):
Some multiword expressions function as single semantic units. The challenge is
deciding when to treat them as a single token.
Examples:
•New York (a proper noun) vs. rock ’n’ roll (a descriptive phrase).
 Tokenization in NLTK

[Link]: Matches sequences like U.S.A. where letters are separated by periods.
[Link] with Hyphens: Captures words like poster-print with optional hyphenation.
[Link] and Percentages: Handles numbers with optional dollar signs, decimals, or percentage symbols.
[Link]: Matches sequences like ....
[Link]: Captures punctuation characters such as commas, semicolons, and brackets as separate tokens.

Bird, Loper and Klein (2009), Natural Language Processing with Python. O’Reilly
Tokenization in Languages Without Spaces
Languages like Chinese, Japanese, and Thai do not use spaces to separate
words, making tokenization significantly more complex compared to
space-separated languages like English.
Challenges:
[Link] Boundaries:
•It is not always clear where one word ends, and another begins.
Example (Chinese): 我喜欢学习 (I like studying).
•Tokenization can vary:
•我 | 喜欢 | 学习 (I | like | study).
•Or, depending on context, could be different.
[Link] Clear Markers:
•Unlike punctuation or spaces in English, there are no explicit indicators for separating words.
 Solution
[Link]-Based Tokenization:
•Uses a pre-built dictionary to match possible words.
•Example: A tokenizer might segment 喜欢 as like if it's in the dictionary.
[Link] Models:
•Machine learning models trained on large datasets can predict word
boundaries.
•Example: A model might predict likely splits based on prior probabilities.
[Link] Network Approaches:
•Modern NLP methods (e.g., BERT) can dynamically identify boundaries using
contextual embeddings.
[Link]-Based Systems:
•Apply language-specific rules, like separating based on syllables or predefined
patterns.
 Word Tokenization in Chinese
Chinese words are particularly complex due to the lack of spaces and the use of
characters ("hanzi" or "zi"), which represent morphemes. The discussion about
average word length (e.g., 2.4 characters per word) and the challenges of deciding
what counts as a word is a standard topic in computational linguistics and Chinese
NLP.

Example Sentence:
姚明进入总决赛
"Yao Ming reaches the finals"
 Word Tokenization in Chinese
Interpretation Options:
3 Words:
1. Tokenized as:
姚明 (YaoMing) | 进入 (reaches) | 总决赛 (finals)
2. Interpretation: Combines proper nouns and phrases to reduce granularity.
5 Words:
1. Tokenized as:
姚 (Yao) | 明 (Ming) | 进入 (reaches) | 总 (overall) | 决赛 (finals)
2. Interpretation: Breaks the sentence into smaller units, maintaining granularity for meaning.
7 Characters (No Words):
1. Tokenized as:
姚 (Yao) | 明 (Ming) | 进 (enter) | 入 (enter) | 总 (overall) | 决 (decision) | 赛 (game)
2. Interpretation: Treats each character as an individual token, which is common in character-based models.
Word Tokenization in Chinese
 Word Tokenization in Urdu

• Definition:
Word tokenization is the process of splitting a sentence or text into individual words (tokens) to
facilitate text analysis and processing.
• Example:
Sentence: "‫"میں نے آج بہت سی کتابیں خریدی ہیں۔‬
Tokens: ["‫"میں‬, "‫"نے‬, "‫"آج‬, "‫"بہت‬, "‫"سی‬, "‫"کتابیں‬, "‫"خریدی‬, "‫"ہیں‬, "‫]"۔‬
 Challenges in Urdu Word Tokenization
Complex word structures (e.g., compound words like "‫ "کررہا‬and "‫)"چل ریہ‬
Ambiguity in spacing (e.g., "‫ "گھر جا‬vs. "‫)"گھرجا‬
Inflected words (e.g., "‫"کتابیں" → "کتاب‬, "‫)"لڑےک" → "لڑکا‬
Diacritics and different writing styles
Lack of standardized datasets for Urdu NLP
Other text tokenization
 Other text tokenization

Subword Tokenization(Three Common Algorithms)

[Link]-Pair Encoding (BPE) (Sennrich et al., 2016):
•Concept: Starts with individual characters as tokens and iteratively merges the most frequent adjacent pairs until a
vocabulary size is met.
Input: "lowering"
Steps:
•Initial tokens: l, o, w, e, r, i, n, g
•Merge l + o → lo, then lo + w → low, and so on.
•Final tokens: low, er, ing.
 Byte Pair Encoding (BPE) Token Learner
How It Works:
[Link] Vocabulary:
•Start with all individual characters as tokens.
•Example: {A, B, C, a, b, c}.
[Link] the Process:
•Step 1: Find the two adjacent symbols (characters or tokens) that appear most
frequently together in the training corpus.
Example: If AB occurs 10 times and BC occurs 8 times, choose AB.
•Step 2: Merge the pair into a single token (AB) and add it to the vocabulary.
Updated vocabulary: {A, B, C, AB, a, b, c}.
•Step 3: Replace every occurrence of AB in the corpus with the merged token AB.
[Link]:
•Repeat until the desired number of merges (k) is completed or until no frequent
pairs are left.
 Byte Pair Encoding (BPE) Token Learner
Steps:
[Link] Tokens: {l, o, w, e, r}. Corpus: l o w l o w l o w e r l o w e
r.
[Link] Merge:
•Most frequent pair: lo.
•Merge: lo.
•Updated Vocabulary: {l, o, w, e, r, lo}.
•Updated Corpus: lo w lo w lo w e r lo w e r.
[Link] Merge:
•Most frequent pair: low.
•Merge: low.
•Updated Vocabulary: {l, o, w, e, r, lo, low}.
•Updated Corpus: low low low e r low e r.
[Link] Merge:
•Most frequent pair: er.
•Merge: er.
•Updated Vocabulary: {l, o, w, e, r, lo, low, er}.
•Updated Corpus: low low lower lower.
[Link] Tokens: {low, er}.
Applying BPE
Applying BPE
 Properties of BPE Tokens
[Link] Words:
•BPE tokens usually include common whole words, e.g., "the," "and."
[Link] Subwords:
•Also include smaller units like subwords, which are meaningful parts of words.
•Examples:
•Suffixes like -est, -er.
•Prefixes like un-.
[Link] Definition:
•A morpheme is the smallest meaning-bearing unit of a language.
•Example:
•The word unlikeliest consists of 3 morphemes:
•un- (prefix)
•likely (root word)
•-est (suffix).
Applications:
•BPE tokens adapt well to unseen or rare words by breaking them into meaningful subwords or morphemes.