[Link]: AM4501 SUB.

NAME: NATURAL LANGUAGE PROCESSING

UNIT I: Introduction to Natural Language Processing

Natural Language Processing - Linguistic Background - Mathematical Foundations –Morphological

Analysis - Tokenization - Stemming - Lemmatization - Boundary Determination.

 To understand and apply fundamental NLP concepts and techniques.

 To grasp essential linguistic theories relevant to NLP, including syntax and semantics.
 To utilize mathematical principles like probability and statistics in NLP applications.
 To perform morphological analysis to break down and interpret word structures.
 To implement tokenization to segment text into individual tokens.
 To use lemmatization to normalize words to their base or dictionary form.
 To accurately determine and implement sentence and word boundary detection.

Natural Language Processing

What is NLP?

 NLP stands for Natural Language Processing

 Branch of artificial intelligence (AI) that focuses on the interaction between computers and
human language
 Helps computers to understand human language and also allows machines to communicate
with us.
 Algorithms and models that enable computers to understand, interpret, and generate human
language in a meaningful way.
 For instance, Google’s keyboard suggests auto-corrects, and word predicts in email
writing (words that would be used).
 Translation systems use language modeling to work efficiently with multiple languages.

How does Natural Language processing work?

• Converting unstructured data into computer-readable language by NLP attributes.

• Complex algorithms to break down any text content to extract meaningful information from it.

• Collected data is then used to further teach machines the logic of natural language

• Uses syntactic and semantic analysis to guide machines by identifying and recognizing data patterns.
Why natural language processing is important?

• Facilitates Human-Computer Interaction: users to communicate with devices, applications, and

systems using spoken or written language

• Text Understanding and Analysis: enables organizations to analyze customer feedback, social media
content, news articles, and other textual information to understand trends, sentiments, and patterns

• Automates Routine Tasks: asks such as text summarization, sentiment analysis, document
classification, and information extraction, improving efficiency and productivity.

• Enhances Search and Information Retrieval: search engines and information retrieval systems by
understanding user queries and returning relevant results

• Supports Multilingual Communication: machine translation and language understanding capabilities,

NLP facilitates communication across different languages

• Enables Sentiment Analysis and Opinion Mining: sentiment analysis analyzes and interprets the
sentiment or emotion expressed in text data

• Empowers Chat bots and Virtual Assistants: understand user queries, provide information, perform
tasks, and offer personalized recommendations, enhancing customer service and user experience.

History of NLP
Components of NLP

1. Natural Language Understanding (NLU)

• NLU focuses on interpreting and extracting meaning from human language input.

• It involves techniques such as text parsing, entity recognition, sentiment analysis, and intent
detection.

• NLU systems aim to comprehend the content of text or speech input to extract relevant information
and understand the user's intentions or queries.

• Examples of NLU applications include chatbots that understand user queries, sentiment analysis
tools that analyze emotions in text, and voice assistants that interpret spoken commands

2. Natural Language Generation (NLG)

• NLG, on the other hand, deals with creating human-like text or speech output based on structured
data or input from NLU systems.

• NLG systems generate coherent and contextually relevant text or speech by combining linguistic
rules, templates, and sometimes machine learning models.

• NLG applications include text summarization, language translation, chatbot responses, and
content generation for news articles or reports.

NLP Applications,

 Translating the languages

 Text processing in various languages
 Automatic Text Summarization
 Analyzing sentiments
 Speech Recognition
 Named Entity Recognition
 Phrase Extraction
 Tense Identification
 Relationship Extraction and so on.
 NLP Pipeline

Phases of NLP
 Things We need to know for learning NLP
• Text processing techniques
• Word Embedding
• Deep Learning Network for NLP (CNN, LSTM, GRUs, Encoder and Decoder)
• Attention mechanism, Transfer learning in NLP
• Transformers (BERT, GPT, ALBERT and so on) • Fine Tuning NLP task
• Large Language Model (LLM)

Linguistic Background

Understanding the linguistic background is crucial for anyone working in Natural Language Processing
(NLP) because it provides the foundational knowledge of how human languages are structured and
function. Here are the basic linguistic concepts relevant to NLP, along with explanations and examples:

1. Phonetics and Phonology:

o Phonetics is the study of the sounds of human speech. Phonology deals with how those
sounds are organized in particular languages.
o Example: In English, the sounds /p/ and /b/ are distinct phonemes, meaning they can
change the meaning of a word (e.g., "pat" vs. "bat").
2. Morphology:
o Morphology is the study of the structure of words. It involves understanding how words
are formed from morphemes, which are the smallest units of meaning.
o Example: The word "unhappiness" consists of three morphemes: "un-" (a prefix meaning
"not"), "happy" (the root), and "-ness" (a suffix indicating a state or condition).
3. Syntax:
o Syntax is the study of how words combine to form sentences. It involves understanding
the rules and patterns that govern sentence structure.
o Example: In English, a typical sentence structure follows the Subject-Verb-Object
(SVO) order, such as "The cat (subject) chased (verb) the mouse (object)."
4. Semantics:
o Semantics is the study of meaning in language. It examines how words and sentences
convey meaning.
o Example: The sentence "The cat is on the mat" conveys a specific meaning about the
location of the cat in relation to the mat.
5. Pragmatics:
o Pragmatics is the study of how context influences the interpretation of meaning. It looks
at how language is used in communication and how meaning is constructed in different
situations.
 o Example: The phrase "Can you pass the salt?" is understood as a request rather than a
literal question about someone's ability to pass the salt.
6. Lexicology:
o Lexicology is the study of words, their nature, and meaning, including their classification
and structure.
o Example: Understanding the different meanings and uses of the word "bank" (financial
institution vs. the side of a river).
7. Discourse Analysis:
o Discourse analysis involves studying language use across longer texts or conversations,
looking at how coherence and cohesion are maintained.
o Example: Analyzing a political speech to understand how the speaker builds an argument
and engages the audience.

Example Application in NLP

Sentence: "The quick brown fox jumps over the lazy dog."

1. Phonetics and Phonology:

o Analyze the individual sounds (phonemes) in the sentence and their phonetic
characteristics.
2. Morphology:
o Identify the morphemes: "quick" (root), "brown" (root), "fox" (root), "jump-s" (root +
plural suffix), "over" (root), "the" (determiner), "lazy" (root), "dog" (root).
3. Syntax:
o Parse the sentence structure: [The quick brown fox] (subject) [jumps] (verb) [over the
lazy dog] (prepositional phrase).
4. Semantics:
o Determine the meaning: A fast, brown-colored fox performs the action of jumping over a
slow or unenergetic dog.
5. Pragmatics:
o Consider the context: This sentence is often used because it contains all the letters of the
English alphabet, which is useful for testing fonts and keyboards.
6. Lexicology:
o Study the meanings of individual words and their use in different contexts (e.g., "fox" as
an animal vs. "fox" as a term for a cunning person).
7. Discourse Analysis:
o Although this sentence is standalone, in a broader text, analyze how it fits within the
paragraph and contributes to the overall message.

Mathematical Foundations

Mathematical concepts are essential in Natural Language Processing (NLP) as they provide the tools and
techniques to model and analyze language computationally.

1. Probability and Statistics:

o Probability and statistics help in modeling the uncertainty and variability of language
data.
o Example: Estimating the probability of a word sequence using n-grams. For a bigram
model (n=2), the probability of a sentence "The cat" can be approximated as
 P("The cat")=P("cat"∣"The")×P("The")P(\text{"The cat"}) = P(\text{"cat"}|\text{"The"})
\times P(\text{"The"})P("The cat")=P("cat"∣"The")×P("The").
2. Linear Algebra:
o Linear algebra provides the framework for handling large datasets and performing
operations on vectors and matrices, which are fundamental in many NLP tasks.
o Example: Representing words as vectors in a high-dimensional space using word
embeddings (e.g., Word2Vec, GloVe). Each word is a vector, and operations like
addition and dot product can be performed on these vectors.
3. Calculus:
o Calculus is used in optimization problems, particularly in training machine learning
models where gradient descent is employed to minimize a loss function.
o Example: Training a neural network for sentiment analysis involves calculating the
gradient of the loss function with respect to the model parameters and updating the
parameters using gradient descent.
4. Linear Regression:
o Linear regression models the relationship between a dependent variable and one or more
independent variables.
o Example: Predicting the sentiment score of a review based on the frequency of positive
and negative words. The model could be a simple linear function where the sentiment
score is predicted as a weighted sum of word frequencies.
5. Optimization:
o Optimization techniques are used to find the best parameters for a model that minimize
(or maximize) a given objective function.
o Example: Using stochastic gradient descent (SGD) to optimize the weights in a logistic
regression model for binary text classification.
6. Information Theory:
o Information theory deals with quantifying information, entropy, and the efficiency of
encoding schemes.
o Example: Calculating the entropy of a language model to measure its uncertainty or
using mutual information to identify word associations.

Example Application in NLP

Task: Sentiment Analysis of Movie Reviews

1. Probability and Statistics:

o Example: Using a Naive Bayes classifier for sentiment analysis.
o Explanation: Estimate the probability of a review being positive or negative based on the
frequency of words. For instance, if "great" appears frequently in positive reviews,
P("positive"∣"great")P(\text{"positive"}|\text{"great"})P("positive"∣"great") will be high.
2. Linear Algebra:
o Example: Word Embeddings.
o Explanation: Convert words to vectors using embeddings like Word2Vec. Perform
operations such as finding the cosine similarity between vectors to measure word
similarity.
3. Calculus:
o Example: Gradient Descent.
o Explanation: During the training of a neural network, use calculus to compute gradients
and update weights to minimize the loss function. For instance, in back propagation, the
gradient of the loss with respect to each weight is computed.
 4. Linear Regression:
o Example: Predicting Sentiment Scores.
o Explanation: Model the sentiment score of a review as a linear function of the frequency
of certain keywords. For instance,
sentiment score=w1×frequency("good")+w2×frequency("bad")+b\text{sentiment score}
= w_1 \times \text{frequency}(\text{"good"}) + w_2 \times
\text{frequency}(\text{"bad"}) + bsentiment score=w1×frequency("good")+w2
×frequency("bad")+b.
5. Optimization:
o Example: Tuning Model Parameters.
o Explanation: Use optimization algorithms like SGD to minimize the loss function during
the training of a logistic regression model for classifying reviews as positive or negative.
6. Information Theory:
o Example: Entropy in Language Models.
o Explanation: Calculate the entropy of a language model to understand its uncertainty.
For example, lower entropy indicates a more confident model.

Morphological Analysis

 Morphological analysis is a crucial aspect of Natural Language Processing (NLP) that involves
understanding and processing the structure of words.
 Morphology deals with the internal structure of words and how they can be modified to express
different grammatical categories such as tense, case, voice, aspect, person, number, gender, and
mood.

Basic Concepts in Morphological Analysis

1. Morphemes:
o The smallest units of meaning in a language. Morphemes can be roots, prefixes, suffixes,
or infixes.
o Example: In the word "unhappiness", "un-", "happy", and "-ness" are morphemes.
2. Inflection:
o The modification of a word to express different grammatical categories without changing
its core meaning.
o Example: Adding "-s" to "dog" to form "dogs" (plural form).
3. Derivation:
o The process of creating a new word by adding a prefix or suffix, which often changes the
grammatical category or meaning of the word.
o Example: Adding "un-" to "happy" to form "unhappy" (opposite meaning).
4. Compounding:
o The combination of two or more words to create a new word with a specific meaning.
o Example: Combining "tooth" and "brush" to form "toothbrush".
5. Stemming:
o The process of reducing a word to its base or root form. Stemming is often a crude
heuristic that chops off common prefixes and suffixes.
o Example: Reducing "running", "runs", and "runner" to "run".
6. Lemmatization:
o The process of reducing a word to its base or dictionary form, known as the lemma.
Lemmatization is more sophisticated than stemming and considers the context.
o Example: Reducing "was" to "be", "better" to "good", and "running" to "run".
Example of Morphological Analysis in NLP

Let's take a sample sentence and perform morphological analysis on it:

Sentence: "The cats were running quickly towards the playground."

1. Tokenization:
o Step: Split the sentence into individual words (tokens).
o Tokens: ["The", "cats", "were", "running", "quickly", "towards", "the", "playground"]
2. Morphological Analysis:
o Tokens: ["The", "cats", "were", "running", "quickly", "towards", "the", "playground"]
o Analysis:
 "The": Determiner, no morphological change.
 "cats":
 Root: "cat"
 Inflection: "-s" (plural form)
 "were":
 Lemma: "be"
 Tense: Past
 "running":
 Root: "run"
 Inflection: "-ing" (present participle)
 "quickly":
 Root: "quick"
 Derivation: "-ly" (adverb form)
 "towards": Preposition, no morphological change.
 "the": Determiner, no morphological change.
 "playground":
 Compound: "play" + "ground"
3. Stemming:
o Tokens: ["The", "cats", "were", "running", "quickly", "towards", "the", "playground"]
o Stems: ["The", "cat", "were", "run", "quick", "toward", "the", "playground"]
4. Lemmatization:
o Tokens: ["The", "cats", "were", "running", "quickly", "towards", "the", "playground"]
o Lemmas: ["The", "cat", "be", "run", "quickly", "towards", "the", "playground"]

Applications of Morphological Analysis in NLP

1. Text Normalization:
o Example: Converting various forms of a word to a single form for consistency in text
processing (e.g., "running", "runs", and "runner" all normalized to "run").
2. Information Retrieval:
o Example: Improving search results by matching different forms of a word (e.g.,
searching for "run" should also return documents containing "running" and "ran").
3. Machine Translation:
o Example: Accurately translating words with correct inflections and derivations to
maintain grammatical correctness in the target language.
4. Part-of-Speech Tagging:
 oExample: Identifying the correct grammatical category of words based on their
morphological structure (e.g., distinguishing between "running" as a verb and "runner" as
a noun).
5. Named Entity Recognition (NER):
o Example: Recognizing named entities in text, such as person names, locations, and
organizations, which may have inflected forms.

Boundary Determination

 Boundary determination, also known as boundary detection or segmentation, is a fundamental

task in Natural Language Processing (NLP) that involves identifying the boundaries of linguistic
units such as sentences, clauses, phrases, or words within a text.
 Accurate boundary determination is crucial for many downstream NLP tasks, including
tokenization, parsing, machine translation, and information retrieval.

Basic Concepts in Boundary Determination

1. Sentence Boundary Detection (SBD):

o The process of identifying where sentences begin and end in a text.
o Challenges: Handling abbreviations, titles, and punctuation that do not mark sentence
boundaries (e.g., " [Link] went to Washington. He stayed there for 2 weeks.").
o Example: In the text "I saw [Link]. He said, 'Hello!'", detecting that there are two
sentences: "I saw [Link]." and "He said, 'Hello!'".
2. Word Boundary Detection (Tokenization):
o The process of identifying individual words or tokens in a text.
o Challenges: Handling compound words, contractions, and languages without clear word
boundaries (e.g., Chinese).
o Example: In the sentence "I can't do this.", detecting the tokens: ["I", "ca", "n't", "do",
"this", "."].
3. Phrase Boundary Detection:
o The process of identifying phrases or clauses within a sentence.
o Challenges: Dealing with nested structures and varying phrase lengths.
o Example: In the sentence "The quick brown fox jumps over the lazy dog," identifying
phrases like ["The quick brown fox", "jumps over", "the lazy dog"].

Example Application in NLP

Let’s consider an example to illustrate boundary determination in practice:

Text: "Hello! My name is [Link]. I live in New York. How can I help you?"

1. Sentence Boundary Detection:

o Input: "Hello! My name is [Link]. I live in New York. How can I help you?"
o Output: ["Hello!", "My name is [Link].", "I live in New York.", "How can I help
you?"]
2. Word Boundary Detection (Tokenization):
o Input: "Hello! My name is [Link]. I live in New York. How can I help you?"
o Output: ["Hello", "!", "My", "name", "is", "Mr.", "Raja", ".", "I", "live", "in", "New",
"York", ".", "How", "can", "I", "help", "you", "?"]
Techniques for Boundary Determination

1. Rule-Based Approaches:
o Using predefined rules and heuristics to detect boundaries.
o Example: A rule-based system might define a sentence boundary after a period followed
by a space and an uppercase letter.
2. Machine Learning-Based Approaches:
o Training models to learn patterns in text data to predict boundaries.
o Example: Using labeled training data, a machine learning model can be trained to
identify sentence boundaries based on features like punctuation, capitalization, and
context.
3. Hybrid Approaches:
o Combining rule-based and machine learning techniques to improve accuracy.
o Example: A hybrid system might first apply basic rules to segment text and then refine
the boundaries using a trained model.

Example Algorithm: Punkt Sentence Tokenizer

 The Punkt Sentence Tokenizer is a popular unsupervised algorithm used for sentence boundary
detection.
 It works by analyzing text for common patterns and structures without needing a pre-labeled
dataset.
Example:
import nltk
[Link]('punkt')
from [Link] import sent_tokenize
text = "Hello! My name is [Link].’I live in New York. How can I help you?"
sentences = sent_tokenize(text)
print(sentences)

Output:
['Hello!', 'My name is Mr. Raja.', 'I live in New York.', 'How can I help you?']

Applications of Boundary Determination

1. Text Preprocessing:
o Example: Preparing text for further analysis by segmenting it into sentences and tokens.
2. Information Retrieval:
o Example: Enhancing search engines to return more accurate results by understanding the
structure of text.
3. Machine Translation:
o Example: Improving translation quality by accurately segmenting text into sentences and
phrases.
4. Speech Recognition:
o Example: Transcribing spoken language into written text with correct sentence and word
boundaries.
5. Text Summarization:
o Example: Identifying key sentences to generate summaries of documents.