P P Savani University

School of Engineering

Institute of Computer Science and Application

Department of Computer Application
Course Code: SSCS3512

Course Name: Artificial Intelligence

SECTION I

UNIT 3 : Natural Language Processing (NLP)

OUTLINE
• Introduction to Natural Language Processing

• Techniques in NLP: Tokenization, Stemming and lemmatization

• NLP Applications: Sentiment analysis, Text classification, Named Entity

Recognition (NER).
Introduction to Natural Language
Processing
• Natural language processing (NLP) is the intersection of computer science, linguistics
and machine learning. The field focuses on communication between computers and humans
in natural language and NLP is all about making computers understand and generate human
language. Applications of NLP techniques include voice assistants like Amazon’s Alexa and
Apple’s Siri, but also things like machine translation and text-filtering.
• Natural language processing has heavily benefited from recent advances in machine learning,
especially from deep learning techniques. The field is divided into the three parts:
• Speech recognition — the translation of spoken language into text.
• Natural language understanding — a computer’s ability to understand language.
• Natural language generation — the generation of natural language by a computer.
Introduction to Natural Language
Processing
Applications of NLP
• The applications of Natural Language Processing are as follows:
• Text and speech processing like-Voice assistants – Alexa, Siri, etc.
• Text classification like Grammarly, Microsoft Word, and Google Docs
• Information extraction like-Search engines like DuckDuckGo, Google
• Chatbot and Question Answering like:- website bots
• Language Translation like:- Google Translate
• Text summarization
Introduction to Natural Language
Processing
Introduction to Natural Language
Processing
Natural Language Processing (NLP) is a field of artificial intelligence (AI) focused on the
interaction between computers and humans through natural language. The ultimate goal of NLP
is to enable computers to understand, interpret, and generate human languages in a way that is
both meaningful and useful.
Tokenization:
• Breaking down text into individual units such as words or phrases.
Part-of-Speech Tagging (POS):
• Identifying the grammatical category of words (nouns, verbs, adjectives, etc.).
Named Entity Recognition (NER):
• Recognizing and classifying named entities in text (such as names of people, organizations,
locations, dates).
Introduction to Natural Language
Processing
Syntax and Parsing:
• Analyzing the grammatical structure of sentences.
Sentiment Analysis:
• Determining the sentiment expressed in text (positive, negative, neutral).
Machine Translation:
• Translating text from one language to another.
Text Classification:
• Categorizing text into predefined categories.
Topic Modeling:
• Identifying topics within a set of texts.
Speech Recognition:
• Converting spoken language into text.
Text Generation:
• Generating coherent and contextually relevant text.
Introduction to Natural Language
Processing
Techniques in NLP:
• Rule-Based Methods: Using predefined linguistic rules.
• Statistical Methods: Leveraging statistical models to predict and analyze text.
• Machine Learning: Using algorithms to learn from data (e.g., Naive Bayes, SVM).
• Deep Learning: Utilizing neural networks (e.g., RNNs, LSTMs, Transformers).
Techniques in NLP: Tokenization,
Stemming and lemmatization
• Tokenization. Tokenization substitutes sensitive information with non-sensitive
information, or a token. Tokenization is often used in payment transactions to protect credit
card data.
• Lemmatization and stemming. Lemmatization groups together different inflected versions
of the same word. For example, the word "walking" would be reduced to its root form, or
stem, "walk" to process.

• Stemming Example Lemmatization Example

• Examples: Examples:
• "running" -> "run“ "running" -> "run" (verb form)
• "jumps" -> "jump“ "better" -> "good" (comparative form)
• "happiness" -> "happi“ "was" -> "be" (verb form)
Tokenization
• Tokenization is the process of dividing a text into smaller units known as tokens.
• Tokens are typically words or sub-words in the context of natural language processing.
• Tokenization is a critical step in many NLP tasks, including text processing, language
modelling, and machine translation.
• Tokenization involves using a tokenizer to segment unstructured data and natural language
text into distinct chunks of information, treating them as different elements.
• The tokens within a document can be used as vector, transforming an unstructured text
document into a numerical data structure suitable for machine learning.
• This rapid conversion enables the immediate utilization of these tokenized elements by a
computer to initiate practical actions and responses.
Tokenization
Types of Tokenization
• Tokenization can be classified into several types based on how the text is segmented. Here
are some types of tokenization:
• Word Tokenization:
• Word tokenization divides the text into individual words. Many NLP tasks use this approach,
in which words are treated as the basic units of meaning.
• Example:
Input: "Tokenization is an important NLP task."
Output: ["Tokenization", "is", "an", "important", "NLP", "task", "."]
Tokenization
Sentence Tokenization:
• The text is segmented into sentences during sentence tokenization. This is useful for tasks
requiring individual sentence analysis or processing.
• Example:

Input: "Tokenization is an important NLP task. It helps break down text into smaller units."
Output: ["Tokenization is an important NLP task.", "It helps break down text into smaller
units."]
Tokenization
Subword Tokenization:
• Subword tokenization entails breaking down words into smaller units, which can be
especially useful when dealing with morphologically rich languages or rare words.
• Example:

Input: "tokenization"
Output: ["token", "ization"]

• Character Tokenization:
• This process divides the text into individual characters. This can be useful for modelling
character-level language.
• Example:
Input: "Tokenization"
Output: ["T", "o", "k", "e", "n", "i", "z", "a", "t", "i", "o", "n"]
Stemming
Stemming is a method in text processing that eliminates prefixes and suffixes from words,
transforming them into their fundamental or root form,
The main objective of stemming is to streamline and standardize words, enhancing the
effectiveness of the natural language processing tasks.
• For example, “chocolates” becomes “chocolate” and “retrieval” becomes “retriev.”
• This is crucial for pipelines for natural language processing, which use tokenized words that
are acquired from the first stage of dissecting a document into its constituent words.
Types of Stemmer in NLTK(Natural
Language Toolkit)
1. Porter’s Stemmer
Developed by Martin Porter in 1980
Uses a set of rules to iteratively strip suffixes from words.
Example: EED -> EE means “if the word has at least one vowel and
consonant plus EED ending, change the ending to EE” as ‘agreed’
becomes ‘agree’.
from [Link] import PorterStemmer
stemmer = PorterStemmer()
print([Link]('running')) # Output: 'run'
Types of Stemmer in NLTK(Natural
Language Toolkit)
2. Lovins Stemmer
developed by Julie Beth Lovins in 1968
The Lovins Stemmer uses the longest match principle, where the
longest suffix is removed first.
The Lovins Stemmer uses a large set of context-sensitive rules to strip
suffixes from words.
Example: sitting -> sitt -> sit
Types of Stemmer in NLTK(Natural
Language Toolkit)
3. Dawson Stemmer
developed by John Dawson in 1974
It is an extension of Lovins stemmer in which suffixes are stored in the reversed order indexed
by their length and last letter.
extends the number of suffixes and rules, making it more comprehensive. The Dawson Stemmer
significantly expands the list of suffixes compared to the Lovins Stemmer, with around 1200
suffixes included.
Example “nationalization” -> “national” -> “nation”
Types of Stemmer in NLTK(Natural
Language Toolkit)
4. Krovetz Stemmer
developed by Robert Krovetz in the early 1990s, is a well-known
stemming algorithm that takes a different approach compared to
traditional rule-based stemmers like the Porter or Lovins Stemmer.
Instead of just removing suffixes based on predefined rules, the
Krovetz Stemmer is designed to ensure that the resulting stem is a
valid word in the English language.
Example:
Universities -> universit -> university
Types of Stemmer in NLTK(Natural
Language Toolkit)
5. Xerox Stemmer
originally developed by Xerox's Palo Alto Research Center (PARC)
recognized for its efficiency and accuracy, particularly in handling complex morphological
structures found in various languages.
Capable of processing extensive datasets and generating valid words, it has a tendency to over-
stem, primarily due to its reliance on lexicons (dictionaries), making it language-dependent.
This reliance means that the effectiveness of the stemming process is closely tied to the quality
and comprehensiveness of the lexicon used.
This constraint implies that its effectiveness is limited to specific languages.
Example:
‘children’ -> ‘child’
‘understood’ -> ‘understand’
‘whom’ -> ‘who’
‘best’ -> ‘good’
Types of Stemmer in NLTK(Natural
Language Toolkit)
6. N-Gram Stemmer
The algorithm, aptly named n-grams (typically n=2 or 3), involves breaking words into
segments of length n and then applying statistical analysis to identify patterns or substrings
within words.
The word is split into overlapping sequences of n characters, called n-grams.
Example:
•Word: "running"
•3-Grams: run, unn, nni, nin, ing
Use Cases: N-Gram Stemmers are often used in information retrieval, text mining, and other
NLP tasks where language independence is crucial, or where traditional stemming approaches
might struggle with irregular or unknown word forms.
Types of Stemmer in NLTK(Natural
Language Toolkit)
7. Snowball Stemmer
developed by Martin Porter as a more efficient and flexible successor to the original Porter
Stemmer.
The Snowball Stemmer is implemented within the Snowball framework, a programming
language designed for writing stemming algorithms.
The Snowball Stemmer, compared to the Porter Stemmer, is multi-lingual as it can handle non-
English words.
Original words: ['running', 'jumped', 'happily', 'quickly', 'foxes']
Stemmed words: ['run', 'jump', 'happi', 'quick', 'fox']
Lemmatization
Lemmatization is a fundamental text pre-processing technique widely applied in natural
language processing (NLP) and machine learning.
lemmatization seeks to distil words to their foundational forms. In this linguistic refinement,
the resultant base word is referred to as a “lemma.”
Lemmatization is the process of grouping together the different inflected forms of a word so
they can be analyzed as a single item.
Lemmatization is similar to stemming but it brings context to the words.
So, it links words with similar meanings to one word.
Text pre processing includes both Stemming as well as lemmatization. Lemmatization is
preferred over Stemming because lemmatization does morphological analysis of the words.
Examples of lemmatization:
-> rocks : rock
-> corpora : corpus
-> better : good
Types of Lemmatization
1. Rule-Based Lemmatization:
This technique uses a set of predefined rules to convert inflected forms of a word into its
lemma.
The rules are based on the grammatical structure of the language, such as verb conjugations,
noun pluralizations, etc.
For example, a rule-based lemmatizer might convert "running" to "run" by removing the "-ing"
suffix.
Types of Lemmatization
2. Dictionary-Based Lemmatization:
This approach relies on a comprehensive dictionary or lexical database that maps inflected
forms to their corresponding lemmas.
The lemmatizer looks up each word in the dictionary and retrieves its lemma.
For instance, "better" would be mapped to "good.“
WordNet is a popular lexical database often used in dictionary-based lemmatization.
Types of Lemmatization
3. Machine Learning-Based Lemmatization:
Machine learning models can be trained to perform lemmatization by learning from a large
corpus of text where words are labeled with their lemmas.
These models can be more flexible and accurate, especially for languages with complex
morphology.
They consider the word’s part of speech and surrounding context to predict the correct lemma.
Techniques like Conditional Random Fields (CRFs) and neural networks can be used in this
approach.
Example of Lemmatization:
• Let's consider the word "better":
• Input: "better"
• Rule-Based Output: Might not handle "better" correctly without
context.
• Dictionary-Based Output: "good" (based on a dictionary lookup).
• Machine Learning-Based Output: "good" (based on learned patterns
from data).
Sentiment analysis
• Sentiment analysis
Sentiment analysis is the process of analyzing digital text to determine if the emotional tone of the
message is positive, negative, or neutral.
Today, companies have large volumes of text data like emails, customer support chat transcripts,
social media comments, and reviews.
Sentiment analysis tools can scan this text to automatically determine the author’s attitude towards
a topic.
Companies use the insights from sentiment analysis to improve customer service and increase
brand reputation.
It focuses not only on polarity (positive, negative & neutral) but also on emotions (happy, sad,
angry, etc.).
It uses various Natural Language Processing algorithms such as Rule-based, Automatic, and
Hybrid.
Sentiment analysis
Sentiment analysis is important
• Customer Feedback Analysis: Businesses can analyze customer reviews, comments, and
feedback to understand the sentiment behind them helping in identifying areas for
improvement and addressing customer concerns, ultimately enhancing customer satisfaction.
• Brand Reputation Management: Sentiment analysis allows businesses to monitor their
brand reputation in real-time. By tracking mentions and sentiments on social media, review
platforms, and other online channels, companies can respond promptly to both positive and
negative sentiments, mitigating potential damage to their brand.
Sentiment analysis
• Product Development and Innovation: Understanding customer sentiment helps identify
features and aspects of their products or services that are well-received or need
improvement. This information is invaluable for product development and innovation,
enabling companies to align their offerings with customer preferences.
• Competitor Analysis: Sentiment Analysis can be used to compare the sentiment around a
company’s products or services with those of competitors.
Businesses identify their strengths and weaknesses relative to competitors, allowing for
strategic decision-making.
• Marketing Campaign Effectiveness
Businesses can evaluate the success of their marketing campaigns by analyzing the sentiment
of online discussions and social media mentions.
Positive sentiment indicates that the campaign is resonating with the target audience, while
negative sentiment may signal the need for adjustments.
Text classification
Text classification
• Text classification also known as text tagging or text categorization is the process of
categorizing text into organized groups. By using Natural Language Processing (NLP), text
classifiers can automatically analyze text and then assign a set of pre-defined tags or
categories based on its content.
• Text Classification is a fundamental task in Natural Language Processing (NLP) that involves
categorizing text into organized groups. This task can be performed using various techniques,
ranging from simple rule-based systems to advanced machine learning models.
• Key Techniques in Text Classification
• Rule-Based Approaches:
• Definition: Use predefined linguistic rules to classify text.
• Example: Keyword matching where specific words or phrases are mapped to categories.
Text classification
Machine Learning Models:
• Supervised Learning: Models are trained on labeled datasets where the category of each
text is known.
• Naive Bayes Classifier: Uses probabilities of words belonging to specific categories.
• Support Vector Machines (SVM): Finds a hyperplane that best separates categories.
• Decision Trees and Random Forests: Tree-based models that split data based on
feature values.
Text classification
Deep Learning Models:
• Convolutional Neural Networks (CNNs): Typically used for image data but effective in text
classification by capturing local patterns.
• Recurrent Neural Networks (RNNs): Good for sequential data like text, capturing
dependencies between words.
• Long Short-Term Memory Networks (LSTMs): A type of RNN that effectively manages long-
term dependencies.
• Transformers (e.g., BERT, GPT): Advanced models that understand context by processing
entire sequences simultaneously.
• Pretrained Models and Transfer Learning:
• BERT (Bidirectional Encoder Representations from Transformers): Context-aware
model that can be fine-tuned for specific tasks.
• GPT (Generative Pre-trained Transformer): Useful for generating text and can be fine-
tuned for classification.
Text classification
Steps in Text Classification
1. Data Collection
2. Preprocessing
3. Feature Extraction
4. Model Training
5. Evaluation
6. Deployment and Monitoring
Applications of Text Classification
• Spam Detection: Classifying emails as spam or non-spam.
• Sentiment Analysis: Categorizing text based on sentiment (positive, negative, neutral).
• Topic Labeling: Assigning topics to documents, news articles, or social media posts.
• Language Detection: Identifying the language of a given text.
• Customer Support: Categorizing customer queries to route them to the appropriate
department.
Named Entity Recognition (NER)
Named Entity Recognition (NER) is a crucial task in Natural Language Processing (NLP) that
involves identifying and classifying named entities mentioned in unstructured text into
predefined categories such as person names, organizations, locations, dates, and other proper
nouns.
• Steps in NER
• Data Collection:
• Collect or access a large corpus of text with annotated entities.
• Data Preprocessing:
• Tokenization: Split text into individual tokens (words).
• Normalization: Convert text to a standard format (lowercasing, removing special
characters).
• Feature Extraction (for traditional ML models):
• Extract features like part-of-speech tags, capitalization, word embeddings, etc.
Named Entity Recognition (NER)
• Model Training:
• Train NER models using annotated datasets.
• Commonly used datasets: CoNLL-2003, OntoNotes, etc.
• Evaluation:
• Precision: The percentage of correctly identified entities among the identified entities.
• Recall: The percentage of correctly identified entities among all entities present in the
text.
• F1-Score: The harmonic mean of precision and recall.
• Deployment and Monitoring:
• Deploy the NER model in a production environment.
• Continuously monitor its performance and update with new data as necessary.
Named Entity Recognition (NER)
Applications of NER
• Information Extraction: Extracting structured information from unstructured text (e.g.,
extracting names of people and organizations from news articles).
• Content Categorization: Tagging content with relevant entities for better organization and
retrieval.
• Question Answering Systems: Improving the accuracy of systems by understanding entity-
related queries.
• Healthcare: Extracting patient information, diagnoses, medications from clinical notes.
• Finance: Analyzing financial reports, news, and documents to identify relevant entities like
company names, financial terms.