UNIT 5 NATURAL LANGUAGE PROCESSING

Learning outcomes:
Comprehend the complexities of natural languages and elaborate on the need for
NLP techniques for machines to understand various natural languages effectively.
Explore the various applications of NLP in everyday life, such as , voice assistants,
auto generated captions, language translation, sentiment analysis, text classification
and keyword extraction.
Understand the concepts like lexicon, syntax, semantics, and logical analysis of
input text.
Understand the concept of chatbot and the differences between smartbots and script
bots.
Learn about the Text Normalization technique used in NLP and the popular
NLP model - Bag-of-Words.

Main points:
1. Features of Natural languages.
Natural language, like English, is a human language unlike artificial
or machine languages.
Natural languages follow rules but are also flexible, changing over
time and with use. Natural languages show creativity and
efficiency, helping us communicate in new ways.
2. Introduction to Natural Language Processing.
Natural Language Processing (NLP) is a field that combines computer
science, artificial intelligence and language studies.
It helps computers understand, process and create human language
in a way that makes sense and is useful.
3. Various real-life applications of NLP
Sentiment analysis, chatbots, machine translation, and text
summarization.
4. Explore the various stages of NLP that involve in understanding and
processing human language.
Lexical Analysis
Syntactic Analysis
Semantics Analysis
Discourse Integration
Pragmatic Analysis
5. Text Normalization
Tokenization
Lowercasing
Stop word removal
Stemming and lemmatization
 6. Bag of Words
A fundamental technique used to represent text data as
numerical vectors.
It essentially treats a document as a collection (or "bag") of words,
ignoring word order and grammar, and focuses on the frequency of
each word.
This allows machine learning algorithms to process text data more
easily.
Multiple Choice Questions (MCQs)

1. Which feature of NLP helps in understanding the emotions of the people

mentioned with the feedback?
(a) Virtual assistants (b) Sentiment analysis
(c) Text classification (d) Automatic summarization

2. Which of the following is used for finding the frequency of words in some
given text sample?
(a) Stemming (b) Lemmatisation
(c) Bag of words (d) None of the above

3. Machine translation feature converts .

(a) One language to another
(b) Human language to machine language
(c) Any human language to Programming
(d) Machine language to human language

4. Which of the following comes under NLP?

(a) Chatbots (b) Price comparison websites
(c) Facial recognition (d) All of the above

5. Chatbots are AI systems which

(a) Interact with humans through text or speech
(b) Are able to offer round the clock responses and handle multiple queries
simultaneously
(c) Both (a) and (b) (d) Neither (a) nor (b)

6. What do we call the process of dividing a string into component words?

(a) Regression (b) Word Tokenisation
(c) Classification (d) Clustering

7. Sentence segment is the step for building the NLP model.

(a) First (b) Second (c) Third (d) Fourth
8. Which of these is not a stopword?
(a) This (b) Things
(c) Is (d) Do

9. What is the stem of the word

(a) Mak (b) Make
(c) Making (d) Maker

10. What is the lemma of the word

(a) Mak (b) Make
(c) Making (d) Maker

11. Which algorithms result in two things, a vocabulary of words and frequency
of the words in the corpus?
(a) Sentence segmentation (b) Tokenisation
(c) Bag of words (d) Text normalisation

12. Which of the following is the type of data used by NLP applications?
(a) Images (b) Numerical data
(c) Graphical data (d) Text and Speech

13. A corpus contains 12 documents. How many document vectors will be there for
that corpus?
(a) 12 (b) 1 (c) 24 (d)1 / 12

14. This real life application of NLP is used to provide an overview of a news item or
blog post, while avoiding redundancy from multiple sources and maximising the
diversity of content obtained. Which is this application?
(a) Chatbot (b) Virtual Assistant
(c) Sentiment Analysis (d) Automatic Summarisation

15. Which of the following words represent an example of a lemma resulting

Processing (NLP)?
(a) Care (b) Cared
(c) Cares (d) Car

16. Bag of Words is a model which helps in extracting features out of

the text which can be helpful in machine learning algorithms.
(a) Data Science (DS) (b) Virtual Reality (VR)
 (c) Natural Language Processing (NLP) (d) Computer Vision (CV)

17. Select the correct features of Smart Bot

(a) Smart-bots are flexible and powerful
(b) Coding is required to take this up on board
(c) Smart bots work on bigger databases and other resources directly
(d) All of the above

18. For the whole corpus is divided into sentences. Each sentence is taken as
a different data so now the whole corpus gets reduced to sentences.
(a) Text Regulation (b) Sentence Segmentation
(c) Tokenization (d) Stemming

19. Assertion (A): Stemming is a technique used to reduce an inflected word

down to its word stem.
Reason (R): For example, the words and

(a) Both A and R are correct and R is the correct explanation of A

(b) Both A and R are correct but R is not the correct explanation of A
(c) A is correct but R is not correct
(d) A is not correct but R is correct

20. Assertion (A): TF-IDF is a natural language processing (NLP) technique

used to evaluate the importance of different words in a sentence.
Reason (R): useful in text classification and for helping a machine
learning model read words.
(a) Both A and R are correct and R is the correct explanation of A
(b) Both A and R are correct but R is not the correct explanation of A
(c) A is correct but R is not correct
(d) A is not correct but R is correct

Multiple Choice Questions (MCQs) - Answers:

1. b 2. c 3. a 4. a 5. c

6. b 7. a 8. b 9. a 10. b

11. c 12. d 13. a 14. d 15. a

 16. c 17. d 18. b 19. a 20. a

Short answer type questions:

1. What is the meaning of syntax and semantics in NLP? Answer:
Syntax refers to the grammatical structure of a sentence. Semanticss refers to the meaning
of the sentence.

2. What is the difference between stemming and lemmatization? Answer:

Stemming is a technique used to extract the base form of the words by removing affixes from
them. It is just like cutting down the branches of a tree to its stems. For example, the stem of
the words eating, eats, eaten is eat. Lemmatization is the grouping together of different forms
of the same word. In search queries, lemmatization allows end users to query any version of a
base word and get relevant results.
3. What is a document vector table?
Answer:
Document Vector Table is used while implementing Bag of Words algorithm. In a document
vector table, the header row contains the vocabulary of the corpus and other rows correspond
to different documents. If the document contains a particular word it is represented by 1 and
absence of word is represented by 0 value.
4. What do you mean by corpus?
Answer:
In Text Normalization, we undergo several steps to normalize the text to a lower level. That is,
we will be working on text from multiple documents and the term used for the whole textual
data from all the documents altogether is known as corpus.
5. Differentiate between a script-bot and a smart-bot. (Any 2 differences) Answer:
Script-bot
A scripted chatbot carry even a glimpse of A.I
Script bots are easy to make
Smart-bot
Smart bots are built on NLP and ML.
Smart bots are comparatively difficult to make.
6. What is inverse document frequency?
Answer:
Document Frequency is the number of documents in which the word occurs irrespective of
how many times it has occurred in those documents. In case of inverse document frequency,
we need to put the document frequency in the denominator while the total number of documents
is the numerator. For example, if the document frequency of a word is 2 in a particular
document then its inverse document frequency will be 3/2. (Here no. of documents is 3).
7. What is the significance of converting the text into a common case? Answer:
In Text Normalization, we undergo several steps to normalize the text to a lower level. After
the removal of stop words, we convert the whole text into a similar case, preferably lower case.
This ensures that the case-sensitivity of the machine does not consider same words as different
just because of different cases.
8. Mention some applications of Natural Language Processing. Answer:
Natural Language Processing Applications-
Sentiment Analysis.
Chatbots & Virtual Assistants.
Text Classification.
Text Extraction.
Machine Translation
Text Summarization
Market Intelligence
Auto-Correct
9. What are stop words? Explain with the help of examples. Answer:
These
words do not carry important meaning and are usually removed from texts. It is possible to
remove stop words using Natural Language Toolkit (NLTK), a suite of libraries and programs
for symbolic and statistical natural language processing.
10. Explain the concept of Bag of Words.
Answer:
Bag of Words is a Natural Language Processing model which helps in extracting features out
of the text which can be helpful in machine learning algorithms. In bag of words, we get the
occurrences of each word and construct the vocabulary for the corpus. Bag of Words just
creates a set of vectors containing the count of word occurrences in the document (reviews).
Bag of Words vectors are easy to interpret. The bag of words gives us two things:
i) A vocabulary of words for the corpus

ii) The frequency of these words (number of times it has occurred

in the whole corpus).

Long answer type questions:

1. Why are human languages complicated for a computer to understand? Explain. Answer:
The communications made by the machines are very basic and simple. Human communication
is complex. There are multiple characteristics of the human language that might be easy for a
human to understand but extremely difficult for a computer to understand. For machines it is
difficult to understand our language. Arrangement of the words and meaning - There are rules
in human language. There are nouns, verbs, adverbs, adjectives. A word can be a noun at one
time and an adjective some other time. This can create difficulty while processing by
computers. Multiple Meanings of a word - In natural language, it is important to understand
that a word can have multiple meanings, and the meanings fit into the statement according to
the context of it.
Perfect Syntax, no Meaning - Sometimes, a statement can have a perfectly correct syntax but it
does not mean anything. In Human language, a perfect balance of syntax and semantics is
important for better understanding. These are some of the challenges we might have to face if
we try to teach computers how to understand and interact in human language
2. What are the steps of Text Normalization? Explain them briefly. Answer:
Text Normalization: In Text Normalization, we undergo following steps to normalize the
text to a lower level.
i) Sentence Segmentation - Under sentence segmentation, the whole corpus is divided into
sentences. Each sentence is taken as a different data so now the whole corpus gets reduced to
sentences.
ii) Tokenisation - After segmenting the sentences, each sentence is then further divided into
tokens. Tokens is a term used for any word or number or special character occurring in a
sentence. Under tokenisation, every word, number and special character is considered
separately and each of them is now a separate token.
iii) Removing Stop words, Special Characters and Numbers - In this step, the tokens which
are not necessary are removed from the token list.
iv) Converting text to a common case -After the stop words removal, we convert the whole text
into a similar case, preferably lower case. This ensures that the case-sensitivity of the machine
does not consider same words as different just because of different cases.
v) Stemming - In this step, the remaining words are reduced to their root words.

In other words, stemming is the process in which the affixes of words are removed and the words
are converted to their base form.
vi) Lemmatization - In lemmatization, the word we get after affix removal (also known as
lemma) is a meaningful one. With this we have normalized our text to tokens which are the
simplest form of words present in the corpus. Now it is time to convert the tokens into numbers.
For this, we would use the Bag of Words algorithm.
3. Through a step-by-step process, calculate TFIDF for the given corpus and mention the
word(s) having highest value.
Document 1: We are going to Mumbai Document 2:
Mumbai is a famous place. Document 3: We are going to
a famous place. Document 4: I am famous in Mumbai.
Answer:
Term frequency is the frequency of a word in one document. Term frequency can easily be found
from the document vector table as in that table we mention the frequency of each word of
vocabulary in each document.
We are going to Mumbai is a famous place I am in
1 1 1 1 1 0 0 0 0 0 0 0
0 0 0 0 1 1 1 1 1 0 0 0
1 1 1 1 0 0 1 1 1 0 0 0
0 0 0 0 1 0 0 1 0 1 1 1
Document Frequency is the number of documents in which the word occurs irrespective of how
many times it has occurred in those documents. The document frequency would be:
 We are going to Mumbai is a famous place I am in
2 2 2 2 3 1 2 3 2 1 1 1
Here, the total number of documents are 4, hence inverse document frequency becomes:
We are going to Mumbai is a famous place I am in
4/2 4/2 4/2 4/2 4/3 4/1 4/2 4/3 4/2 4/1 4/1 4/1
The formula of TFIDF for any word W becomes:
TFIDF (W) = TF(W) * log (IDF(W))

4. Explain how AI can play a role in sentiment analysis of human beings? Answer:
The goal of sentiment analysis is to identify sentiment among several posts or
even in the same post where emotion is not always explicitly expressed. Companies use Natural
Language Processing applications, such as sentiment analysis, to identify opinions and
sentiment online to help them understand what customers think about their products and

behind an expressed opinion, which can be extremely relevant in understanding and driving
purchasing decisions.
5. Through a step-by-step process, perform text normalization for the given Corpus:
Raj and Vijay are best friends. They play together with other friends. Raj likes to play football
but Vijay prefers to play online games. Raj wants to be a footballer. Vijay wants to become an
online gamer.
Answer:
Sentence segmentation:
1. Raj and Vijay are best friends.
2. They play together with other friends.
3. Raj likes to play football but Vijay prefers to play online games.
4. Raj wants to be a footballer.
5. Vijay wants to become an online gamer.

Tokenization:
Raj and Vijay are Best friends.
They play together with Other friends.

Raj likes to play Football but Vijay prefers to play online games.

Raj wants to be A footballer.

Vijay wants to become An online gamer.

Removing punctuation and stop words and converting to common case:

Raj and Vijay Best friends.
They play together with Other friends.

Raj likes play Football but Vijay prefers to play online games.
Raj wants footballer.

Vijay wants become online gamer.

Stemming and lemmatization:

Word Stem Word Stem
play play likes like
wants want friends friend
games Game gamer game

Unit 6-MIND MAP