Chapter 6 Notes
Natural Language Processing
Natural Language
A natural language is a human language, such as French, Spanish, English, Japanese, etc.
Features of Natural Languages are:
• They are governed by set rules that include syntax, lexicon, and semantics.
• All natural languages are redundant, i.e., the information can be conveyed in multiple ways.
• All natural languages change over time.
Computer Language
Computer languages are languages used to interact with a computer, such as Python, C++, Java, HTML, etc.
Computers require a specific set of instructions to understand human input called programs. Computers
can only process electronic signals in the form of binary language. Natural Language Processing helps to
convert natural language to binary form.
Natural Language Processing
Natural Language Processing or NLP is the sub-field of AI that is focused on enabling computers to analyse,
understand and process human languages to derive meaningful information from human language.
Applications of NLP
• Voice assistants: Voice assistants take our natural speech, process it, and give us an output. They use
NLP to understand natural language and execute tasks efficiently.
For example: Hey Google, set an alarm at 3.30 pm
• Autogenerated captions: Captions are generated by turning natural speech into text in real-time. It is
enhances the accessibility of video content.
For example: Auto-generated captions on YouTube
• Language Translation: This involves the conversion of text or speech from one language to another,
facilitating cross-linguistic communication and fostering global connectivity.
For example: Google Translate
• Sentiment Analysis: Sentiment Analysis is a tool to express an opinion, whether the underlying
sentiment is positive, negative, or neutral. Customer sentiment analysis helps in the automatic
detection of emotions when customers interact with the products, services, or brand. Voice of the
customer analysis helps to analyze customer feedback and gain actionable insights from it. It measures
the gap between what customers expect and what they actually experience when they use the
products or services.
• Text Classification: Text classification is a tool which classifies a sentence or document category-wise.
For example, we can observe news articles containing information on various sectors, including Food,
Sports, and Politics, being categorized through the text classification process.
• Keyword Extraction: Keyword extraction is a tool that automatically extracts the most used, important
words and expressions from a text. It can give valuable insights into people’s opinions about any
business on social media. Customer Service can be improved by using a Keyword extraction tool.
Chatbots
A chatbot is a computer program that's designed to simulate human conversation through voice
commands or text chats or both. It can learn over time how to best interact with humans. It can answer
questions and troubleshoot customer problems, evaluate and qualify prospects, generate sales leads and
increase sales on an ecommerce site.
Stages of Natural Language Processing
The steps involved in NLP are:
Step 1: Lexical Analysis - It is the process of dividing a large chunk of words into structural paragraphs,
sentences, and words. It uses a lexicon which stands for a collection of the various words and phrases used
in a language.
Step 2: Syntactic Analysis (Parsing) - It is the process of checking the grammar of sentences and phrases. It
forms a relationship among words and eliminates logically incorrect sentences.
Step 3: Semantic Analysis - The input text is now checked for meaning, and every word and phrase is
checked for meaningfulness. NLP must understand the context in which words are used.
Step 4: Discourse Integration - It is the process of forming the story of the sentence. Every sentence
should have a relationship with its preceding and succeeding sentences.
Step 5: Pragmatic Analysis - In this stage, sentences are checked for their relevance in the real world.
Pragmatic means practical or logical. It means to discard the actual word meaning/literal meaning and to
take the intended meaning.
Text Processing
Human languages are complicated. Natural Language Processing makes it possible for machines to
understand and speak in Natural Languages just like humans. The language of computers is Numerical. The
conversion of natural language to numerical form takes a few steps.
Text Normalisation
In Text Normalisation, we undergo several steps to normalise the text into tokens which are the simplest
form of words. It includes the following 6 steps.
Step 1 - Sentence Segmentation
The term used for the whole textual data from all the documents altogether is known as corpus. Under
sentence segmentation, the whole corpus is divided into sentences. Each sentence is taken as a different
data.
Step 2 - Tokenization
After segmenting the sentences, each sentence is then further divided into tokens. Token is a term used
for any word or number or special character occurring in a sentence. Under tokenisation, every word,
number and special character is considered separately and each of them is now a separate token.
Step 3 - Removing Stop words, Special Characters and Numbers
In this step, the tokens which are not necessary are removed from the token list.
Stop words are the words which occur very frequently in the corpus but do not add any value to it.
Humans use grammar to make their sentences meaningful for the other person to understand. But
grammatical words do not add any essence to the information which is to be transmitted through the
statement hence they come under stop words. Some examples of stop words are:
The corpus might also have special characters and/or numbers. It depends on the type of corpus that we
are working on whether we should keep them in it or not. For example, if you are working on a document
containing email IDs, then you might not want to remove the special characters and numbers whereas in
some other textual data if these characters do not make sense, then you can remove them along with the
stop words.
Step 4 - Converting Text to a Common Case
After the removal of stop words, we convert the whole text into a similar case, preferably lowercase. This
ensures that the case sensitivity of the machine does not consider the same words with different cases as
different.
Step 5 - Stemming
In this step, the remaining words are reduced to their root words. In other words, stemming is the process
in which the affixes of words are removed and the words are converted to their base form. Stemming does
not take into account whether the stemmed word is meaningful or not. It just removes the affixes hence it
is faster.
For example, studies was reduced to studi after the affix removal which is not a meaningful word.
Step 6 - Lemmatization
Stemming and lemmatization both are alternative processes to each other as the role of both the
processes is same – removal of affixes. But the difference between both of them is that in lemmatization,
the word we get after affix removal (also known as lemma) is a meaningful one. Lemmatization makes
sure that a lemma is a word with meaning and hence it takes a longer time to execute than stemming.
The difference between stemming and lemmatization can be summarized by this example:
Bag of Words Algorithm
Bag of Words is a Natural Language Processing model which helps in extracting features out of the text
which can be helpful in machine learning algorithms. In the bag of words, we get the occurrences of each
word and construct the vocabulary for the corpus.
Calling this algorithm a “bag” of words symbolises that the sequence of sentences or tokens does not
matter. In this case, all we need are the unique words and their frequency.
Step 1 - Text Processing
Here we collect data and pre-process it.
Example: Consider 3 documents having one sentence each.
Document 1: Aman and Avni are stressed
Document 2: Aman went to a therapist
Document 3: Avni went to download a health chatbot
After text normalisation, the text becomes:
Document 1: [aman, and, avni, are, stressed]
Document 2: [aman, went, to, a, therapist]
Document 3: [avni, went, to, download, a, health, chatbot]
Step 2 - Create a Dictionary
Make a list of all the unique words occurring in the corpus (vocabulary) and create a dictionary.
Even though some words are repeated in different documents, they are all written just once as while
creating the dictionary, we create the list of unique words.
Step 3 - Create document vectors
For each document in the corpus, find out how many times the word from the unique list of words has
occurred. In this step, the vocabulary is written in the top row. For each word in the document, if it
matches the vocabulary, put a 1 under it. If the same word appears again, add 1. And if the word does not
occur in that document, put a 0.
Step 4 - Create document vectors for all the documents
This step gives us the document vector table for our corpus.
In the table below, the header row contains the vocabulary of the corpus and three rows correspond to
three different documents.
Term Frequency
Term frequency is the frequency of a word in one document. Term frequency can easily be found in the
document vector table as in that table we mention the frequency of each word of the vocabulary in each
document.
For example, in a document on air pollution, air and pollution would be the words which occur many times
in it. And these words are valuable as they give us some context around the document. These words which
have adequate occurrence in the corpus are said to have some amount of value and are termed as
frequent words. These words mostly talk about the document’s subject and their occurrence is adequate
in the corpus.
But suppose we have 10 documents and all of them talk about different issues. Then words like ‘and’,
‘this’, ‘is’, ‘the’, etc. are the words which occur the most in almost all the documents. Though they are
important for humans as they make the statements understandable to us, for the machine they are a
complete waste as they do not provide us with any information regarding the corpus. Hence, these are
termed as stop words and are mostly removed at the pre-processing stage only.
As the occurrence of words drops further, the value of such words rises. These words are termed as rare or
valuable words. These words occur the least but add the most value to the corpus.
The graph given is a plot of the occurrence of words
versus their value. If the words have the highest
occurrence in all the documents of the corpus (stop
words), they are said to have negligible value. The
frequent words which have adequate occurrence in
the corpus are said to have some amount of value.
Rare or valuable words occur the least but add the
most value to the corpus.
Inverse Document Frequency
Document Frequency is the number of documents in which the word occurs.
Example: Consider 3 documents having one sentence each.
Document 1: Aman and Avni are stressed
Document 2: Aman went to a therapist
Document 3: Avni went to download a health chatbot
The document frequency would be:
We get the inverse document frequency, when we put the document frequency in the denominator while
the total number of documents is the numerator. The inverse document frequency would be:
TFIDF: Term Frequency & Inverse Document Frequency
The formula of TFIDF for any word W is given as
TFIDF(W) = TF(W) * log(IDF(W))
TFIDF converts the words into numbers. These numbers help the computer understand which words are to
be considered while processing the natural language. The higher the value, the more important the word is
for a given corpus. Words that occur in all the documents with high term frequencies have the lowest
values and are considered to be the stop words. For a word to have a high TFIDF value, the word needs to
have a high term frequency but less document frequency which shows that the word is important for one
document but is not a common word for all documents.
Applications of TFIDF
1. Document Classification - Helps in classifying the type and genre of a document.
2. Topic Modelling - It helps in predicting the topic for a corpus.
3. Information Retrieval System - To extract the important information out of a corpus.
4. Stop word filtering - Helps in removing unnecessary words from a text body.