Natural Language Processing

Natural Language Processing

Natural Language Processing, or NLP, is the sub-field of AI that is focused on enabling computers to understand and
process human languages. AI is a subfield of Linguistics, Computer Science, Information Engineering, and Artificial
Intelligence concerned with the interactions between computers and human (natural) languages, in particular how to
program computers to process and analyse large amounts of natural language data.
Applications of Natural Language Processing
 Automatic Summarization: Information overload is a real problem when we need to access a specific,
important piece of information from a huge knowledge base. Automatic summarization is relevant not only for
summarizing the meaning of documents and information, but also to understand the emotional meanings within
the information, such as in collecting data from social media. Automatic summarization is especially relevant
when used to provide an overview of a news item or blog post, while avoiding redundancy from multiple
sources and maximizing the diversity of content obtained.

 Sentiment Analysis: The goal of sentiment analysis is to identify sentiment among several posts or even in the
same post where emotion is not always explicitly expressed. Companies use Natural Language Processing
applications, such as sentiment analysis, to identify opinions and sentiment online to help them understand
what customers think about their products and services (i.e., “I love the new iPhone” and, a few lines later “But
sometimes it doesn’t work well” where the person is still talking about the iPhone) and overall indicators of their
reputation. Beyond determining simple polarity, sentiment analysis understands sentiment in context to help
better understand what’s behind an expressed opinion, which can be extremely relevant in understanding and
driving purchasing decisions.

 Text classification: Text classification makes it possible to assign predefined categories to a document and
organizes it to help you find the information you need or simplify some activities. For example, an application of
text categorization is spam filtering in email.

 Virtual Assistants: Nowadays Google Assistant, Cortana, Siri, Alexa, etc have become an integral part of our
lives. Not only can we talk to them but they also have the abilities to make our lives easier. By accessing our
data, they can help us in keeping notes of our tasks, make calls for us, send messages and a lot more. With the
help of speech recognition, these assistants can not only detect our speech but can also make sense out of it.
According to recent researches, a lot more advancements are expected in this field in the near future.

 Chatbots: A chatbot is a computer program that simulates human conversation through voice commands or
text chats or both. Chatbot, short for chatterbot, is an Artificial Intelligence (AI) feature that can be embedded
and used through any major messaging applications. There are a number of synonyms for chatbot,
including "talkbot," "bot," "IM bot," "interactive agent" or "artificial conversation entity."
Difference between Script-bot and Smart-bot
Script-bot Smart-bot
Script bots are easy to make Smart-bots are flexible and powerful
Script bots work around a script which is Smart bots work on bigger databases and other
programmed in them resources directly
Mostly they are free and are easy to integrate to a Smart bots learn with more data
messaging platform
No or little language processing skills Coding is required to take this up on board
Limited functionality Wide functionality
1
Artificial Intelligence
By: Ajay Mishra Sir
Difference between Human Language and Computer Language
Fluency is a powerful and straightforward concept to express your ability with a language. When it comes to human
languages, it’s the ability to fully communicate your intentions to a human; for computer languages, the ability to fully
communicate your intentions to a computer. For both kinds of languages, the levels are the same:
one cannot communicate; can poorly do it (beginner); can do it (native/fluent), or can brilliantly do it (think book
authors, public speakers, and Software Engineers in a senior level). Both consist
of syntax, semantics and pragmatics functions, found in the core of theoretical linguistics.

 Syntax is the set of rules that govern how language elements are combined to form a valid expression. In human
language, the basic rule is that elements like nouns, verbs and punctuation marks must follow the subject-verb-
object order to produce a valid sentence. In a programming language, to produce a valid conditional statement,
elements such as if, parenthesis, braces, comparison operators (===, <=) must be combined in a certain order
too.

 Semantics is what the language expression evaluates to. In human languages, expressions evaluate to thoughts,
questions, and answer; while computers evaluate expressions to CPU instructions that comprise a program’s
flow.
 Pragmatics is what the language expression evaluates to within its interactional context – which might totally
affect semantics and syntax. In a natural language, the context is essentially built around the cultural aspects of
the speaker, place, time, manner, and several others unpredictable aspects that humans introduce when
communicating. For computer languages, we can see pragmatics as the methodology or implementation used
when approaching a problem: composition vs. inheritance, functional vs. imperative, picking a Design Pattern
etc.

Data Processing
Data processing transforms raw data into meaningful information. Data processing services require skilled
professionals to apply different techniques for analyzing and processing data. For every business organization,
data has become the most important tool to make critical decisions. Technology intervention has helped to
increase the credibility of data analysis. Newer technologies such as Machine Learning are heavily dependent
upon bulk data. As data is the core of these technologies, it has to be presented in a way or format which
these technologies understand. If not provided in the correct format, the algorithms would provide incorrect
analysis and data will not be comprehended correctly. The organization needs to understand the concept of
analysis and needs to focus time on data collection, cleaning, exploring and changing it in the format which is
needed. For the benefit of people who are working on Artificial Intelligence, ML and other advanced
technologies there is a hierarchy which needs to be followed in proper data processing.
Steps to process out the data
1. Text Normalisation: In Text Normalisation, we undergo several steps to normalise the text to a lower
level. Before we begin, we need to understand that in this section, we will be working on a collection of
written text. That is, we will be working on text from multiple documents and the term used for the
whole textual data from all the documents altogether is known as corpus. Not only would we go
through all the steps of Text Normalisation, we would also work them out on a corpus. Let us take a
look at the steps:
 Sentence Segmentation: Under sentence segmentation, the whole corpus is divided into sentences.
Each sentence is taken as a different data so now the whole corpus gets reduced to sentences.

2
Artificial Intelligence
By: Ajay Mishra Sir
  Tokenisation: After segmenting the sentences, each sentence is then further divided into tokens.
Tokens is a term used for any word or number or special character occurring in a sentence. Under
tokenisation, every word, number and special character is considered separately and each of them is
now a separate token.

 Removing Stopwords, Special Characters and Numbers: Stopwords are the words which occur very
frequently in the corpus but do not add any value to it. Humans use grammar to make their sentences
meaningful for the other person to understand. But grammatical words do not add any essence to the
information which is to be transmitted through the statement hence they come under stopwords. Some
examples of stopwords are:

These words occur the most in any given corpus but talk very little or nothing about the context or the
meaning of it. Hence, to make it easier for the computer to focus on meaningful terms, these words are
removed. Along with these words, a lot of times our corpus might have special characters and/or
numbers. Now it depends on the type of corpus that we are working on whether we should keep them
in it or not. For example, if you are working on a document containing email IDs, then you might not
want to remove the special characters and numbers whereas in some other textual data if these
characters do not make sense, then you can remove them along with the stopwords.

3
Artificial Intelligence
By: Ajay Mishra Sir
  Converting text to a common case: After the stopwords removal, we convert the whole text into a
similar case, preferably lower case. This ensures that the case-sensitivity of the machine does not
consider same words as different just because of different cases.

 Stemming: In this step, the remaining words are reduced to their root words. In other words, stemming
is the process in which the affixes of words are removed and the words are converted to their base
form.

Note that in stemming, the stemmed words (words which are we get after removing the affixes) might
not be meaningful. Here in this example as you can see: healed, healing and healer all were reduced to
heal but studies was reduced to studi after the affix removal which is not a meaningful word. Stemming
does not take into account if the stemmed word is meaningful or not. It just removes the affixes hence it
is faster.

 Lemmatization: Stemming and lemmatization both are alternative processes to each other as the role
of both the processes is same – removal of affixes. But the difference between both of them is that in
lemmatization, the word we get after affix removal (also known as lemma) is a meaningful one.
Lemmatization makes sure that lemma is a word with meaning and hence it takes a longer time to
execute than stemming.

As you can see in the same example, the output for studies after affix removal has become study instead
of studi.

Difference between stemming and lemmatization can be summarized by this example:

4
Artificial Intelligence
By: Ajay Mishra Sir
 2. Bag of Words: Bag of Words is a Natural Language Processing model which helps in extracting features out of
the text which can be helpful in machine learning algorithms. In bag of words, we get the occurrences of each
word and construct the vocabulary for the corpus.

This image gives us a brief overview about how bag of words works. Let us assume that the text on the left in
this image is the normalised corpus which we have got after going through all the steps of text processing. Now,
as we put this text into the bag of words algorithm, the algorithm returns to us the unique words out of the
corpus and their occurrences in it. As you can see at the right, it shows us a list of words appearing in the corpus
and the numbers corresponding to it shows how many times the word has occurred in the text body. Thus, we
can say that the bag of words gives us two things:
 A vocabulary of words for the corpus
 The frequency of these words (number of times it has occurred in the whole corpus).
Here calling this algorithm “bag” of words symbolises that the sequence of sentences or tokens does not matter
in this case as all we need are the unique words and their frequency in it.

The step-by-step approach to implement bag of words algorithm

1) Text Normalisation: Collect data and pre-process it.
2) Create Dictionary: Make a list of all the unique words occurring in the corpus. (Vocabulary)
3) Create document vectors: For each document in the corpus, find out how many times the word from
the unique list of words has occurred.
4) Create document vectors for all the documents.

Step 1: Text Normalisation: Collecting data and pre-processing it.

 Document 1: Aman and Anil are stressed
 Document 2: Aman went to a therapist
 Document 3: Anil went to download a health chatbot
Here are three documents having one sentence each. After text normalisation, the text becomes:
 Document 1: [aman, and, anil, are, stressed]
 Document 2: [aman, went, to, a, therapist]
 Document 3: [anil, went, to, download, a, health, chatbot]
Note that no tokens have been removed in the stopwords removal step. It is because we have very little
data and since the frequency of all the words is almost the same, no word can be said to have lesser
value than the other.

Step 2: Create Dictionary: Go through all the steps and create a dictionary i.e., list down all the words which
occur in all three documents:
5
Artificial Intelligence
By: Ajay Mishra Sir
 Dictionary

Note that even though some words are repeated in different documents, they are all written just once as while
creating the dictionary, we create the list of unique words.

Step 3: Create document vector: In this step, the vocabulary is written in the top row. Now, for each word in the
document, if it matches with the vocabulary, put a 1 under it. If the same word appears again, increment the
previous value by 1. And if the word does not occur in that document, put a 0 under it.

Since in the first document, we have words: aman, and, anil, are, stressed. So, all these words get a value of 1
and rest of the words get a 0 value.

Step 4: Repeat for all documents: Same exercise has to be done for all the documents. Hence, the table
becomes:

In this table, the header row contains the vocabulary of the corpus and three rows correspond to three different
documents. Take a look at this table and analyse the positioning of 0s and 1s in it. Finally, this gives us the
document vector table for our corpus. But the tokens have still not converted to numbers. This leads us to the
final steps of our algorithm: TFIDF.

TFIDF: Term Frequency & Inverse Document Frequency:

Bag of words algorithm gives us the frequency of words in each document we have in our corpus. It gives us an
idea that if the word is occurring more in a document, its value is more for that document. For example, if I have
a document on air pollution, air and pollution would be the words which occur many times in it. And these
words are valuable too as they give us some context around the document. But let us suppose we have 10
documents and all of them talk about different issues. One is on women empowerment, the other is on
unemployment and so on. Now air and pollution would not be one of the most occurring words in the whole
and it would not have the highest frequency in all of them.

And, this, is, the, etc. are the words which occur the most in almost all the documents. But these words do not
talk about the corpus at all. Though they are important for humans as they make the statements

6
Artificial Intelligence
By: Ajay Mishra Sir
 understandable to us, for the machine they are a complete waste as they do not provide us with any
information regarding the corpus. Hence, these are termed as stopwords and are mostly removed at the pre-
processing stage only.

Take a look at this graph. It is a plot of occurrence of words versus their value. As you can see, if the words have
highest occurrence in all the documents of the corpus, they are said to have negligible value hence they are
termed as stop words. These words are mostly removed at the pre-processing stage only. Now as we move
ahead from the stopwords, the occurrence level drops drastically and the words which have adequate
occurrence in the corpus are said to have some amount of value and are termed as frequent words. These
words mostly talk about the document’s subject and their occurrence is adequate in the corpus. Then as the
occurrence of words drops further, the value of such words rises. These words are termed as rare or valuable
words. These words occur the least but add the most value to the corpus.
TFIDF stands for Term Frequency and Inverse Document Frequency. TFIDF helps un in identifying the value for
each word. Let us understand each term one by one.

 Term Frequency: Term frequency is the frequency of a word in one document. Term frequency can
easily be found from the document vector table as in that table we mention the frequency of each word
of the vocabulary in each document.

Here, you can see that the frequency of each word for each document has been recorded in the table.
These numbers are nothing but the Term Frequencies!

 Inverse Document Frequency: Now, let us look at the other half of TFIDF which is Inverse Document
Frequency. For this, let us first understand what does document frequency mean. Document Frequency
is the number of documents in which the word occurs irrespective of how many times it has occurred in
those documents. The document frequency for the exemplar vocabulary would be:

7
Artificial Intelligence
By: Ajay Mishra Sir
 Here, you can see that the document frequency of ‘aman’, ‘anil’, ‘went’, ‘to’ and ‘a’ is 2 as they have
occurred in two documents. Rest of them occurred in just one document hence the document
frequency for them is one.
Talking about inverse document frequency, we need to put the document frequency in the denominator
while the total number of documents is the numerator. Here, the total number of documents are 3,
hence inverse document frequency becomes:

Summarising the concept, we can say that:

 Words that occur in all the documents with high term frequencies have the least values and are
considered to be the stopwords.
 For a word to have high TFIDF value, the word needs to have a high term frequency but less document
frequency which shows that the word is important for one document but is not a common word for all
documents.
 These values help the computer understand which words are to be considered while processing the
natural language. The higher the value, the more important the word is for a given corpus.

Applications of TFIDF
 Document Classification: Helps in classifying the type and genre of a document.
 Topic Modelling: It helps in predicting the topic for a corpus.
 Information Retrieval System: To extract the important information out of a corpus.
 Stop word filtering: Helps in removing the unnecessary words out of a text body.

8
Artificial Intelligence
By: Ajay Mishra Sir