Natural Language Processing

Language is a method of communication with the help of which

we can speak, read and write.
Natural Language Processing (NLP) is a subfield of Computer
Science that deals with Artificial Intelligence (AI), which
enables computers to understand and process human language.
NLP stands for Natural Language Processing, which is a part
of Computer Science, Human language, and Artificial
Intelligence.
It is the technology that is used by machines to understand,
analyse, manipulate, and interpret human's languages.
It helps developers to organize knowledge for performing tasks
such as translation, automatic summarization, Named Entity
Recognition (NER), speech recognition, relationship
extraction, and topic segmentation.

NLP is a component of AI. More than 50 years of existence. It

has its roots in linguistics. NLP uses AI to take real world input,
process and make sense of it.
Two main phases of NLP
1. Data pre-processing
2. Algorithm development

Data pre-processing
It involves cleaning the data and putting the data in workable
format.
Highlighting the features
Step 1: Tokenization
Breaking the text into smaller units.
Step 2: Stop word removal
Removing the words that have no useful information such as
articles, prepositions and other such words.
Step 3: Lemmatiztion or stemming
Words are reduced to their root forms.
4. POS tagging
Words are marked as parts of speech
Example 1:
Janna went back to university
1. Tokenization
[“Janna”,”went”,”back”,”to”,”university”]
2. Stop word removal
[“Janna”, “went”, “university”]

3. Lemmatization or stemming
[“Janna”, “go”, “univers”]

4. PoS tagging
Janna/NN, go/VB, Univers/OBJ
Eg2:
Machine learning is cool
1. Tokenization
[“Machine”, “learning” , “is”, “cool”]

[Link] word removal

[“machine”, “learning”, “cool”]

[Link]
[“machine”, “learn”, “cool”]
Stemming
[“machin”, “learn”, “cool”]

4. PoS tagging
Machine/NN, learn/VB, cool/JJ

PoS tagging for:

John likes animals
John/NN likes/VBZ animals/NNS
NNS- plural noun
Once data has been pre-processed algorithm can be developed.

Various NLP algorithms exist. Two main algorithms are:

1. Rule-based algorithm
[Link] learning based algorithm

Techniques for NLP are:

1. Syntax analysis
2. Semantic analysis
Syntax analysis
It analyses the arrangement of words according to grammar.
a)Parsing – grammatical analysis
input(Dog barked)  NLP algorithmoutput(Dog – noun,
barked-verb)

b)Word segmentation
It takes the text and derives the word forms from it.
Scanned material or handwritten textM/CRecognize words by
white spaces.
c)Sentence breaking
Sentences are broken with the help of periods(dots). Dots acts as
sentence boundaries.
Eg:
The dog barked.| I woke up
d)Morphological analysis
Divides the given word into smallest parts called morphemes.
Eg:
Untestably
Un, test, able, ly - morphemes
This is useful in machine translation and speech recognition.
5) Stemming
Translate a word into its root form
Eg:
Walk, Walked, Walking --- stemmer----walk
Semantic analysis
It analyses the meaning behind the words. NLP algorithm
understand the meaning and structure of the sentence.
a) Word sense disambiguation
It derives the meaning of the word based on the context.
Example:
The sheep is in the pen.
In this sentence, pen means folding , it does not mean the pen used
for writing.
The meaning is fixed as folding by understanding the context
“sheep” in the above sentence.
b)Named entity recognition
Able to differentiate between words that are visually same.
Eg:
Daniel McDonald’s son went to McDonald and ordered a happy
meal.

Person Restaurant
Advantages of NLP
oNLP helps users to ask questions about any subject and get a
direct response within seconds.
oNLP offers exact answers to the question means it does not
offer unnecessary and unwanted information.
oNLP helps computers to communicate with humans in their
languages.
oIt is very time efficient.
oMost of the companies use NLP to improve the efficiency of
documentation processes, accuracy of documentation, and
identify the information from large databases.
Disadvantages of NLP
A list of disadvantages of NLP is given below:
o NLP may not show context.
o NLP is unpredictable
o NLP is unable to adapt to the new domain, and it has a
limited function that's why NLP is built for a single and
specific task only.
Machine Translation
Machine translation is used to translate text or speech from one
natural language to another natural language. Machine translation
is the process of automatically translating content from one
language (the source) to another (the target) without any
human input. Translation was one of the first applications of
computing power, starting in the 1950s.
Machine Translation is evaluated on the BLEU(BiLingual
Evaluation Understudy) score. BLEU is a metric for automatically
evaluating machine-translated text. The score is between 0-1, the
higher the score the better the machine translation.
Machine Translation has been present in the industry for several
decades. Beginning in the 1970s, there were projects to achieve
automatic translation. Over the years three major approaches have
emerged:
 Rule-based Machine Translation (RBMT): 1970-1990
 Statistical Machine Translation (SMT): 1990-2010
 Neural Machine Translation (NMT): 2014-Present
Rule-based Machine Translation (RBMT): 1970-1990
RBMT systems are built on linguistic principles that allow words
to be put in multiple locations and have varied meanings
depending on the context. The RBMT approach is applied to the
following language rules: transfer, analysis, and creation. Human
language experts and programmers develop these rules.
Advantages:
 You do not need bilingual text
 Complete control (new law applicable in all cases)
 Reusable (existing language rules can be transferred when
paired with new languages)
Disadvantages:
 Need good dictionaries
 Manually set rules (requires expertise)

Statistical Machine Translation (SMT): 1990-2010

SMT (Statistical Machine Translation) mainly uses or gets
trained on existing human translation known as bilingual text
corpus. As we have seen above the RBMT system mainly focuses
on word-based translation but the SMT system focuses on
phrase-based translation. The goal of phrase-based translation is
to remove the constraints of word-based translation by translating
entire sequences of words of varying lengths.
The word sequences are called phrases, however, they are
generally not linguistic phrases, but rather phrases discovered
using statistical approaches from bilingual text corpora.
Advantages:
 One SMT is suitable for two languages.
 Offline translation without a dictionary: with a proper
language model, the translation is very smooth.
Disadvantages:
 Requires a bilingual corpus
 It is not appropriate for pairs of languages that have
significant differences in word order.
Neural Machine Translation (NMT): 2014-Present
NMT is a popular and widely used translation service that
incorporates an end-to-end approach for automatic translation
which overcomes the weaknesses of RBMT and SMT methods.
NMT uses the most recent deep learning methods to produce
better translation output than other traditional Machine Translation
solutions.
It is the most recent type of machine translation that employs a
neural network that is closely related to the neurons of the
human brain, allowing it to categorize data into various
groups and layers. NMT is a language translation approach that
tries to incorporate the context of the sentences or paragraphs
rather than individual words.
The NMT system is made up of current multilingual databases
and automated learning mechanisms that contribute to
continuous improvement.
Advantages:
 End-to-end models (no pipeline of specific tasks)
Disadvantages:
 Requires a bilingual corpus
 Sometimes unable to resolve unfamiliar words.
Considerations for Machine Translation
Here are some factors you should consider when choosing an MT
tool for your project:
 Budget:
Neural machine translation is sometimes more expensive to
train than SMT, but the translation quality improvement can
justify the expense.
 Industry:
Some industries involve translating complex and technical
language, which requires the more sophisticated processing
that NMT provides.
 Language Pairs:
SMT works best for certain language pairs. For example,
Latin-based languages with similar syntax and linguistic rules
are the most compatible with machine translation.

 Amount of Content:
NMT requires large quantities of source text to process and
learn from, so it’s not a good fit for small projects.

Customer-Facing vs. Internal Content:

Customer-facing content, such as sales or marketing
materials that reflect brand quality, needs the most
sophisticated combination of machine translation and
human post-editing with qualified translators.
 When cost and time are factors, basic internal documentation
or employee communications can be translated with basic MT.

Which Machine Translation Engine Is Best?

Prominent tech players like Google, Amazon, and Microsoft
use NMT to power their machine translation engines (MTEs).
When we compare different engines, it is essential to understand
that they are constantly learning and improving.
Google Translate

Google Translate was the first MT engine to use neural

language processing and employ machine learning from
repeated use. It’s generally considered one of the leading
machine translation engines based on usage, number of
languages, and integration with searches.
 Amazon Translate

Amazon Translate is closely integrated with Amazon Web

Services (AWS). Some evidence suggests Amazon Translate
provides more accurate translations of certain languages,
notably Chinese.
 Microsoft Translator

Microsoft Translator integrates with products like MS Office

and Skype. This feature provides instant access to
translation in documents and compatible programs.
 Watson Language Translator

The Watson Language Translator is the MT tool from IBM.

It integrates with IBM Watson Data and IBM Watson
Studio. These tools help manage data and build AI models.


 DeepL Translate

DeepL Translate is an independent MT engine produced by

a small company in Germany. Thanks to the company’s
proprietary neural AI, DeepL provides natural-sounding and
nuanced translations. Worldwide use of Deepl has
vastly increased in recent years.
Benefits of Machine Translation (MT)
Machine translation is an indispensable tool in the translation
process. It can be used alone or in combination with
human post-editing. MT offers three primary benefits for your
translation workflows:
Fast Translation Speed
Machine translation can translate millions of words for high-
volume translation projects. But speed isn’t the only benefit! MT
uses AI to get smarter as more content is translated. Plus, MT
can work with a TMS to manage and tag high-volume content.
This helps you stay organized when you need to quickly
translate content into multiple languages.
Excellent Language Selection
Most major machine language translation providers can translate
50-100 languages. These programs are powerful enough to
translate multiple languages at once so you can roll out global
products and documentation updates. MT is well-suited to
language pairs such as English to French or English to Spanish.
Reduced Costs
Even when human translators are needed for post-editing, MT
cuts translation delivery times and costs. MT takes care of the
initial heavy lifting by producing basic but useful translations,
which a human translator can refine and edit. This way, the fini
shed versions will adhere more closely to the text’s original
intent, and the content can be effectively localized.
 Planning and Moving in AI
 The planning in Artificial Intelligence is about the decision
making tasks performed by the robots or computer
programs to achieve a specific goal. The task of coming up
with a sequence of actions that will achieve a goal is called
planning.

 Planning is a search problem that requires to find an efficient

sequence of actions that transform a system from a given
starting state to the goal state

 The execution of planning is about choosing a sequence of actions

with a high likelihood to complete the specific task.
Classical Planning Environments
we consider only environments that are fully observable,
deterministic,
finite, static (change happens only when the agent acts), and
discrete (in time, action, objects,
and effects). These are called classical planning environments.
In contrast, non-classical planning is for partially observable or
stochastic environments and involves a different set of
algorithms and agent designs.
Action Schema
An action schema represents a number of different actions that
can be derived by instantiating the variables p, from, and to to
different constants.
In general, an action schema consists of three parts:

The action name and parameter list-

(1)

for example, Fly(p, from, to) - serves to identify the action.

The precondition is a conjunction of function-free positive

(2)

literals stating what must be true in a state before the action can
be executed. Any variables in the precondition must also appear
in the action's parameter list.
 The effect is a conjunction of function-free literals describing
(3)

how the state changes when the action is executed. A positive

literal P in the effect is asserted to be true in
the state resulting from the action, whereas a negative literal P
is asserted to be false. Variables in the effect must also appear
in the action's parameter list.

Blocks-World planning problem

 The blocks-world problem is known as Sussman Anomaly.
 Non-interleaved planners of the early 1970s were unable to solve
this problem, hence it is considered as anomalous.
 When two subgoals G1 and G2 are given, a non-interleaved
planner produces either a plan for G1 concatenated with a plan for
G2, or vice-versa.
 In blocks-world problem, three blocks labeled as 'A', 'B', 'C' are
allowed to rest on the flat surface. The given condition is that only
one block can be moved at a time to achieve the goal.
 The start state and goal state are shown in the following diagram.
 Components of Planning System
The planning consists of following important steps:
 Choose the best rule for applying the next rule based on the best
available heuristics.
 Apply the chosen rule for computing the new problem state.
 Detect when a solution has been found.
 Detect dead ends so that they can be abandoned and the system’s
effort is directed in more fruitful directions.
 Detect when an almost correct solution has been found.
Goal stack planning
This is one of the most important planning algorithms, which is
specifically used by STRIPS.
 The stack is used in an algorithm to hold the action and satisfy
the goal. A knowledge base is used to hold the current state,
actions.
 Goal stack is similar to a node in a search tree, where the
branches are created if there is a choice of an action.

The important steps of the algorithm are as stated below:

i. Start by pushing the original goal on the stack. Repeat

this until the stack becomes empty. If stack top is a compound
goal, then push its unsatisfied sub goals on the stack.
ii. If stack top is a single unsatisfied goal then, replace it by an
action and push the action’s precondition on the stack to satisfy
the condition.
iii. If stack top is an action, pop it from the stack, execute it and
change the knowledge base by the effects of the action.
iv. If stack top is a satisfied goal, pop it from the stack.
Non-linear planning
This planning is used to set a goal stack and is included in the
search space of all possible subgoal orderings. It handles the goal
interactions by interleaving method.
Advantage of non-Linear planning
Non-linear planning may be an optimal solution with respect to
plan length (depending on search strategy used).
 Disadvantages of Nonlinear planning
It takes larger search space, since all possible goal orderings are
taken into consideration.
 Complex algorithm to understand.
 Algorithm
1. Choose a goal 'g' from the goalset
2. If 'g' does not match the goal state, then
 Choose an operator 'o' whose add-list matches goal g
 Push 'o' on the opstack
 Add the preconditions of 'o' to the goalset
3. While all preconditions of operator on top of opstack are met in
state
 Pop operator o from top of opstack
 state = apply(o, state)
 plan = [plan; o]