Natural Language Processing

BE-NLP
Halima sadia Sadia md Saghir 1
Assistant Professor
Dept. of CSE AIML
Textbooks:
T1. Daniel Jurafsky, James H. and Martin, Speech and Language Processing, Second
Edition, Prentice Hall, 2008.
[Link] D. Manning and Hinrich Schutze, Foundations of Statistical Natural
Language, Processing, MIT Press,1999.
References:
R1. Siddiqui and Tiwary U.S., Natural Language Processing and Information Retrieval, Oxford University
Press, 2008.
R2. Daniel M Bikel and ImedZitouni ― Multilingual natural language processing applications: from theory
to practice, IBM Press, 2013.
R3. Alexander Clark, Chris Fox, Shalom Lappin ― The Handbook of Computational Linguistics and
Natural
Language Processing, John Wiley and Sons, 2012. 2
R4. Nitin Indurkhya and Fred J. Damerau, ―Handbook of Natural Language Processing, Second
Edition,
Chapman and Hall/CRC Press, 2010.
R5. Niel J le Roux and SugnetLubbe, A step by step tutorial: An introduction into R application and
programming.
R6. Steven Bird, Ewan Klein and Edward Loper, Natural language processing with Python: Analyzing text
Course Outline
•Introduction
•Word Level Analysis
•Syntax analysis
•Semantic Analysis
•Pragmatics
•Applications(preferably for Indian regional languages)
3
 Motivation for NLP
• Understand language analysis & generation
• Communication
• Language is a window to the mind
• Data is in linguistic form
• Data can be in Structured (table form), Semi structured (XML form),
Unstructured (sentence form).

4
 Goal of NLP

Fundamental Scientific goal:

Deep understanding of the broad languages

Engineering Goal:
Design, Implement and test systems that can process natural language and
used in wide variety of practical applications
5
NLP stands for Natural Language Processing, which is a part of Computer Science, Human
language, and Artificial Intelligence. It is the technology that is used by machines to
understand, analyse, manipulate, and interpret human's languages. It helps developers to
organize knowledge for performing tasks such as translation, automatic summarization,
Named Entity Recognition NER, speech recognition, relationship extraction, and topic
segmentation.

6
7
8
9
10
Components of NLP
1. Natural Language Understanding (NLU)
Understanding involves the following tasks − (What user says, its intention and meaning) ( AI
Hard problem )
-Mapping the given input in natural language into useful representations.
- Analyzing different aspects of the language.
2. Natural Language Generation (NLG)- Reply should be intelligent and conversational
It is the process of producing meaningful phrases and sentences in the form of natural language
from some internal representation.
It involves −
[Link] planning − It includes retrieving the relevant content from knowledge base.
[Link] planning − It includes choosing required words, forming meaningful phrases,
setting tone of the sentence. 11

[Link] Realization − It is mapping sentence plan into sentence structure.

The NLU is harder than NLG.
12
Module 1: Introduction to NLP
By,
Prachi Shahane
Assistant Professor
Department of Computer Engineering

13
 Topics to be covered
1. Origin & History of NLP; Language, Knowledge and Grammar in language
processing;
2. Stages in NLP; Ambiguities and its types in English and Indian Regional
Languages;
3. Challenges of NLP; Applications of NLP

14
History of NLP
Here, are important events in the history of Natural Language Processing:
1950-NLP started when Alan Turing published an article called “Machine and Intelligence.”
1950-Attempts to automate translation between Russian and English
1960-The work of Chomsky and others on formal language theory and generative syntax
1990-Probabilistic and data-driven models had become quite standard
2000-A Large amount of spoken and textual data become available32

15
Early enthusiasm (1950’s): Machine Translation
Bar-Hillel report (1960) concluded that fully-automatic high-quality translation could not be
accomplished without knowledge (Dictionary + Encyclopedia)

Less ambitious applications (late 1960’s & early 1970’s): Limited success, failed to scale up
Speech recognition
Dialogue (Eliza)

Real world evaluation (late 1970’s – now)

Story understanding (late 1970’s & early 1980’s)

Large scale evaluation of speech recognition, text retrieval, information extraction (1980 – now)
Statistical approaches enjoy more success (first in speech recognition & retrieval,
16 later others)

Current trend: Boundary between statistical and symbolic approaches is disappearing.

We need to use all the available knowledge
Application-driven NLP research (bioinformatics, Web, Question answering
 Knowledge in Speech and Language Processing

• By SLP, we have in mind those computational techniques that process spoken and written
human language, as language.
• What distinguishes these language processing applications from other data processing systems
is their use of knowledge of language.
• Unix wc program
• When used to count bytes and lines, wc is an ordinary data processing application.
• However, when it is used to count the words in a file it requires knowledge about what it means
17
to be a word, and thus becomes a language processing system.
• Both the tasks of being capable of analyzing an incoming audio signal and recovering the
exact sequence of words and generating its response require knowledge about phonetics and
phonology, which can help model how words are pronounced in colloquial speech
• Producing and recognizing the variations of individual words (e.g., recognizing that doors is
plural) requires knowledge about morphology, which captures information about the shape
and behavior of words in context

18
 Language Processing
• Level 1 – Speech sound (Phonetics & Phonology)

• Level 2 – Words & their forms (Morphology, Lexicon)

• Level 3 – Structure of sentences (Syntax, Parsing)

• Level 4 – Meaning of sentences (Semantics)

• Level 5 – Meaning in context & for a purpose (Pragmatics)

• Level 6 – Connected sentence processing in a larger body

19

of text (Discourse)
 Examples of Levels
• L1 : sound
• L2 : Dog - Dog(s), Dog(ged)
Lady – Lad(ies)
Should we store all forms of words in the lexicon?
• L3 : Ram goes to market (right)
goes Ram to the market (wrong)
• L4 : translation from unstructured to structured representation
go : (event)
agent : Ram
20
source : ?
destination : market
• L5 : User situation & context

“Is that water?” – the action to be performed is different in a chemistry lab and
on a dining table.

• L6 : Backward & forward references –

• Coreference resolution

“The man went near the dog. It bit him.”

Often co reference & ambiguity go together as in – 21

“The dog went near the cat. It bit it.”

22
Stages of NLP

1. Phonology and Phonetics: (Processing of sound)

Two words are said to be homophones of each other when they sound the same, spelling may or
may not be the same, but their meanings are different. Such words are understood as part of the
context they are used in a particular sentence.
For Example:
1. Homophones with the same spellings:
Mean: (mathematical average) and Mean: (not nice)
Book: (something to read from) and Book: (making a reservation)
2. Homophones with different spellings:
Week: (a period of seven days) and Weak: (having little physical strength or energy)
Cell: (a prison cell) and Sell: (to give or hand over something in 23
exchange for money)
3. Word boundary detection is a challenge in case of rapid speech:
Please get me a part from it.
As a unit, a part is ambiguous, meaning a piece of something or to get apart (separated).
2. Morphology: (Processing of word forms)
This form of ambiguity arises due to the further processing carried out on the
lexemes (root words) to make them of use in a particular sentence.
For Example:
“I like people who make me nuts.”
Here, the phrase make me nuts is ambiguous as it can be interpreted as two different
situations:
1) I like people who make me nuts (some dish made from peanuts
24
that I like to eat)
or 2) I like people who make me nuts (to make someone very annoyed).
3. Lexical Analysis− It involves identifying and analyzing the structure of words. Lexicon of a
language means the collection of words and phrases in a language. Lexical analysis is dividing
the whole chunk of text into paragraphs, sentences, and words.

4. Syntactic Analysis (Parsing)− It involves analysis of words in the sentence for grammar
and arranging words in a manner that shows the relationship among the words. The sentence
such as “The school goes to boy” is rejected by English syntactic analyzer.

25
5. Semantic Analysis− It draws the exact meaning or the dictionary meaning from the text. The
text is checked for meaningfulness. It is done by mapping syntactic structures and objects in the
task domain. The semantic analyzer disregards sentence such as “hot ice-cream”.
6. Discourse Integration− The meaning of any sentence depends upon the meaning of the
sentence just before it. In addition, it also brings about the meaning of immediately
succeeding sentence.

7. Pragmatic Analysis− During this, what was said is re-interpreted on what it actually
meant. It involves deriving those aspects of language which require real world knowledge.

26
Why NLP is hard

• Natural language is designed to make human communication efficient. Therefore,

• We omit a lot of “common sense” knowledge, which we assume the hearer/reader
possesses
• We keep a lot of ambiguities, which we assume the hearer/reader knows how to
resolve
• This makes EVERY step in NLP hard
27
Ambiguity is a “killer”!
Common sense reasoning is pre-required
John persuaded Bill to buy a TV for himself

28
will will will wills will

29
 Ambiguity in Natural language
Natural language is highly ambiguous .

Ambiguity

• Presence of two or more possible meanings in a passage

• Deliberately creating confusion by using a word or phrase with two or more
meanings in an argument
• Doubtfulness as regards interpretation 30
• Something of doubtful meaning

Ambiguity in speech recognition

The performance of NLP systems including machine translation, automatic question
answering, information retrieval depends on the correct meaning of the text. Biggest
challenge is ambiguity i.e., unclear or open meaning depending on the context of
usage.

Types of Ambiguity are:

• Lexical Ambiguity
• Syntactic Ambiguity

The lexical ambiguity of a word or phrase means having more than one meaning of the
word in the language.
31
e.g. 1. "Aam" in Hindi
2. She bagged two silver medals (Noun); She made a silver speech (Adjective);
His worries had silvered his hair (Verb)
 Syntactic ambiguity is a situation where a sentence may be explained in more
than one way due to ambiguous sentence structure.

Ambiguity in preposition attachment

E.g. I saw the man on the hill with a telescope

Possible Meanings
1. “I saw a man, who was on a hill and had a telescope.”
2. “I used a telescope to see a man who was on a hill.”
3. “I was on a hill, and saw a man who had a telescope.”
4. “I was on a hill, and used a telescope to see a man.”
32
5. “I saw a man, who was on a hill, and the hill had a telescope on it.”
6. “I saw a man, while I was on a hill which had a telescope on it.”
Simplified versions of the above list

1. Man on hill, man has telescope.

2. I have telescope, man on hill.
3. I'm on a hill, man has telescope.
4. I'm on a hill, I use telescope to see man.
5. Man on hill, telescope on hill, I see both.
6. I'm on a hill, telescope is on a hill, I see man.
33
Ambiguity in Natural Language Processing can be removed using:
•Word Sense Disambiguation
•Part of Speech Tagger
•HMM (Hidden Markov Model) Tagger
•Hybrid combination of taggers with machine learning techniques.
Word sense disambiguation (WSD) aims to identify the intended meanings of words
(word senses) in a given context. For a given word and its possible meanings, WSD
categorizes an occurrence of the word in context into one or more of its sense classes.
The features of the context (such as neighbouring words) provide the evidence for
classification. 34

Statistical methods for ambiguity resolution include concepts of Part of Speech (POS)
tagging and the usage of probabilistic grammars in parsing.
flying planes can be dangerous

made her duck

Jokes
why is the teacher wearing sun glasses

because the class is so bright

35
This is a simple sentence

36
What can we do about ambiguity?
• non-probabilistic methods (FSMs for morphology, CKY parsers for syntax) return all possible
analyses.
• probabilistic models (HMMs for POS tagging, PCFGs for syntax) and algorithms (Viterbi,
probabilistic CKY) return the best possible analysis.
But the “best” analysis is only good if our probabilities are accurate.
Where do they come from?

Statistical NLP
• more robust than earlier rule-based methods. 37

• Relevant statistics/probabilities are learned from data.

• Normally requires lots of data about any particular phenomenon.
Ambiguity in NLP
NL has an extremely rich form and structure.
It is very ambiguous. There can be different levels of ambiguity −
[Link] ambiguity−
[Link] Level ambiguity−
[Link] ambiguity−
[Link] ambiguity
38
[Link] ambiguity−
Lexical ambiguity − It is at very primitive level such as word level.
For example, treating the word “board” as noun or verb?

Words that sound the same but have a different meaning

•Write and Right

•Knows and Nose
•Hear and Here
•Red and Read
•Flower and Flour 39

•I and Eye
•Weight and Wait
•A part and Apart
40
41
Syntactic Ambiguity

•They were milking cows.”. Are they Cows that provided Milk, or were they doing
the activity, Milking Cows?
•“They are visiting relatives.“. Are they the persons who visit relatives, or are they
acquainted to each other and visiting?
•“Speaking to you as a child…“. Is the listener a child or the one who speaks?

42
Semantic ambiguity-(Meaning)
Once the word is formed and it’s structure has been detected, sentence processing devotes itself to
meaning extraction.
This occurs when the meaning of the words themselves can be misinterpreted
Even after the syntax and the meanings of the individual words have been resolved, there are two
ways of reading the sentence, which results in an ambiguity

exmple --
“ Car hit the pole while it was moving”
“Aditya and Shraddha got married last month”
“Seema loves her mother and Sriya does too”.( The interpretations can be Sriya loves Seema’s mother
or Sriya likes her own mother)
If the meaning of the words are misinterpreted
“Car hit the pole while it was moving”. 43

They went with the charming men and women”. Were there only
charming men? Were the men and women charming?

The older men and women came to the party.

Ambiguity in NLP
4. Pragmatic ambiguity− (Multiple interpretation)
Pragmatic Analysis is part of the process of extracting information from text. Specifically, it’s the portion
that focuses on taking a structures set of text and figuring out what the actual meaning was.
Pragmatic ambiguity refers to a situation where the context of a phrase gives it multiple
interpretation
Pragmatic ambiguity arises when the statement is not specific, and the context does not provide the
information needed to clarify the statement. Information is missing, and must be inferred.
The below mentioned examples of humorous exchange illustrates the nature of the problem.
Tourist (checking out of the hotel): Waiter, go upstairs to my room and see if my sandals are there; do not be
late; I have to catch the train in 15 minutes. 44

Waiter (running upstairs and coming back panting): Yes sir,they are there.
Clearly, the waiter is falling short of the expectation of the tourist, since he does not understand the
pragmatics of the situation.
5. Referential ambiguity− Referring to something using pronouns. For example, Rima went to Gauri.
She said, “I am tired.” − Exactly who is tired?
One input can mean different meanings.
Many inputs can mean the same thing.
6. Discourse: (Processing of connected text)
Discourse is a coherent structured group of sentences. In this, based on the previous set of sentences, a
hypothesis is built and applied on the upcoming sentences. For example:
Sentence-1: Abhishek was coming dejected from the school
(who is Abhishek: most likely a student?)
Sentence-2: He could not control the class
(who is Abhishek now? Most likely the teacher?) 45

Sentence-3: Teacher should not have made him responsible

(who is Abhishek now? Most likely a student again, albeit a special student-the monitor?)
7. Textual Humour and Ambiguity:
Ambiguity arises in Humour due to the difference in the mindset and sarcastic behaviour. Some
people understand the pun involved in the sentence while other take it seriously and try to get out some
sense out of it logically.
Example of Humour and lexical ambiguity:
Salena: What do you do?
Aditya: I race cars.
Salena: Do you win many races?
Aditya: No, the cars are must faster.
The ambiguity of the words "race cars" (driving cars vs running against cars) 46
and the
two different meanings picked by Salena and Aditya gave rise to the humour.
Natural Language Processing (NLP) Challengess:

•Contextual words and phrases and homonyms

•Synonyms
•Irony and sarcasm
•Ambiguity
•Errors in text or speech
•Colloquialisms and slang
47
•Domain-specific language
•Low-resource languages
•Lack of research and development
Contextual words and phrases and homonyms
The same words and phrases can have different meanings according the context of a sentence and
many words – especially in English – have the exact same pronunciation but totally different
meanings.
For example:
• I ran to the store because we ran out of milk.
• Can I run something past you real quick?
• The house is looking really run down.
• These are easy for humans to understand because we read the context of the sentence and we
understand all of the different definitions.
• while NLP language models may have learned all of the definitions, differentiating
48 between them
in context can present problems.
• ambiguity: a word that could be used as a verb, noun, or adjective.
Homonyms
• two or more words that are pronounced the same but have different definitions – can be problematic
for question answering and speech-to-text applications because they aren’t written in text form.
• Usage of their and there, for example, is even a common problem for humans.
Synonyms
• Synonyms can lead to issues similar to contextual understanding because we use many different
words to express the same idea.
• Furthermore, some of these words may convey exactly the same meaning, while some may be
levels of complexity (small, little, tiny, minute) and different people use synonyms to denote
slightly different meanings within their personal vocabulary.
• So, for building NLP systems, it’s important to include all of a word’s possible meanings and all
possible synonyms. Text analysis models may still occasionally make mistakes, but the more
relevant training data they receive, the better they will be able to understand synonyms.
Irony and sarcasm
Irony and sarcasm present problems for machine learning models because they 49 generally use words
and phrases that, strictly by definition, may be positive or negative, but actually connote the opposite.

Models can be trained with certain cues that frequently accompany ironic or sarcastic phrases, like
“yeah right,” “whatever,” etc.,
Errors in text and speech
Misspelled or misused words can create problems for text analysis.
With spoken language, mispronunciations, different accents, stutters, etc., can be difficult for a
machine to understand.
Colloquialisms and slang
• Informal phrases, expressions, idioms, and culture-specific lingo present a number of problems
for NLP – especially for models intended for broad use.
• Because as formal language, colloquialisms may have no “dictionary definition” at all, and
these expressions may even have different meanings in different geographic areas.
• Furthermore, cultural slang is constantly morphing and expanding, so new words pop up every
day. 50

• This is where training and regularly updating custom models can be helpful, although it
oftentimes requires quite a lot of data.
Domain-specific language
• Different businesses and industries often use very different language.
• An NLP processing model needed for healthcare, for example, would be very different than one
used to process legal documents.
• there are a number of analysis tools trained for specific fields, but extremely niche industries may
need to build or train their own models.
Low-resource languages Many languages, especially those spoken by people with less access to
technology often go overlooked and under processed. For example, by some estimations, there are
over3,000 languages in Africa, alone. There simply isn’t very much data on many of these languages.
Lack of research and development
Machine learning requires A LOT of data to function to its outer limits – billions
51 of pieces of

training data. The more data NLP models are trained on, the smarter they become. That said, data
(and human language!) is only growing by the day, as are new machine learning techniques and
custom algorithms. Al l of the problems above will require more research and new techniques
Challenges in NLP
•Information overload
•Data ambiguities
•Extraction of the relevant and correct information
What does an NLP system need to “know”?
• Language consists of many levels of structure
• Humans fluently integrate all of these in producing/understanding language
• Ideally, so would a computer!

53
Knowledge about language
Knowledge about the world
A way to combine knowledge resources

54
Anaphora Ambiguity

“Seema loves her mother and Sriya does too”.

“The horse ran up the hill. It was very steep. It soon got tired ”

55
Pragmatic Ambiguity

Anaphora Ambiguity
“Seema loves her mother and Sriya does too”.
“The horse ran up the hill. It was very steep. It soon got tired ”

56
Thank You!

57