NATURAL LANGUAGE PROCESSING

Module-V

Mr Gunti Spandan
Assistant Professor
Department of CSE
GITAM School of Technology (GST)
Email: sgunti@[Link]

Department of CSE, GST CSEN4141: NLP 1

 TEXT BOOKS and REFERENCES
TEXT BOOKS:

1. Daniel Jurafsky, James H Martin, “Speech and Language Processing: An introduction to Natural Language
Processing, Computational Linguistics and Speech Recognition”, 2/e, Prentice Hall, 2008.
2. C. Manning, H. Schutze, “Foundations of Statistical Natural Language Processing”, MIT Press. Cambridge, MA,
1999.
3. Jacob Eisenstein, Introduction to Natural Language Processing, MIT Press, 2019.

REFERENCE BOOK:

1. Jalaj Thanaki, Python Natural Language Processing: Explore NLP with machine Learning and deep learning
Techniques, Packt, 2017.

Department of CSE, GST CSEN4141: NLP 2

 21. Coreference Resolution
Book 1: Chapter 21: 21.1 Pg.415-423
• An important component of language understanding is knowing who is being talked about in a text.

• Consider the following passage:

(1) Victoria Chen, CFO of Megabucks Banking, saw her pay jump to $2.3 million, as the 38-year-old became the
company’s president. It is widely known that she came to Megabucks from rival Lotsabucks.

• Each of the underlined phrases refer to a person named Victoria Chen.

• We call linguistic expressions like her or Victoria Chen mentions or referring expressions, and
the discourse entity that is referred to (Victoria Chen) the referent.

• Two or more referring expressions that are used to refer to the same discourse entity are said to corefer;
thus, Victoria Chen and she corefer in (1).

Department of CSE, GST CSEN4141: NLP 3

 Continued…
• Coreference is an important component of natural language understanding.

• A dialogue system that has just told the user:

“There is a 2pm flight on United and a 4pm one on Cathay Pacific”
must know which flight the user means by “I’ll take the second one”.

• A question answering system that uses Wikipedia to answer a question about Marie Curie
must know who she was in the sentence “She was born in Warsaw”.

Discourse Model:
This is used by Natural language understanding systems (and humans) to interpret linguistic expressions.

• It is a mental model (Fig. 21.1) that the understander builds incrementally when interpreting a text.
• It contains representations of the entities referred to in the text, as well as
properties of the entities and relations among them.

Department of CSE, GST CSEN4141: NLP 4

 Continued...
• When a referent is first mentioned in a discourse, we say that a representation for it is evoked into the model.
• Upon subsequent mention, this representation is accessed from the model.

Anaphora:
Reference in a text to an entity that has been previously introduced into the discourse is Anaphora and
the referring expression used is said to be an anaphor (or anaphoric).

Department of CSE, GST CSEN4141: NLP 5

 Continued...
Examples of Anaphoric: the pronouns she and her and the definite NP the 38-year-old (in passage (1))
Antecedent:
It is the anaphor that corefers with a prior mention (in this case Victoria Chen).
Not every referring expression is an antecedent.

Singleton:
An entity that has only a single mention in a text (like Lotsabucks in (1)) is called a singleton.

Coreference Resolution:
Coreference resolution is the task of determining whether two mentions corefer, by which we mean they
refer to the same entity in the discourse model (the same discourse entity).

Coreference chain or a cluster: The set of coreferring expressions.

Ex: In processing (1), a coreference resolution algorithm would need to find at least four coreference
chains, corresponding to the four entities in the discourse model in Fig. 21.1.

Department of CSE, GST CSEN4141: NLP 6

 Continued...
1. { Victoria Chen, her, the 38-year-old, She }
2. { Megabucks Banking, the company, Megabuck }
3. { her pay }
4. { Lotsabucks }

• Mentions can be nested

Ex: The mention her is syntactically part of another mention, her pay,
referring to a completely different discourse entity.

• Coreference resolution thus comprises two tasks (although they are often performed jointly):
(1) identifying the mentions, and (2) clustering them into coreference chains/discourse entities.

• Two mentions corefered if they are associated with the same discourse entity.
• We have to decide which real world entity is associated with this discourse entity.
Ex: the mention Washington might refer to the US state, or the capital city, or
the person George Washington (interpretation of the sentence will be very different for each of these).
Department of CSE, GST CSEN4141: NLP 7
 21.1 Coreference Phenomena: Linguistic Background
Referring expressions:
They are used to evoke and access entities in the discourse model, and talk about linguistic features of
the anaphor/antecedent relation (like number/gender agreement, or properties of verb semantics).

(21.1.1) Types of Referring Expressions:

There are four types.

(1) Indefinite Noun Phrases:

Usually it is marked with the determiner a (or an), but can also be marked by a quantifier
such as some or even the determiner this.
Indefinite reference generally introduces into the discourse context entities that are new to the hearer.

Exs:
a. Mrs. Martin was so very kind as to send Mrs. Goddard a beautiful goose.
b. He had gone round one day to bring her some walnuts.
c. I saw this beautiful cauliflower today.

Department of CSE, GST CSEN4141: NLP 8

 Continued...
(2) Definite Noun Phrases:
• It uses the which refers to an entity that is identifiable to the hearer.
• An entity can be identifiable to the hearer because it has been mentioned previously in the text and thus is
already represented in the discourse model:
Ex:
It concerns a white stallion which I have sold to an officer. But the pedigree of the white stallion was not fully
established.

(3) Pronouns: Another form of definite reference is pronominalization, used for entities that are extremely salient in
the discourse.
Exs:
(a) Emma smiled and chatted as cheerfully as she could.

 Pronouns can also participate in cataphora, in which they are mentioned before their referents are, as in (b).
(b) Even before she saw it, Dorothy had been thinking about the Emerald City every day.
Here, the pronouns she and it both occur before their referents are introduced.

Department of CSE, GST CSEN4141: NLP 9

 Continued…
 Pronouns also appear in quantified contexts in which they are considered to be bound, as in (c).
(c) Every dancer brought her left arm forward.

• Under the relevant reading, her does not refer to some woman in context,
but instead behaves like a variable bound to the quantified expression every dancer.

(4) Demonstrative Pronouns: this and that

They can appear either alone or as determiners (ex: this ingredient, that spice).

(a) I just bought a copy of Thoreau’s Walden. I had bought one five years ago.
That one had been very tattered; this one was in much better condition.

 this NP is ambiguous; in colloquial spoken English, it can be indefinite, as in (21.6), or definite, as in (21.14).

Department of CSE, GST CSEN4141: NLP 10

 Continued...
(21.1.2) Information Status (or Information Structure):
It is the way referring expressions are used to evoke new referents into the discourse (introducing new
information), or access old entities from the model (old information).

Entities can be discourse-new or discourse-old, and indeed it is common to distinguish at least

three kinds of entities informationally.

new NPs:
brand new NPs: these introduce entities that are discourse-new and hearer-new like a fruit or some
walnuts.
unused NPs: these introduce entities that are discourse-new but hearer-old like Hong Kong, Marie Curie,
or
the New York Times.
old NPs:
also called evoked NPs, these introduce entities that already in the discourse model, hence are both
discourse-old and hearer-old, like it in “I went to a new restaurant. It was...”.

Department of CSE, GST CSEN4141: NLP 11

 Continued...
inferrables:
these introduce entities that are neither hearer-old nor discourse-old, but the hearer can infer their
existence by reasoning based on other entities that are in the discourse.
Exs:
(i) I went to a superb restaurant yesterday. The chef had just opened it.
(ii) Mix flour, butter and water. Knead the dough until shiny.

 Neither the chef nor the dough were in the discourse model based on the first sentence of either example,
but the reader can make a bridging inference that these entities should be added to the discourse model
and associated with the restaurant and the ingredients,
based on world knowledge that restaurants have chefs and dough is the result of mixing flour and liquid.

(21.1.3) Complications: Non-Referring Expressions

• Many noun phrases or other nominals are not referring expressions, although they may bear a confusing
superficial resemblance.
Department of CSE, GST CSEN4141: NLP 12
 Continued...
Ex: The NP a car in the following example does not create a discourse referent:
(21.20) Janet doesn’t have a car.
and cannot be referred back to by anaphoric it or the car:
(21.21) *It is a Toyota.
(21.22) *The car is red.

The four common types of structures that are not counted as mentions in coreference tasks:
(and hence complicate the task of mention-detection)

(i) Appositives:

• An appositional structure is a noun phrase that appears next to a head noun phrase, describing the head.
(21.23) Victoria Chen, CFO of Megabucks Banking, saw ...
(21.24) United, a unit of UAL, matched the fares.
• Appositional NPs are not referring expressions, instead functioning as supplementary description of the head NP.

Department of CSE, GST CSEN4141: NLP 13

 Continued...
(ii) Predicative and Prenominal NPs:
• Predicative or attributive NPs describe properties of the head noun.
• In United is a unit of UAL, the NP a unit of UAL describes a property of United, rather than referring to a distinct
entity.
• Thus they are not marked as mentions in coreference tasks.
From the earlier example:
the NPs $2.3 million and the company’s president are Predicate NPs (describing the properties of
(i) her pay jumped to $2.3 million her pay and the 38-year old)
(ii) the 38-year-old became the company’s president

(iii) Expletives:
 Many uses of pronouns like it in English and corresponding pronouns in other languages are not referential.
 Such expletive or pleonastic cases include it is raining, in idioms like hit it off, or in particular syntactic situations
like clefts (i) or extraposition (ii):
(i) It was Emma Goldman who founded Mother Earth.
(ii) It surprised me that there was a herring hanging on her wall.

Department of CSE, GST CSEN4141: NLP 14

 Continued...
(iv) Generics: Generics in non-referring expressions are generic statements or noun phrases that do not refer to a
specific individual or object, but instead express generalizations about entire kinds, categories, or types
Exs:
(i) I love mangos. They are very tasty.
(they refers to the class of mangos in general, not to a particular mango or set of mangos)
(ii) In July in San Francisco you have to wear a jacket.
(the pronoun you can also be used generically)

(21.1.4) Linguistic Properties of the Coreference Relation

(i) Number Agreement:

 Referring expressions and their referents must generally agree in number.

(English she/her/he/him/his/it are singular, we/us/they/them are plural, and
you is unspecified for number.)
 So a plural antecedent like the chefs cannot generally corefer with a singular anaphor like she.
 However, algorithms cannot enforce number agreement too strictly.

Department of CSE, GST CSEN4141: NLP 15

 Continued...
First, semantically plural entities can be referred to by either it or they:
Ex : IBM announced a new machine translation product yesterday. They have been working on it for 20 years.

Second, singular they has become much more common, in which they is used to describe singular individuals,
often useful because they is gender neutral.

(ii) Person Agreement:

 English distinguishes between first, second, and third person, and a pronoun’s antecedent must agree with the
pronoun in person.

 Thus a third person pronoun (he, she, they, him, her, them, his, her, their) must have a third person antecedent
(one of the above or any other noun phrase).

 However, phenomena like quotation can cause exceptions.

In the below, I, my, and she are coreferent:
Ex: “I voted for Nader because he was most aligned with my values,” she said.

Department of CSE, GST CSEN4141: NLP 16

 Continued...
(iii) Gender or Noun Class Agreement:
 In many languages, all nouns have grammatical gender or noun class and pronouns generally agree with the
grammatical gender of their antecedent.

 In English this occurs only with third-person singular pronouns, which distinguish between male (he, him, his),
female (she, her), and non-personal (it) grammatical genders.

 Non-binary pronouns like ze or hir may also occur in more recent texts.

 Knowing which gender to associate with a name in text can be complex, and may require world knowledge about
the individual.
Exs:
(i) Maryam has a theorem. She is exciting. (she=Maryam, not the theorem)
(ii) Maryam has a theorem. It is exciting. (it=the theorem, not Maryam)

Department of CSE, GST CSEN4141: NLP 17

 Continued...
(iv) Binding Theory Constraints:
 The binding theory is a name for syntactic constraints on the relations between a mention and an antecedent in
the same sentence.

• Oversimplifying a bit, reflexive pronouns like himself and herself corefer with the subject of the most immediate
clause that contains them (Ex: (i)), whereas non-reflexives cannot corefer with this subject (Ex: (2)).
Exs:
(i) Janet bought herself a bottle of fish sauce. [herself = Janet]
(ii) Janet bought her a bottle of fish sauce. [her ≠ Janet]

(v) Recency:
Entities introduced in recent utterances tend to be more salient than those introduced from utterances further back.
Thus, in the below, the pronoun it is more likely to refer to Jim’s map than the doctor’s map.

Ex: The doctor found an old map in the captain’s chest. Jim found an even older map hidden on the shelf.
It described an island.

Department of CSE, GST CSEN4141: NLP 18

 Continued...
(vi) Grammatical Role:
 Entities mentioned in subject position are more salient than those in object position, which are in turn more
salient than those mentioned in oblique (indirect) positions.

 The following sentences express roughly the same propositional content,

but the preferred referent for the pronoun he varies with the subject—John in (i) and Bill in (ii).

(i) Billy Bones went to the bar with Jim Hawkins. He called for a burger. [ he = Billy ]
(ii) Jim Hawkins went to the bar with Billy Bones. He called for a burger. [ he = Jim ]

(vii) Verb Semantics:

• Some verbs semantically emphasize one of their arguments, biasing the interpretation of subsequent pronouns.

Compare (i) and (ii).

(i) John telephoned Bill. He lost the laptop.
(ii) John criticized Bill. He lost the laptop.

Department of CSE, GST CSEN4141: NLP 19

 Continued...
• These examples differ only in the verb used in the first sentence,
yet “he” in (i) is typically resolved to John, whereas “he” in (ii) is resolved to Bill.
• This may be due to the link between implicit causality and saliency:
the implicit cause of a “criticizing” event is its object, whereas
the implicit cause of a “telephoning” event is its subject.
• In such verbs, the entity which is the implicit cause is more salient.

(viii) Selectional Restrictions:

 Many other kinds of semantic knowledge can play a role in referent preference.
Ex: The selectional restrictions that a verb places on its arguments can help eliminate referents, as in (i).
(i) I ate the soup in my new bowl after cooking it for hours.

• There are two possible referents for it, the soup and the bowl.
• The verb eat requires that its direct object denote something edible, and
this constraint can rule out bowl as a possible referent.

Department of CSE, GST CSEN4141: NLP 20

 22. Discourse Coherence
From Book-1: Chapter 22
 Movies may start with some event and may have flashbacks. They may not follow the events linearly.
 But coherent structure is not just a fact about movies or works of art.
 Like movies, language does not normally consist of isolated, unrelated sentences, but instead of collocated,
structured, coherent groups of sentences.

 We refer to such a coherent structured group of sentences as a discourse, and

we use the word coherence to refer to the relationship between sentences that makes real discourses
different than just random assemblages of sentences.
Examples of Discourse:
The chapter in a book, a news article, a conversation, a thread on social media, a Wikipedia page, and
your favorite novel.

Coherent discourse:
Creating a text by taking random sentences each from many different sources and pasting them together,
is not a coherent discourse.

Department of CSE, GST CSEN4141: NLP 21

 Continued...
 Real discourses exhibit both local coherence and global coherence.

The three ways in which real discourses are locally coherent:

(1) Sentences or clauses in real discourses are related to nearby sentences in systematic ways.

Ex: John took a train from Paris to Istanbul. He likes spinach.

• This sequence is incoherent because it is unclear to a reader why the second sentence follows the first;
what does liking spinach have to do with train trips?
• In fact, a reader might go to some effort to try to figure out how the discourse could be coherent;
perhaps there is a French spinach shortage?

By contrast, in the following coherent example:

Jane took a train from Paris to Istanbul. She had to attend a conference.

Department of CSE, GST CSEN4141: NLP 22

 Continued...
 The second sentence gives a REASON for Jane’s action in the first sentence.
 Structured relationships like REASON that hold between text units are called coherence relations,
and coherent discourses are structured by many such coherence relations.

(2) In a coherent discourse, some entities are salient and the discourse focuses on them and doesn’t go back and
forth between multiple entities.
 This is called entity-based coherence.
Ex: An incoherent passage (the salient entity seems to wildly swing from
John to Jenny to the piano store to the living room, back to Jenny, then the piano
again).
John wanted to buy a piano for his living room.
Jenny also wanted to buy a piano.
He went to the piano store.
It was nearby.
The living room was on the second floor.
She didn’t find anything she liked.
The piano he bought was hard to get up to that floor.
Department of CSE, GST CSEN4141: NLP 23
 Continued...
 Entity-based coherence models measure this kind of coherence by tracking salient entities across a discourse.

(3) Discourses can be locally coherent by being topically coherent.

 Nearby coherent sentences are generally about the same topic and use the same or similar vocabulary to discuss
these topics.
 This is called lexical cohesion (the sharing of identical or semantically related words in nearby sentences).

Ex: The fact that the words house, chimney, garret, closet, and window—
all of which belong to the same semantic field— appear in the two sentences (given below),
or that they share the identical word shingled, is a cue that the two are tied together as a discourse:

 Before winter I built a chimney, and shingled the sides of my house...

I have thus a tight shingled and plastered house... with a garret and a closet, a large window on each side....

Department of CSE, GST CSEN4141: NLP 24

 Continued...
Global Coherence:

• In addition to the local coherence between adjacent or nearby sentences, discourses also exhibit
global coherence.

Examples:

• Many genres of text are associated with particular conventional discourse structures.

• Academic articles might have sections describing the Methodology or Results.

• Stories might follow conventional plotlines or motifs (ideas or themes).

Department of CSE, GST CSEN4141: NLP 25

 18. Word Senses and WordNet
From Book-1, Chapter: 18
 Words are ambiguous: the same word can be used to mean different things.
Exs: mouse, bank, etc.

Word sense: A sense (or word sense) is a discrete representation of one aspect of the meaning of a word.

WordNet:

• A large online thesaurus —a database that represents word senses—with versions in many languages.
• It also represents relations between senses.
Ex: There is an IS-A relation between dog and mammal (a dog is a kind of mammal) and
a part-whole relation between engine and car (an engine is a part of a car).

Necessity of WordNet:
 Knowing the relation between two senses can play an important role in language understanding.

Department of CSE, GST CSEN4141: NLP 26

 Continued...
Consider the antonymy relation:
 Two words are antonyms if they have opposite meanings, like long and short, or up and down.
 Distinguishing these is quite important for language understanding.
(if a user asks a dialogue agent to turn up the music, it would be unfortunate to instead turn it down).

18.1 Word Senses

• A sense (or word sense) is a discrete representation of one aspect of the meaning of a word.
• We represent each sense with a superscript:
mouse1 and mouse2 , bank1 and bank2

mouse1 : .... a mouse controlling a computer system in 1968.

mouse2 : .... a quiet animal like a mouse
bank1 : ...a bank can hold the investments in a custodial account ...
bank2 : ...as agriculture burgeons on the east bank, the river ...

Department of CSE, GST CSEN4141: NLP 27

 Continued...
18.1 Defining Word senses:
• Dictionaries or thesauruses give textual definitions for each sense called glosses.

Here are the glosses for two senses of bank:

1. financial institution that accepts deposits and channels the money into lending activities
2. sloping land (especially the slope beside a body of water)

• Glosses are not a formal meaning representation; they are just written for people.

(1) Dictionaries:
Definitions of right, left, red, and blood from the American Heritage Dictionary:

right adj. located nearer the right hand esp. being on the right when facing the same direction as the observer.
left adj. located nearer to this side of the body than the right.
red n. the color of blood or a ruby.
blood n. the red liquid that circulates in the heart, arteries and veins of animals.
Department of CSE, GST CSEN4141: NLP 28
 Continued...
Circularity in these definitions:
 Definition of right: makes two direct references to itself
Definition of left: contains an implicit self-reference in the phrase this side of the body,
which presumably means the left side
Definitions of red and blood: refer to each other

 For humans, such entries are useful since the user of the dictionary has sufficient grasp of these other terms.
 Dictionaries often give example sentences along with glosses, and these can again be used to help build a sense
representation.

(2) Thesauruses:

 They define a sense through its relationship with other senses.

• The above definitions make it clear that:
right and left are similar kinds of lemmas that stand in some kind of alternation, or opposition, to one
another.

Department of CSE, GST CSEN4141: NLP 29

 Continued...
• Similarly, we can glean (bring together) that red is a color and that blood is a liquid.

• Sense relations of this sort (IS-A, or antonymy) are explicitly listed in on-line databases like WordNet.
• Given a sufficiently large database of such relations, many applications are quite capable of performing
sophisticated semantic tasks about word senses (even if they do not really know their right from their left).

18.1.2 How many senses do words have?

• Two senses are discrete -- if they have independent truth conditions, different syntactic behavior,
and independent sense relations, or if they exhibit antagonistic meanings.

Consider the meaning of ‘serve’ in the following sentences:

(i) They rarely serve red meat, preferring to prepare seafood.

(ii) He served as U.S. ambassador to Norway in 1976 and 1977.
(iii) He might have served his time, come out and led an upstanding life.

Department of CSE, GST CSEN4141: NLP 30

 Continued...
 There are three distinct senses of serve.
 One practical technique for determining if two senses are distinct is:
to conjoin two uses of a word in a single sentence;
this kind of conjunction of antagonistic readings is called zeugma.

A zeugma is a literary term for using one word to modify two other words, in two different ways.
Ex: “She broke his car and his heart.”
When you use one word to link two thoughts, you're using a zeugma.

18.4 Word Sense Disambiguation

• The task of selecting the correct sense for a word is called word sense disambiguation, or WSD.
• WSD algorithms take as input a word in context and a fixed inventory of potential word senses and
outputs the correct word sense in context.

Department of CSE, GST CSEN4141: NLP 31

 Continued...
18.4.1 WSD: The Task and Datasets
Task setup for WSD:
• The inventory of sense tags depends on the task.

Sense tagging context Sense tag inventory

Translation from English to Spanish the set of different Spanish translations.
Indexing of medical articles the set of MeSH (Medical Subject Headings) thesaurus entries.
Or the set of senses from a resource like WordNet

Some examples for the word bass:

Department of CSE, GST CSEN4141: NLP 32

 Continued...
All-words Task: Disambiguating all the words in a text
• The system is given an entire text and a lexicon with an inventory of senses for each entry and
we have to disambiguate every word in the text.
• The all-words task is similar to part-of-speech tagging, except with a much larger set of tags since each lemma has
its own set.

Ex: SemCor corpus showing the WordNet sense numbers of the tagged words
(standard WSD notation is used)

• the SemCor-based WSD task is to choose the correct sense from the possible senses in WordNet.
• For fruit, this would mean choosing between the correct answer from
fruit1n the ripened reproductive body of a seed plant,
fruit2n yield (an amount of a product) and
fruit3n the consequence of some effort or action.

Department of CSE, GST CSEN4141: NLP 33

 Continued...
Fig. 18.8 sketches the task.

Department of CSE, GST CSEN4141: NLP 34

 Continued...
Heuristics:

(1) Choosing the most frequent sense for each word from the senses in a labeled corpus.

For WordNet, this corresponds to the first sense, since senses in WordNet are generally ordered from
most frequent to least frequent based on their counts in the SemCor sense-tagged corpus.

The most frequent sense baseline can be quite accurate, and is therefore often used as a default,

(2) One sense per discourse:

A word appearing multiple times in a text or discourse often appears with the same sense.

Department of CSE, GST CSEN4141: NLP 35

 19. Semantic Role Labeling
Introduction:
Purchase of stock by XYZ Corporation:
 This purchasing event and its participants can be described by a wide variety of surface forms.
 The event can be described by a verb (sold, bought) or a noun (purchase),
and XYZ Corp can be the syntactic subject (of bought), the indirect object (of sold), or
in a genitive or noun compound relation (with the noun purchase) despite having notionally the same
role in all of them:
• XYZ corporation bought the stock.
• They sold the stock to XYZ corporation.
• The stock was bought by XYZ corporation.
• The purchase of the stock by XYZ corporation...
• The stock purchase by XYZ corporation...

 Semantic roles, express the role that arguments of a predicate take in the event, codified in databases like
PropBank and FrameNet.
 Semantic role labeling, the task of assigning roles to spans in sentences.

Department of CSE, GST CSEN4141: NLP 36

 19.1 Semantic Roles
Consider:
Sasha broke the window.
Pat opened the door.
An event representation of these two sentences would be:

• The roles of the subjects of the verbs break and open are Breaker and Opener respectively.
• These deep roles are specific to each event;
Breaking events have Breakers, Opening events have Openers, and so on.

• Breakers and Openers have something in common.

Department of CSE, GST CSEN4141: NLP 37

 Continued...
Thematic roles: These are a way to capture this semantic commonality between Breakers and Openers.
• We say that the subjects of both these verbs are agents.
• Thus, AGENT is the thematic role that represents an abstract idea.
• Similarly, the direct objects of both these verbs, the BrokenThing and OpenedThing, are both prototypically
inanimate objects that are affected in some way by the action.
• The semantic role for these participants is theme.

Department of CSE, GST CSEN4141: NLP 38

What is Machine Translation?
Automatic translation of text or speech from one language to another using computational methods.

• Began in 1950s with Georgetown-IBM experiment

• Enables cross-cultural communication at scale

• Complex due to ambiguity and cultural context

• Evolution: Rule-based → Statistical → Neural

Main MT Approaches
• Word-for-Word: Direct lexical substitution

• Syntactic Transfer: Parse-based transformation

• Semantic Transfer: Meaning-based translation

• Interlingua: Language-independent representation

• Statistical: Probabilistic models from data

 Transfer Metaphor: Syntactic Transfer
Transforms source syntax into target syntax through structured representations.

• This approach performs translation by analysing the source language syntax and transforming it into the target

language’s structure.

• It typically involves parsing the source sentence (using a context-free grammar), mapping its tree structure into

the equivalent target structure, and generating the target text.

• For example, in English (SVO: Subject-Verb-Object) versus Japanese (SOV: Subject-Object-Verb), syntactic

transfer ensures correct word order.

 Semantic Transfer
Represents source meaning semantically, then generates target text.

• Translation occurs at the level of meaning.

• The source sentence's meaning is represented using semantic frames or logical forms, which are then used
to generate the target text.

• This is effective for resolving structural differences and can handle idiomatic expressions better.

• For example, the German sentence "Ich esse gern" can be represented semantically as LIKE(I, EAT), to
correctly translate as "I like to eat" in English.
 Interlingua Approach

• Translation happens through a universal, language-independent representation called Interlingua.

• The source sentence is first converted to Interlingua, which captures its universal semantic meaning, and
then rendered into the target language.

• Its main advantages include scalability across many languages and preservation of deep meaning.

• Challenges include the difficulty of designing truly universal representations, disambiguating meaning, and
handling varying cultural concepts.
 Statistical Machine Translation
• This approach leverages probabilistic models trained on large parallel corpora (bilingual datasets).

• The translation model learns from example sentence pairs—no hand-crafted linguistic rules are needed.

• Methods like the noisy channel model (using Bayes’ theorem) are central, and large bilingual resources
(such as the Canadian Hansards corpus with 3 million aligned sentences) are important for accuracy.

Why is This Useful?

 Data driven: Rather than writing language-specific rules, the system “figures out” how words and phrases
correspond based on observed data.

 Flexible: As more data is added, the system improves.

 Language models: By using large monolingual corpora for the target language, the translation output
becomes more fluent and natural.
 Statistical Machine Translation

Strengths:
•Can handle language pairs without deep linguistic understanding.

•Adapts to new domains if more data is provided.

Weaknesses:

•Needs large, high-quality bilingual data for training.

•Struggles with rare words, idioms, and long-distance grammatical dependencies.

 Noisy Channel Model

•Key Idea: To find the best English sentence that matches a given French sentence, you use probability
models.

•Main Formula: You want the English e that makes P(e|f) highest, which splits into P(e) (how natural
English is) × P(f|e) (how well the French matches the English).

•Three Components:
• Language Model: Checks if the English is fluent.
• Translation Model: Checks if the translation makes sense.
• Decoder: System that finds the best match.
 Text & Word Alignment
 Sentence Alignment: Matching whole sentences between two languages. Algorithms use things
like sentence length or punctuation.

 Word Alignment: Match up individual words inside matching sentence pairs. This is tricky—what
if one English word matches several French words or vice versa?

 Challenges: Includes tricky cases like idioms ("kick the bucket") or words that don’t translate
(articles, short words).
 IBM Model 1: Foundation
Simplest statistical translation model (Brown et al., 1993).

 The simplest way to build word-to-word translations automatically.

 Every French word matches one English word.

 Model ignores the order of words—just treats them like a bag of words!

 NULL Alignment: Some French words (like “le” or “de”) might not need an English word—so they
align to ‘NULL’.
IBM Model 1: Key Assumptions
 Independence: Each translation happens separately.

 Uniform Alignment: All possible alignments are just as likely to start.

 One-to-One: Each French word has one English match.

 Lexical Only: Only considers word meanings, not word order

IBM Model 1: Mathematical Model

The Formula:
P(f,a∣e)=ε∏t(f∣e)

Probability of French sentence and its alignment, given English sentence.

Uniform alignment , with = English length.

Translation probability = chance to translate French word f from English word e.

IBM Model 1: Example
English (e): the house
French (f): la maison

• Possible alignments: (la→the, maison→house), (la→house, maison→the), etc.

• P(f|e) = Σ P(f, a|e)

• Sum over all possible alignment combinations

• Model learns: t(la|the), t(maison|house) are high

 EM Algorithm: Overview
Expectation-Maximization alternates between two steps to find optimal parameters.

 Goal: Figure out word translations from lots of sentence pairs where alignments aren’t marked.

 Solution: Ask the computer to guess alignments, then improve its guesses over several rounds
(Expectation-Maximization).

 Result: A table of good translations.

Step 1: Initialization
•Start: Assume every English word can translate to any French word with equal
chance.

•Example: If 10,000 French words, every pairing starts at 0.0001 probability.

Step 2: E-Step (Expectation)

•Action: For each sentence, guess how likely each word is aligned based on
current probabilities.

•Formula: ; this is the match score.

Step 3: M-Step (Maximization)
Update translation probabilities by normalizing collected counts

count(f|e) = Σ δ(f, e)

t(f|e) = count(f|e) / Σ count(f'|e)

Action: Update translation probabilities using the guess scores.
Formula: Add up the delta values, then normalize so everything sums to 1 for each
English
. word.

Step 4: Iteration & Convergence

•Repeat E-step & M-step until probabilities stabilize (5-10 rounds).

•End Result: The model’s guess about translations gets really accurate.
Practical Training Example
Corpus: "the house" → "la maison", "the book" → "le livre"

• Iteration 1: All t(f|e) uniform, δ values equal

• Iteration 2: t(la|the) increases, t(maison|house) increases

• Iteration 5: Clear word correspondences emerge

• Convergence: t(la|the) ≈ 0.8, t(maison|house) ≈ 0.9

 IBM Model 1: Limitations
• Bag-of-words: ignores word position and order

• No fertility modeling (one-to-many alignments)

• Cannot capture phrase-level translations

• Foundation for advanced models (IBM 2-5, HMM)

Department of CSE, GST CSEN4141: NLP 56

NLP Applications

Department of CSE, GST CSEN4141: NLP 57

 Sentiment Classification
 Sentiment classification, often called opinion mining, is the process of determining the emotional tone behind a
body of text.
 The goal is to classify the text into predefined sentiment categories (e.g., positive, negative, neutral)
 How it works: Machine learning models are trained on large datasets of labeled text (e.g., movie reviews marked
as "positive" or "negative") to learn patterns associated with different sentiments.

Real-world applications:
 Brand monitoring: Companies analyse social media chatter to understand public perception of their products or
campaigns .
 Customer feedback analysis: Businesses automatically sort through vast amounts of customer reviews and
support interactions to identify areas for improvement.
 Market research: Financial analysts use sentiment analysis on news articles and financial reports to predict
market trends.

Department of CSE, GST CSEN4141: NLP 58

 Text Summarization
• Text summarization is the task of creating a short, accurate summary of a longer document or several
documents
• The goal is to condense the source material while retaining the most important information and meaning.

Types:
• Extractive summarization: Identifies and extracts key sentences or phrases directly from the source text to
form the summary [1].
• Abstractive summarization: Generates new sentences to describe the main points, much like a human would
summarize an article, often requiring more advanced neural network models [1].

Real-world applications:
• News aggregation: Providing a quick snippet of a news story so users can decide if they want to read the full
article.
• Research paper analysis: Helping researchers quickly grasp the core findings of academic papers.
• Meeting transcripts: Generating concise summaries of long meeting notes or recordings.

Department of CSE, GST CSEN4141: NLP 59

 Factoid Question Answering (QA)
• Factoid Question Answering is a specific type of question-answering system designed to retrieve a concise fact
(e.g., a date, a name, a location) in response to a user's question .
• The system analyzes the question to determine the required type of answer (e.g., "who" requires a person's
name) and then searches a knowledge base or a collection of documents to find the precise answer .

Real-world applications:
• Search engines: Providing direct answers at the top of search results pages for factual queries.

• Virtual assistants: Answering quick factual questions asked of devices like smart speakers or phone assistants.

• Enterprise knowledge management: Allowing employees to quickly find specific data points within a
company's vast internal documentation

Department of CSE, GST CSEN4141: NLP 60

Department of CSE, GST CSEN4141: NLP 61