Module 4 –Semantics

Analysis
Semantics - Lexical Semantics- Word Senses - Relations between Senses - Word Sense
Disambiguation (WSD) – Word Similarity Analysis using Thesaurus and Distributional
methods – Word2vec – fastText word Embedding - Lesk Algorithm – Thematic Roles,
Semantic Role labelling - Pragmatics Analysis - Anaphora Resolution.
Semantic Analysis
• Assigning meanings to the structures created by syntactic analysis.
• Mapping words and structures to particular domain objects in way
consistent with our knowledge of the world.
• Semantic can play an import role in selecting among competing
syntactic analyses and discarding logical analyses.
• I robbed the bank -- bank is a river bank or a financial institution
• We have to decide the formalisms- which will be used in the meaning
representation.

2
Aspects of Semantic Analysis
Semantic Analysis
• Agent • Experiencer • Theme • Source
• Patient • Instrument • Location • Goal
metonymy Homonymy

polysemy
WordNet Synset Relationships
• Antonym: front  back
• Attribute: benevolence  good (noun to adjective)
• Pertainym: alphabetical  alphabet (adjective to noun)
• Similar: unquestioning  absolute
• Cause: kill  die
• Entailment: breathe  inhale
• Holonym: chapter  text (part to whole)
• Meronym: computer  cpu (whole to part)
• Hyponym: plant  tree (specialization)
• Hypernym: apple  fruit (generalization)
 Word Sense
Ways that dictionaries and thesauruses offer for defining senses.

Glosses: textual definitions for each sense.

Glosses are not a formal meaning representation; they are just written for
people.

Ex: Two sense for bank

1. financial institution that accepts deposits and channels the money into
lending activities 2. sloping land (especially the slope beside a body of
water)

Dictionary definitions: Defining a sense through its relationship with other

senses. Formal meaning
WordNet: A Database of Lexical Relations
• WordNet lexical database: English WordNet consists of three separate
databases, one each for nouns and verbs and a third for adjectives
and adverbs.
• Each database contains a set of lemmas, each one annotated with a
set of senses.
• The WordNet 3.0 release has 117,798 nouns, 11,529 verbs, 22,479
adjectives, and 4,481 adverbs.
• The average noun has 1.23 senses, and the average verb has 2.16
senses.
 WordNet: A Database of Lexical Relations

• The set of near-synonyms for a WordNet sense is called a synset (for synonym set);
synsets are an important primitive in WordNet. The entry for bass includes synsets like
{bass6 , bass voice1 , basso2}
WordNet: A Database of Lexical Relations

• WordNet also labels each synset with a lexicographic category drawn from a
semantic field for example the 26 categories for nouns
• These categories are often called supersenses, because they act as coarse
semantic categories or groupings of senses which can be useful when word
senses are too fine-grained
WordNet: Sense Relations in WordNet
WordNet: Sense Relations in WordNet
 WSD Methods
• (1) Supervised Machine Learning
• (2) Thesaurus/Dictionary Methods
• (3) Semi-‐Supervised Learning
• (4) Unsupervised Learning

4
Word Sense Disambiguation
• The task of selecting the correct sense for a word is called Word Sense
Disambiguation, or WSD.
• WSD algorithms take a word in context as input and a fixed inventory
word sense disambiguation WSD of potential word senses and
outputs the correct word sense in context.
• So, Sense-Tag inventories for such WSD algorithms depends on the
task and corresponding datasets.
• Eg: (1) Task: Translation
Dataset: WordNet with Supersenses
 Word Sense Disambiguation – Task and dataset

• all-words task: the system is given entire text and a lexicon with an
inventory of senses for each entry and we have to disambiguate every
word in the text (or sometimes just every content word).
 Word Sense Disambiguation – Task and dataset
• Supervised all-word disambiguation tasks are generally trained from a
semantic concordance, a corpus in which each open-class word in each
sentence is labelled semantically with its word sense from a specific
dictionary or thesaurus, most often WordNet, along with POS tags.

• SemCor is subset corpus from Brown Corpus (having > 226036 words) with
manually tagged WordNet senses.
Example:
‘fruit’ can mean different senses from WordNet: ripened body of plant, yield or consequence of an action.

• Other Sense-Tag corpora are built for SENSEVAL and SemEval WSD tasks.
• Eg: SENSEVAL-3 Task 1 English all-words test dataset
SemEval-13 Task 12 dataset
• (1) Supervised All-word Disambiguation:
• Most-frequent sense i.e, the first sense in WordNet for this wordform.
• one sense per discourse i.e., same sense for a word appearing multiple times
in a text or discourse. This is mostly used for homonymy.
(2) WSD Algorithm using Contextual Embeddings
1-Nearest Neighbor Algorithm
• At training time we pass each sentence in the SemCore labeled
dataset through any contextual embedding (e.g., BERT) resulting in a
contextual sense embedding (vs) for each labeled token in SemCore.

• Testing:
(2) WSD Algorithm using Contextual Embeddings
(2) WSD Algorithm using Contextual Embeddings
• After all, the number of senses that appear in SemCor is only a small
fraction of the words in WordNet. So, what do we do for missing
senses in training data?
• sense embeddings for the other members of its synset:
• hypernym-specific synset embeddings:
• lexicographic (supersense-specific) synset embeddings:
Weaver refers to this concept in the context of Machine Translation.
• Feature Vectors refers to the numeric and nominal values that can encode raw
linguistic information. Two types:
• Collocation feature vectors:
• This refers to those word(s) or phrase(s) in a position-specific relationship with the
target word.
• This includes the word itself, its root form and POS tag.
• Such local lexical and grammatical information will help in accurately isolating the
sense of the target word.
• Bag-of-words feature vectors:
• This refers to the context of target word by a vector of binary features, indicating if
the vocabulary word is present or absent in the context.
• This is regardless of the context word’s exact position from the target word.
• Set of simple features:
• POS tags
• Collocation features of words of different lengths (n-grams)
• weighted average of embeddings
• Supervised classifier (Eg: SVM, Decision Trees, Naïve Bayes) is trained on these
feature vectors to choose the sense for each input word.
Word Sense Disambiguation

Classification
Dan
Jurafsky

Classification: definition

• Input:
• a word w and some features f
• a fixed set of classes C = {c1,
c2,…, cJ}

• Output: a predicted class c∈C

Dan
Jurafsky
Classification Methods:
Supervised Machine Learning
• Input:
• a word w in a text window d (which we’ll call a
“document”)
• a fixed set of classes C = {c1, c2,…, cJ}
• A training set of m hand-‐labeled text windows again
called “documents” (d1,c1),....,(dm,cm)
• Output:
• a learned classifier γ:d  c
2
2
Dan
Jurafsky Classification Methods:
Supervised Machine Learning
• Any kind of classifier
• Naive Bayes
• Logistic regression
• Neural Networks
• Support-‐vector machines
• k-‐Nearest Neighbors

•…
(4) WSD - Wikipedia as a source of training data
• Wikipedia can also be used as a source of sense-labeled data.
• The text referring to a concept in a Wikipedia based article, may contain an explicit
link to the concept’s Wikipedia page, which is named by a unique identifier.
• This link can be used as a sense annotation and these sentences are added to the
training data for a supervised system.
• Algorithms map word vectors in WordNet synset, gloss and related senses against
word vectors in Wikipedia page title, outgoing links and page category.
• The resulting mapping is used to create BabelNet, a large sense-annotated
resource.
• Example: 
 Word Sense Disambiguation

Dictionary and Thesaurus Methods

The Simplified Lesk algorithm
• Labelled corpora is expensive and difficult.
• An alternative class of WSD algorithms, knowledge-based algorithms, rely
on knowledge-based WordNet or dictionary or Thesaurus or such
resources and don’t require labeled data.
• While supervised algorithms generally work better, knowledge-based
methods can be used in languages or domains where only thesauruses or
dictionaries, and no sense-labeled corpora are available.
• The lesk algorithm is the oldest and most powerful knowledge-based WSD
method, and is a useful baseline.
• Lesk is really a family of algorithms that choose the sense whose
dictionary gloss or definition shares the most words with the target word’s
neighborhood OR corpora context.
The Simplified Lesk algorithm
Extended Lesk Algorithm
Extended Lesk Algorithm
• This is also called as minimally supervised learning.
• Commonly used bootstrapping algorithm is Yarowsky algorithm (1995).
Word Sense Disambiguation

Unsupervised Learning
(1) WSD -Word-in-Context Evaluation
• This is an unsupervised WSD algorithm.

• We can think of WSD as a kind of contextualized similarity task, since our

goal is to be able to distinguish the meaning of a word like bass in one
context (playing music) from another context (fishing).

• Here the system is given two sentences, each with the same target word
but in a different sentential context.

• The system must decide whether the target words are used in the same
sense in the two sentences or in a different sense.
 (1) WSD -Word-in-Context Evaluation
• In Word-in-context, first cluster the word senses into coarser clusters,
so that the two sentential contexts for the target word are marked as T
if the two senses are in the same cluster.
(1) WSD -Word-in-Context Evaluation
• WiC clusters all pairs of senses if they are first degree connections in
the WordNet semantic graph, including sister senses, or if they belong
to the same supersense.
• The baseline algorithm to solve the WiC task uses contextual
embeddings like BERT with a simple thresholded cosine.
• The contextual embeddings for the target word is first computed in
each of the two sentences, and then the cosine between them is
computed.
• If it’s above a threshold tuned on a devset, True (the two senses are
the same) flag is returned, else False.
(2) WSD – Word Sense Induction (WSI)
• Another unsupervised WSD disambiguation approach, where senses are
not named, just referred as jth sense of word w.
• Training:

• Testing:

• Frequent clustering method: Agglomerative Clustering for SemEval bakeoff datasets.

WORD SIMILARITY
WORD SIMILARITY
• Vector semantics is the standard way to represent word meaning in NLP

• vector semantics is to represent a word as a point in a multidimensional semantic

space that is derived from the distributions of embeddings of words and word
neighbors.

• Valence: Pleasantness of the stimulus.

• Arousal: Intensity of emotion provoked by the stimulus.
• Dominance: degree of control exerted by the stimulus.

Eg: meaning of heartbreak can be represented as the vector point [2.45, 5.65, 3.58]

Embedding: Vectors for representing words in high-dimensional space.

WORD SIMILARITY
• For example, suppose you didn’t know the meaning of the word ongchoi (a
recent borrowing from Cantonese) but you see it in the following contexts:
(6.1) Ongchoi is delicious sauteed with garlic.
• (6.2) Ongchoi is superb over rice.
• (6.3) Ongchoi leaves with salty sauces...
WORD SIMILARITY
• For example, suppose you didn’t know the meaning of the word ongchoi (a
recent borrowing from Cantonese) but you see it in the following contexts:
(6.1) Ongchoi is delicious sauteed with garlic.
• (6.2) Ongchoi is superb over rice.
• (6.3) Ongchoi leaves with salty sauces...

• And suppose that you had seen many of these context words in other
contexts:
• (6.4) ...spinach sauteed with garlic over rice...
• (6.5) ...chard stems and leaves are delicious...
• (6.6) ...collard greens and other salty leafy greens
WORD SIMILARITY
• For example, suppose you didn’t know the meaning of the word ongchoi (a
recent borrowing from Cantonese) but you see it in the following contexts:
(6.1) Ongchoi is delicious sauteed with garlic.
• (6.2) Ongchoi is superb over rice.
• (6.3) Ongchoi leaves with salty sauces...

• And suppose that you had seen many of these context words in other
contexts:
• (6.4) ...spinach sauteed with garlic over rice...
• (6.5) ...chard stems and leaves are delicious...
• (6.6) ...collard greens and other salty leafy greens
ongchoi is a leafy green similar to these other leafy greens
t-SNE (t-distributed Stochastic Neighbour Embeddings)
 Vectors and documents
• a term-document matrix: each row represents a word in the vocabulary and
each term-document matrix column represents a document from some
collection of documents

DOCUMENTS

WORDS
• A vector space is a collection of vectors, characterized by their dimension
dimension.
• The ordering of the numbers in a vector space indicates different meaningful
dimensions on which documents vary.
• Term-document matrices were originally defined as a means of finding similar
documents for the task of document information retrieval.

• Two documents that are similar will tend to have similar words, and if two
documents have similar words their column vectors will tend to be similar.

• The vectors for the comedies As You Like It [1,114,36,20] and Twelfth Night
[0,80,58,15] look a lot more like each other (more fools and wit than battles)
than they look like Julius Caesar [7,62,1,2] or Henry V [13,89,4,3].
Words as vectors: document dimensions

wit, [20,15,2,3]; battle, [1,0,7,13]; and good = ?; fool= ?

Words as vectors: document dimensions

wit, [20,15,2,3]; battle, [1,0,7,13]; and good [114,80,62,89]; fool, [36,58,1,4]

Words as vectors: word dimensions
Word-word matrix or the term-context matrix: The columns are labeled by words
rather than documents
• Steps to Calculate PMI
“this is a foo bar bar black sheep foo bar bar black sheep foo bar bar
black sheep shep bar bar black sentence”
Step 1: Convert it to tokens
[1] "this" "is" "a" "foo" "bar" "bar" "black" "sheep"
[9] "foo" "bar" "bar" "black"
[ ... and 11 more ]
• Step 2: Count of Words
• In this step we need to calculate the number of times a word occur in
the text. It's simply a raw count.
• this is a foo bar black sheep shep sentence
• 1 11 3 8 4 3 1 1
• Step 3: Create Co-occurence matrix
• Co-occurence matrix shows how many times words co-occur within
the text. The matrix below is 9x9 considering all possible combination
of the "forward" co-occurrences. For example words "foo" and "bar"
appeared 3 times together within the text.
• Compute PMI score: Let's concentrate on one word pair "foo" and
"bar". These two words come together 3 times. Number of words are
23. Hence numerator would be (3/23). Jump to denominator - "foo"
has 3 instances and "bar" has 8. Now we calculate product of their
individual probabilities. Final score is 1.523562.
Retrieval in vector space model
• Query q is represented in the same way or slightly
differently.
• Relevance of di to q: Compare the similarity of query q
and document di.
• Cosine similarity (the cosine of the angle between the
two vectors)

• Cosine is also commonly used in text clustering

107
Cosine for measuring similarity

• To measure the similarity between two target words v and w, we need a metric
that takes two vectors (of the same dimensionality, either both with words as
dimensions OR documents as dimensions).

• By far the most common similarity metric is the cosine of the angle between the
vectors
Cosine for measuring similarity

Vector length
Cosine for measuring similarity
Cosine for measuring similarity
An Example
• A document space is defined by three terms:
• hardware, software, users
• the vocabulary
• A set of documents are defined as:
• A1=(1, 0, 0), A2=(0, 1, 0), A3=(0, 0, 1)
• A4=(1, 1, 0), A5=(1, 0, 1), A6=(0, 1, 1)
• A7=(1, 1, 1) A8=(1, 0, 1). A9=(0, 1, 1)
• If the Query is “hardware and software”
• what documents should be retrieved?

112
An Example (cont.)
• In Boolean query matching:
• document A4, A7 will be retrieved (“AND”)
• retrieved: A1, A2, A4, A5, A6, A7, A8, A9 (“OR”)
• In similarity matching (cosine):
• q=(1, 1, 0)
• S(q, A1)=0.71, S(q, A2)=0.71, S(q, A3)=0
• S(q, A4)=1, S(q, A5)=0.5, S(q, A6)=0.5
• S(q, A7)=0.82, S(q, A8)=0.5, S(q, A9)=0.5
• Document retrieved set (with ranking)=
• {A4, A7, A1, A2, A5, A6, A8, A9}

113
 Word Similarity analysis
using Thesaurus and Distributional methods
Deer - elk
WORDNET