NATURAL LANGUAGE PROCESSING

Module-IV

Mr. Gunti Spandan

Assistant Professor
Department of CSE
GITAM School of Technology (GST)
Bangalore – 561203
Email: sgunti@[Link]

Department of CSE, GST CSEN4141: NLP 1

 TEXT BOOKS and REFERENCES
TEXT BOOKS:

1. Daniel Jurafsky, James H Martin, “Speech and Language Processing: An introduction to Natural Language
Processing, Computational Linguistics and Speech Recognition”, 2/e, Prentice Hall, 2008.
2. C. Manning, H. Schutze, “Foundations of Statistical Natural Language Processing”, MIT Press. Cambridge, MA,
1999.
3. Jacob Eisenstein, Introduction to Natural Language Processing, MIT Press, 2019.

REFERENCE BOOK:

1. Jalaj Thanaki, Python Natural Language Processing: Explore NLP with machine Learning and deep learning
Techniques, Packt, 2017.

Department of CSE, GST CSEN4141: NLP 2

 Introduction
A Vector is a numerical representation of a word or text that captures its meaning and context, allowing machine
learning models to process it

The vector representations are called embeddings, capture semantic relationships based on how words co-occur in
large text corpora.

Vector semantics: The principle that the meaning of a word can be derived from the contexts (neighboring words)
in which it appears.

Vector semantics in NLP represents words as dense vectors in a high-dimensional space, where words with similar
meanings are placed closer together.

Lexical semantics in NLP is the study of the meaning of individual words and their relationships, forming the first
step in semantic analysis.

Department of CSE, GST CSEN4141: NLP 3

 6. Vector Semantics and Embeddings
From Book-1: Chapter 6

• Words that occur in similar contexts tend to have similar meanings.

• This link between similarity in how words are distributed and similarity in what they mean is called the
distributional hypothesis.
• Vector semantics instantiates linguistic hypothesis by learning representations of the meaning of words, called
embeddings, directly from their distributions in texts.

6.1 Lexical Semantics

 Lexical semantics (also known as lexicosemantics), as a subfield of linguistic semantics, is the study of word
meanings.

• We want a model of word meaning to do all sorts of things for us.

Department of CSE, GST CSEN4141: NLP 4

 Continued…
• The model of word meaning should tell us that
some words have similar meanings (cat is similar to dog),
others are antonyms (cold is the opposite of hot),
some have positive connotations (happy)
while others have negative connotations (sad).

• It should represent the fact that the meanings of buy, sell, and pay offer
differing perspectives on the same underlying purchasing event.

Ex: If I buy something from you, you’ve probably sold it to me, and I likely paid you.

• More generally, a model of word meaning should allow us

to draw inferences to address meaning-related tasks like question-answering or dialogue.

Department of CSE, GST CSEN4141: NLP 5

 Continued...
1. Lemmas and Senses:
Ex: Two definitions of the word ‘mouse’ from a dictionary
mouse (N)
1. any of numerous small rodents...
2. a hand-operated device that controls a cursor...
• Here the form mouse is the lemma (also called the citation form).
• The form mouse would also be the lemma for the word mice.
(dictionaries don’t have separate definitions for inflected forms like mice)

• Similarly sing is the lemma for sing, sang, sung.

• In many languages the infinitive form is used as the lemma for the verb,
so Spanish dormir “to sleep” is the lemma for duermes “you sleep”.

• The specific forms sung or carpets or sing or duermes are called wordforms.

Department of CSE, GST CSEN4141: NLP 6

 Continued...
• So, each lemma can have multiple meanings.
(the lemma mouse can refer to the rodent or the cursor control device)
• We call each of these aspects of the meaning of mouse a word sense.

• Lemmas can be polysemous (have multiple senses) and they can make interpretation difficult.
(is someone who types “mouse info” into a search engine looking for a pet or a tool?).

2. Synonymy:

 This is the relationship between word senses.

 For example when one word has a sense whose meaning is identical to a sense of another word, or
nearly identical, we say the two senses of those two words are synonyms.

 Synonyms include such pairs as

couch/sofa vomit/throw up filbert/hazelnut car/automobile
Department of CSE, GST CSEN4141: NLP 7
 Continued...
Formal definition of synonymy (between words rather than senses):
• Two words are synonymous
if they are substitutable for one another in any sentence without changing the truth conditions of the
sentence, the situations in which the sentence would be true.

• We say that the two words have the same propositional meaning.

• Synonyms like car / automobile or water / H2O are truth preserving, but they are still not identical in meaning.
• Indeed, no two words are absolutely identical in meaning.

• Principle of contrast states that a difference in linguistic form is always associated with some difference in
meaning.
Ex: The word H2O is used in scientific contexts and it would be inappropriate
where the word water has to be used.

• The word synonym is therefore used to describe a relationship of approximate or rough synonymy.

Department of CSE, GST CSEN4141: NLP 8

 Continued...
3. Word Similarity:
 While words don’t have many synonyms, most words do have lots of similar words.
 Cat is not a synonym of dog, but cats and dogs are certainly similar words.

 In moving from synonymy to similarity, it will be useful to shift

from talking about relations between word senses (like synonymy)
to relations between words (like similarity).
• Knowing how similar two words are can help in computing
how similar the meaning of two phrases or sentences are.
(this a very important component of natural language understanding tasks like:
question answering, paraphrasing, and summarization).
 Paraphrasing (express the meaning of (something written or spoken) using different words, especially to achieve
greater clarity)
Ex:
Original: Giraffes like Acacia leaves and hay and they can consume 75 pounds of food a day.
Paraphrase: A giraffe can eat up to 75 pounds of Acacia leaves and hay every day.
Department of CSE, GST CSEN4141: NLP 9
 Continued...
• One way of getting values for word similarity is to ask humans to judge how similar one word is to another.
• The SimLex-999 dataset gives values on a scale from 0 to 10,
which range from near-synonyms (vanish, disappear) to pairs that scarcely seem to have anything in
common (hole, agreement):
vanish disappear 9.8
belief impression 5.95
muscle bone 3.65
modest flexible 0.98
hole agreement 0.3

4. Word Relatedness:
The meaning of two words can be related in ways other than similarity.
Ex: coffee and cup (not similar, don’t share practically any features)
-- Coffee is a plant and cup is a manufactured object.
-- They are clearly related (associated by co-participating in an everyday event).

Department of CSE, GST CSEN4141: NLP 10

 Continued...
• The relatedness between words can be observed if they belong to the same semantic field.
• Semantic field is a set of words which cover a particular semantic domain and bear structured relations with each
other.
Ex: Words might be related by being in the semantic field of
hospitals (surgeon, scalpel, nurse, anesthetic, hospital),
restaurants (waiter, menu, plate, food, chef), or
houses (door, roof, kitchen, family, bed).

5. Semantic Frames and Roles:

• A semantic frame is a set of words that denote perspectives or participants in a particular type of event.

Ex: A commercial transaction

-- A kind of event in which one entity trades money to another entity in return for some good or service,
after which the good changes hands or perhaps the service is performed.

Department of CSE, GST CSEN4141: NLP 11

 Continued...
• This event can be encoded lexically by using verbs like
buy (the event from the perspective of the buyer),
sell (from the perspective of the seller),
pay (focusing on the monetary aspect), or nouns like buyer.

• Frames have semantic roles (like buyer, seller, goods, money), and words in a sentence can take on these roles.

• Knowing that buy and sell have this relation makes it possible for a system to know that

a sentence like Sam bought the book from Ling could be paraphrased as
Ling sold the book to Sam, and that
Sam has the role of the buyer in the frame and Ling the seller.

 Being able to recognize such paraphrases is important for question answering.

Department of CSE, GST CSEN4141: NLP 12

 Continued…
6. Connotations:
 Words have affective meanings or connotations.

 The word connotation has different meanings in different fields, but here we use it to mean
the aspects of a word’s meaning that are related to a writer or reader’s emotions, sentiment, opinions, or
evaluations.

Ex: Some words have positive connotations (happy) while others have negative connotations (sad).
Blue is a color, but it is also a word used to describe a feeling of sadness, as in: “She's feeling blue”.
Literal meaning (denotation) is blue. The connotation is sad (for blue).

 Even words whose meanings are similar in other ways can vary in connotation.
Ex: Consider the difference in connotations between fake, knockoff, forgery, on the one hand, and
copy, replica, reproduction on the other, or
innocent (positive connotation) and naive (negative connotation).

Department of CSE, GST CSEN4141: NLP 13

 Continued...
 Some words describe positive evaluation (great, love) and others negative evaluation (terrible, hate).
 Positive or negative evaluation language is called Sentiment.

 Word sentiment plays a role in important tasks like:

sentiment analysis, stance detection, and applications of NLP to the language of
politics and consumer reviews.

 Words vary along three important dimensions of affective meaning:

valence: the pleasantness of the stimulus
arousal: the intensity of emotion provoked by the stimulus
dominance: the degree of control exerted by the stimulus
Exs:
Happy or satisfied are high on valence, while unhappy or annoyed are low on valence.
Excited is high on arousal, while calm is low on arousal.
Controlling is high on dominance, while awed or influenced are low on dominance.

Department of CSE, GST CSEN4141: NLP 14

 Continued...
• Each word is thus represented by three numbers, corresponding to its value on each of the three dimensions:
Valence Arousal Dominance
courageous 8.05 5.5 7.38
music 7.67 5.57 6.5
heartbreak 2.45 5.65 3.58
cub 6.71 3.95 4.24

• Osgood et al. (1957) noticed that in using these 3 numbers to represent the meaning of a word,
the model was representing each word as a point in a three dimensional space,
a vector whose three dimensions corresponded to the word’s rating on the three scales.

• The idea that word meaning could be represented as a point in space

(e.g., that part of the meaning of heartbreak can be represented as the point [2:45;5:65;3:58])
was the first expression of the vector semantics models.

Department of CSE, GST CSEN4141: NLP 15

 6.2 Vector Semantics
• Vectors semantics is the standard way to represent word meaning in NLP, helping us model many of the aspects of
word meaning.

• Osgood’s idea:
Use a point in three-dimensional space to represent the connotation of a word.
Proposal by linguists like Joos, Harris, and Firth:
Define the meaning of a word by its distribution in language use, meaning its neighboring words or
grammatical environments.

• Their idea was that two words that occur in very similar distributions (whose neighbouring words are similar)
have similar meanings.
• For example, suppose you didn’t know the meaning of the word ongchoi
but you see it in the following contexts:
(1) Ongchoi is delicious sauteed with garlic.
(2) Ongchoi is superb over rice.
(3) ...ongchoi leaves with salty sauces...

Department of CSE, GST CSEN4141: NLP 16

 Continued...
And suppose that you had seen many of these context words in other contexts:
(4) ...spinach sauteed with garlic over rice...
(5) ...chard stems and leaves are delicious...
(6) ...collard greens and other salty leafy greens

• The fact that ongchoi occurs with words like rice and garlic and delicious and salty,
as do words like spinach, chard, and collard greens might suggest that
ongchoi is a leafy green similar to these other leafy greens.

Vector Semantics:
Represent a word as a point in a multidimensional semantic space that is derived from the distributions
of word neighbours.

 Vectors for representing words are called embeddings.

Department of CSE, GST CSEN4141: NLP 17

 Continued...
• Fig. 6.1 shows a visualization of embeddings learned for sentiment analysis,
showing the location of selected words projected down from 60-dimensional space into a two
dimensional space.

• The distinct regions containing positive words, negative words, and neutral function words.

Department of CSE, GST CSEN4141: NLP 18

 Continued...
• NLP applications like the sentiment classifiers depend on the same words appearing in the training and test sets.
• But by representing words as embeddings, classifiers can assign sentiment as long as it sees some words with
similar meanings.
The most commonly used models:
(1) tf-idf model:
The meaning of a word is defined by a simple function of the counts of nearby words.
This method results in very long vectors that are sparse, i.e. mostly zeros (since most words
simply never occur in the context of others).

(2) word2vec model:

Used for constructing short, dense vectors that have useful semantic properties.

(3) Cosine:
Used to compute semantic similarity, between two words, two sentences, or two documents.
(an important tool in practical applications like question answering, summarization, or automatic essay grading.)

Department of CSE, GST CSEN4141: NLP 19

 6.3 Words and Vectors
6.3.1 Vectors and documents:
Term-document matrix:
Each row represents a word in the vocabulary and each column represents a document from some collection of
documents.

• The term-document matrix of Fig. 6.2 was first defined as part of the vector space model of information retrieval .

• In this model, a document is represented as a count vector, a column in Fig. 6.3.

Department of CSE, GST CSEN4141: NLP 20

 Continued...

• Vector -- just a list or array of numbers.

 ‘As You Like It’ is represented as the list [1,114,36,20] (the first column vector in Fig. 6.3) and
‘Julius Caesar’ is represented as the list [7,62,1,2] (the third column vector).

 A vector space is a collection of vectors, characterized by their dimension.

 In the example in Fig. 6.3, the document vectors are of dimension 4.

 In real term-document matrices, the vectors representing each document would have
dimensionality |V|, the vocabulary size.
Department of CSE, GST CSEN4141: NLP 21
 Continued...
• The ordering of the numbers in a vector space indicates different meaningful dimensions on which documents
vary.
• Thus the first dimension for both these vectors corresponds to the number of times the word battle occurs, and
we can compare each dimension,
for example that the vectors for As You Like It and Twelfth Night have similar values (1 and 0, respectively)
for the first dimension.
• Fig. 6.4 shows a visualization in two dimensions corresponding to the words battle and fool.

Department of CSE, GST CSEN4141: NLP 22

 Continued...
Term-document matrices:
• Defined as a means of finding similar documents for the task of document information retrieval.
• Two documents that are similar will tend to have similar words, and hence their column vectors will tend to be
similar.
• The vectors for the comedies As You Like It [1,114,36,20] and Twelfth Night [0,80,58,15] look a lot more like each
other (more fools and wit than battles) than they look like Julius Caesar [7,62,1,2] or Henry V [13,89,4,3].

• More generally, the term-document matrix has

|V| rows (one for each word type in the vocabulary) and
D columns (one for each document in the collection).
(vocabulary sizes are generally in the tens of thousands, and the number of documents can be enormous (think
about all the pages on the web)).

Information retrieval (IR): The task of finding the document d from the D retrieval documents in some collection
that best matches a query q.
• IR requires comparison processes like: the tf-idf term weighting, and the cosine similarity metric.

Department of CSE, GST CSEN4141: NLP 23

 Continued...
6.3.2 Words as Vectors: Document Dimensions
• Documents can be represented as vectors in a vector space.
• Vector semantics can also be used to represent the meaning of words.
• We do this by associating each word with a word vector— a row vector rather than a column vector, hence with
different dimensions (Fig. 6.5).
• The four dimensions of the vector for fool, [36,58,1,4], correspond to the four Shakespeare plays.
• Word counts in the same four dimensions are used to form the vectors for the other 3 words:
wit, [20,15,2,3]; battle, [1,0,7,13]; and good [114,80,62,89].

Department of CSE, GST CSEN4141: NLP 24

 Continued...
• For documents,
similar documents have similar vectors, because similar documents tend to have similar words.

• This same principle applies to words:

similar words have similar vectors because they tend to occur in similar documents.

• The term-document matrix thus lets us represent the meaning of a word by the documents it tends to occur in.

6.3.3 Words as Vectors: Word Dimensions

 An alternative to using the term-document matrix:
Represent words as vectors of document counts, in which the columns are labeled by words rather than
documents. (called term-term matrix, word-word matrix or the term-context matrix)

 This matrix is thus of dimensionality |V| x |V| and each cell records the number of times the row (target) word
and the column (context) word co-occur in some context in some training corpus.

Department of CSE, GST CSEN4141: NLP 25

 Continued...
• The context could be the document, in which case the cell represents the number of times the two words appear
in the same document.

• It is most common to use smaller contexts, generally a window around the word,
for example of 4 words to the left and 4 words to the right, in which case the cell represents the number
of times (in some training corpus) the column word occurs in such a ± 4 word window around the row word.

Ex: Some words in their windows:

is traditionally followed by cherry pie, a traditional dessert

often mixed, such as strawberry rhubarb pie. Apple pie
computer peripherals and personal digital assistants. These devices usually
a computer. This includes information available on the internet

Department of CSE, GST CSEN4141: NLP 26

 Continued...
• If we then take every occurrence of each word (say strawberry) and count the context words around it, we get a
word-word co-occurrence matrix.
Fig. 6.6
A simplified subset of the word-word co-occurrence matrix for these four words computed from the Wikipedia
corpus.

• cherry and strawberry are more similar to each other (both pie and sugar tend to occur in their window)
than they are to other words like digital.

• Conversely, digital and information are more similar to each other than to strawberry.

Department of CSE, GST CSEN4141: NLP 27

 6.4 Cosine for measuring similarity
• To measure similarity between two target words v and w,
we need a metric that takes two vectors (of the same dimensionality,
either both with words as dimensions, hence of length |V|,
or both with documents as dimensions as documents, of length |D| and gives a measure of their
similarity.

• The most common similarity metric is the cosine of the angle between the vectors.
• The cosine is based on the dot product operator from linear algebra, also called the inner product:

(1)

• Most metrics for similarity between vectors are based on the dot product.
• The dot product acts as a similarity metric because it will tend to be high just when the two vectors have large
values in the same dimensions.
• Orthogonal vectors (vectors that have zeros in different dimensions) will have a dot product of 0, representing
their strong dissimilarity.
Department of CSE, GST CSEN4141: NLP 28
 Continued...
• This raw dot product, however, has a problem as a similarity metric: it favours long vectors.

The vector length is defined as: ( (2)

• The dot product is higher if a vector is longer, with higher values in each dimension.
• More frequent words have longer vectors, since they tend to co-occur with more words and have higher
co-occurrence values with each of them.
• The raw dot product thus will be higher for frequent words.
• But this is a problem; we’d like a similarity metric that tells us how similar two words are regardless of their
frequency.

Normalized dot product:

We modify the dot product to normalize for the vector length by dividing the dot product by the lengths
of each of the two vectors.

Department of CSE, GST CSEN4141: NLP 29

 Continued...
• This normalized dot product is the same as the cosine of the angle between the two vectors,
following from the definition of the dot product between two vectors a and b:

(3)

• The cosine similarity metric between two vectors v and w is:

Department of CSE, GST CSEN4141: NLP 30

 Continued...
• The cosine value ranges from 1 (for vectors pointing in the same direction),
through 0 (for orthogonal vectors),
to -1 (for vectors pointing in opposite directions).
But since raw frequency values are non-negative, the cosine for these vectors ranges from 0–1.
Computation of Cosine: Computing which of the words cherry or digital is closer in meaning to information,
just using raw counts from the following shortened table:

Department of CSE, GST CSEN4141: NLP 31

 Continued...
The model decides that information is way closer to digital than it is to cherry, a result that seems sensible.

 The model decides that information is way closer to digital than it is to cherry.

Department of CSE, GST CSEN4141: NLP 32

 6.5 TF-IDF: Weighing terms in the vector
TF-IDF: term frequency-inverse document frequency
• The co-occurrence matrices (above) represent each cell by frequencies,
either of words with documents (Fig. 6.5), or words with other words (Fig. 6.6).

• But raw frequency is not the best measure of association between words.
• Raw frequency is very skewed (unbalanced) and not very discriminative.

• Words that occur nearby frequently (maybe pie nearby cherry) are more important than words that only appear
once or twice.
Yet words that are too frequent like the or good— are unimportant.

• Two solutions to balance these two conflicting constraints:

1. tf-idf algorithm (usually used when the dimensions are documents)

2. PPMI algorithm (usually used when the dimensions are words)

Department of CSE, GST CSEN4141: NLP 33

 Continued...
• The tf-idf algorithm is the product of two terms.

 The first is the term frequency: the frequency of the word t in the document d.
• We can just use the raw count as the term frequency:
tft,d = count(t, d) (6.11)

• We squash (flatten or compress) the raw frequency a bit, by using the log10 of the frequency instead.

• That is a word appearing 100 times in a document doesn’t make that word 100 times more likely to be relevant to
the meaning of the document.

• Because we can’t take the log of 0, we normally add 1 to the count:

tft,d = log10(count(t, d)+1) (6.12)

Department of CSE, GST CSEN4141: NLP 34

 Continued...
 Then term occurrence in a document:
0 times -- tf = log10(1) = 0
10 times -- tf = log10 (11)=1.4
100 times -- tf = log10 (101) = 2.004
1000 times -- tf = 3.00044, and so on.

 The second factor in tf-idf is document frequency dft :

dft of a term t is the number of documents it occurs in.

 Idea is to give a higher weight to words that occur only in a few documents.

• Terms that are limited to a few documents are useful for discriminating those documents from the rest of the
collection; terms that occur frequently across the entire collection aren’t as helpful.

• Document frequency is not the same as the collection frequency of a term.

(collection frequency: total number of times the word appears in the whole collection in any document)
Department of CSE, GST CSEN4141: NLP 35
 Continued...
• Consider the two words Romeo and action in the collection of Shakespeare’s 37 plays.

• They have identical collection frequencies (they both occur 113 times in all the plays) but
very different document frequencies (since Romeo only occurs in a single play).
Collection frequency Document frequency
Romeo 113 1
action 113 31

• If our goal is to find documents about the romantic tribulations of Romeo,

the word Romeo should be highly weighted, but not action.

• We emphasize discriminative words like Romeo via the inverse document frequency (or idf term weight).
idf = N/dft
where N is the total number of documents in the collection, and
dft is the number of documents in which term t occurs.

Department of CSE, GST CSEN4141: NLP 36

 Continued...
• The lowest weight of 1 is assigned to terms that occur in all the documents.

• It’s usually clear what counts as a document:

-- in Shakespeare we would use a play;

-- when processing a collection of encyclopedia articles like Wikipedia, the document is a Wikipedia
page;
-- in processing newspaper articles, the document is a single article.

• Because of the large number of documents in many collections, this measure too is usually squashed with a log
function.

(6.13)

Department of CSE, GST CSEN4141: NLP 37

 Continued...
From Shakespeare corpus:

The words ‘good’ and ‘sweet’ occur in all the 37 documents.

Hence the respective idf values are zero.

• The tf-idf weighted value wt,d for word t in document d thus combines
term frequency tft,d (from 6.11 or 6.12) with idf from 6.13:

wt,d = tft,d x idft (6.14)

Department of CSE, GST CSEN4141: NLP 38

 Continued...
• In Fig. 6.9, Eq. 6.12 is used.
• The tf-idf values for the dimension corresponding to the word good have now all become 0;
since this word appears in every document, the tf-idf algorithm leads it to be ignored.

• Similarly, the word fool, which appears in 36 out of the 37 plays, has a much lower weight.

Department of CSE, GST CSEN4141: NLP 39

 Continued...
References for Creating TF-IDF model for a Sample Text (a Collection of Documents):

1. [Link]

2. [Link]
339a4089

3. [Link]

***

Department of CSE, GST CSEN4141: NLP 40

 6.8 Word2vec
Embeddings:
These are short dense vectors with number of dimensions d ranging from 50-1000, rather than the much larger
vocabulary size |V| or number of documents D.

Skip-gram with Negative Sampling (SGNS):

This algorithm is one of two algorithms in a software package called word2vec
(sometimes the algorithm is referred to as word2vec).

The intuition (insight) of skip-gram:

1. Treat the target word and a neighbouring context word as positive examples.
2. Randomly sample other words in the lexicon to get negative samples.
3. Use logistic regression to train a classifier to distinguish those two cases.
4. Use the learned weights as the embeddings.

Department of CSE, GST CSEN4141: NLP 41

 Continued...
6.8.1 The classifier:
• Consider the following sentence with a target word apricot, and assume we’re using a window of 2 context words:
... lemon, a [tablespoon of apricot jam, a] pinch ...
c1 c2 w c3 c4

• Our goal is to train a classifier such that, given a tuple (w, c)

of a target word w paired with a candidate context word c (for example (apricot, jam), or
perhaps (apricot, aardvark)) it will return the probability that c is a real context word
(true for jam, false for aardvark):

P(+|w, c) (6.24)

• The probability that word c is not a real context word for w is just 1 minus Eq. 6.24:
P(- |w, c) = 1 - P(+ | w, c) (6.25)

Department of CSE, GST CSEN4141: NLP 42

 Continued...
• Computing the probability P by Classifier:
This is based on embedding similarity:
a word is likely to occur near the target if its embedding is similar to the target embedding.

• We rely on the intuition that two vectors are similar if they have a high dot product.
Similarity (w, c) ≈ c . w (6.26)
• The dot product c . w is not a probability, it’s just a number ranging from - infinity to infinity.

To turn the dot product into a probability, we’ll use the logistic or sigmoid function σ (x), the fundamental core of
logistic regression:

(6.27)
Now probability that word c is a real context word for target word w as:

(6.28)

Department of CSE, GST CSEN4141: NLP 43

 Continued...
• The sigmoid function returns a number between 0 and 1,
• To make it a probability we’ll also need the total probability of the two possible events
(c is a context word, and c isn’t a context word) to sum to 1.

• We thus estimate the probability that word c is not a real context word for w as:

(6.29)

• Equation 6.28 gives us the probability for one word, but there are many context words in the window.
• Skip-gram makes the simplifying assumption that all context words are independent, allowing us to just multiply
their probabilities:

Department of CSE, GST CSEN4141: NLP 44

 Continued...
• In summary, skip-gram trains a probabilistic classifier that, given a test target word w and its context window of
L words c1:L, assigns a probability based on how similar this context window is to the target word.
• The probability is based on applying the logistic (sigmoid) function to the dot product of the embeddings
of the target word with each context word.
• To compute this probability, we just need embeddings for each target word and context word in the vocabulary.

Department of CSE, GST CSEN4141: NLP 45

 Continued...
• Skip-gram actually stores two embeddings for each word,
one for the word as a target, and one for the word considered as context.
• The parameters need to learn are two matrices W and C,
each containing an embedding for every one of the |V| words in the vocabulary V.

6.8.2 Learning Skip-gram Embeddings

• Skip-gram learns embeddings by starting with random embedding vectors and then
iteratively shifting the embedding of each word w to be
more like the embeddings of words that occur nearby in texts, and
less like the embeddings of words that don’t occur nearby.

• Consider a single piece of training data:

... lemon, a [tablespoon of apricot jam, a] pinch ...
c1 c2 w c3 c4

Department of CSE, GST CSEN4141: NLP 46

 Continued...
• We have a target word w (apricot), and 4 context words in the L = ±2 window, resulting in 4 positive training
instances (on the left below):

positive examples + negative examples –

w cpos w cneg w cneg
apricot tablespoon apricot aardvark apricot seven
apricot of apricot my apricot forever
apricot jam apricot where apricot dear
apricot a apricot coaxial apricot if

• For training a binary classifier, we also need negative examples.

• In fact skip-gram with negative sampling (SGNS) uses more negative examples than positive examples
(with the ratio between them set by a parameter k).

• So for each of these (w, cpos) training instances, we’ll create k negative samples,
each consisting of the target w plus a ‘noise word’ cneg.
Department of CSE, GST CSEN4141: NLP 47
 Continued...
• A noise word is a random word from the lexicon, constrained not to be the target word w.
• The right above shows the setting where k = 2, so for each positive example w, cpos ,
we’ll have 2 negative examples in the negative training set.

• The noise words are chosen according to their weighted unigram frequency pα(w), where α is a weight.

• If we were sampling according to unweighted frequency p(w), it would mean that

with unigram probability p(“the”), we would choose the word the as a noise word,
with unigram probability p(“aardvark”), we would choose aardvark, and so on.

• But in practice it is common to set α = 0.75, i.e. use the weighting p3/4 (w):

(6.32)

Department of CSE, GST CSEN4141: NLP 48

 Continued...
• Setting α = 0.75 gives better performance because it gives rare noise words slightly higher probability: for rare
words, Pα(w) > P(w).

• Consider the probabilities with two events, P(a) = 0.99 and P(b) = 0.01:

(6.33)

• Given the set of positive and negative training instances, and an initial set of embeddings,
the goal of the learning algorithm is to adjust those embeddings to:

Maximize the similarity of the target word, context word pairs (w, cpos ) drawn from the positive examples
Minimize the similarity of the (w, cneg) pairs from the negative examples.

Department of CSE, GST CSEN4141: NLP 49

 Continued...
• If we consider one word/context pair (w, cpos ) with its k noise words cneg1,……,cnegk,
we can express these two goals as the following loss function L to be minimized (hence the - symbol).
Here the first term expresses that:
we want the classifier to assign the real context word cpos a high probability of being a neighbour, and
the second term expresses that:
we want to assign each of the noise words cnegi a high probability of being a non-neighbour,
all multiplied because we assume independence:

(6.34)

Department of CSE, GST CSEN4141: NLP 50

 Continued...
• We minimize this loss function using stochastic gradient descent.
• Fig. 6.14 shows the intuition of one step of learning.

Department of CSE, GST CSEN4141: NLP 51

 Continued...
• The skip-gram model learns two separate embeddings for each word i:
the target embedding wi and the context embedding ci , stored in two matrices,
the target matrix W and the context matrix C.

• It’s common to just add them together, representing word i with the vector wi + ci .

See also:
[Link]

Department of CSE, GST CSEN4141: NLP 52

 6.9 Visualizing Embeddings
• Visualizing embeddings is an important goal in helping understand, apply, and improve these models of word
meaning.

• But how can we visualize a (for example) 100-dimensional vector?

(1) To visualize the meaning of a word w embedded in a space is to list the most similar words to w by sorting the
vectors for all words in the vocabulary by their cosine with the vector for w.

Ex: The 7 closest words to frog using the GloVe embeddings are:
frogs, toad, litoria, leptodactylidae, rana, lizard, and eleutherodactylus.

(2) To use a clustering algorithm to show a hierarchical representation of which words are similar to others in the
embedding space.

Department of CSE, GST CSEN4141: NLP 53

 Continued...
• The below figure uses hierarchical clustering of some embedding vectors for nouns as a visualization method.

(3) To project the 100 dimensions of a word down into 2 dimensions. (Fig. 6.1)

Department of CSE, GST CSEN4141: NLP 54

 6.10 Semantic properties of embeddings
(1) Different types of similarity or association:
 One parameter of vector semantic models is the size of the context window used to collect counts
(relevant to both sparse tf-idf vectors and dense word2vec vectors).
• This is generally between 1 and 10 words on each side of the target word (for a total context of 2-20 words).

• The choice depends on the goals of the representation.

• Shorter context windows tend to lead to representations that are a bit more syntactic, since the information is
coming from immediately nearby words.

 When the vectors are computed from short context windows, the most similar words to a target word w tend to
be semantically similar words with the same parts of speech.

 When vectors are computed from long context windows, the highest cosine words to a target word w tend to be
words that are topically related but not similar.

Department of CSE, GST CSEN4141: NLP 55

 Continued...
• For example Levy and Goldberg showed that using skip-gram
with a window of 2, the most similar words to the word Hogwarts (from the Harry Potter series) were
names of other fictional schools: Sunnydale (from Buffy the Vampire Slayer) or
Evernight (from a vampire series). (from other stories)
with a window of 5, the most similar words to Hogwarts were other words topically related
to the Harry Potter series: Dumbledore, Malfoy, and half-blood. (from the same series)

Two kinds of similarity or association between words:

(1) Two words have first-order co-occurrence (called syntagmatic association)
if they are typically nearby each other.
Ex: wrote is a first-order associate of book or poem.

(2) Two words have second-order co-occurrence (called paradigmatic association)

if they have similar neighbours.
Ex: wrote is a second-order associate of words like said or remarked.

Department of CSE, GST CSEN4141: NLP 56

 Continued...
(2) Analogy/Relational Similarity:
• Another semantic property of embeddings is their ability to capture relational meanings.
The parallelogram model:
Solves simple analogy problems of the form a is to b as a* is to what?.
Problem to a system: apple : tree :: grape : ? (answer is vine)

In the parallelogram model (Fig. 6.15), the vector from the word apple to the word tree (= apple – tree) is added
to the vector for grape; the nearest word to that point is returned.

Department of CSE, GST CSEN4141: NLP 57

 Continued...
Expression Result
(vector king) – vector man + vector woman a vector close to queen
(vector Paris) – vector France + vector Italy a vector close Rome.

• The embedding model extracts representations of relations like MALE-FEMALE, or CAPITAL-CITY-OF, or even
COMPARATIVE/SUPERLATIVE.

• For a a:b::a*:b* problem (given a, b and a*, find b*), the parallelogram method is thus:

with the distance function defined either as cosine or as Euclidean distance.

Department of CSE, GST CSEN4141: NLP 58