Neural NLP Tutorial
Language Modeling:
Models of P(text)
Are These Sentences
• OK?
Jane went to the store.
• store to Jane went the.
• Jane went store.
• Jane goed to the store.
• The store went to Jane.
Engineering Solu-
• tions
Jane went to the
store.
• store to Jane went
•
the.
Jane went } Create a grammar
of the language
store.
• Jane goed to the Consider
store. } morphology and excep-
• The store went to tions
Semantic categories,
Jane. } prefer-
ences
What Can we Do w/
• LMs?
Score sentences:
Jane went to the store . →
high store to Jane went the .
→ low
(same as calculating loss for train-
ing)
• Generate sentences:
while didn’t choose end-of-sentence
symbol:
calculate probability
Calculating the Probability of
a Sentence
Next Con-
Word text
This is a classification problem over the next word!
How do we do
this?!?!
Count-based Language
Models
• Count up the frequency and di-
vide:
[Link]
[Link]/ngrams
Any problem?
Problems w/ Count-based Mod-
els
• Cannot share strength among similar words
she bought a car she bought a bicycle
she purchased a she purchased a bicy-
car cle
• Cannot handle long-distance dependencies
for tennis class he wanted to buy his own racquet
for programming class he wanted to buy his own computer
A Slight Simplification:
Sentence Classification
Models of P(label |
text)
An Example Prediction Problem:
Sentence Classification
very
good
I hate this good
neutral
bad
movie very
bad
very
good
I this movie good
neutral
love bad
very bad
Word Representation
(task-specific)
• Each word has its own 5 elements corresponding
to [very good, good, neutral, bad, very bad]
• “hate” will have a high value for “very bad”, etc.
A First Try:
Bag of Words (BOW)
I hate this
movie
lookup lookup lookup lookup bias scores
+ + + + =
probs
softmax
Adversarial Examples
very
good
I don’t love this good
neutral
movie bad
very bad
very
good
There’s nothing I don’t good
love about this movie neutral
bad
very bad
Combination Fea-
tures
• Does it contain “don’t” and “love”?
• Does it contain “don’t”, “i”, “love”, and
“nothing”?
Idea 1: Neural Networks
I hate this
movie
lookup lookup lookup lookup
scores
some complicated func-
tion to extract combina-
tion features probs
(neural net) softmax
Idea 2: Continuous Bag of
Words (CBOW)
I hate this
movie
lookup lookup lookup lookup
+ +
+
=
W + =
bias scores
What do Our Vectors
Represent?
• Each vector has “features” (e.g. is this an ani-
mate
object? is this a positive word, etc.)
• We sum these features, then use these to
make “discriminative” predictions– P(c|d)
• Still no combination features: only the expres-
sive power of a linear model, but dimension re-
duced
Revisiting LM as a classification
LM as a classification problem into words
CBO Language Model:
Table Model
(Mikolov et al. 2013)
W
• Predict word based on sum of surrounding embeddings
giving a *** at the
lookup lookup lookup
lookup
+ + +
= talk
W = softmax loss
scores probs
Soft-
•
max
Convert scores into probabilities by taking
the exponent and normalizing (softmax)
-3.2 0.002
-2.9 0.003
s= 1.0
2.2 p= 0.329
0.444
0.6 0.090
… …
Skip-gram
(Mikolov et al. 2013)
• Predict each word in the context given the word
talk giving
lookup
a
W =
at
loss
the
Word Embeddings Nice byproduct of
LM
• “What is the female equivalent of king?” is not
easily accessible in many traditional resources
Representing SW in 4 bit
MBTI
I ? ? ? ?
I ? ? ?
23
Representing SW in 1 real numbers?
BMI
obese
BMI>25
BMI
Why not 2*w? h^3? Black art of feature
engineering
24
What makes MBTI a good representation?
25
Visualization of Embeddings
• Reduce high-dimensional embeddings into 2/3D
for visualization (e.g. Mikolov et al. 2013)
Capital?
(Korea,?, Seoul) =
True
If h+?=t
Time Delay Neural Net-
(Waibel et al. 1989)
works
I hate this movie
tanh( tanh( tanh(
W*[x1;x2] W*[x2;x3] W*[x3;x4] These are soft 2-grams!
+b) +b) +b)
probs
soft-
pooling max(
W*h +
b)
Convolutional Networks
(LeCun et al. 1997)
Parameter extraction performs a 2D sweep, not
1D
Pool-
• ingfunction feature-wise
Calculate some reduction
• Max pooling: “Did you see this feature anywhere in
the range?” (most common)
• Average pooling: “How prevalent is this feature over the
entire range”
• k-Max, Dynamic, ...
Weaknesses of
CNNs
• CNNs are great for short-distance feature extractors
• But don't have holistic view of the sentence to cap-
ture long-distance dependencies
Long-distance Dependencies
in Language
• Agreement in number, gender, etc.
He does not have very much confidence in himself.
She does not have very much confidence in her-
self.
• Selectional preference
The reign has lasted as long as the life of the
queen. The rain has lasted as long as the life of the
clouds.
Remember History w/ RNNs
• What does processing a sequence look like?
I hate this
movie
RNN RNN RNN RNN
predict
label
Weaknesses of
RNNs
• Indirect passing of information, credit assignment
more difficult
• Made better by LSTMs/GRUs/etc. but not per-
fect
• Can be slow, due to incremental processing