-->

Chapter 4: Representation Learning — Vector

Semantics and Neural Networks
Covered Material Value

Chapters Chapters 6 and 7 (Jurafsky & Martin)

Related Week Week 4

Related Slides / Files [Link], [Link], [Link]

Index

Index (this list)

Brief Content Summary

Abstract

Keywords

1 Introduction

1.1 Motivation for Representation Learning

1.2 Chapter Overview

2 Lexical Semantics

2.1 Words and Word Senses

2.2 Synonymy, Similarity, and Relatedness

2.3 The Distributional Hypothesis

3 Vector Space Models

3.1 Term-Document Matrices

3.2 Term-Term (Word-Word) Matrices

3.3 Weighting Schemes: TF-IDF

4 Measuring Semantic Similarity

4.1 Cosine Similarity

4.2 Properties of Cosine Distance

5 Pointwise Mutual Information

5.1 PMI Definition and Intuition

 5.2 Positive PMI (PPMI)

5.3 Worked Example: Computing PPMI

6 Word Embeddings

6.1 Dense Vector Representations

6.2 Word2Vec and Skip-Gram

6.3 Skip-Gram with Negative Sampling (SGNS)

6.4 Learning the Embeddings

6.5 Properties of Learned Embeddings

6.6 Bias in Embeddings

7 Neural Network Fundamentals

7.1 The Neural Unit

7.2 Activation Functions

7.3 The XOR Problem and Multi-Layer Networks

8 Feedforward Neural Networks

8.1 Architecture

8.2 The Hidden Layer

8.3 The Output Layer and Softmax

8.4 Classification with Neural Networks

9 Training Neural Networks

9.1 Loss Functions: Cross-Entropy

9.2 Gradient Descent

9.3 Computation Graphs

9.4 Backpropagation
9.5 Practical Considerations

10 Neural Language Models

10.1 Feedforward Language Models

10.2 Forward Inference
10.3 Training Neural LMs

11 Conclusion
12 References
Appendix A: Mathematical Derivations
 Glossary

Brief Content Summary

This chapter addresses the fundamental question of how to represent meaning computationally. Beginning with
lexical semantics and the distributional hypothesis ("you shall know a word by the company it keeps"), the
chapter develops vector space models for representing words as high-dimensional vectors based on co-
occurrence patterns. Central weighting schemes including Term Frequency-Inverse Document Frequency (TF-IDF)
and Positive Pointwise Mutual Information (PPMI) are derived and illustrated. The chapter then transitions to
dense, learned representations—word embeddings—focusing on the Word2Vec Skip-Gram model with Negative
Sampling (SGNS). The second half introduces neural network fundamentals: the computational unit, activation
functions (sigmoid, tanh, ReLU), and multi-layer feedforward architectures. Training via cross-entropy loss,
gradient descent, and backpropagation through computation graphs is formally presented. The chapter
concludes with feedforward neural language models, demonstrating how these components integrate into
systems that predict upcoming words from prior context.

Abstract
This chapter provides a rigorous treatment of representation learning in Natural Language Processing (NLP),
spanning both count-based vector semantics and neural network-based approaches. The exposition begins with
lexical semantics, establishing foundational notions of word meaning, synonymy, similarity, and the distributional
hypothesis that underpins modern distributional semantics. Vector space models are constructed via term-
document and term-term matrices, with formal derivations of TF-IDF weighting and cosine similarity as the
primary similarity metric. Pointwise Mutual Information (PMI) and its positive variant (PPMI) are introduced as
association measures superior to raw counts. The chapter then develops Word2Vec's Skip-Gram architecture,
formulating embedding learning as a binary classification task that distinguishes true context words from noise
samples. The transition to neural networks covers computational units with nonlinear activations, the XOR
problem as motivation for hidden layers, and feedforward network architectures for text classification. Training
procedures including cross-entropy loss, stochastic gradient descent, and error backpropagation via computation
graphs are presented with mathematical precision. The chapter culminates with feedforward neural language
models that learn to predict the next word while simultaneously acquiring useful word representations.

Keywords

Vector Semantics; Distributional Hypothesis; TF-IDF; Cosine Similarity; Pointwise Mutual Information; PPMI; Word
Embeddings; Word2Vec; Skip-Gram; Negative Sampling; Neural Networks; Activation Functions; Feedforward
Networks; Softmax; Cross-Entropy Loss; Backpropagation; Computation Graphs; Neural Language Models

1 Introduction
This section establishes the motivation for learning vector representations of words and provides an overview of
the chapter's structure.

1.1 Motivation for Representation Learning

A fundamental challenge in Natural Language Processing is determining how to represent the meaning of words
in a form amenable to computational processing. Traditional symbolic approaches, where words are represented
as atomic symbols with no inherent structure, suffer from a critical limitation: they cannot capture the
relationships between words. For instance, such representations provide no mechanism to determine that "cat"
and "dog" are more similar to each other than either is to "democracy."

The solution developed in this chapter relies on the insight that word meaning can be inferred from usage
patterns. Words that appear in similar contexts tend to have similar meanings—a principle known as the
distributional hypothesis. This insight motivates the construction of vector representations where semantic
relationships manifest as geometric relationships in a vector space.

Two paradigms for constructing such representations are explored:

1. Count-based methods: Words are represented by vectors of co-occurrence counts (or weighted
transformations thereof) with other words or documents. These methods are transparent and interpretable
but yield sparse, high-dimensional vectors.

2. Prediction-based methods: Dense, low-dimensional vectors (embeddings) are learned by training neural
networks to predict words from their contexts. These methods produce representations that capture subtle
semantic and syntactic regularities.

The second half of the chapter introduces the neural network machinery required to understand prediction-based
embeddings and, more broadly, modern deep learning approaches to NLP.

1.2 Chapter Overview

The chapter proceeds as follows:

Sections 2–4 develop count-based vector semantics, covering lexical semantics foundations, vector space
construction via term-document and term-term matrices, TF-IDF weighting, and cosine similarity.

Section 5 introduces Pointwise Mutual Information (PMI) as an association measure that addresses
limitations of raw co-occurrence counts.

Section 6 transitions to dense embeddings, presenting the Word2Vec Skip-Gram model and its training via
negative sampling.

Sections 7–9 provide a self-contained introduction to neural networks, covering computational units,
activation functions, feedforward architectures, and training via backpropagation.

Section 10 synthesizes the preceding material in the context of feedforward neural language models.
2 Lexical Semantics
This section introduces foundational concepts from lexical semantics that inform the computational treatment of
word meaning.

2.1 Words and Word Senses

The study of word meaning requires distinguishing between several related concepts:

Lemmas and Wordforms. A lemma is a canonical form representing a set of inflected wordforms. For example,
the lemma sing encompasses the wordforms sing, sang, sung, and singing. Lemmas are typically what appear as
headwords in dictionaries.

Word Senses. A single lemma may have multiple distinct meanings, each called a sense. The word bank has at
least two senses:

A financial institution
The sloping land beside a body of water

Words with multiple senses are termed polysemous. The phenomenon of polysemy presents challenges for
computational models, as a single wordform may require different representations depending on context.

2.2 Synonymy, Similarity, and Relatedness

Several distinct relationships hold between word meanings:

Synonymy. Two words are synonyms if they have the same meaning in some or all contexts. Perfect synonymy is
rare; most synonyms differ in nuance, register, or collocational preferences. For example, big and large are near-
synonyms but exhibit different usage patterns (big sister vs. ?large sister).

The formal criterion for synonymy is substitutability: two words are synonymous if substituting one for the other
preserves the truth conditions of any sentence.

Similarity. Words may be similar without being synonymous. Similarity is often defined via shared features or
taxonomic proximity. For instance, cat and dog are similar (both are mammals, pets, quadrupeds) but are not
synonyms.

Relatedness (Association). A broader notion captures words that co-occur or are thematically connected without
necessarily being similar. Coffee and cup are related but not similar—they belong to different semantic categories
but frequently co-occur.

These distinctions matter because different vector space models capture different aspects of meaning. Co-
occurrence-based models often capture relatedness (association), while taxonomic similarity requires additional
structure.

2.3 The Distributional Hypothesis

The theoretical foundation for vector semantics is the distributional hypothesis, articulated by linguists including
Zellig Harris (1954) and John Firth (1957). Firth's formulation is often quoted:

"You shall know a word by the company it keeps."

The hypothesis asserts that words occurring in similar linguistic contexts tend to have similar meanings. This
principle enables the inference of semantic properties from observable distributional patterns, without requiring
access to referents or world knowledge.

Formally, if w1 and w2 appear with overlapping sets of context words, we infer that w1 and w2 are semantically
​ ​ ​ ​

related. The strength of this inference scales with the degree of contextual overlap.

3 Vector Space Models

This section develops the construction of vector representations from co-occurrence statistics.

3.1 Term-Document Matrices

The simplest vector space model represents words based on their occurrence in documents. Given a vocabulary
V and a collection of D documents, the term-document matrix M has dimensions ∣V ∣ × D, where entry Mij ​

records the frequency of word i in document j .

Each row of M is a vector representation of a word, where dimensions correspond to documents. Words
appearing in similar documents will have similar row vectors.

Example. Consider four words across four Shakespeare plays:

As You Like It Twelfth Night Julius Caesar Henry V

battle 1 0 7 13

soldier 2 0 12 36

fool 37 58 1 5

crown 5 117 0 0

The vectors for battle and soldier are similar (high values for the history plays), while fool has a different
distribution (concentrated in the comedies).

Limitations. Term-document matrices conflate topical similarity with semantic similarity. Documents on related
topics will induce similarity between words that co-occur with those topics, even if the words themselves are not
semantically related.

3.2 Term-Term (Word-Word) Matrices

A more direct approach to capturing word meaning constructs vectors based on co-occurrence with other words,
yielding a term-term matrix (also called a word-word matrix or word-context matrix).
Given vocabulary V , the term-term matrix M has dimensions ∣V ∣ × ∣V ∣, where entry Mij records how often
​

word i occurs in the context of word j . Context is typically defined by a window of ±k words around the target.

Each row is now a vector of length ∣V ∣, where dimensions correspond to context words. Words with similar co-
occurrence patterns will have similar vectors.

Example. With a context window of ±7 words in a corpus:

aardvark computer data pinch result sugar

apricot 0 0 0 1 0 1

pineapple 0 0 0 1 0 1

digital 0 2 1 0 1 0

information 0 1 6 0 4 0

The vectors for apricot and pineapple are identical, reflecting their similar usage in contexts involving food
preparation (pinch, sugar). The vectors for digital and information are similar and distinct from the fruit words.

3.3 Weighting Schemes: TF-IDF

Raw co-occurrence counts are problematic because frequent words (e.g., the, of) dominate the counts without
contributing discriminative information. Term Frequency-Inverse Document Frequency (TF-IDF) weighting
addresses this by upweighting informative terms and downweighting common terms.

Term Frequency (TF)

Raw term frequency tft,d is the count of term t in document d. To reduce the impact of very high frequencies, a
​

log transformation is applied:

tft,d = {
1 + log10 (count(t, d)) if count(t, d) > 0
(3.1)
​

0 otherwise
​ ​ ​

Inverse Document Frequency (IDF)

Terms appearing in many documents are less informative for distinguishing documents. Let dft denote the​

document frequency of term t (the number of documents containing t), and let N be the total number of
documents. The inverse document frequency is:

idft = log10 ( )
N
(3.2)
dft
​ ​ ​

Rare terms have high IDF; terms appearing in all documents have idf = 0.

Combined TF-IDF Weight

The TF-IDF weight combines both factors:

wt,d = tft,d × idft

​ ​ (3.3)
This weighting scheme is foundational in information retrieval and serves as a preprocessing step for many NLP
tasks.

4 Measuring Semantic Similarity

This section develops cosine similarity as the standard metric for comparing word vectors.

4.1 Cosine Similarity

Given two word vectors, we require a measure of their similarity. The dot product is a natural candidate:

N
v ⋅ w = ∑ vi w i ​ ​ ​ (4.1)
i=1

However, the dot product is unbounded and scales with vector magnitude—longer vectors yield larger products
regardless of directional similarity.

Cosine similarity normalizes by vector lengths, measuring the cosine of the angle between vectors:

v⋅w ∑N
i=1 vi wi
cosine(v, w) = = (4.2)
​ ​ ​

∣v∣ ∣w∣
​ ​

N N
∑i=1 ​ vi2 ​ ​ ∑i=1 ​ wi2 ​ ​

where ∣v∣ = ∑i vi2 is the Euclidean (ℓ2 ) norm.

​ ​ ​

Cosine similarity ranges from −1 to +1:

cosine = 1: vectors point in the same direction (maximally similar)

cosine = 0: vectors are orthogonal (no similarity)
cosine = −1: vectors point in opposite directions

For non-negative vectors (e.g., count-based representations), cosine similarity ranges from 0 to 1.

Worked Example. Consider computing the similarity between apricot and digital using the term-term matrix from
Section 3.2:

vapricot = [0, 0, 0, 1, 0, 1]
​

vdigital = [0, 2, 1, 0, 1, 0]
​

Dot product:

vapricot ⋅ vdigital = (0)(0) + (0)(2) + (0)(1) + (1)(0) + (0)(1) + (1)(0) = 0

​ ​

The vectors are orthogonal—the words share no context words and are thus unrelated according to this
representation.

Now compare apricot and pineapple:

 vpineapple = [0, 0, 0, 1, 0, 1]
​

0+0+0+1+0+1 2
cosine(vapricot , vpineapple ) = = =1
2⋅ 2 2
​ ​ ​ ​

​ ​

The vectors are identical, yielding maximal similarity.

4.2 Properties of Cosine Distance

The cosine distance is defined as:

dcosine (v, w) = 1 − cosine(v, w)

​
(4.3)

This transforms similarity to distance (dissimilarity).

Cosine similarity is preferable to Euclidean distance for word vectors because:

1. Scale invariance: Cosine similarity is unaffected by vector magnitude. A word appearing twice as often in a
corpus does not become "more similar" to itself.

2. Sparsity handling: In high-dimensional sparse vectors, most dimensions are zero. Euclidean distance is
dominated by these zeros, while cosine focuses on the dimensions where both vectors have non-zero values.

3. Interpretability: The angular interpretation provides geometric intuition—similar words cluster in similar
directions from the origin.

5 Pointwise Mutual Information

This section introduces PMI and PPMI as association measures that address limitations of raw co-occurrence
counts.

5.1 PMI Definition and Intuition

Raw co-occurrence counts are biased toward frequent words. The word the co-occurs frequently with nearly
every word, not because of semantic affinity but because of its high frequency. We seek a measure that asks:
"Does this word co-occur with this context more than we would expect by chance?"

Pointwise Mutual Information (PMI) measures the discrepancy between the observed co-occurrence probability
and the probability expected under independence:

P (w, c)
PMI(w, c) = log2 (5.1)
P (w)P (c)
​ ​

where:

P (w, c) is the probability of observing word w and context word c together

P (w) is the marginal probability of w
 P (c) is the marginal probability of c

Interpretation:

PMI > 0: The words co-occur more often than expected under independence (positive association)
PMI = 0: Co-occurrence matches the independence expectation
PMI < 0: The words co-occur less often than expected (negative association, or avoidance)

5.2 Positive PMI (PPMI)

Negative PMI values are problematic:

1. Unreliability: Reliably estimating that two words avoid each other requires observing their absence of co-
occurrence across a large corpus. Such estimates are noisy.

2. Sparsity: Most word pairs never co-occur, yielding PMI = −∞ (since log 0 = −∞).

The standard solution is Positive PMI (PPMI), which clamps negative values to zero:

P (w, c)
PPMI(w, c) = max (log2 , 0) (5.2)
P (w)P (c)
​ ​

PPMI matrices are sparse and non-negative, making them suitable for cosine similarity computations.

5.3 Worked Example: Computing PPMI

Consider a small corpus yielding the following co-occurrence matrix with a context window of ±1 word:

computer data result pie sugar

cherry 0 0 0 5 10

strawberry 0 0 0 3 8

digital 10 20 5 0 0

information 15 25 10 0 0

Step 1: Compute probabilities.

Total count: N = 0 + 0 + 0 + 5 + 10 + 0 + 0 + 0 + 3 + 8 + 10 + 20 + 5 + 0 + 0 + 15 + 25 + 10 +
0 + 0 = 111

Marginal probabilities for words (rows):

15 11 35 50
P (cherry) = , P (strawberry) = , P (digital) = , P (information) =
111 111 111 111
​ ​ ​ ​

Marginal probabilities for contexts (columns):

25 45 15
P (computer) = , P (data) = , P (result) =
111 111 111
​ ​ ​
 8 18
P (pie) = , P (sugar) =
111 111
​ ​

Step 2: Compute PMI for selected pairs.

For (cherry, sugar):

10
P (cherry, sugar) =
111
​

10/111 10 ⋅ 111 1110

PMI(cherry, sugar) = log2 = log2 = log2 = log2 4.11 ≈ 2.04
(15/111)(18/111) 15 ⋅ 18 270
​ ​ ​ ​ ​ ​ ​

For (digital, sugar):

0
P (digital, sugar) = =0
111
​

PMI(digital, sugar) = log2 0 = −∞ ​

Thus PPMI(digital, sugar) = 0.

For (information, data):

25
P (information, data) =
111
​

25/111 25 ⋅ 111 2775

PMI(information, data) = log2 = log2 = log2 = log2 1.23 ≈ 0.30
(50/111)(45/111) 50 ⋅ 45 2250
​ ​ ​ ​ ​ ​ ​

Observation: PPMI assigns high values to word-context pairs that co-occur more than expected (cherry-sugar:
2.04), moderate values to expected associations (information-data: 0.30), and zero to pairs that never co-occur
(digital-sugar: 0).

6 Word Embeddings

This section transitions from sparse, count-based representations to dense, learned embeddings.

6.1 Dense Vector Representations

PPMI matrices, while effective, have significant drawbacks:

1. High dimensionality: Vectors have ∣V ∣ dimensions, where vocabulary sizes can reach hundreds of
thousands.

2. Sparsity: Most entries are zero, wasting storage and computational resources.

3. Limited generalization: The representation cannot leverage similarity between context words.

Word embeddings address these issues by representing words as dense vectors of much lower dimensionality
(typically 50–300 dimensions). These representations are learned from data such that:
 Similar words receive similar embeddings
Semantic and syntactic regularities emerge as algebraic relationships

The term "embedding" reflects that words are embedded as points in a continuous vector space.

6.2 Word2Vec and Skip-Gram

Word2Vec (Mikolov et al., 2013) is a family of algorithms for learning word embeddings. The Skip-Gram model,
the more widely used variant, learns embeddings by training a classifier to predict context words given a target
word.

The key insight is that rather than counting co-occurrences explicitly, we train a model whose parameters encode
the same distributional information implicitly. The trained parameters become the word embeddings.

Skip-Gram Objective. Given a target word w and a context word c, the Skip-Gram model learns to estimate:

P (c ∣ w) (6.1)

For a window of ±L words, the model predicts each context word independently:

P (c−L , … , c−1 , c+1 , … , c+L ∣ w) =

​ ​ ​ ​ ∏ ​ P (cj ∣ w)
​ (6.2)
−L≤j≤L,j=0


Representation. Each word w is associated with two vectors:

Target embedding w ∈ Rd : used when w is the target word

Context embedding c ∈ Rd : used when w is a context word

The probability is modeled using the sigmoid of the dot product:

1
P (+ ∣ w, c) = σ(c ⋅ w) = (6.3)
1 + e−c⋅w
​

where P (+∣w, c) is the probability that c is a true context word of w .

6.3 Skip-Gram with Negative Sampling (SGNS)

Training a classifier to distinguish true context words from all other words is computationally prohibitive (softmax
over ∣V ∣ classes). Negative sampling transforms this into a tractable binary classification problem.

Training Setup:

1. Positive examples: (w, c) pairs from the actual corpus, labeled +

2. Negative examples: (w, cneg ) pairs where cneg is sampled randomly, labeled −
​ ​

For each positive pair, we sample k negative examples (typically k = 5–15).

Loss Function. For a single training instance with target w , positive context cpos , and k negative samples
​

cneg1 , … , cnegk :
​
​

​
​
 k
L = − [log σ(cpos ⋅ w) + ∑ log σ(−cnegi ⋅ w)]
​ ​

​
​
(6.4)
i=1

Minimizing this loss:

Increases the dot product cpos ​ ⋅ w (making true pairs similar)

Decreases the dot product cneg ​
⋅ w (making noise pairs dissimilar)

Negative Sampling Distribution. Negative samples are drawn according to:

count(w)α
Pα (w) = (6.5)
∑w′ count(w′ )α
​ ​

where α = 0.75 is typical. This raises the probability of rare words relative to uniform sampling while still
favoring frequent words.

6.4 Learning the Embeddings

The embeddings are learned via stochastic gradient descent. Given the loss in Equation 6.4, we compute
gradients with respect to both w and c.

Gradient for target embedding:

k
∂L
= [σ(cpos ⋅ w) − 1] cpos + ∑ σ(cnegi ⋅ w) cnegi (6.6)
∂w
​ ​ ​ ​ ​ ​

​ ​

i=1

Gradient for context embedding:

∂L
= [σ(cpos ⋅ w) − 1] w (6.7)
∂cpos
​ ​

∂L
= σ(cnegi ⋅ w) w (6.8)
∂cnegi
​ ​

Update rules (with learning rate η ):

∂L
w(t+1) = w(t) − η (6.9)
∂w
​

∂L
c(t+1) = c(t) − η (6.10)
∂c
​

After training, the target embedding matrix W is typically used as the word embedding matrix. Alternatively, W
and C can be summed or concatenated.

6.5 Properties of Learned Embeddings

Word2Vec embeddings exhibit remarkable properties:
Semantic Clustering. Words with similar meanings cluster together in the embedding space. Nearest neighbors
of "France" include "Spain," "Belgium," and "Germany."

Analogical Reasoning. Embeddings capture relational structure such that:

vking − vman + vwoman ≈ vqueen

​ ​ ​ ​
(6.11)

This suggests that the vector offset vking ​ − vman encodes a gender-neutral "royalty" concept.
​

Parallelogram Structure. Analogies manifest geometrically as parallelograms:

vking − vqueen ≈ vman − vwoman

​ ​ ​ ​
(6.12)

These properties emerge from the training objective without explicit supervision for analogies.

6.6 Bias in Embeddings

Because embeddings are learned from text corpora, they inherit biases present in those corpora. Studies have
documented:

Gender stereotypes: The vector vdoctor ​

− vman + vwoman is closer to "nurse" than to "doctor"
​ ​

Racial biases: African American names cluster with negative sentiment words

Occupational biases: Female-associated terms cluster with domestic roles

These biases can propagate to downstream applications (e.g., resume screening, sentiment analysis). Research on
debiasing embeddings seeks to identify and mitigate these effects while preserving useful semantic information.

Implications. The presence of bias in embeddings is a reminder that distributional semantics captures usage
patterns, not normative meaning. Models trained on biased text will learn biased representations.

7 Neural Network Fundamentals

This section introduces the building blocks of neural networks.

7.1 The Neural Unit

The fundamental computational element is the neural unit (also called a node or neuron). A neural unit
computes a weighted sum of its inputs, adds a bias term, and applies a nonlinear activation function.

Given an input vector x = [x1 , x2 , … , xn ]T , weight vector w = [w1 , w2 , … , wn ]T , and bias b, the unit
​ ​ ​ ​ ​ ​

computes:
n
z = w ⋅ x + b = ∑ w i xi + b ​ ​ ​ (7.1)
i=1

y = f (z) (7.2)
where f is the activation function and y is the unit's output.

The bias b allows the unit to shift its activation threshold. Without bias, the decision boundary must pass through
the origin.

7.2 Activation Functions

The choice of activation function determines the unit's nonlinear behavior.

Sigmoid. The sigmoid function squashes real values to the range (0, 1):

1
σ(z) = (7.3)
1 + e−z
​

Properties:

σ(z) → 1 as z → +∞
σ(z) → 0 as z → −∞
σ(0) = 0.5
Derivative: dσ
dz ​ = σ(z)(1 − σ(z))

Sigmoid is used for binary classification outputs but suffers from the vanishing gradient problem: gradients
approach zero for large ∣z∣, slowing learning.

Hyperbolic Tangent (tanh). Tanh maps inputs to the range (−1, 1):

ez − e−z
tanh(z) = (7.4)
ez + e−z
​

Tanh is zero-centered, which can improve gradient flow. It is related to sigmoid by:

tanh(z) = 2σ(2z) − 1 (7.5)

Derivative: d tanh
dz
​
= 1 − tanh2 (z)

Rectified Linear Unit (ReLU). ReLU is the most widely used activation in modern deep learning:

ReLU(z) = max(0, z) (7.6)

Derivative:

={
dReLU 0 if z < 0
(7.7)
1 if z ≥ 0
​ ​ ​

dz

ReLU advantages:

Computationally efficient (no exponentials)

Constant gradient for positive inputs (mitigates vanishing gradients)

Induces sparsity (negative inputs produce zero output)
ReLU disadvantage: "Dead neurons" can occur when units get stuck at zero output.

7.3 The XOR Problem and Multi-Layer Networks

A single neural unit implements a linear classifier, which cannot represent nonlinearly separable functions. The
canonical example is XOR (exclusive or):

x1 x2 x1 XOR x2
​ ​ ​ ​

0 0 0

0 1 1

1 0 1

1 1 0

No single line can separate the + class (where output is 1) from the − class. XOR is not linearly separable.

Solution: Multi-Layer Networks. Adding a hidden layer between input and output creates a two-layer network
that can represent XOR.

Consider a network with:

Input layer: x1 , x2 ​ ​

Hidden layer: 2 units with ReLU activation

Output layer: 1 unit with sigmoid activation

One valid solution:

Hidden unit 1: h1 = max(0, x1 + x2 − 0.5)

​ ​ ​

Hidden unit 2: h2 = max(0, −x1 − x2 + 1.5)

​ ​ ​

Output: y = σ(h1 + h2 − 1) ​ ​

The hidden layer transforms the input space such that the output becomes linearly separable.

Universal Approximation. A feedforward network with a single hidden layer containing enough units can
approximate any continuous function to arbitrary precision (Cybenko, 1989). This theoretical result motivates the
use of hidden layers for modeling complex functions.

8 Feedforward Neural Networks

This section develops the architecture and operation of feedforward networks.

8.1 Architecture
A feedforward neural network (also called a multi-layer perceptron or MLP) consists of:

1. Input layer: Receives the feature vector x ∈ Rn 0 ​

 2. Hidden layers: One or more layers of processing units

3. Output layer: Produces the final prediction

In a fully connected (dense) network, every unit in layer l connects to every unit in layer l + 1. Information flows
forward from input to output with no cycles.

Notation. For a network with L layers:

W[l] : weight matrix for layer l (dimensions nl × nl−1 ) ​ ​

b[l] : bias vector for layer l (dimensions nl × 1)

​

a[l] : activation vector for layer l

z[l] : pre-activation (weighted sum) for layer l

8.2 The Hidden Layer

The hidden layer transforms the input representation. For a single hidden layer:

z[1] = W[1] x + b[1] (8.1)

h = g(z[1] ) (8.2)

where g is the activation function applied element-wise.

The hidden layer learns an internal representation of the input that is useful for the task. These learned
representations are the "deep" features that give deep learning its power.

Dimensionality. If the input has n0 dimensions and the hidden layer has n1 units:
​ ​

W[1] ∈ Rn1 ×n0 ​ ​

b[1] ∈ Rn1 ​

h ∈ Rn 1 ​

The hidden layer can expand (n1 ​ > n0 ), compress (n1 < n0 ), or preserve dimensionality.
​ ​

8.3 The Output Layer and Softmax

For classification tasks, the output layer produces a probability distribution over K classes.

Binary Classification. A single output unit with sigmoid activation:

y^ = σ(w ⋅ h + b)
​ (8.3)

^∈
where y ​ (0, 1) is interpreted as P (y = 1∣x).

Multi-class Classification. The softmax function generalizes sigmoid to K classes:

z = Uh + b[2] (8.4)
 exp(zk )
y^k = softmax(zk ) = (8.5)
​

K
​ ​ ​ ​

∑j=1 exp(zj )
​ ​

Softmax properties:

^k
Outputs sum to 1: ∑k y ​ ​ ​ =1
^k
All outputs are positive: y ​ ​ >0
Exponential amplifies differences: larger zk receive disproportionately higher probability
​

^
The output y ​
= [y^1 , … , y^K ]T is a probability distribution over classes.
​ ​ ​ ​

8.4 Classification with Neural Networks

Complete Forward Pass. For a 2-layer network (1 hidden layer):

h = g(W[1] x + b[1] ) (8.6)

z = W[2] h + b[2] (8.7)

y^ = softmax(z)
​ (8.8)

Using Pre-trained Embeddings. For text classification, the input x is often derived from word embeddings.
Common strategies:

1. Embedding lookup: For single words, retrieve the embedding vector

2. Embedding average: Average the embeddings of all words in a document

3. Concatenation: For fixed-length sequences, concatenate embeddings

Example: Sentiment Classification. Given a sentence, we:

1. Look up embeddings for each word

2. Average the embeddings to get x

3. Pass through hidden layer: h = ReLU(Wx + b)

^=
4. Compute output: y ​ softmax(Uh)

The network learns weights that project the embedding space into a sentiment-predictive representation.

9 Training Neural Networks

This section presents the theory and algorithms for training neural networks.

9.1 Loss Functions: Cross-Entropy

Training requires a loss function that quantifies the discrepancy between predictions and ground truth. For
classification, the standard choice is cross-entropy loss.
Binary Cross-Entropy. For binary classification with true label y ∈ {0, 1} and predicted probability y^ = P (y =
​

1∣x):

LCE (y^, y) = − [y log y^ + (1 − y) log(1 − y^)]

​ ​ ​ ​ (9.1)

This loss:

Equals − log y
^ when y
​
= 1 (penalizes low confidence in positive class)
Equals − log(1 − y
^) when y
​ = 0 (penalizes high confidence in negative class)

Multi-class Cross-Entropy. For K classes with true label vector y (one-hot encoded) and predicted probabilities
y^:
​

K
LCE (y^, y) = − ∑ yk log y^k
​ ​ ​ ​ ​ ​ (9.2)
k=1

Since y is one-hot (only one yk ​ = 1), this simplifies to:

LCE = − log y^c

​ ​ ​ (9.3)

where c is the index of the correct class. Minimizing cross-entropy is equivalent to maximizing the log probability
of the correct class.

9.2 Gradient Descent

Neural networks are trained by minimizing the loss function using gradient descent. The gradient indicates the
direction of steepest increase; we move in the opposite direction.

Update Rule. Given parameters θ and learning rate η :

θ(t+1) = θ(t) − η∇θ L ​ (9.4)

where ∇θ L is the gradient of the loss with respect to parameters.

​

Stochastic Gradient Descent (SGD). Computing the gradient over the entire training set is expensive. SGD
approximates the full gradient using a single example (or mini-batch):

θ(t+1) = θ(t) − η∇θ L(x(i) , y (i) )

​ (9.5)

SGD is noisy but computationally efficient and often escapes local minima.

Mini-batch Gradient Descent. A compromise uses batches of B examples:

B
1
θ (t+1)
=θ (t)
− η ∑ ∇θ L(x(i) , y (i) )
​ ​ ​ (9.6)
B i=1

Typical batch sizes range from 16 to 256.

9.3 Computation Graphs

A computation graph represents the forward computation as a directed acyclic graph (DAG) where:

Nodes represent variables (inputs, intermediate values, outputs)

Edges represent operations

Computation graphs enable systematic gradient computation via the chain rule.

Example. Consider L(a, b, c) = c ⋅ (a + 2b).

Decompose into elementary operations:

d = 2b
e=a+d
L=c⋅e

The graph flows from inputs (a, b, c) through intermediate nodes (d, e) to output L.

For a = 3, b = 1, c = −2:

d=2
e=5
L = −10

9.4 Backpropagation
Backpropagation (backward differentiation) computes gradients by applying the chain rule from output to input
along the computation graph.

Chain Rule. For a composite function L = L(e(a, d), c) where e = a + d and d = 2b:
∂L ∂L ∂e
= ⋅ (9.7)
∂a ∂e ∂a
​ ​ ​

∂L ∂L ∂e ∂d
= ⋅ ⋅ (9.8)
∂b ∂e ∂d ∂b
​ ​ ​ ​

Local Gradients. At each node, we compute the local gradient (derivative of output with respect to input):

For L = c ⋅ e:
∂L ∂L
= c, =e
∂e ∂c
​

For e = a + d:
∂e ∂e
= 1, =1
∂a ∂d
​ ​

For d = 2b:

∂d
 ∂d
=2
∂b
​

Backward Pass. Starting from the output:

∂L
1. ∂L ​ = 1 (base case)
∂L
2. ∂e ​ = c = −2
3. ∂L
∂c ​ =e=5
4. ∂L
∂a
​ = ∂L
∂e
​ ⋅ ∂e
∂a
​ = (−2)(1) = −2
∂L ∂L ∂e
5. ∂d ​
= ∂e
​
⋅ ∂d
​
= (−2)(1) = −2
6. ∂L
∂b
​ = ∂L
∂d
​ ⋅ ∂d
∂b
​ = (−2)(2) = −4
∂L ∂L ∂L
The gradients ∂a ​ = −2, ∂b
​ = −4, ∂c
​ = 5 can be verified analytically.

Neural Network Backpropagation. For a neural network, the same principle applies. The computation graph
includes:

Matrix multiplications (Wx)

Bias additions (+b)

Activation functions (σ , ReLU)

Loss computation

The gradients of activation functions are:

Sigmoid:

dσ(z)
= σ(z)(1 − σ(z))
​ (9.9)
dz
Tanh:

d tanh(z)
= 1 − tanh2 (z)
​ (9.10)
dz
ReLU:

={
dReLU(z) 0 z<0
(9.11)
1 z≥0
​ ​ ​

dz

Gradient with respect to output layer. For cross-entropy loss with softmax output and correct class c:

∂L
= y^k − yk (9.12)
∂zk
​ ​ ​ ​

This elegant result simplifies implementation: the gradient is simply the difference between predicted and true
probabilities.
9.5 Practical Considerations
Several techniques improve training:

Weight Initialization. Unlike logistic regression, neural networks cannot be initialized with all zeros (all units
would compute identical gradients). Weights are initialized with small random values, typically drawn from:

2
N (0, )
nin + nout
​ ​

​ ​

where nin and nout are the input and output dimensions.
​ ​

Regularization. To prevent overfitting:

L2 regularization: Add λ∥θ∥2 to the loss

Dropout: Randomly set unit activations to zero during training with probability p

Dropout (Hinton et al., 2012) is particularly effective. During training, each unit is "dropped" with probability p
(typically 0.5). At test time, all units are used but outputs are scaled by (1 − p).

Hyperparameters. Key hyperparameters include:

Learning rate η

Mini-batch size

Number of hidden layers and units per layer

Dropout rate

Choice of optimizer (SGD, Adam, etc.)

Hyperparameters are tuned on a validation set, not the training set.

Modern Frameworks. Deep learning frameworks (PyTorch, TensorFlow) automate gradient computation via
automatic differentiation. The user specifies the forward pass; gradients are computed automatically.

10 Neural Language Models

This section synthesizes embeddings and neural networks in the context of language modeling.

10.1 Feedforward Language Models

A neural language model uses a neural network to estimate the probability of the next word given previous
words:

P (wt ∣w1 , … , wt−1 )

​ ​ ​ (10.1)

Like n-gram models, feedforward neural LMs approximate using a fixed context window:
 P (wt ∣w1 , … , wt−1 ) ≈ P (wt ∣wt−N +1 , … , wt−1 )
​ ​ ​ ​ ​ ​ (10.2)

Advantages over n-gram models:

Generalization via embeddings: Words with similar embeddings yield similar predictions
Continuous representations: No discrete probability tables

Longer effective context: Modern architectures extend beyond fixed windows

Disadvantages:

Higher computational cost

Requires more training data

Less interpretable

Example. An n-gram model trained on "the cat gets fed" cannot predict "fed" after "the dog gets" without
observing that exact trigram. A neural LM, knowing that "cat" and "dog" have similar embeddings, generalizes
appropriately.

10.2 Forward Inference

Given context words wt−N +1 , … , wt−1 , forward inference produces a probability distribution over the next
​ ​

word.

Architecture. For a context window of size 3:

1. Embedding lookup: Each context word is represented as a one-hot vector and multiplied by the embedding
matrix E ∈ Rd×∣V ∣ :

ei = Exi
​ ​
(10.3)

2. Concatenation: The embeddings are concatenated to form the embedding layer:

e = [et−3 ; et−2 ; et−1 ]

​ ​ ​
(10.4)

yielding a vector of dimension 3d.

3. Hidden layer: Apply weights and activation:

h = g(We + b) (10.5)

4. Output layer: Project to vocabulary size and apply softmax:

z = Uh (10.6)

y^ = softmax(z)
​
(10.7)

^i is the predicted probability that the next word is vocabulary item i:

The output y ​ ​

y^i = P (wt = Vi ∣wt−3 , wt−2 , wt−1 )

​ ​ ​ ​ ​ ​
10.3 Training Neural LMs
Training uses self-supervision: the next word in the corpus serves as the label.

Loss Function. For a single prediction, the loss is cross-entropy:

L = − log P (wt ∣wt−N +1 , … , wt−1 ) = − log y^wt

​ ​ ​

​
​ (10.8)

Parameter Updates. The parameters θ = {E, W, b, U} are updated via SGD:

∂L
θ(t+1) = θ(t) − η (10.9)
∂θ
​

Gradients are computed via backpropagation through the network, including into the embedding matrix E.

Freezing vs. Fine-tuning Embeddings. Two strategies exist:

1. Frozen embeddings: Initialize E with pre-trained embeddings (e.g., Word2Vec) and do not update during
training. Only W, b, U are learned.

2. Fine-tuning: Initialize with pre-trained embeddings but continue updating E during training. This adapts
embeddings to the specific task.

Fine-tuning is generally preferred when sufficient training data is available. Freezing is useful for small datasets
where updating E risks overfitting.

Perplexity. Language models are evaluated using perplexity:

N
1
PP(W ) = P (w1 , … , wN ) −1/N
= N
∏ (10.10)
P (wi ∣w1 , … , wi−1 )
​ ​ ​ ​ ​

​ ​ ​

i=1

Lower perplexity indicates a better model. Perplexity can be interpreted as the weighted average number of
choices the model considers at each position.

11 Conclusion
This chapter has developed the theoretical foundations for representing word meaning and learning from text
data.

Count-based methods construct word vectors from co-occurrence statistics. TF-IDF weighting and PPMI address
the bias toward frequent words, yielding sparse but interpretable representations. Cosine similarity serves as the
standard metric for comparing these vectors.

Word embeddings (Word2Vec, Skip-Gram with Negative Sampling) learn dense representations by training a
classifier to distinguish true context words from noise samples. The resulting embeddings capture semantic and
syntactic regularities, manifesting as algebraic relationships in the vector space.
Neural networks provide the computational machinery for learning embeddings and performing downstream
tasks. The feedforward architecture, with its hidden layers and nonlinear activations, can represent complex
functions. Training via gradient descent and backpropagation adjusts weights to minimize cross-entropy loss.

Neural language models synthesize these components, using embeddings as input to networks that predict
upcoming words. These models generalize better than n-gram models because similar words yield similar
predictions.

The transition from count-based to neural methods reflects a broader shift in NLP: from hand-crafted features to
learned representations. The embeddings and network weights learned in this chapter form the foundation for
the deep learning architectures (RNNs, Transformers) covered in subsequent chapters.

12 References
Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). A neural probabilistic language model. Journal of
Machine Learning Research, 3, 1137–1155.

Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and
Systems, 2(4), 303–314.

Firth, J. R. (1957). A synopsis of linguistic theory 1930–1955. In Studies in Linguistic Analysis, 1–32. Oxford:
Blackwell.

Goldberg, Y. (2017). Neural Network Methods for Natural Language Processing. Morgan & Claypool Publishers.

Harris, Z. S. (1954). Distributional structure. Word, 10(2–3), 146–162.

Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012). Improving neural networks
by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580.

Jurafsky, D., & Martin, J. H. (2024). Speech and Language Processing (3rd ed. draft). Chapters 6–7.

Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In Proceedings of ICLR.

Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space.
In Proceedings of ICLR Workshop.

Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed representations of words and
phrases and their compositionality. In Advances in Neural Information Processing Systems, 3111–3119.

Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global vectors for word representation. In Proceedings
of EMNLP, 1532–1543.

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors.
Nature, 323(6088), 533–536.
Appendix A: Mathematical Derivations

A.1 Derivation of Sigmoid Derivative

1
Given σ(z) = 1+e−z
:
​

dσ d −1 −2 e−z
= (1 + e−z ) = − (1 + e−z ) ⋅ (−e−z ) =
(1 + e−z )2
​ ​ ​

dz dz

Rewriting:

dσ 1 e−z 1 + e−z − 1
= ⋅ = σ(z) ⋅ = σ(z)(1 − σ(z))
1 + e−z 1 + e−z 1 + e−z
​ ​ ​ ​

dz

A.2 Gradient of Cross-Entropy with Softmax

ezk
For multi-class cross-entropy L = − ∑k yk log y^k with softmax outputs y^k = :
​

∑j e z j
​ ​ ​ ​ ​ ​ ​ ​

Taking derivative with respect to zi : ​

∂L ∂ log y^k 1 ∂ y^k

= − ∑ yk = − ∑ yk
​ ​ ​ ​

∂zi ∂zi y^k ∂zi

​ ​ ​ ​ ​ ​ ​ ​

​ ​ ​ ​ ​

k k

The derivative of softmax:

={ k
∂ y^k
​ y^ (1 − y^k ) k = i
​ ​ ​ ​ ​

∂zi −y^k y^i k=i

​ ​ ​

​ ​ ​ ​

Substituting and simplifying (where c is the correct class with yc ​ = 1):

∂L
= y^i − yi
∂zi
​ ​ ​ ​

A.3 PPMI as a Matrix Factorization

Levy and Goldberg (2014) showed that Skip-Gram with Negative Sampling implicitly factorizes a shifted PMI
matrix:

WT C ≈ MPMI − log k

where k is the number of negative samples. This connection unifies count-based and prediction-based methods.

Glossary

Term Definition

Activation Function Nonlinear function applied element-wise to transform neural unit outputs
 Term Definition

Backpropagation Algorithm for computing gradients by applying the chain rule backward through a computation graph

Bias Scalar parameter added to the weighted sum in a neural unit, allowing shift of the activation threshold

Computation Graph Directed acyclic graph representing the sequence of operations in a neural network

Cosine Similarity Similarity measure based on the cosine of the angle between two vectors

Cross-Entropy Loss Loss function measuring the difference between predicted probability distribution and true distribution

Distributional Hypothesis The principle that words occurring in similar contexts have similar meanings

Dropout Regularization technique that randomly sets unit activations to zero during training

Embedding Dense, low-dimensional vector representation of a word

Feedforward Network Neural network where connections flow in one direction from input to output with no cycles

Gradient Descent Optimization algorithm that iteratively updates parameters in the direction of steepest descent

Hidden Layer Layer between input and output that learns internal representations

IDF Inverse Document Frequency; downweights terms appearing in many documents

Lemma Canonical dictionary form of a word, representing a set of inflected forms

Loss Function Function quantifying the discrepancy between predictions and ground truth

Negative Sampling Training technique that contrasts true context words with randomly sampled "noise" words

One-Hot Vector Sparse vector with a single 1 at the index corresponding to a word and 0s elsewhere

Perplexity Evaluation metric for language models; the inverse probability normalized by sequence length

PMI Pointwise Mutual Information; measures association between word pairs relative to independence

PPMI Positive PMI; PMI clamped to non-negative values

ReLU Rectified Linear Unit; activation function max(0, z)

Self-Supervision Training paradigm where labels are derived from the data itself (e.g., predicting the next word)

Sigmoid S-shaped activation function mapping real values to (0, 1)

Skip-Gram Word2Vec architecture that predicts context words from a target word

Softmax Function that converts a vector of real values to a probability distribution

TF-IDF Term Frequency-Inverse Document Frequency; weighting scheme balancing term frequency and document frequency

Word2Vec Family of algorithms for learning word embeddings from text corpora