Why Do We Need Deep Learning?
A Toy Sentiment Analysis Example
Toy Problem: Sentiment Classification
Goal: classify sentences as Positive
Words Hierarchy helps:
or Negative.
?
Examples: Meaning Words → Phrases
• This movie is good (sentiment)
Phrases → Meaning
• This movie is bad
• This movie is not good Key challenge: negation
• This movie is not bad changes meaning.
1
Representation: Bag of Words
We use three binary features:
x1 = ⊮(“good”), x2 = ⊮(“bad”), x3 = ⊮(“not”)
Sentence
Sentence x1 x2 x3 encode
good 1 0 0 x = (x1 , x2 , x3 )
bad 0 1 0
not good 1 0 1 Bag of words:
not bad 0 1 1 ignores word order
2
Single Layer Model: One Neuron
One neuron computes:
x1
w1
z = w1 x1 +w2 x2 +w3 x3 , y = step(z) w2
x2 y
Choose:
w3
x3 single layer
w1 = 2, w2 = −2, w3 = −1 (linear)
Interpretation:
• “good” pushes positive
• “bad” pushes negative
• “not” pushes slightly negative
3
Single Layer Failure: Negation Interaction
“not good” ⇒ (1, 0, 1)
Linear model:
z = 2(1) + (−2)(0) + (−1)(1) = 1 adds word effects
cannot
Prediction: Positive (wrong) But negation means:
word interaction
“not bad” ⇒ (0, 1, 1)
z = 2(0)+(−2)(1)+(−1)(1) = −3 We want a model that can build:
“not good” as a unit
Prediction: Negative (wrong)
Core issue: word meanings are not independent; they combine.
4
Two Layer Network: Words → Phrases → Sentiment
Hidden layer (phrases)
Inputs
good
x1 h1 Output
without not
x2 bad y
h2
without not
x3 not
h3
good
not
h4
bad
Layer 1 builds phrase detectors. Layer 2 combines phrase meaning.
5
Layer 1: Phrase Detectors (Weights + Biases)
Inputs: x = (x1 , x2 , x3 )
Define hidden units using
thresholding: Hidden x1 x2 x3 b
h1 1 0 -1 -0.5
h1 = step(x1 −x3 −0.5) (good withouth2not) 0 1 -1 -0.5
h3 1 0 1 -1.5
h2 = step(x2 −x3 −0.5) (bad without hnot)
4 0 1 1 -1.5
h3 = step(x1 + x3 − 1.5) (not good)
h4 = step(x2 + x3 − 1.5) (not bad)
These units explicitly represent interactions like “not + good”.
6
Layer 2: Combine Phrase Meaning into Sentiment
We now combine hidden units:
z = 2h1 − 2h2 − 2h3 + 2h4 y = step(z)
Interpretation:
• h1 supports Positive
• h2 supports Negative
• h3 supports Negative (because “not good”)
• h4 supports Positive (because “not bad”)
Phrase featuresweighted combine
Sentiment
h = (h1 , h2 , h3 , h4 ) y
7
Worked Examples (Forward Pass)
Sentence x1 x2 x3 h1 h2 h3 h4 y
good 1 0 0 1 0 0 0 1
bad 0 1 0 0 1 0 0 0
not good 1 0 1 0 0 1 0 0
not bad 0 1 1 0 0 0 1 1
This is the key story: the network first constructs phrase meaning,
then decides sentiment.
8
Why This Explains Deep Learning
Single layer:
• adds independent word contributions
• fails on interaction patterns like negation
Two layers:
• Layer 1: words → phrases (features)
• Layer 2: phrases → sentiment (decision)
Deep learning is powerful when meaning depends on
compositions:
tokens → phrases → meaning
9
ReLU Activation Function
Definition: [ ReLU(x) = max(0, x)]
Behavior:
• x < 0 ⇒ ReLU(x) = 0
• x ≥ 0 ⇒ ReLU(x) = x
Examples:
• ReLU(−3) = 0
• ReLU(2) = 2
Why use ReLU?
• Introduces non-linearity
• Simple and fast to compute
• Works well in deep neural networks
10
Example 1: Handwritten Digit Recognition
Task: Classify images of digits (0–9)
Input:
• Image of size 28 × 28
• Flattened into vector: [ x ∈ R784 ]
Neural Network: [ x → ReLU → ReLU → Output (10 classes)]
What each layer learns:
• Layer 1: edges (lines, strokes)
• Layer 2: shapes (curves, loops)
• Layer 3: digits (0–9)
11
Example 2: Text Classification
Task: Classify text (e.g., sentiment)
Input representation: [
x1 = ⊮(good), x2 = ⊮(bad), x3 = ⊮(not)]
Example: [ “not good” → (1, 0, 1)]
Neural Network: [ x → ReLU → ReLU → Output]
What each layer learns:
• Layer 1: word features
• Layer 2: phrase features (e.g., “not good”)
• Layer 3: sentiment (positive/negative)
12
Practice Problem 1: Why Single Layer Fails
Consider the same features: [
x1 = ⊮(good), x2 = ⊮(bad), x3 = ⊮(not)]
A single neuron uses: [ y = step(2x1 − 2x2 − x3 )]
Tasks:
1. Compute the output for:
• ‘good”
• ‘bad”
• ‘not good”
• ‘not bad”
2. Which predictions are incorrect?
3. Explain in one sentence: Why does a single-layer model fail
here?
13
Practice Problem 2: Forward Pass with ReLU
Consider a 2-layer neural network:
Layer 1: [ h1 = ReLU(x1 − x2 )][h2 = ReLU(x2 − x1 )]
Layer 2: [ y = h1 − h2 ]
Tasks:
1. Compute (h1 , h2 , y ) for:
• (x1 , x2 ) = (1, 0)
• (x1 , x2 ) = (0, 1)
• (x1 , x2 ) = (1, 1)
2. What function is this network computing?
14
Solution 1: Single Layer Failure
Given: [ y = step(2x1 − 2x2 − x3 )]
Compute outputs:
• “good” (1, 0, 0): [ z = 2(1) - 0 - 0 = 2 ⇒ y = 1 (correct)]
• “bad” (0, 1, 0): [ z = 0 - 2 - 0 = -2 ⇒ y = 0 (correct)]
• “not good” (1, 0, 1): [ z = 2 - 0 - 1 = 1 ⇒ y = 1 (wrong)]
• “not bad” (0, 1, 1): [ z = 0 - 2 - 1 = -3 ⇒ y = 0 (wrong)]
Conclusion:
• Model adds word contributions independently
• Cannot capture interactions like “not + good”
15
Solution 2: ReLU Forward Pass
Layer 1: [ h1 = ReLU(x1 − x2 ), h2 = ReLU(x2 − x1 )]
Layer 2: [ y = h1 − h2 ]
Case 1: (1, 0) [
h1 = ReLU(1) = 1, h2 = ReLU(−1) = 0][y = 1 − 0 = 1]
Case 2: (0, 1) [
h1 = ReLU(−1) = 0, h2 = ReLU(1) = 1][y = 0 − 1 = −1]
Case 3: (1, 1) [ h1 = ReLU(0) = 0, h2 = ReLU(0) = 0][y = 0]
16