EELE 0531: Machine Learning
Lecture 9
Instructor: Waleed Ejaz
wejaz@[Link]
Slides adopted from Prof. Andrew Ng Founder of [Link], Co-founder Coursera, Adjunct Professor at Stanford University 1
Handwritten digit recognition using Forward Propagation
▪ Binary Classification problem: Is this the digit
‘0’ or ‘1’
Probability of being a handwritten ‘1’
2
Handwritten digit recognition using Forward Propagation
▪ Digit images 0 and 1
Probability of
being a
handwritten ‘1’
3
Activation Functions in Neural Networks
▪ We have used the sigmoid activation function in all the hidden and
output nodes.
▪ We started this way because we were building neural networks by
taking logistic regression, creating many logistic regression units, and
stringing them together.
▪ You can make neural networks more powerful by using other
activation functions.
4
Demand Prediction Example
▪ Given the price, shipping cost, marketing, and material, we tried to
predict if something is highly affordable.
▪ If there's good awareness and high perceived quality and based on that
try to predict it was a top seller.
▪ This assumes that awareness is maybe binary as either people are aware,
or they are not.
▪ In reality, the degree to which possible buyers are aware of the T-shirt
you're selling may not be binary. They can be a little bit aware,
somewhat aware, extremely aware, or it could have gone completely
viral.
▪ Thus, we can estimate the probability of awareness or rather than
modeling awareness, is just a number between 0 and 1.
▪ Maybe awareness should be any non-negative number because there
can be any non-negative value of awareness going from 0 up to very,
very large numbers.
5
Examples of Activation Functions
▪ A very common choice of activation function in neural networks is
the ReLU.
6
Choosing Activation Functions
▪ Binary Classification: Sigmoid (predicts probability y=1)
▪ Regression (positive/negative values): Linear (allows for any output range)
▪ Regression (non-negative values): ReLU (output is always >= 0)
▪ For hidden Layers: ReLU is commonly used because of faster learning and
efficiency
• Computational Efficiency: ReLU is faster to compute than sigmoid.
• Faster Learning: ReLU avoids the "vanishing gradient" problem common with
sigmoid.
• Less Prone to Saturation: ReLU only saturates (becomes flat) in one direction.
7
Summary of Choosing Activation Functions
▪ TensorFlore Implementation
8
Other Activation Functions
▪ tanh: Similar to sigmoid, but output ranges from -1 to 1.
▪ Leaky ReLU: A variation of ReLU that allows a small gradient for
negative inputs.
▪ Swish: A newer activation function that can sometimes outperform
ReLU.
9
Why activation functions?
▪ Without activation functions, neural networks would just be linear models.
▪ Activation functions introduce non-linearity: This allows neural networks to
learn complex patterns.
▪ Demand Prediction Example
▪ Using linear activation functions throughout a neural network reduces it to a
linear regression model.
▪ This limits the network's ability to learn complex patterns.
10
G +
1
Example
▪ Consider a simple network: input x, one hidden unit (a1), one output unit (a2). ↑
▪ With linear activation (g(z) = z), the network's output becomes a linear function of the input.
[1]
𝑎[1] = 𝑤1 𝑥 + 𝑏1
[1]
&
[2]
𝑎[2] = 𝑤1 𝑎[1] + 𝑏1
[2]
a(z)
=
W
(2]
,
[i]
,
b)
[2] [1] [1] [2]
𝑎[2] = 𝑤1 (𝑤1 𝑥 + 𝑏1 ) + 𝑏1 +
[2] [1] [2] [1] [2]
𝑎[2] = 𝑤1 𝑤1 𝑥 + 𝑤1 𝑏1 + 𝑏1
w b
▪ This is equivalent to 𝑎[2] = 𝑤𝑥 + 𝑏 (a linear equation).
11
Linear Functions of Linear Functions
▪ A linear function of a linear function is always a linear function.
▪ Multiple layers with linear activation functions do not increase the
network's complexity.
▪ Linearity in Larger Networks
• Even in larger networks, using linear activation in all layers results in a model
equivalent to linear regression.
• Using linear activation in hidden layers and a logistic activation in the output
layer is equivalent to logistic regression.
All layers use linear Output activation is
activation including output sigmoid (hidden layers are
layer equivalent to linear still linear) equivalent to
regression logistic regression
12
The Solution: Non-Linear Activations
▪ Avoid linear activation functions in hidden layers.
▪ ReLU activation function is a good default choice.
▪ Non-linear activations enable the network to learn complex, non-
linear patterns.
13
Multiclass Classification
14
Multiclass Classification
▪ Classification problems with more than two possible output labels.
▪ Examples: handwritten digit classification, disease classification,
visual defect inspection.
▪ Binary versus Multiclass Classification
• Binary classification: two possible classes (e.g., 0 or 1).
• Multiclass classification: more than two possible classes (e.g., 0, 1, 2, ..., 9).
15
Softmax Regression
▪ A generalization of logistic regression for multiclass classification.
▪ Estimates the probability of each class for a given input.
▪ Uses the softmax activation function to ensure that the probabilities
sum to 1.
▪ Neural Networks for Multiclass Classification
• Can be used for multiclass classification by adding a softmax layer as the
output layer.
• Can be trained using backpropagation to minimize the loss function.
16
Logistic Regression Recap
▪ Binary classification (y = 0 or 1)
𝑧 = 𝑤 ∙ 𝑥Ԧ + 𝑏
1
𝑎=𝑔 𝑧 = −𝑧
= 𝑃(𝑦 = 1|𝑥)
Ԧ
1+𝑒
▪ Example: If 𝑃(𝑦 = 1|𝑥)Ԧ = 0.71, then 𝑃(𝑦 = 0|𝑥)
Ԧ = 0.29 (must sum to 1)
▪ Logistic Regression: A different perspective
• 𝑎1 = 𝑃 𝑦 = 1 𝑥Ԧ
• 𝑎2 = 𝑃 𝑦 = 0 𝑥Ԧ = 1 − 𝑎1
• 𝑎1 + 𝑎2 = 1
17
Softmax Regression (Example with 4 classes)
▪ Class 1: 𝑧1 = 𝑤1 ∙ 𝑥Ԧ + 𝑏1
𝑒 𝑧1
𝑎1 = = 𝑃(𝑦 = 1|𝑥)
Ԧ
𝑒 𝑧1 + 𝑒 𝑧2 + 𝑒 𝑧3 + 𝑒 𝑧4
▪ Class 2: 𝑧2 = 𝑤2 ∙ 𝑥Ԧ + 𝑏2
𝑒 𝑧2
𝑎1 = = 𝑃(𝑦 = 2|𝑥)
Ԧ
𝑒 𝑧1 + 𝑒 𝑧2 + 𝑒 𝑧3 + 𝑒 𝑧4
▪ Class 3: 𝑧3 = 𝑤3 ∙ 𝑥Ԧ + 𝑏3
𝑒 𝑧3
𝑎1 = = 𝑃(𝑦 = 3|𝑥)
Ԧ
𝑒 𝑧1 + 𝑒 𝑧2 + 𝑒 𝑧3 + 𝑒 𝑧4
▪ Class 4: 𝑧4 = 𝑤4 ∙ 𝑥Ԧ + 𝑏4
𝑒 𝑧4
𝑎1 = = 𝑃(𝑦 = 4|𝑥)
Ԧ
𝑒 𝑧1 + 𝑒 𝑧2 + 𝑒 𝑧3 + 𝑒 𝑧4
18
Softmax for ‘n’ classes
▪ For class j
𝑧𝑗 = 𝑤𝑗 ∙ 𝑥Ԧ + 𝑏𝑗
▪ where j=1…N (N possible outputs)
▪ Parameters 𝑤1 , 𝑤2 … 𝑤𝑁
𝑒 𝑧𝑗
𝑎𝑗 = = 𝑃(𝑦 = 𝑗|𝑥)
Ԧ
σ𝑁
𝑘=1 𝑒 𝑧𝑘
𝑎1 + 𝑎2 + ⋯ + 𝑎𝑁 = 1
19
Loss Function for Logistic Regression
20
Lost Function Visualization
▪ Plot the graph of -log(a_j) versus a_j
▪ Loss is minimized when a_j is close to 1.
▪ The algorithm is encouraged to predict the correct class with high
probability.
21
Neural Network with Softmax Output
▪ Binary Classification problem: Is this the digit ‘0’ or ‘1’
▪ Handwritten digit classification: Is this the digit ‘0’, ‘1’,…., ‘9’`
Softmax
22
ReLu
Neural Network with Softmax Output
▪ Specify the model
▪ Specify loss and cost
▪ Train
23
Improving Numerical Stability in Softmax
▪ Avoiding Round-off Errors for Accurate Computations
▪ Computers use floating-point numbers with limited precision.
▪ Different calculation methods for the same value can lead to varying
round-off errors.
▪ For example: Calculating 2/10,000 directly vs. (1 + 1/10,000) - (1 -
1/10,000)
▪ Try it in python
24
Round-off Errors and the Softmax Cost Function
▪ Traditional softmax calculation involves intermediate steps that can
accumulate round-off errors.
▪ These errors can become significant, especially with very large or
very small exponent values in the softmax formula.
25
Numerical Roundoff Error: Take Home
▪ What does from_logits=True do?
[Link](loss=SparseCategoricalCrossEntropy(from_logits=True))
26
More Numerically accounte implementation
Model now gives z
27
Logistic Regression (more numerically accurate)
Model now gives from z_1, z_2,…,z_10
28
Multi-class versus multi-label
▪ Multi-class: One label per input.
• Example: Handwritten digit classification (0-9).
▪ Multi-label: Multiple labels per input.
• Example: Image tagging (car, pedestrian, bus).
29
Multi-label Classification
30
Neural Network Architecture
▪ Train individual networks for each label (car, bus, pedestrian).
Car
Bus
Pedestrian
Sigmoid
31
Advanced Optimization
▪ Gradient descent uses a single learning rate (alpha) for all
parameters.
▪ Inefficiency: Can be slow to converge, especially with complex cost
functions.
▪ Sensitivity to Learning Rate:
• Too small, and learning is slow.
• Too large, and it may overshoot the minimum.
32
33
Adam Algorithm Intuition
▪ Adaptive Learning Rates: Adam calculates individual learning rates for each
parameter.
▪ Momentum: Considers past gradients to smooth out the optimization process.
▪ Faster Convergence: Typically converges faster than traditional gradient descent.
34
35
TensorFlow Implementation
36