Finish Backpropagation and then

a DNN PyTorch Example

CSE 5526 - Intro to Neural Networks
Lecture 9

Instructor: Donald S. Williamson

@TheASPIREGrpOSU

theaspiregroup_osu

The ASPIRE Group @ OSU

1
Agenda and Learning Outcomes
This Week’s Topic(s)

• Topic(s): DNN Python Example

• Finish Backpropogation using vectors
• DNN Python example
• PyTorch coding
• Softmax Layer for Multi-class classi cation
• Announcements:
• Exam I will be on Thursday, February 19th (more info on this later)

2
fi
 (H+1)×1
ỹh ∈ ℝ

Backprop: Output Layer (cont.)

O×1
vo, ϕ(vo) ∈ ℝ

Wo ∈ ℝ(H+1)×O
Stochastic Case for Weight Matrix
yo ∈ ℝO×1
O×1
d∈ℝ
1 T T T 1 T T T
Ex(n) = (d d − 2d yo + yo yo) = (d d − 2d ϕ(vo) + ϕ(vo) ϕ(vo))
2 2
1 T T T T T T
= (d d − 2d ϕ(Wo ỹh) + ϕ(Wo ỹh) ϕ(Wo ỹh))
2

Must compute gradient w.r.t. the weight in the output layer

T

∂Wo ( ∂vo )
∂vo ∂yo This results in a Jacobian matrix, but due to independence
∇Wo E = −(d − yo) ⊙ of the output neurons (i.e. no cross-derivative terms)
∂yo A vector of
= diag(Φ′(vo)) ≡ ϕ′(vo) derivatives, one for
∂vo each output neuron


3
Jacobian Matrix
∂yo,i
[ ∂vo ]
∂yo
The Jacobian matrix is de ned element wise as: =
• ∂vo,j
ij

• Hence, a matrix of partial derivatives between the i-th output neuron and the j-th
neuron’s activation potential

• Because the i-th activation is only given to the i-the neuron

∂yo,i
If i = j, = ϕ′(vo,i) Thus,
• ∂vo,j ϕ′(vo,1) 0 ⋯ 0

[ ∂vo ]
∂yo 0 ϕ′(vo,2) ⋯ 0
∂yo,i = = diag(Φ′(vo))

If i ≠ j, =0 ⋮ ⋮ ⋱ ⋮
• ∂vo,j ij
0 0 ⋯ ϕ′(vo,O)




fi
 (H+1)×1
ỹh ∈ ℝ

Backprop: Output Layer (cont.)

O×1
vo, ϕ(vo) ∈ ℝ

Wo ∈ ℝ(H+1)×O
Stochastic Case for Weight Matrix
yo ∈ ℝO×1
O×1
d∈ℝ
1 T T T 1 T T T
Ex(n) = (d d − 2d yo + yo yo) = (d d − 2d ϕ(vo) + ϕ(vo) ϕ(vo))
2 2
1 T T T T T T
= (d d − 2d ϕ(Wo ỹh) + ϕ(Wo ỹh) ϕ(Wo ỹh))
2

Must compute gradient w.r.t. the weight in the output layer

∂vo
(−(d − yo) ⊙ ϕ̄′(vo))
T
∇Wo E =
∂Wo
∂yo A vector of
Alternatively, could just use the = diag(Φ′(vo)) ≡ ϕ̄′(vo) derivatives, one for
Jacobian matrix, and have ∂vo each output neuron
matrix multiplication instead



5
 (H+1)×1
ỹh ∈ ℝ

Backprop: Output Layer (cont.)

O×1
vo, ϕ(vo) ∈ ℝ

Wo ∈ ℝ(H+1)×O
Stochastic Case for Weight Matrix
yo ∈ ℝO×1
O×1
d∈ℝ
1 T T T 1 T T T
Ex(n) = (d d − 2d yo + yo yo) = (d d − 2d ϕ(vo) + ϕ(vo) ϕ(vo))
2 2
1 T T T T T T
= (d d − 2d ϕ(Wo ỹh) + ϕ(Wo ỹh) ϕ(Wo ỹh))
2

Must compute gradient w.r.t. the weight in the output layer

T
∂vo ∂vo ∂(Wo ỹh)
(−(d − yo) ⊙ ϕ̄′(vo))
T
∇Wo E = =
∂Wo ∂Wo ∂Wo

6
 (H+1)×1
ỹh ∈ ℝ

Backprop: Output Layer (cont.)

O×1
vo, ϕ(vo) ∈ ℝ

Wo ∈ ℝ(H+1)×O
Stochastic Case for Weight Matrix
yo ∈ ℝO×1
O×1
d∈ℝ
1 T T T 1 T T T
Ex(n) = (d d − 2d yo + yo yo) = (d d − 2d ϕ(vo) + ϕ(vo) ϕ(vo))
2 2
1 T T T T T T
= (d d − 2d ϕ(Wo ỹh) + ϕ(Wo ỹh) ϕ(Wo ỹh))
2

Vector Calculus Review: Derivative of a vector transpose scaled by a matrix

T T
• Suppose we have a vector, g , that is scaled by matrix A, i.e., g A
T
• The derivative of g A with respect to g is A

7
 (H+1)×1
ỹh ∈ ℝ

Backprop: Output Layer (cont.)

O×1
vo, ϕ(vo) ∈ ℝ

Wo ∈ ℝ(H+1)×O
Stochastic Case for Weight Matrix
yo ∈ ℝO×1
O×1
d∈ℝ
1 T T T 1 T T T
Ex(n) = (d d − 2d yo + yo yo) = (d d − 2d ϕ(vo) + ϕ(vo) ϕ(vo))
2 2
1 T T T T T T
= (d d − 2d ϕ(Wo ỹh) + ϕ(Wo ỹh) ϕ(Wo ỹh))
2

Must compute gradient w.r.t. the weight in the output layer

T
∂vo ∂vo ∂(Wo ỹh)
(−(d − yo) ⊙ ϕ̄′(vo))
T
∇Wo E = = = ỹh
∂Wo ∂Wo ∂Wo

8
 (H+1)×1
ỹh ∈ ℝ

Backprop: Output Layer (cont.)

O×1
vo, ϕ(vo) ∈ ℝ

Wo ∈ ℝ(H+1)×O
Stochastic Case for Weight Matrix
yo ∈ ℝO×1
O×1
d∈ℝ
1 T T T 1 T T T
Ex(n) = (d d − 2d yo + yo yo) = (d d − 2d ϕ(vo) + ϕ(vo) ϕ(vo))
2 2
1 T T T T T T
= (d d − 2d ϕ(Wo ỹh) + ϕ(Wo ỹh) ϕ(Wo ỹh))
2

Must compute gradient w.r.t. the weight in the output layer

∇Wo E = ỹh (−(d − yo) ⊙ ϕ̄′(vo)) = − ỹh ((d − yo) ⊙ ϕ̄′(vo))

T T

= − ỹh (e ⊙ ϕ̄′(vo))
T

T
=− ỹhδo



9
 (H+1)×1
ỹh ∈ ℝ

Backprop: Output Layer (cont.)

O×1
vo, ϕ(vo) ∈ ℝ

Wo ∈ ℝ(H+1)×O
Stochastic Case for Weight Matrix
yo ∈ ℝO×1
O×1
d∈ℝ

• Substituting this back into the weight update equation

Wo(n + 1) = Wo(n) − η ∇Wo Ex
T
= Wo(n) + ηỹh(n)δo (n) δ (delta) rule

where δo(n) = (eo ⊙ ϕ′(vo))

• Each neuron in the output layer will have it’s own delta (e.g., it is a vector)

10
Questions

11
Backprop: Hidden Layer
Stochastic Case for Weight Matrix

• Weight update equation Wh(n + 1) = Wh(n) − η ∇Wh Ex

• Via Chain Rule:
= − x̃ ((Wo δo) ⊙ ϕ̄′h(vh))
∂E ∂yo ∂vo ∂ỹh ∂vh T
∇Wh Ex =
∂yo ∂vo ∂ỹh ∂vh ∂Wh
T (m+1)×H
= − x̃δh ∈ ℝ

T
Wh(n + 1) = Wh(n) + ηx̃δh

12
Extension to More Hidden layers
Straightforward idea

• In general, for any layer with input (to that layer) ã and local error signal δ:
T
Δwab(n) = − ãδ

• The delta rule applies to the output layer, while the generalized delta rule
applies to hidden layers.

• For hidden layers, this is done in a layer-by-layer fashion, from the output end.
• The entire procedure is call backdrop because the error is propagated
backwards from the end to the beginning.

13
 Backprop (cont.)
Remarks on Backprop xi
wj i

• Illustration of the generalized δ rule,

δ1
• Backpropagation learning is local: δj
concerns only “pre-synaptic” and “post- xi j k δk
synaptic” neurons only

• Local minima are possible due to

nonlinearity:
• The generalized δ rule gives a solution to the credit (blame
• Infrequent when use online update assignment problem

• Helps to randomize order of input

presentation during each epoch

Part III
14
Questions

15
DNN Python Example

16
Digit Recognition Example
DNN Training and Testing Pipeline

• Suppose you are a Data Scientist at the post o ce. Your boss wants you to
build a classi cation model that converts handwritten digits to their digital form

• Data has been collected, which contains several handwritten examples of

each of the digits (e.g. 0 through 9)
• MNIST dataset will be used (classic baseline classi cation dataset)
• Contains 70,000 images of digits handwritten by high school students and
employees of the US Census Bureau
• Each image has a label with the digit it represents
• How can this be done using a deep neural network?
17
fi
ffi
fi
Collect and Normalize Data

• Torchvision (from PyTorch) has helper Downloads and normalizes the

functions to download popular datasets, MNIST dataset and returns a
which includes MNIST dictionary

• The code below gets the MNIST data

18
Split the Data
• Divide the data into training, validation, and testing sets
• MNIST has already been divided into training and testing sets.
• 60,000 images are for training and validation
• 10,000 images are for testing

19
Visualize the data
Plot example images

• Get images then de-

normalize them to their
proper range of values

• Use imshow() to plot the

images

20
Questions to consider
DNN Training

• Now that you have your data, let’s train a DNN classi er.
• What do I need to consider (e.g. design decisions)?:
• How many layers does my DNN need?
• How many neurons are needed in each layer?
• What type of activation function in each layer?
• How are my network weights initialized?
• How do I measure performance?
• …
21
fi
Define Hyper-parameters
It’s all Empirical.
• The number of hidden layers and the number of units in
these layers is a design decision
1. May start with one hidden layer, train/test performance.
Then add one layer at a time until performance gets worse
2. May use the same number of units in each layer.
3. May “funnel” number of units down from high to lower
values for each layer

• Three layer DNN (two hidden layers)

• 300 units in rst hidden layer. 100
units in 2nd hidden layer

22
fi
Data Loader for Mini-Batch Processing
Generally, you’re working with LARGE datasets with DNNs

• DataLoader is an e cient way to perform Mini-batch learning. It e ectively

manages batches and getting the input and label for each sample in each
batch

• This will change depending on the dataset

Check documentation of DataLoader

for description of all possible attributes

23
ffi
ff
 Define the Neural Network

• Need to extend [Link] to have all

necessary functionality
• Then need to override the __init__() function
• The __init__() function is where you de ne
the basic architecture (e.g. types of layers,
units per layer, number of layers, needed
functions,…)

You also determine weight/bias

initialization here (more on this to come)

• De ne the forward pass of the DNN

• Use desired activation function in
each layer
24
fi
fi
 Define the Neural Network

• Once the DNN architecture is de ned, create an instance of it.

• Then de ne the optimizer (e.g. Stochastic Gradient Descent, ADAM, ADAGRAD, …) (more on these
later)

• Also de ne the loss function. We are using Cross Entropy, since we are performing multi-class
classi cation

25
fi
fi
fi
fi
 DNNs as Multi-Class Classifiers
Softmax layer

• Many of the activation functions are for regression and not

classi cation? How do we handle classi cation problems with DNNs?

• For binary classi cation, we can merely threshold the output neuron
(e.g., a sigmoid or tanh)

• For multi-class classi cation (when the classes are exclusive), each Softmax
neuron in the output layer corresponds to a single class.

• The output layer is then modi ed to replace the individual activation

functions with a shared soft-max function

• Outputs represent estimated probability of the corresponding class

vk
e Cross Entropy Loss to train network
yk = pyk = ϕk(v) = K vj K
∑j=1 e
∑
Ece = − dk log(pyk)
26
k=1
fi
fi
fi
fi
fi
DNNs as Multi-Class Classifiers
Softmax layer - A Simple Example

• We want to minimize cross entropy.

• For a 4-class classi cation problem suppose the
label, d = [0,1,0,0]
• If yk = [0.2,0.3,0.4,0.1], the CE loss is 1.2
Softmax

• If yk = [0.1,0.8,0.05,0.05], the CE loss is 0.22 (e.g.,

better)
• Hence, vk became larger for the true label class Cross Entropy Loss to train network
vk
e K
yk = pyk = ϕk(v) =
∑
K
∑j=1 e vj Ece = − dk log(pyk)
27
k=1
fi
DNN Training
Training prediction and evaluation

• We can now begin the training

process, since everything has
been de ned and initialized

• We need to do a few other things:

• Keep track of training losses
• Iterate over each epoch
• Perform mini-batch gradient
descent

• Update the weights

28
fi
 DNN Training
Validation Loss for Performance Evaluation

• We need to always be
aware of the potential to
over t

• Hence, we should evaluate

the model as it trains, using
the validation/development
data

• DO NOT update network

based on this

29
fi
Test Performance for MNIST Digit Recognition
Generalization
• Results

30
Questions

31