Neural networks

Learning
Forward propagation
• for one example (x, y)
a[0] = x a[0]=x a[1] a[2] h=a[3]
z[1] = W[1] a[0] + b[1]
a[1] = g(z[1])
z[2] = W[2] a[1] + b[2]
a[2] = g(z[2])
z[3] = W[3] a[2] + b[3]
layer 0 layer 1 layer 2 layer 3
a[3] = g(z[3]) = h(x)
Minimizing cost function
• now, we have the prediction (h(x)) and the desired output (y)
• we have to calculate
• the value of 𝐽(𝑤)
𝜕
• partial derivatives [𝑙] 𝐽(𝑤) to compute the gradient
𝑤𝑖𝑗
Calculating partial derivatives of J
• it can be mathematically proven that
𝜕 [𝑙−1] [𝑙] 𝜕 [𝑙]
[𝑙] 𝐽 𝑤 = 𝑎𝑗 𝛿𝑖 , [𝑙] 𝐽 𝑤 = 𝛿𝑖 (regularization ignored now)
𝑤𝑖𝑗 𝑏𝑗

[𝑙] 𝜕
𝛿𝑗 = [𝑙] 𝐽 𝑤 … “error” of node j in layer l
𝑧𝑗
• we have to calculate all δj[l], i.e., how different are the desired activation
values (in fact, the values of z) based on the training set and values
calculated by the NN
• one we have delta terms, the gradients for weights and biases can be
computed
Backpropagation algorithm
• for the last layer (l=3)*:
• δj[3] = aj[3] – yj (vectorized version: δ[3] = a[3] – y)

a[0]=x a[1] a[2] h=a[3]

*if the cross entopy loss is used together with the sigmoid, see later
 a[0]=x a[1] a[2] h=a[3]

Backpropagation algorithm
• for previous layers
• δ[2] = (W[3])T δ[3] .* g’(z[2]) g’(z[2]) = a[2] .* (1 – a[2])*
• δ[1] = (W[2])T δ[2] .* g’(z[1]) g’(z[1]) = a[1] .* (1 – a[1])
.* … element-wise multiplication (Hadamard product)

• each neuron receives some error from the layer above through W and
considers the local slope of the cost function (g’); unit j contributes to the
errors in the following layer, depending on the weights and the slope of J:
[𝑙] [𝑙+1] [𝑙+1] [𝑙]
𝛿𝑗 = ෍ 𝑤𝑘𝑗 𝛿𝑘 𝑔′(𝑧𝑗 )
𝑘

* if g is the sigmoid, g’(z) = g(z)(1-g(z)), g(z)=a

Backpropagation algorithm
• there is no δ[0] because the first layer represents inputs and there are
no errors associated with them

a[0]=x a[1] a[2] h=a[3]

Backpropagation algorithm
Training set {(x(1), y(1)), (x(2), y(2)), …, (x(m), y(m)))
Set Δij[l] = 0 for all i, j, l
for i = 1 to m
set a[0] = x(i)
perform forward propagation to compute a[l] for l = 1, 2, …, L
compute δ[L] = a[L] – y(i)
compute δ[L-1], δ[L-2], …, δ[1]
Δij[l] = Δij[l] + aj[l-1]δi[l] (can be vectorized: Δ[l] = Δ[l] + δ[l](aj[l-1])T)
Dij[l] = 1/m Δij[l] + λwij[l] if j ≠ 0
Dij[l] = 1/m Δij[l] if j = 0 (bias terms)
Backpropagation algorithm
single value
Training set {(x(1), y(1)), (x(2), y(2)), …, (x(m), y(m))) entire layer
Set Δij[l] = 0 for all i, j, l
for i = 1 to m
set a[0] = x(i)
perform forward propagation to compute a[l] for l = 1, 2, …, L
compute δ[L] = a[L] – y(i)
compute δ[L-1], δ[L-2], …, δ[1]
Δij[l] = Δij[l] + aj[l-1]δi[l] (can be vectorized: Δ[l] = Δ[l] + δ[l](aj[l-1])T)
Dij[l] = 1/m Δij[l] + λwij[l] if j ≠ 0
Dij[l] = 1/m Δij[l] if j = 0 (bias terms)
Backpropagation algorithm
single value
Training set {(x(1), y(1)), (x(2), y(2)), …, (x(m), y(m))) entire layer
Set Δij[l] = 0 for all i, j, l
for i = 1 to m
set a[0] = x(i)
perform forward propagation to compute a[l] for l = 1, 2, …, L
compute δ[L] = a[L] – y(i)
compute δ[L-1], δ[L-2], …, δ[1]
Δij[l] = Δij[l] + aj[l-1]δi[l] (can be vectorized: Δ[l] = Δ[l] + δ[l](aj[l-1])T)
Dij[l] = 1/m Δij[l] + λwij[l] if j ≠ 0 𝝏 (𝒍)
[l]
Dij = 1/m Δij [l] if j = 0 (bias terms) 𝒍
𝑱 𝒘 = 𝑫𝒊𝒋
𝒘𝒊𝒋
a neural network for two inputs and one output

x1

x2

input layer 1 layer 2 output

adding bias terms

+1 +1 +1

x1

x2

input layer 1 layer 2 output

 forward calculations

+1 +1 +1

x1 z1[1] → a1[1] z1[2] → a1[2] z1[3] → a1[3]

x2 z2[1] → a2[1] z2[2] → a2[2]

input layer 1 layer 2 output

 𝟏
a[1] = a [1]
forward calculations 1
a2[1]

w [2] w11[2] w12[2]

10
+1 +1 +1 w[2] = w [2] w21[2] w22[2]
20
w10[2]
w20[2]
w [2] z[2] = w[2]a[1]
11
x1 z1[1] → a1 [1] z1[2] → a1[2] z1[3] → a1[3]
w21[2]
w12[2] a[2] = g(z[2])
x2 z2[1] → a2[1] z2[2] → a2[2]
w22[2]
input layer 1 layer 2 output
 𝟏
a[2] = a [2]
forward calculations 1
a2[2]

w [2] w11[2] w12[2]

10
+1 +1 +1 w[2] = w [2] w21[2] w22[2]
20
w10[2]
w11[2] z[2] = w[2]a[1]
x1 z1[1] → a1[1] z1[2] → a1[2] z1[3] → a1[3]

w12[2] a[2] = g(z[2])

x2 z2[1] → a2[1] z2[2] → a2[2]

input layer 1 layer 2 output

 𝟏
a[2] = a [2]
forward calculations 1 z1[2] = w10[2]*1 + w11[2]*a1[1] + w12[2]*a2[1]
a2[2]

w [2] w11[2] w12[2]

10
+1 +1 +1 w[2] = w [2] w21[2] w22[2]
20
w10[2]
w11[2] z[2] = w[2]a[1]
x1 z1[1] → a1[1] z1[2] → a1[2] z1[3] → a1[3]

w12[2] a[2] = g(z[2])

x2 z2[1] → a2[1] z2[2] → a2[2]

input layer 1 layer 2 output

 errors calculations

+1 +1 +1

x1 z1[1] → a1[1] z1[2] → a1[2] z1[3] → a1[3]

δ1[3] = a1[3] – y1

x2 z2[1] → a2[1] z2[2] → a2[2]

input layer 1 layer 2 output

 errors calculations

+1 +1 +1

x1 z1[1] → a1[1] z1[2] → a1[2] z1[3] → a1[3]

δ1[2] δ1[3]

x2 z2[1] → a2[1] z2[2] → a2[2]

δ2[2]
input layer 1 layer 2 output
 errors calculations

+1 +1 +1

x1 z1[1] → a1[1] z1[2] → a1[2] z1[3] → a1[3]

δ1[1] δ1[2] δ1[3]

x2 z2[1] → a2[1] z2[2] → a2[2]

δ2[1] δ2[2]
input layer 1 layer 2 output
 w [2] w [2] w12[2] [2])
[2] 10 11
w = w [2] w [2] δ
w22[2] δ[2] = 1 [2]
errors calculations 20 21 δ2
δ[1] = (w[2])T δ[2] .*(a[1] .* (1 - a[1]))
(remove δ [1])
+1 +1 +1 0

w11[2]
x1 z1[1] → a1[1] z1[2] → a1[2] z1[3] → a1[3]
w [2]
21
δ1[1] δ1[2] δ1[3]
w12[2]
x2 z2[1] → a2[1] z2[2] → a2[2]
w22[2]
δ2[1] δ2[2]
input layer 1 layer 2 output
 w [2] w [2] w12[2]
[2] 10 11
w = w [2] w [2] w22[2]
errors calculations 20 21

δ[1] = (w[2])T δ[2] …

+1 +1 +1 δ2[1] = w12[2] δ1[2] + w22[2] δ2[2] …

x1 z1[1] → a1[1] z1[2] → a1[2] z1[3] → a1[3]

δ1[1] w12[2] δ1[2] δ1[3]

x2 z2[1] → a2[1] z2[2] → a2[2]

w22[2]
δ2[1] δ2[2]
input layer 1 layer 2 output
Computational graph
• function J depends on three variables: J = 3(a + bc)
• a computational graph represents the variables and operations and
describes the computations

a
v=a+u J = 3v 3
b
a
u = bc J
c + v *
b
u
*
c
Computational graph

a=5

b=3 v=a+u J = 3v

u = bc v = 11 J = 33
c=2
u=6
Computational graph

a=5

b=3 v=a+u J = 3v

u = bc v = 11 J = 33
c=2
u=6 v = 11 → 11.001
J = 33 → 33.003
𝑑𝐽
=3
𝑑𝑣
Computational graph

a=5

b=3 v=a+u J = 3v

u = bc v = 11 J = 33
c=2
u=6 a = 5 → 5.001
v = 11 → 11.001
v = 11 → 11.001
J = 33 → 33.003
𝑑𝑣
=1 𝑑𝐽
𝑑𝑎 =3
𝑑𝑣
Computational graph a = 5 → 5.001
v = 11 → 11.001 𝑑𝐽 𝑑𝐽 𝑑𝑣
= = 3*1 = 3
𝑑𝑎 𝑑𝑣 𝑑𝑎
J = 33 → 33.003
a=5

b=3 v=a+u J = 3v

u = bc v = 11 J = 33
c=2
u=6 a = 5 → 5.001
v = 11 → 11.001
v = 11 → 11.001
J = 33 → 33.003
𝑑𝑣
=1 𝑑𝐽
𝑑𝑎 =3
𝑑𝑣
Chain rule
• if a variable z depends on the variable y, which depends on the
variable x , then z depends on x via the intermediate variable y
𝑑𝑧 𝑑𝑧 𝑑𝑦
=
𝑑𝑥 𝑑𝑦 𝑑𝑥
• also describes the composition of functions:
𝑑
𝑓 𝑔(𝑥) = 𝑓 ′ 𝑔 𝑥 𝑔′ (𝑥)
𝑑𝑥
Computational graph a = 5 → 5.001
v = 11 → 11.001 𝑑𝐽 𝑑𝐽 𝑑𝑣
= = 3*1 = 3
𝑑𝑎 𝑑𝑣 𝑑𝑎
J = 33 → 33.003
a=5

b=3 v=a+u J = 3v

u = bc v = 11 J = 33
c=2
u=6 a = 5 → 5.001
v = 11 → 11.001
v = 11 → 11.001
J = 33 → 33.003
𝑑𝑣
u = 6 → 6.001 =1 𝑑𝐽
v = 11 → 11.001
𝑑𝑎 =3
𝑑𝑣
J = 33 → 33.003
𝑑𝐽 𝑑𝐽 𝑑𝑣
= =3*1=3
𝑑𝑢 𝑑𝑣 𝑑𝑢
 Computational graph a = 5 → 5.001
v = 11 → 11.001 𝑑𝐽 𝑑𝐽 𝑑𝑣
= = 3*1 = 3
𝑑𝑎 𝑑𝑣 𝑑𝑎
J = 33 → 33.003
a=5

b=3 v=a+u J = 3v

u = bc v = 11 J = 33
c=2
u=6 a = 5 → 5.001
v = 11 → 11.001
v = 11 → 11.001
J = 33 → 33.003
𝑑𝑣
b = 3 → 3.001 u = 6 → 6.001 =1 𝑑𝐽
u = 6 → 6.002 v = 11 → 11.001
𝑑𝑎 =3
𝑑𝑣
J = 33 → 33.003 J = 33 → 33.003
𝑑𝐽 𝑑𝐽 𝑑𝑣 𝑑𝑢 𝑑𝐽 𝑑𝐽 𝑑𝑣
= = 3*1*2=6 = =3*1=3
𝑑𝑏 𝑑𝑣 𝑑𝑢 𝑑𝑏 𝑑𝑢 𝑑𝑣 𝑑𝑢
Logistic regression derivatives
w1
x1
w2 z = w1x1 + w2x2 + b a = g(z) J(a, y)
x2
b
Logistic regression derivatives
J(a,y) = − 𝑦 log 𝑎 − 1 − 𝑦 log 1 − 𝑎
w1
x1
w2 z = w1x1 + w2x2 + b a = g(z) J(a, y)
x2
b
𝑑𝐽(𝑎, 𝑦) 𝑦 1−𝑦
=− +
𝑑𝑎 𝑎 1−𝑎
Logistic regression derivatives
w1
x1
w2 z = w1x1 + w2x2 + b a = g(z) J(a, y)
x2
b
𝑑𝑎 𝑑𝐽(𝑎, 𝑦) 𝑦 1−𝑦
= 𝑎(1 − 𝑎) =− +
𝑑𝑧 𝑑𝑎 𝑎 1−𝑎
Logistic regression derivatives
w1
x1
w2 z = w1x1 + w2x2 + b a = g(z) J(a, y)
x2
b
𝑑𝑎 𝑑𝐽(𝑎, 𝑦) 𝑦 1−𝑦
= 𝑎(1 − 𝑎) =− +
𝑑𝑧 𝑑𝑎 𝑎 1−𝑎

𝑑𝐽 𝑑𝐽 𝑑𝑎
= =𝑎−𝑦
𝑑𝑧 𝑑𝑎 𝑑𝑧
Logistic regression derivatives
w1
x1
w2 z = w1x1 + w2x2 + b a = g(z) J(a, y)
x2
b
𝑑𝑧 𝑑𝑎 𝑑𝐽(𝑎, 𝑦) 𝑦 1−𝑦
= 𝑥1 = 𝑎(1 − 𝑎) =− +
𝑑𝑤1 𝑑𝑧 𝑑𝑎 𝑎 1−𝑎
𝑑𝑧
= 𝑥2
𝑑𝑤2
𝑑𝑧 𝑑𝐽 𝑑𝐽 𝑑𝑎
=1 = =𝑎−𝑦∗
𝑑𝑏 𝑑𝑧 𝑑𝑎 𝑑𝑧

*this is the delta term for the last layer in a NN

Logistic regression derivatives
w1
x1
w2 z = w1x1 + w2x2 + b a = g(z) J(a, y)
x2
b
𝑑𝑧 𝑑𝑎 𝑑𝐽(𝑎, 𝑦) 𝑦 1−𝑦
= 𝑥1 = 𝑎(1 − 𝑎) =− +
𝑑𝑤1 𝑑𝑧 𝑑𝑎 𝑎 1−𝑎
𝑑𝑧
= 𝑥2
𝑑𝑤2
𝑑𝑧 𝑑𝐽 𝑑𝐽 𝑑𝑎
=1 = =𝑎−𝑦
𝑑𝑏 𝑑𝑧 𝑑𝑎 𝑑𝑧

𝑑𝐽 𝑑𝐽 𝑑𝐽 𝑑𝐽 𝑑𝐽 𝑑𝐽
= 𝑥1 = 𝑥2 =
𝑑𝑤1 𝑑𝑧 𝑑𝑤2 𝑑𝑧 𝑑𝑏 𝑑𝑧
Logistic regression on m examples
• average cost is calculated:
1 𝑚
J(w) = σ𝑖=1 𝑐𝑜𝑠𝑡(ℎ(𝑖) , 𝑦 (𝑖) )
𝑚
𝜕𝐽
• partial derivatives are also averages of individual loss terms:
𝜕𝑤𝑖
𝑚
𝜕𝐽(w) 1 𝜕
= ෍ 𝑐𝑜𝑠𝑡(ℎ(𝑖) , 𝑦 (𝑖) )
𝜕𝑤𝑖 𝑚 𝜕𝑤𝑖
𝑖=1
 Neural network derivatives
W[2]
x b[2]

W[1] z[1]=W[1]x+b[1] a[1]=g(z[1]) z[2]=W[2]a[1]+b[2] a[2]=g(z[2]) J(a[2], y)

b[1]
 Neural network derivatives
W[2]
x b[2]

W[1] z[1]=W[1]x+b[1] a[1]=g(z[1]) z[2]=W[2]a[1]+b[2] a[2]=g(z[2]) J(a[2], y)

da[2]
b[1]
 Neural network derivatives
W[2]
x b[2]

W[1] z[1]=W[1]x+b[1] a[1]=g(z[1]) z[2]=W[2]a[1]+b[2] a[2]=g(z[2]) J(a[2], y)

dz[2] da[2]
b[1]
𝑑𝑧 [2] = 𝑎[2] − 𝑦
(dz and da usually calculated in one step)
 Neural network derivatives
dW[2]
𝑑𝑊 [2] = 𝑑𝑧 [2] 𝑎 1 𝑇
W[2]
x b[2] 𝑑𝑏 [2] = 𝑑𝑧 [2]
db[2]
W[1] z[1]=W[1]x+b[1] a[1]=g(z[1]) z[2]=W[2]a[1]+b[2] a[2]=g(z[2]) J(a[2], y)
dz[2] da[2]
b[1]
𝑑𝑧 [2] = 𝑎[2] − 𝑦
 Neural network derivatives
dW[2]
𝑑𝑊 [2] = 𝑑𝑧 [2] 𝑎 1 𝑇
W[2]
x b[2] 𝑑𝑏 [2] = 𝑑𝑧 [2]
db[2]
W[1] z[1]=W[1]x+b[1] a[1]=g(z[1]) z[2]=W[2]a[1]+b[2] a[2]=g(z[2]) J(a[2], y)
dz[1] da[1] dz[2] da[2]
b[1]
𝑑𝑧 [1] 2 𝑇 𝑑𝑧 [2] .∗ 1 ′ (𝑧 1
=𝑊 𝑔 ) 𝑑𝑧 [2] = 𝑎[2] − 𝑦
(dz and da computed in one step)
da[1] = W[2]T dz[2] dz[1] = da[1] .* g[1]’(z[1])
 Neural network derivatives
dW[2]
𝑑𝑊 [2] = 𝑑𝑧 [2] 𝑎 1 𝑇
W[2]
x b[2] 𝑑𝑏 [2] = 𝑑𝑧 [2]
db[2]
dW[1]
W[1] z[1]=W[1]x+b[1] a[1]=g(z[1]) z[2]=W[2]a[1]+b[2] a[2]=g(z[2]) J(a[2], y)
dz[1] da[1] dz[2] da[2]
b[1] db[1]

𝑑𝑧 [1] 2 𝑇 𝑑𝑧 [2] .∗ 1 ′ (𝑧 1
=𝑊 𝑔 ) 𝑑𝑧 [2] = 𝑎[2] − 𝑦

𝑑𝑊 [1] = 𝑑𝑧 [1] 𝑥 𝑇
𝑑𝑏 [1] = 𝑑𝑧 [1]
 Neural network derivatives (vectorized)
1
W[2] dW[2]
𝑑𝑊 [2] = 𝑑𝑍 [2] 𝐴 1 𝑇
𝑚 1
X b[2] 𝑑𝑏 = 𝑠𝑢𝑚 𝑜𝑓 𝑟𝑜𝑤𝑠(𝑑𝑍 [2] )
[2]
dW[1]
db[2] 𝑚
W[1] Z[1]=W[1]X+b[1] A[1]=g(Z[1]) Z[2]=W[2]A[1]+b[2] A[2]=g(Z[2]) J(A[2], Y)
dz[1] da[1] dz[2] da[2]
b[1] db[1]

𝑑𝑍 [1] 2 𝑇 𝑑𝑍 [2] .∗ 1 ′ (𝑍 1
=𝑊 𝑔 ) 𝑑𝑍 [2] = 𝐴[2] − 𝑌

[1]
1
𝑑𝑊 = 𝑑𝑍 [1] 𝑋 𝑇
𝑚
1
𝑑𝑏 [1] = 𝑠𝑢𝑚 𝑜𝑓 𝑟𝑜𝑤𝑠(𝑑𝑍 [1] )
𝑚
The role of δ
• computing derivatives using the chain rule involves repeating some
computations, e.g.,
[𝟐] [𝟐] [𝟏] [1]
𝑑𝐽 𝒅𝑱 𝒅𝒂 𝒅𝒛 𝒅𝒂𝟏 𝑑𝑧1
[1]
= [𝟐] [𝟐] [𝟏] [𝟏] [1]
𝑑𝑤11 𝒅𝒂 𝒅𝒛 𝒅𝒂𝟏 𝒅𝒛𝟏 𝑑𝑤11
[𝟐] [𝟐] [𝟏] [1]
𝑑𝐽 𝒅𝑱 𝒅𝒂 𝒅𝒛 𝒅𝒂𝟏 𝑑𝑧1
[1]
= [𝟐] [𝟐] [𝟏] [𝟏] [1]
𝑑𝑤12 𝒅𝒂 𝒅𝒛 𝒅𝒂𝟏 𝒅𝒛𝟏 𝑑𝑤12
• δ stores the chain rule computation up to the neuron and can be used to
easily compute derivatives with respect to all incoming weights, e.g.,
[1] [1]
𝑑𝐽 𝑑𝐽 𝑑𝑧1 [1] 𝑑𝑧1 [1] [0]
[1] = [1] [1] = 𝛿1 [1] = 𝛿1 𝑎1
𝑑𝑤11 𝑑𝑧1 𝑑𝑤11 𝑑𝑤11
Derivatives
derivative = the slope of the tangent line at the given point

f(x)=3x f(x)=x2

1 2 x 1 2 x

df/dx = 3 df/dx = 2x
Numerical estimation of gradients
• the gradient = the slope of a function in a given point

J(w)

real
slope

w w
Numerical estimation of gradients
• the gradient = the slope of a function in a given point

J(w)
J(w+ε)

J(w-ε)

w-ε w w+ε w
Numerical estimation of gradients
• the gradient = the slope of a function in a given point

J(w)
estimated
J(w+ε)
slope
J(w-ε)

w-ε w w+ε w
Numerical estimation of gradients
• the gradient = the slope of a function in a given point

J(w)
estimated
J(w+ε)
slope
J(w-ε)
real
slope

w-ε w w+ε w
Numerical estimation of gradients
• the gradient = the slope of a function in a given point

J(w)
estimated
J(w+ε)
slope =
J(w-ε) J(w+ε) −J(w−ε)
real 2ε
slope

w-ε w w+ε w
Numerical estimation of gradients
• can be applied also to partial derivations

𝜕 𝐽 𝑤1 + 𝜀, 𝑤2 , 𝑤3 , … , 𝑤𝑛 − 𝐽 𝑤1 − 𝜀, 𝑤2 , 𝑤3 , … , 𝑤𝑛
𝐽 𝑤 ≈
𝜕𝑤1 2𝜀
𝜕 𝐽 𝑤1 , 𝑤2 + 𝜀, 𝑤3 , … , 𝑤𝑛 − 𝐽 𝑤1 , 𝑤2 − 𝜀, 𝑤3 , … , 𝑤𝑛
𝐽 𝑤 ≈
𝜕𝑤2 2𝜀
⋮
𝜕 𝐽 𝑤1 , 𝑤2 , 𝑤3 , … , 𝑤𝑛 + 𝜀 − 𝐽 𝑤1 , 𝑤2 − 𝜀, 𝑤3 , … , 𝑤𝑛 − 𝜀
𝐽 𝑤 ≈
𝜕𝑤𝑛 2𝜀
Numerical estimation of gradients
• numerical estimation is very slow
• can be used to check whether the implementation of
backpropagation is correct
• the gradient calculated using backprop should be close to the estimated one
• after the check, numerical estimation should be disabled
Initialization of weights
• some values of w must be specified at the beginning
Zero initialization
• after forward propagation, a1[1] = a2[1]
• this means that δ1[1] = δ2[1]
𝜕 𝜕 , 𝜕 𝜕 , 𝜕 𝜕
=> [1] 𝐽 𝑤 = [1] 𝐽 𝑤 [1] 𝐽 𝑤 = [1] 𝐽 𝑤 [1] 𝐽 𝑤 = [1] 𝐽 𝑤
𝑤10 𝑤20 𝑤11 𝑤21 𝑤12 𝑤22
=> w10[1] = w20[1] etc. after update
=> a1[1] remains equal to a2[1] etc. +1 +1
(this is redundant)
x1 a1[1]

x2 a2[1]
Initialization of weights
• random initialization enables so called symmetry breaking
• each wij[l] is initialized to a value from [-ε; +ε]
Putting it together
• choosing the architecture of a NN
• number of input units = number of features
• number of output units = number of classes
• reasonable default
• one hidden layer
• if more hidden layers, they have the same number of units
• the number of units comparable to the number of inputs
Putting it together – training
• randomly initialize weights
• implement forward propagation to get hw(x(i)) for any x(i)
• implement the computation of J(w)
• implement backpropagation to compute errors
Putting it together – training
• for i = 1 to m
• perform forward and backpropagation for (x(i), y(i)), i.e., get activations a[l] and delta
terms δ[l]
• update DELTA terms, i.e., Δ[l] = Δ[l] + δ[l+1](a[l])T
𝜕
• compute partial derivatives Dij[l] = [𝑙] 𝐽 𝑤
𝑤𝑖𝑗
Dij[l] = 1/m Δij[l] + λwij[l] if j ≠ 0
Dij[l] = 1/m Δij[l] if j = 0 (bias terms)
• possibly use gradient checking
• use gradient descent (or any other optimization method) to find optimal
weights (that minimize J)