Basics of Neural

Network Programming

Binary Classification
[Link]
Binary Classification

1 (cat) vs 0 (non cat)

Blue
Green 255 134 93 22
Red 255 134
123 202
94 22
83 2
255 231
123 42
34 22
94 83 4 30
44 187
123 94
34 83
44 2 192
34 187
76 232 124
34 4434 187
76 92
67 232 34 142
83 194
34 7667 232
83 124
194 94
67 83 194 202
Andrew Ng
Notation

Andrew Ng
 Basics of Neural
Network Programming

Logistic Regression
[Link]
Logistic Regression

Andrew Ng
 Basics of Neural
Network Programming

Logistic Regression
[Link]
cost function
Logistic Regression cost function
! "
!" = % & ' + ) , where % * = "#$ +,

Given (' (.) , ! (.) ),…,(' (1) , ! (1) ) , want !" (2) ≈ ! 2 .
Loss (error) function:

Andrew Ng
 Basics of Neural
Network Programming

Gradient Descent
[Link]
Gradient Descent
' ,
Recap: !" = % & ( + * , % + = ,-. /0
6 6
1 1
1 &, * = 4
5 ℒ(!" 7 , ! (7) ) =
− 5
4 ! (7) log !" 7 + (1 − ! (7) ) log(1 − !" 7 )
78, 78,

Want to find &, * that minimize 1 &, *

*
& Andrew Ng
Gradient Descent

Andrew Ng
 Basics of Neural
Network Programming

Derivatives
[Link]
Intuition about derivatives
! " = 3"

"
Andrew Ng
 Basics of Neural
Network Programming

More derivatives
[Link]
examples
Intuition about derivatives
! " = "$

"
Andrew Ng
More derivative examples

Andrew Ng
 Basics of Neural
Network Programming

Computation Graph
[Link]
Computation Graph

Andrew Ng
 Basics of Neural
Network Programming

Derivatives with a
[Link] Computation Graph
Computing derivatives
&=5
11 33
"=3 6 $ =&+! ) = 3$
!="#
#=2

Andrew Ng
Computing derivatives
&=5
11 33
"=3 6 $ =&+! ) = 3$
!="#
#=2

Andrew Ng
 Basics of Neural
Network Programming

Logistic Regression
[Link]
Gradient descent
Logistic regression recap

! = $%& + (
)* = + = ,(!)
ℒ +, ) = −() log(+) + (1 − )) log(1 − +))

Andrew Ng
Logistic regression derivatives
&%
$%
&( ! = $% &% + $( &( + ) * = +(!) ℒ(a, 1)
$(
b

Andrew Ng
 Basics of Neural
Network Programming

Gradient descent
[Link]
on m examples
Logistic regression on m examples

Andrew Ng
Logistic regression on m examples

Andrew Ng
 Basics of Neural
Network Programming

Vectorization
[Link]
What is vectorization?

Andrew Ng
 Basics of Neural
Network Programming

More vectorization
[Link]
examples
Neural network programming guideline
Whenever possible, avoid explicit for-loops.

Andrew Ng
Vectors and matrix valued functions
Say you need to apply the exponential operation on every element of a
matrix/vector.

!$
!= ⋮
!&

u = [Link]((n,1))
for i in range(n):
u[i]=[Link](v[i])

Andrew Ng
Logistic regression derivatives
J = 0, dw1 = 0, dw2 = 0, db = 0
for i = 1 to n:
! (") = " $ # (") + %
&(") = '(! (") )
* += − - (") log -1 " + (1 − - " ) log(1 − -1 " )
d! (") = &(") (1 − &(") )
(")
d"% += #% d! (")
(")
d"' += #' d! (")
db += d! (")
J = J/m, d"% = d"% /m, d"' = d"' /m, db = db/m

Andrew Ng
 Basics of Neural
Network Programming

Vectorizing Logistic
[Link]
Regression
Vectorizing Logistic Regression
! (#) = & ' ( (#) + * ! (-) = & ' ( (-) + * ! (.) = & ' ( (.) + *
+(#) = ,(! (#) ) +(-) = ,(! (-) ) +(.) = ,(! (.) )

Andrew Ng
 Basics of Neural
Network Programming

Vectorizing Logistic
[Link] Regression’s Gradient
Computation
Vectorizing Logistic Regression

Andrew Ng
Implementing Logistic Regression

J = 0, d!! = 0, d!" = 0, db = 0
for i = 1 to m:
" ($) = ! & # ($) + %
&($) = '(" ($) )
* += − - ($) log & $ + (1 − - $ ) log(1 − & $ )
d" ($) = &($) −- ($)
($)
d!! += #! d" ($)
($)
d!" += #" d" ($)
db += d" ($)
J = J/m, d!! = d!! /m, d!" = d!" /m
db = db/m
Andrew Ng
 Basics of Neural
Network Programming

Broadcasting in
[Link]
Python
Broadcasting example
Calories from Carbs, Proteins, Fats in 100g of different foods:
Apples Beef Eggs Potatoes
Carb 56.0 0.0 4.4 68.0
Protein 1.2 104.0 52.0 8.0
Fat 1.8 135.0 99.0 0.9

cal = [Link](axis = 0)
percentage = 100*A/([Link](1,4))
Broadcasting example
1 101
2 100 102
+ =
3 103
4 104

1 2 3 100 200 300 101 202 303

1 2 3 100 101 102 103

Explanation of logistic
[Link] regression cost function
(Optional)
Logistic regression cost function

Andrew Ng
Logistic regression cost function
If $ = 1: ( $ ) = $*
If $ = 0: ( $ ) = 1 − $*

Andrew Ng
Cost on m examples

Andrew Ng