Intro to Deep Learning
Geena Kim
*Some of the slide/diagram adopted from CMU deep learning course
Why Deep Learning?
C a rs
d riving
Sel f-
Why Deep Learning?
[Link]
Why Deep Learning?
Medical Imaging,
AI in Medicine
P. Rajpurkar et el.,
Why Deep Learning?
Shopping,
e-commerce
Computer vision in Robotics
Why Deep Learning?
Voice
Recog
Smar
t Dev nition,
ices
Deep learning in game playing
Deep learning face generation
Deep learning sound generation
[Link]
Deep learning explaining pictures
[Link]
The Deep Learning Age
Neurons in Biological Systems
Neurons in Biological Systems
What is Artificial Neuron (Perceptron)
History of neural networks
[Link]
History of neural networks
“Connectionism”
Connectionism is an approach in the field of cognitive science that
hopes to explain mental phenomena using artificial neural networks.
-Wikipedia
Alexander Bain: He worked on modeling of
brain and how memory works using neural
networks. Mind and body (1873)
History of neural networks
“Perceptron”
History of neural networks
“Perceptron”
Perceptron
• Binary Threshold (Step function)
Activation functions
• Sigmoid
• Binary Threshold (Step function) • Tanh
Perceptron
What can a perceptron do?
Perceptron- quiz
A perceptron does not need to have binary input,
but let’s consider a perceptron as a Boolean logic gate
X1 and X2 are binary inputs (either 0 or 1)
X1 W1 and W2 are weights set to 1
W1
What should the step activation function
Y
th=?
threshold “th” in order to have AND gate
output?
W2
X2
Perceptron- quiz
A perceptron does not need to have binary input,
but let’s consider a perceptron as a Boolean logic gate
X1 and X2 are binary inputs (either 0 or 1)
X1 W1 and W2 are weights set to 1
W1
What should the step activation function
Y
th=?
threshold “th” in order to have OR gate
output?
W2
X2
Perceptron
A perceptron can do more than just a binary logic gate.
It can also do classification or regression tasks.
Learning in perceptron
• Perceptron rule
• Delta rule (Gradient Descent)
Perceptron
Perceptron rule
Perceptron
Perceptron algorithm
Perceptron
Perceptron algorithm
Misclassified Positive Instance
Perceptron
Perceptron algorithm
Update weight (add to W) for the misclassified positive instance
Perceptron
Perceptron algorithm
Perceptron
Perceptron algorithm
Misclassified negative instance
Perceptron
Perceptron algorithm
Update weight (subtract from W) for the misclassified negative instance
Perceptron
Perceptron algorithm
Perceptron
Delta rule (Gradient Descent)
Perceptron
Learning in perceptron- conclusion
• Perceptron rule
• Delta rule (Gradient Descent)
Perceptron
w=1
X
OR X ˅Y
Y w=1 th=1
w=-1
X NOT ¬X
w=1
X 1
th=0
AND X ˄Y
Y w=1
th=2
Perceptron
Not possible to determine weights and threshold for XOR gate!
X
XOR X Y
Y
Perceptron
What’s special about XOR operation?
How do we handle linearly inseparable cases?
MLP
How do we handle linearly inseparable cases?
MLP
Neural networks are universal function approximator
Voice Texts image Caption
States Actions
MLP
We can make an arbitrary shape decision boundary
MLP
The colored area means that the neurons activate in the
region
MLP
MLP
MLP
MLP
MLP
MLP
Neural networks can be used for regression tasks
..
…
Multi-layer Perceptron (Neural network)
Design Parameters
- Architecture
- Number of layers
- Number of neurons in a layer
- Activation functions
Artificial Neural Network facts
Can approximate any non-linear function
-> Nonlinearity comes from nonlinear activation function
Deeper layer and larger number of neurons in a layer
-> more model capacity
MLP: fully-connected neural network
Weights for every connection
Weights = parameters
(cf) Hyperparameters = design parameters
Next time: How Neural Network Training Works
hj = f(wij * Xi + bj)
j k
i
0
y_pred y_target
1
…
2
wij wjk wk error/loss
input hidden hidden output
X_train
layer layer 1 layer 2 layer