PART I
Deep Learning Basics
Slides Borrowed From:
Angjoo Kanazawa (University of Maryland, College Park)
First of all what is Deep Learning?
● Composition of non-linear transformation of
the data.
● Goal: Learn useful representations, aka
features, directly from data.
● Many varieties, can be unsupervised or supervised.
● Today is about ConvNets, which is a supervised deep
learning method.
Recap: Supervised Learning
Slide: M. Ranzato
Supervised Learning: Examples
Slide: M. Ranzato
Supervised Deep Learning
So deep learning is about learning
feature representation in a
compositional manner.
But wait,
why learn features?
The Black Box in a
Traditional Recognition Approach
Feature Post-processing Classifier
Preprocessing Extraction (Feature selection, (SVM,
(HOG, SIFT, etc) MKL etc) boosting, etc)
The Black Box in a
Traditional Recognition Approach
Hand
Engin
eered
Feature Post-processing Classifier
Preprocessing Extraction (Feature selection, (SVM,
(HOG, SIFT, etc) MKL etc) boosting, etc)
Feature Post-processing
Preprocessing Extraction (Feature selection,
(HOG, SIFT, etc) MKL etc)
● Most critical for accuracy
● Most time-consuming in development
● What is the best feature???
● What is next?? Keep on crafting better features?
⇒ Let’s learn feature representation directly
from data.
Learn features and classifier
together
⇒ Learn an end-to-end recognition system.
A non-linear map that takes raw pixels directly
to labels.
Slide: M. Ranzato
Building a complicated function
Each box is a simple nonlinear function
Building a complicated function
Slide: M. Ranzato
Building a complicated function
● Composition is at the core of deep learning methods
● Each “simple function” will have parameters subject to
learning Slide: M. Ranzato
Intuition behind Deep Neural Nets
Slide: M. Ranzato
Intuition behind Deep Neural Nets
Slide: M. Ranzato
Intuition behind Deep Neural Nets
Layer 1 Layer 2 Layer 3 Layer 4
The final layer outputs a probability distribution of categories.
Slide: M. Ranzato
A simple single layer Neural Network
Consists of a linear combination of input
through a nonlinear function:
W is the weight parameter to be learned.
x is the output of the previous layer
f is a simple nonlinear function. Popular choice is max(x,0),
called ReLu (Rectified Linear Unit)
1 layer: Graphical Representation
f h is called a neuron,
hidden unit or feature.
f
f
Joint training architecture overview
Neural Net Training
Slide: M. Ranzato
Neural Net Training
Slide: M. Ranzato
Neural Net Training
Slide: M. Ranzato
Neural Net Training
Slide: M. Ranzato
Neural Net Training
Slide: M. Ranzato
Neural Net Training
Slide: M. Ranzato
Neural Net Training
Slide: M. Ranzato
PART II
ImageNet Classification with
Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton,
Advances in Neural Information Processing Systems 2012
Slides Borrowed From:
Barnabás Póczos & Aarti Singh
(Carnegie Mellon University)
28
ImageNet
15M images
22K categories
Images collected from Web
Human labelers (Amazon’s Mechanical Turk crowd-sourcing)
ImageNet Large Scale Visual Recognition Challenge (ILSVRC-2010)
o 1K categories
o 1.2M training images (~1000 per category)
o 50,000 validation images
o 150,000 testing images
RGB images
Variable-resolution, but this architecture scales them to 256x256 size
29
ImageNet
Classification goals:
Make 1 guess about the label (Top-1 error)
make 5 guesses about the label (Top-5 error)
30
The Architecture
Typical nonlinearities:
Here, however, Rectified Linear Units (ReLU) are used:
Empirical observation: Deep convolutional neural networks with
ReLUs train several times faster than their equivalents with tanh units
A four-layer convolutional neural
network with ReLUs (solid line)
reaches a 25% training error rate on
CIFAR-10 six times faster than an
equivalent network with tanh neurons
(dashed line)
31
The Architecture
The first convolutional layer filters the 224×224×3 input image with
96 kernels of size 11×11×3 with a stride of 4 pixels (this is the distance
between the receptive field centers of neighboring neurons in the kernel
map. 224/4=56
The pooling layer: form of non-linear down-sampling. Max-pooling
partitions the input image into a set of rectangles and, for each such sub-
region, outputs the maximum value
32
The Architecture
Trained with stochastic gradient descent
on two NVIDIA GTX 580 3GB GPUs
for about a week
650,000 neurons
60,000,000 parameters
630,000,000 connections
5 convolutional layer, 3 fully connected layer
Final feature layer: 4096-dimensional
33
Dropout
We know that combining different models can be very useful
(Mixture of experts, majority voting, boosting, etc)
Training many different models, however, is very time consuming.
The solution:
Dropout: set the output of each hidden neuron to zero w.p. 0.5.
35
Dropout
Dropout: set the output of each hidden neuron to zero w.p. 0.5.
The neurons which are “dropped out” in this way do not contribute to
the forward pass and do not participate in backpropagation.
So every time an input is presented, the neural network samples a
different architecture, but all these architectures share weights.
This technique reduces complex co-adaptations of neurons, since a
neuron cannot rely on the presence of particular other neurons.
It is, therefore, forced to learn more robust features that are useful in
conjunction with many different random subsets of the other neurons.
Without dropout, our network exhibits substantial overfitting.
Dropout roughly doubles the number of iterations required to converge.
36
The first convolutional layer
96 convolutional kernels of size 11×11×3 learned by the first
convolutional layer on the 224×224×3 input images.
The top 48 kernels were learned on GPU1 while the bottom 48 kernels
were learned on GPU2
Looks like Gabor wavelets, ICA filters… 37
Results
Results on the test data:
top-1 error rate: 37.5%
top-5 error rate: 17.0%
ILSVRC-2012 competition:
15.3% accuracy
2nd best team: 26.2% accuracy
38
Results
39
Results: Image similarity
six training images that produce feature vectors in
Test column the last hidden layer with the smallest Euclidean distance
from the feature vector for the test image. 40