Deep Learning for Computer Vision
MIT 6.S191
Ava Soleimany
January 29, 2019
6.S191 Introduction to Deep Learning
1/29/19
[Link]
What Computers “See”
Images are Numbers
6.S191 Introduction to Deep Learning
[1] 1/29/19
[Link]
Images are Numbers
6.S191 Introduction to Deep Learning
[1] 1/29/19
[Link]
Images are Numbers
What the computer sees
An image is just a matrix of numbers [0,255]!
i.e., 1080x1080x3 for an RGB image
6.S191 Introduction to Deep Learning
[1] 1/29/19
[Link]
Tasks in Computer Vision
Lincoln 0.8
Washington 0.1
classification
Jefferson 0.05
Obama 0.05
Input Image Pixel Representation
- Regression: output variable takes continuous value
- Classification: output variable takes class label. Can produce probability of belonging to a particular class
6.S191 Introduction to Deep Learning
1/29/19
[Link]
High Level Feature Detection
Let’s identify key features in each image category
Nose, Wheels, Door,
Eyes, License Plate, Windows,
Mouth Headlights Steps
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Manual Feature Extraction
Detect features
Domain knowledge Define features
to classify
Problems?
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Manual Feature Extraction
Detect features
Domain knowledge Define features
to classify
6.S191 Introduction to Deep Learning
[2] 1/29/19
[Link]
Manual Feature Extraction
Detect features
Domain knowledge Define features
to classify
6.S191 Introduction to Deep Learning
[2] 1/29/19
[Link]
Learning Feature Representations
Can we learn a hierarchy of features directly from the data
instead of hand engineering?
Low level features Mid level features High level features
Edges, dark spots Eyes, ears, nose Facial structure
6.S191 Introduction to Deep Learning
[3] 1/29/19
[Link]
Learning Visual Features
Fully Connected Neural Network
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Fully Connected Neural Network
Input: Fully Connected:
• 2D image • Connect neuron in hidden
• Vector of pixel values layer to all neurons in input
layer
• No spatial information!
• And many, many parameters!
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Fully Connected Neural Network
Input: Fully Connected:
• 2D image • Connect neuron in hidden
• Vector of pixel values layer to all neurons in input
layer
• No spatial information!
• And many, many parameters!
How can we use spatial structure in the input to inform the architecture of the network?
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Using Spatial Structure
Input: 2D image. Idea: connect patches of input
Array of pixel values to neurons in hidden layer.
Neuron connected to region of
input. Only “sees” these values.
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Using Spatial Structure
Connect patch in input layer to a single neuron in subsequent layer.
Use a sliding window to define connections.
How can we weight the patch to detect particular features?
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Applying Filters to Extract Features
1) Apply a set of weights – a filter – to extract local features
2) Use multiple filters to extract different features
3) Spatially share parameters of each filter
(features that matter in one part of the input should matter elsewhere)
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Feature Extraction with Convolution
- Filter of size 4x4 : 16 different weights
- Apply this same filter to 4x4 patches in input
- Shift by 2 pixels for next patch
This “patchy” operation is convolution
1) Apply a set of weights – a filter – to extract local features
2) Use multiple filters to extract different features
3) Spatially share parameters of each filter
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Feature Extraction and Convolution
A Case Study
X or X?
Image is represented as matrix of pixel values… and computers are literal!
We want to be able to classify an X as an X even if it’s shifted, shrunk, rotated, deformed.
6.S191 Introduction to Deep Learning
[4] 1/29/19
[Link]
Features of X
6.S191 Introduction to Deep Learning
[4] 1/29/19
[Link]
Filters to Detect X Features
filters
6.S191 Introduction to Deep Learning
[4] 1/29/19
[Link]
The Convolution Operation
1 X 1 =1
element wise
add outputs
multiply
= 9
6.S191 Introduction to Deep Learning
[4] 1/29/19
[Link]
The Convolution Operation
Suppose we want to compute the convolution of a 5x5 image and a 3x3 filter:
filter
image
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs…
6.S191 Introduction to Deep Learning
[5] 1/29/19
[Link]
The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:
filter feature map
6.S191 Introduction to Deep Learning
[5] 1/29/19
[Link]
The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:
filter feature map
6.S191 Introduction to Deep Learning
[5] 1/29/19
[Link]
The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:
filter feature map
6.S191 Introduction to Deep Learning
[5] 1/29/19
[Link]
The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:
filter feature map
6.S191 Introduction to Deep Learning
[5] 1/29/19
[Link]
The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:
filter feature map
6.S191 Introduction to Deep Learning
[5] 1/29/19
[Link]
The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:
filter feature map
6.S191 Introduction to Deep Learning
[5] 1/29/19
[Link]
The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:
filter feature map
6.S191 Introduction to Deep Learning
[5] 1/29/19
[Link]
The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:
filter feature map
6.S191 Introduction to Deep Learning
[5] 1/29/19
[Link]
The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:
filter feature map
6.S191 Introduction to Deep Learning
[5] 1/29/19
[Link]
Producing Feature Maps
Original Sharpen Edge Detect “Strong” Edge
Detect
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Feature Extraction with Convolution
1) Apply a set of weights – a filter – to extract local features
2) Use multiple filters to extract different features
3) Spatially share parameters of each filter
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Convolutional Neural Networks (CNNs)
CNNs for Classification
1. Convolution: Apply filters with learned weights to generate feature maps.
2. Non-linearity: Often ReLU.
3. Pooling: Downsampling operation on each feature map.
Train model with image data.
Learn weights of filters in convolutional layers.
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Convolutional Layers: Local Connectivity
For a neuron in hidden layer:
- Take inputs from patch
- Compute weighted sum
- Apply bias
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Convolutional Layers: Local Connectivity
For a neuron in hidden layer:
- Take inputs from patch
- Compute weighted sum
- Apply bias
' '
4x4 filter: matrix 1) applying a window of weights
$ $ !"# (")*,#), + .
of weights !"# 2) computing linear combinations
"%& #%&
3) activating with non-linear function
for neuron (p,q) in hidden layer
6.S191 Introduction to Deep Learning
1/29/19
[Link]
CNNs: Spatial Arrangement of Output Volume
depth
Layer Dimensions:
ℎ"#"$
where h and w are spatial dimensions
d (depth) = number of filters
height
Stride:
Filter step size
Receptive Field:
Locations in input image that
width a node is path connected to
6.S191 Introduction to Deep Learning
[3] 1/29/19
[Link]
Introducing Non-Linearity
- Apply after every convolution operation (i.e., after Rectified Linear Unit (ReLU)
convolutional layers)
- ReLU: pixel-by-pixel operation that replaces all negative
values by zero. Non-linear operation!
! " = max ( 0 , " )
6.S191 Introduction to Deep Learning
[5] 1/29/19
[Link]
Pooling
1) Reduced dimensionality
2) Spatial invariance
How else can we downsample and preserve spatial invariance?
6.S191 Introduction to Deep Learning
[3] 1/29/19
[Link]
Representation Learning in Deep CNNs
Low level features Mid level features High level features
Edges, dark spots Eyes, ears, nose Facial structure
Conv Layer 1 Conv Layer 2 Conv Layer 3
6.S191 Introduction to Deep Learning
[3] 1/29/19
[Link]
CNNs for Classification: Feature Learning
1. Learn features in input image through convolution
2. Introduce non-linearity through activation function (real-world data is non-linear!)
3. Reduce dimensionality and preserve spatial invariance with pooling
6.S191 Introduction to Deep Learning
1/29/19
[Link]
CNNs for Classification: Class Probabilities
- CONV and POOL layers output high-level features of input + ,-
- Fully connected layer uses these features for classifying input image softmax () =
∑/ + ,0
- Express output as probability of image belonging to a particular class
6.S191 Introduction to Deep Learning
1/29/19
[Link]
CNNs: Training with Backpropagation
Learn weights for convolutional filters and fully connected layers
Backpropagation: cross-entropy loss
! " = $ & (%) log -
,(%)
%
6.S191 Introduction to Deep Learning
1/29/19
[Link]
CNNs for Classification: ImageNet
ImageNet Dataset
Dataset of over 14 million images across 21,841 categories
“Elongated crescent-shaped yellow fruit with soft sweet flesh”
1409 pictures of bananas.
6.S191 Introduction to Deep Learning
[6,7] 1/29/19
[Link]
ImageNet Challenge
Classification task: produce a list of object categories present in image. 1000 categories.
“Top 5 error”: rate at which the model does not output correct label in top 5 predictions
Other tasks include:
single-object localization, object detection from video/image, scene classification, scene parsing
6.S191 Introduction to Deep Learning
[6,7] 1/29/19
[Link]
ImageNet Challenge: Classification Task
30 28.2 2012: AlexNet. First CNN to win.
25.8 - 8 layers, 61 million parameters
classification error %
2013: ZFNet
20
- 8 layers, more filters
16.4 2014:VGG
- 19 layers
11.7 2014: GoogLeNet
10
6.7 - “Inception” modules
5.1 - 22 layers, 5million parameters
3.57
2015: ResNet
0 - 152 layers
10
11
12
13
14
15
an
20
20
20
20
20
20
um
H
6.S191 Introduction to Deep Learning
[6,7] 1/29/19
[Link]
ImageNet Challenge: Classification Task
30 28.2 150
25.8
classification error %
number of layers
20
100
16.4
11.7
10 50
7.3 6.7
5.1
3.57
0 0
10
11
12
13
14
14
15
an
10
11
12
13
14
14
15
um
20
20
20
20
20
20
20
20
20
20
20
20
20
20
H
6.S191 Introduction to Deep Learning
[6,7] 1/29/19
[Link]
An Architecture for Many Applications
An Architecture for Many Applications
Object detection with R-CNNs
Segmentation with fully convolutional networks
Image captioning with RNNs
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Beyond Classification
Semantic Segmentation Object Detection Image Captioning
CAT CAT, DOG, DUCK The cat is in the grass.
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Semantic Segmentation: FCNs
FCN: Fully Convolutional Network.
Network designed with all convolutional layers,
with downsampling and upsampling operations
6.S191 Introduction to Deep Learning
[3,8,9] 1/29/19
[Link]
Driving Scene Segmentation
6.S191 Introduction to Deep Learning Fix reference [10] 1/29/19
[Link]
Driving Scene Segmentation
6.S191 Introduction to Deep Learning
[11, 12] 1/29/19
[Link]
Object Detection with R-CNNs
R-CNN: Find regions that we think have objects. Use CNN to classify.
6.S191 Introduction to Deep Learning
[13] 1/29/19
[Link]
Image Captioning using RNNs
6.S191 Introduction to Deep Learning
[14,15] 1/29/19
[Link]
Image Captioning using RNNs
6.S191 Introduction to Deep Learning
[14,15] 1/29/19
[Link]
Deep Learning for Computer Vision:
Impact and Summary
Data, Data, Data
Airplane
Automobile
Bird
Cat
Deer
Dog
MNIST: handwritten digits
Frog
Horse
Ship
ImageNet:
22K categories. 14M images. Truck
places: natural scenes
CIFAR-10
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Deep Learning for Computer Vision: Impact
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Impact: Face Detection 6.S191 Lab!
6.S191 Introduction to Deep Learning
1/29/19
[Link]
Impact: Self-Driving Cars
6.S191 Introduction to Deep Learning
[16] 1/29/19
[Link]
Impact: Healthcare
Identifying facial phenotypes of genetic disorders using deep learning
Gurovich et al., Nature Med. 2019
6.S191 Introduction to Deep Learning
[17] 1/29/19
[Link]
Deep Learning for Computer Vision: Summary
Foundations CNNs Applications
• Why computer vision? • CNN architecture • Segmentation, object
• Representing images • Application to detection, image
classification captioning
• Convolutions for
feature extraction • ImageNet • Visualization
6.S191 Introduction to Deep Learning
1/29/19
[Link]
References
[Link]/hbLkF6
End of Slides