Deep Learning for Computer Vision

Muhammad Faisal
Research Associate @ VisPro Lab & Intelligent Machines Lab
Information Technology University, Lahore
Outline

■ Deep Learning for Computer Vision

■ Classification Network
■ AlexNet, VGG, GoogleNet, ResNet, DenseNet
■ Segmentation Network
■ SegNet, UNet, PSP-Net, DeepLab
■ Object Detection Network
■ FasterRCNN, SSD, YOLO

■ Deep Learning Frameworks

■ Google Collaboratory
■ Python & PyTorch

2
Image Classification

3
Object Detection

4
Segmentation

5
Image Style Transfer

6
Image Colorization

7
CNN Architectures
Case Studies:
AlexNet  VGG ResNet DenseNet

8
 Review: LeNet-5
[LeCun et al., 1998]

Conv filters were 5x5, applied at stride 1

Subsampling (Pooling) layers were 2x2 applied at stride 2
i.e. architecture is [CONV-POOL-CONV-POOL-FC-FC]

Lecture 9 - 9 9
Case Study: AlexNet
[Krizhevsky et al. 2012]

Architecture:
CONV1
MAX POOL1
NORM1
CONV2
MAX POOL2
NORM2
CONV3
CONV4
CONV5
Max POOL3
FC6
FC7
FC8
Lecture 9 - 1 10
Case Study: AlexNet
[Krizhevsky et al. 2012]

Full (simplified) AlexNet architecture:

[227x227x3] INPUT
[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0
[27x27x96] MAX POOL1: 3x3 filters at stride 2
[27x27x96] NORM1: Normalization layer
[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2
[13x13x256] MAX POOL2: 3x3 filters at stride 2
[13x13x256] NORM2: Normalization layer
[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1
[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1
[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1
[6x6x256] MAX POOL3: 3x3 filters at stride 2
[4096] FC6: 4096 neurons
[4096] FC7: 4096 neurons
11
[1000] FC8: 1000 neurons (class scores)
Case Study: AlexNet
[Krizhevsky et al. 2012]

Full (simplified) AlexNet architecture:

[227x227x3] INPUT
[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0 Details/Retrospectives:
[27x27x96] MAX POOL1: 3x3 filters at stride 2 - first use of ReLU
[27x27x96] NORM1: Normalization layer - used Norm layers (not common anymore)
[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2 - heavy data augmentation
[13x13x256] MAX POOL2: 3x3 filters at stride 2 - dropout 0.5
[13x13x256] NORM2: Normalization layer - batch size 128
[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1 - SGD Momentum 0.9
[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1 - Learning rate 1e-2, reduced by 10
[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1 manually when val accuracy plateaus
[6x6x256] MAX POOL3: 3x3 filters at stride 2 - 7 CNN ensemble: 18.2% -> 15.4%
[4096] FC6: 4096 neurons - L2 weight decay 5e-4
[4096] FC7: 4096 neurons
12
[1000] FC8: 1000 neurons (class scores)
Case Study: AlexNet
[Krizhevsky et al. 2012]

Full (simplified) AlexNet architecture:

[227x227x3] INPUT
[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0
[55x55x48] x 2
[27x27x96] MAX POOL1: 3x3 filters at stride 2
[27x27x96] NORM1: Normalization layer
[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2 Historical note: Trained on GTX 580
[13x13x256] MAX POOL2: 3x3 filters at stride 2 GPU with only 3 GB of memory.
[13x13x256] NORM2: Normalization layer Network spread across 2 GPUs, half
[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1 the neurons (feature maps) on each
[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1 GPU.
[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1
[6x6x256] MAX POOL3: 3x3 filters at stride 2
[4096] FC6: 4096 neurons
[4096] FC7: 4096 neurons
[1000] FC8: 1000 neurons (class scores)
13
Case Study: AlexNet
[Krizhevsky et al. 2012]

Full (simplified) AlexNet architecture:

[227x227x3] INPUT
[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0
[27x27x96] MAX POOL1: 3x3 filters at stride 2
CONV1, CONV2, CONV4, CONV5:
[27x27x96] NORM1: Normalization layer
Connections only with feature maps
[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2
on same GPU
[13x13x256] MAX POOL2: 3x3 filters at stride 2
[13x13x256] NORM2: Normalization layer
[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1
[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1
[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1
[6x6x256] MAX POOL3: 3x3 filters at stride 2
[4096] FC6: 4096 neurons
[4096] FC7: 4096 neurons
[1000] FC8: 1000 neurons (class scores) 14
Case Study: AlexNet
[Krizhevsky et al. 2012]

Full (simplified) AlexNet architecture:

[227x227x3] INPUT
[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0
[27x27x96] MAX POOL1: 3x3 filters at stride 2
CONV3, FC6, FC7, FC8:
[27x27x96] NORM1: Normalization layer
Connections with all feature maps in
[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2
preceding layer, communication
[13x13x256] MAX POOL2: 3x3 filters at stride 2
across GPUs
[13x13x256] NORM2: Normalization layer
[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1
[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1
[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1
[6x6x256] MAX POOL3: 3x3 filters at stride 2
[4096] FC6: 4096 neurons
[4096] FC7: 4096 neurons
15
[1000] FC8: 1000 neurons (class scores)
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners

152 layers 152 layers 152 layers

19 layers 22 layers

shallow 8 layers 8 layers

16
ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
winners

First CNN-based winner 152 layers 152 layers 152 layers

19 layers 22 layers

shallow 8 layers 8 layers

17
ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
winners
ZFNet: Improved
152 layers 152 layers 152 layers
hyperparameters over
AlexNet

19 layers 22 layers

shallow 8 layers 8 layers

18
ZFNet [Zeiler and Fergus, 2013]

AlexNet but:
CONV1: change from (11x11 stride 4) to (7x7 stride 2)
CONV3,4,5: instead of 384, 384, 256 filters use 512, 1024,
512
ImageNet top 5 error: 16.4% -> 11.7%
19
ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
winners

Deeper Networks 152 layers 152 layers 152 layers

19 layers 22 layers

shallow 8 layers 8 layers

20
Case Study: VGGNet
[Simonyan and Zisserman, 2014]

Small filters, Deeper networks

8 layers (AlexNet)
16 - 19 layers (VGG16Net)

Only 3x3 CONV stride 1, pad 1

and 2x2 MAX POOL stride 2

11.7% top 5 error in ILSVRC’13

(ZFNet)
AlexNet VGG16 VGG19
7.3% top 5 error in ILSVRC’14

21
Case Study: VGGNet
[Simonyan and Zisserman, 2014]

Q: Why use smaller filters? (3x3 conv)

AlexNet VGG16 VGG19

22
Case Study: VGGNet
[Simonyan and Zisserman, 2014]

Q: Why use smaller filters? (3x3 conv)

Stack of three 3x3 conv (stride 1) layers

has same effective receptive field as
one 7x7 conv layer

Q: What is the effective receptive field of

three 3x3 conv (stride 1) layers?

AlexNet VGG16 VGG19

23
 Receptive Field

Convolution with 3x3 filter

Receptive field is 3x3

Convolution with 3x3 filter Convolution with 3x3 filter

Receptive field is 3x3 Receptive field is 5x5
24
Case Study: VGGNet
[Simonyan and Zisserman, 2014]

Q: Why use smaller filters? (3x3 conv)

Stack of two 3x3 conv (stride 1) layers
has same effective receptive field as
one 5x5 conv layer

[7x7xC] = 49C parameters

3x(3x3xC) = 27C parameters

But deeper, more non-linearities AlexNet VGG16 VGG19

25
INPUT: [224x224x3] memory: 224*224*3=150K params: 0 (not counting biases)
CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*3)*64 = 1,728
CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*64)*64 = 36,864
POOL2: [112x112x64] memory: 112*112*64=800K params: 0
CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*64)*128 = 73,728
CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*128)*128 = 147,456
POOL2: [56x56x128] memory: 56*56*128=400K params: 0
CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*128)*256 = 294,912 CONV3-
256: [56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824 CONV3-256:
[56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824
POOL2: [28x28x256] memory: 28*28*256=200K params: 0
CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*256)*512 = 1,179,648 CONV3-
512: [28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296 CONV3-512:
[28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296
POOL2: [14x14x512] memory: 14*14*512=100K params: 0
CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-
512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-512:
[14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296
POOL2: [7x7x512] memory: 7*7*512=25K params: 0
FC: [1x1x4096] memory: 4096 params: 7*7*512*4096 = 102,760,448 FC: VGG16
[1x1x4096] memory: 4096 params: 4096*4096 = 16,777,216
FC: [1x1x1000] memory: 1000 params: 4096*1000 = 4,096,000

TOTAL memory: 24M * 4 bytes ~= 96MB / image (for a forward pass)

TOTAL params: 138M parameters 26

INPUT: [224x224x3]memory: 224*224*3=150K params: 0 (not counting biases)
CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*3)*64 = 1,728 Note:
CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*64)*64 = 36,864
POOL2: [112x112x64] memory: 112*112*64=800K params: 0
Most memory is in
CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*64)*128 = 73,728
CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*128)*128 = 147,456 early CONV
POOL2: [56x56x128] memory: 56*56*128=400K params: 0
CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*128)*256 = 294,912 CONV3-
256: [56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824 CONV3-256:
[56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824
POOL2: [28x28x256] memory: 28*28*256=200K params: 0
CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*256)*512 = 1,179,648 CONV3-
512: [28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296 CONV3-512:
[28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296 Most params are
POOL2: [14x14x512] memory: 14*14*512=100K params: 0 in late FC
CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-
512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-512:
[14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296
POOL2: [7x7x512] memory: 7*7*512=25K params: 0
FC: [1x1x4096] memory: 4096 params: 7*7*512*4096 = 102,760,448
FC: [1x1x4096] memory: 4096 params: 4096*4096 = 16,777,216 FC:
[1x1x1000] memory: 1000 params: 4096*1000 = 4,096,000

TOTAL memory: 24M * 4 bytes ~= 96MB / image (only forward! ~*2 for bwd)

TOTAL params: 138M parameters 27

INPUT: [224x224x3]memory: 224*224*3=150K params: 0 (not counting biases)
CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*3)*64 = 1,728
CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*64)*64 = 36,864
POOL2: [112x112x64] memory: 112*112*64=800K params: 0
CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*64)*128 = 73,728 CONV3-
128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*128)*128 = 147,456
POOL2: [56x56x128] memory: 56*56*128=400K params: 0
CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*128)*256 = 294,912 CONV3-256:
[56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824 CONV3-256: [56x56x256]
memory: 56*56*256=800K params: (3*3*256)*256 = 589,824
POOL2: [28x28x256] memory: 28*28*256=200K params: 0
CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*256)*512 = 1,179,648 CONV3-512:
[28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296 CONV3-512: [28x28x512]
memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296
POOL2: [14x14x512] memory: 14*14*512=100K params: 0
CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-512:
[14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-512: [14x14x512]
memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296
POOL2: [7x7x512] memory: 7*7*512=25K params: 0
FC: [1x1x4096] memory: 4096 params: 7*7*512*4096 = 102,760,448 FC: VGG16
[1x1x4096] memory: 4096 params: 4096*4096 = 16,777,216
FC: [1x1x1000] memory: 1000 params: 4096*1000 = 4,096,000 Common names
TOTAL memory: 24M * 4 bytes ~= 96MB / image (only forward! ~*2 for bwd)

TOTAL params: 138M parameters 28

Case Study: VGGNet
[Simonyan and Zisserman, 2014]

Details:
- ILSVRC’14 2nd in classification, 1st in
localization
- Similar training procedure as
Krizhevsky 2012
- No Local Response Normalisation (LRN)
- Use VGG16 or VGG19 (VGG19 only
slightly better, more memory)
- Use ensembles for best results
- FC7 features generalize well to other
tasks
AlexNet VGG16 VGG19

29
ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
winners

Deeper Networks 152 layers 152 layers 152 layers

19 layers 22 layers

shallow 8 layers 8 layers

30
Case Study: GoogLeNet
[Szegedy et al., 2014]

Deeper networks, with computational

efficiency

- 22 layers
- Efficient “Inception” module
- No FC layers
- Only 5 million parameters!
12x less than AlexNet Inception module
- ILSVRC’14 classification
winner (6.7% top 5 error)

31
Case Study: GoogLeNet
[Szegedy et al., 2014]

“Inception module”: design a

good local network topology
(network within a network) and
then stack these modules on
top of each other

Inception module

32
Case Study: GoogLeNet
[Szegedy et al., 2014]

Apply parallel filter operations

on the input from previous
layer:
- Multiple receptive field sizes
for convolution (1x1, 3x3,
5x5)
- Pooling operation (3x3)
Concatenate all filter outputs
Naive Inception module together depth-wise

33
Case Study: GoogLeNet
[Szegedy et al., 2014]

Apply parallel filter operations

on the input from previous
layer:
- Multiple receptive field sizes
for convolution (1x1, 3x3,
5x5)
- Pooling operation (3x3)
Concatenate all filter outputs
Naive Inception module together depth-wise

Q: What is the problem with this?

[Hint: Computational complexity]

34
Case Study: GoogLeNet Q: What is the problem with this?
[Szegedy et al., 2014] [Hint: Computational complexity]

Example:

Module input:
28x28x256

Naive Inception module

35
Case Study: GoogLeNet Q: What is the problem with this?
[Szegedy et al., 2014] [Hint: Computational complexity]

Q1: What is the output size of the

Example:
1x1 conv, with 128 filters?

Module input:
28x28x256

Naive Inception module

36
Case Study: GoogLeNet Q: What is the problem with this?
[Szegedy et al., 2014] [Hint: Computational complexity]

Q1: What is the output size of the

Example:
1x1 conv, with 128 filters?

28x28x128

Module input:
28x28x256

Naive Inception module

37
Case Study: GoogLeNet Q: What is the problem with this?
[Szegedy et al., 2014] [Hint: Computational complexity]

Q2: What are the output sizes of

Example:
all different filter operations?

28x28x128

Module input:
28x28x256

Naive Inception module

38
Case Study: GoogLeNet Q: What is the problem with this?
[Szegedy et al., 2014] [Hint: Computational complexity]

Q2: What are the output sizes of

Example:
all different filter operations?

28x28x128 28x28x192 28x28x96 28x28x256

Module input:
28x28x256

Naive Inception module

39
Case Study: GoogLeNet Q: What is the problem with this?
[Szegedy et al., 2014] [Hint: Computational complexity]

Q3:What is output size after

Example:
filter concatenation?

28x28x128 28x28x192 28x28x96 28x28x256

Module input:
28x28x256

Naive Inception module

40
Case Study: GoogLeNet Q: What is the problem with this?
[Szegedy et al., 2014] [Hint: Computational complexity]

Q3:What is output size after

Example:
filter concatenation?

28x28x(128+192+96+256) = 28x28x672

28x28x128 28x28x192 28x28x96 28x28x256

Module input:
28x28x256

Naive Inception module

41
Case Study: GoogLeNet Q: What is the problem with this?
[Szegedy et al., 2014] [Hint: Computational complexity]

Q3:What is output size after

Example:
filter concatenation?
Conv Ops:
28x28x(128+192+96+256) = 28x28x672 [1x1 conv, 128] 28x28x128x1x1x256
[3x3 conv, 192] 28x28x192x3x3x256
[5x5 conv, 96] 28x28x96x5x5x256
28x28x128 28x28x192 28x28x96 28x28x256 Total: 854M ops

Module input:
28x28x256

Naive Inception module

42
Case Study: GoogLeNet Q: What is the problem with this?
[Szegedy et al., 2014] [Hint: Computational complexity]

Q3:What is output size after

Example:
filter concatenation?
Conv Ops:
28x28x(128+192+96+256) = 28x28x672 [1x1 conv, 128] 28x28x128x1x1x256
[3x3 conv, 192] 28x28x192x3x3x256
[5x5 conv, 96] 28x28x96x5x5x256
28x28x128 28x28x192 28x28x96 28x28x256 Total: 854M ops

Very expensive compute

Module input: Pooling layer also preserves feature

28x28x256 depth, which means total depth after
concatenation can only grow at every
Naive Inception module
layer!

43
Case Study: GoogLeNet Q: What is the problem with this?
[Szegedy et al., 2014] [Hint: Computational complexity]

Q3:What is output size after

Example:
filter concatenation?

28x28x(128+192+96+256) = 529k Solution: “bottleneck” layers

that use 1x1 convolutions to
reduce feature depth
28x28x128 28x28x192 28x28x96 28x28x256

Module input:
28x28x256

Naive Inception module

44
Reminder: 1x1 convolutions

1x1 CONV
56 with 32 filters
56
(each filter has size
1x1x64, and performs a
64-dimensional dot
56 product)
56
64 32

45
Reminder: 1x1 convolutions

1x1 CONV
56 with 32 filters
56

preserves spatial
dimensions, reduces depth!

56 Projects depth to lower 56

64 dimension (combination of 32
feature maps)

46
Case Study: GoogLeNet
[Szegedy et al., 2014]

Naive Inception module

Inception module with dimension reduction

47
Case Study: GoogLeNet
[Szegedy et al., 2014]
1x1 conv “bottleneck”
layers

Naive Inception module

Inception module with dimension reduction

48
Case Study: GoogLeNet Using same parallel layers as
naive example, and adding “1x1
[Szegedy et al., 2014]
conv, 64 filter” bottlenecks:
28x28x480
Conv Ops:
[1x1 conv, 64] 28x28x64x1x1x256
[1x1 conv, 64] 28x28x64x1x1x256
28x28x128 28x28x192 28x28x96 28x28x64
[1x1 conv, 128] 28x28x128x1x1x256
[3x3 conv, 192] 28x28x192x3x3x64
[5x5 conv, 96] 28x28x96x5x5x64
28x28x64 28x28x64 28x28x256 [1x1 conv, 64] 28x28x64x1x1x256
Total: 358M ops

Module input:
Compared to 854M ops for naive version
Bottleneck can also reduce depth after
28x28x256
pooling layer
Inception module with dimension reduction

49
Case Study: GoogLeNet
[Szegedy et al., 2014]

Stack Inception modules

with dimension reduction
on top of each other

Inception module

50
Case Study: GoogLeNet
[Szegedy et al., 2014]

Full GoogLeNet
architecture

- 22 layers
- Efficient “Inception” module
- No FC layers
- 12x less params than AlexNet
- ILSVRC’14 classification
Deeper networks, with computational efficiency winner (6.7% top 5 error) 51
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners
“Revolution of Depth”

152 layers 152 layers

152 layers

19 layers 22 layers

shallow 8 layers 8 layers

52
Case Study: ResNet
[He et al., 2015]

relu
Very deep networks using residual F(x) + x
connections
..
.
- 152-layer model for ImageNet X
F(x)
- ILSVRC’15 classification relu
identity
winner (3.57% top 5 error)
- Swept all classification and
detection competitions in X
ILSVRC’15 and COCO’15! Residual block

53
Case Study: ResNet
[He et al., 2015]

What happens when we continue stacking deeper layers on a “plain”

convolutional neural network?

54
Case Study: ResNet
[He et al., 2015]

What happens when we continue stacking deeper layers on a “plain”

convolutional neural network?

56-layer model performs worse on both training and test error

-> The deeper model performs worse, but it’s not caused by overfitting!

55
Case Study: ResNet
[He et al., 2015]

Hypothesis: the problem is an optimization problem, deeper models are harder

to optimize

56
Case Study: ResNet
[He et al., 2015]

Hypothesis: the problem is an optimization problem, deeper models are harder

to optimize

The deeper model should be able to perform at

least as well as the shallower model.

A solution by construction is copying the learned

layers from the shallower model and setting
additional layers to identity mapping.

57
Case Study: ResNet
[He et al., 2015]

Solution: Use network layers to fit a residual mapping instead of directly trying to fit
a desired underlying mapping
relu
H(x) F(x) + x

F(x) X
relu relu
identity

X X
“Plain” layers Residual block

58
Case Study: ResNet
[He et al., 2015]

Solution: Use network layers to fit a residual mapping instead of directly trying to fit
a desired underlying mapping
H(x) = F(x) + x relu
H(x) F(x) + x
Use layers to
fit residual
F(x) relu
X F(x) = H(x) - x
relu identity instead of
H(x) directly

X X
“Plain” layers Residual block

59
Case Study: ResNet
[He et al., 2015]

Full ResNet architecture:

relu
- Stack residual blocks
F(x) + x
- Every residual block has ..
two 3x3 conv layers .

F(x) X
relu
identity

X
Residual block

60
Case Study: ResNet
[He et al., 2015]

Full ResNet architecture:

relu
- Stack residual blocks
F(x) + x
- Every residual block has ..
two 3x3 conv layers .

- Periodically, double # of 3x3 conv, 128

filters and downsample F(x) X filters, /2
relu spatially with
spatially using stride 2 identity stride 2
(/2 in each dimension)
3x3 conv, 64
filters

X
Residual block

61
Case Study: ResNet
[He et al., 2015]

Full ResNet architecture:

relu
- Stack residual blocks
F(x) + x
- Every residual block has ..
two 3x3 conv layers .

- Periodically, double # of
filters and downsample F(x) X
relu
spatially using stride 2 identity
(/2 in each dimension)
- Additional conv layer at
the beginning X
Residual block

Beginning
conv layer

62
Case Study: ResNet layers
No FC

besides FC
[He et al., 2015] 1000 to
output
classes
Full ResNet architecture: Global
relu average
- Stack residual blocks pooling layer
F(x) + x
- Every residual block has ..
after last
conv layer
two 3x3 conv layers .

- Periodically, double # of
filters and downsample F(x) X
relu
spatially using stride 2 identity
(/2 in each dimension)
- Additional conv layer at
the beginning X
- No FC layers at the end Residual block
(only FC 1000 to output
classes)

63
Case Study: ResNet
[He et al., 2015]

Total depths of 34, 50, 101, or

152 layers for ImageNet ..
.

64
Case Study: ResNet
[He et al., 2015]

For deeper networks

(ResNet-50+), use “bottleneck”
layer to improve efficiency
(similar to GoogLeNet)

65
Case Study: ResNet
[He et al., 2015]

1x1 conv, 256 filters projects

back to 256 feature maps
For deeper networks
(ResNet-50+), use “bottleneck” (28x28x256)
layer to improve efficiency 3x3 conv operates over
(similar to GoogLeNet) only 64 feature maps

1x1 conv, 64 filters

to project to
28x28x64

66
Case Study: ResNet
[He et al., 2015]

Experimental Results
- Able to train very deep
networks without degrading
(152 layers on ImageNet, 1202
on Cifar)
- Deeper networks now achieve
lowing training error as
expected
- Swept 1st place in all ILSVRC
and COCO 2015 competitions
ILSVRC 2015 classification winner
(3.6% top 5 error) -- better than “human
performance”! (Russakovsky 2014)

67
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners

152 layers 152 layers 152 layers

19 layers 22 layers

shallow 8 layers 8 layers

68
 Comparing complexity...

An Analysis of Deep Neural Network Models for Practical Applications, 2017.

Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, 2017. Reproduced with permission.

69
 Comparing complexity... Inception-v4: Resnet + Inception!

An Analysis of Deep Neural Network Models for Practical Applications, 2017.

Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, 2017. Reproduced with permission.

70
 VGG: Highest
Comparing complexity... memory, most
operations

An Analysis of Deep Neural Network Models for Practical Applications, 2017.

Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, 2017. Reproduced with permission.

71
 GoogLeNet:
Comparing complexity... most efficient

An Analysis of Deep Neural Network Models for Practical Applications, 2017.

Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, 2017. Reproduced with permission.

72
 AlexNet:
Comparing complexity... Smaller compute, still memory
heavy, lower accuracy

An Analysis of Deep Neural Network Models for Practical Applications, 2017.

Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, 2017. Reproduced with permission.

73
 ResNet:
Comparing complexity... Moderate efficiency depending on
model, highest accuracy

An Analysis of Deep Neural Network Models for Practical Applications, 2017.

Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, 2017. Reproduced with permission.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - May 1, 2018 74
Beyond ResNets...
Densely Connected Convolutional Networks
[Huang et al. 2017]

- Dense blocks where each layer is

connected to every other layer in
feedforward fashion
- Alleviates vanishing gradient,
strengthens feature propagation,
encourages feature reuse

Lecture 9 - 75
Summary CNN-Architectures

■ VGG, GoogLeNet, ResNet all in wide use, available in model zoos

■ DenseNet current best default, also consider ResNet
■ Trend towards extremely deep networks
■ Significant research centers around design of layer / skip connections and
improving gradient flow

■ Efforts to investigate necessity of depth vs. width and residual connections

76