Degradation of accuracy with higher number of layers

Kaiming He et al: Deep Residual Learning for Image Recognition

How to improve accuracy with increasing layers?
How to improve accuracy with increasing layers?
How to improve accuracy with increasing layers?

ILSVRC 2015: First Prize : 3.6% error.

ILSVRC 2016: First Prize : 3.0% error.

 The Softmax Activation Function:

• Relevant for classification problems with multiple classes (i.e.

multiple outputs)
• For mutually exclusive classes, their probs. should sum to 1, and the
Softmax function is a mechanism to feed this information into the
network
• Recall our ANN terminology:

v1 f y1 • Generally we have yi = f(vi) (as in Sigmoid funcs)

• For Softmax we have yi = f(v1,..., vm)
evi
vi f yi • Specifically, yi = m

e
vj

j =1
vm f ym
yi
• And its derivative: = yi (1 − yi )
vi
 The Softmax Activation Function:

• Illustrations taken from website:

[Link]

e yi
S ( yi ) = m

e
yj

j =1
The Total Architecture:
 First Layer Transformations
• Let us recall something we have seen before

• This shows the 96 filters of size 11x11 in the first convolutional filter layer of AlexNet
after training convergence
• Each 11x11 filter is expressed as an image by scaling the weight values to the range 0-
255 and then combining the 3 colour channels, exactly as the three channels in any
colour image are combined to generate the final image.
 First Layer Transformations
• Note that this is not a characteristic of AlexNet alone. You see similar
representations of first layer filters from completely different CNN architectures,
with filters of different sizes.
First Layer Transformations: Original Image – 10th Filter(3 channels combo) – resultant 10th Activation Map

• The example image shown at right top

• when operated with the corresponding 10th filter shown
in image form at right middle
• generates the result shown at right bottom
• what really happens is that the filter “extracts” its own
unique characteristics out from the original image it
convolves with
• so that, in this example, the red-dominated filter
extracts the red colours powerfully from the original
image while neglecting the other colours
First Layer Transformations: Original Image – 11th Filter(3 channels combo) – resultant 11th Activation Map

• The example image shown at right top

• when operated with the corresponding 11th filter shown
in image form at right middle
• generates the result shown at right bottom
• what really happens is that the filter “extracts” its own
unique characteristics out from the original image it
convolves with
• so that, in this example, the green-dominated filter
extracts the green colours powerfully from the original
image while neglecting the other colours
• Now, how does the convolution operation induce this?
• How does a filter extract its own special characteristics
out from an image?
 Now, how does the convolution operation induce this?
How does a filter extract its own special characteristics out from an image?
• Recall the fundamental operation in a convolution, apart from traversal of filter
• Dot product!
• What really happens in a dot product? A.B = ABcos 
Assume Assume
Image Filter
• The vectors used in these examples F
I1 are in 2-D
Assume • Q: What is the dimension of vectors 
Filter used in actual convolutions?
I2 Assume
 F Image

Dot product: (I2cos ) F

Dot product: (I1cos ) F
• Dot Prod. extracts those characteristics of one vector (IX) that are aligned with a reference vector (F)
• In our context, dot product extracts the characteristics of the image that are aligned
(similar to) with the character of the filter!
 Intermediate Layer Transformations

• Nothing very meaningful can be easily visualized about intermediate layer

transformations, hence we move on to the (pre-) last layer.
Last Layer Transformations
 Last Layer Transformations
• Suppose we forward-pass a large number of images
through a CNN (let it be AlexNet for ease of discussion)
Nearest Neighbours in image • Every image generates a 4096-size vector (1-D) in the
(pixel) space, i.e. as inputs pre-last layer (FC7 in AlexNet), call it the feature
space
• Suppose we apply some clustering technique on all
vectors in this feature space
• These vectors will get aligned into clusters
corresponding to semantically-similar images in the
pixel (input) space
• Implication: At the stage of FC7 itself, the
identification of class has been completed, then the
final Softmax layer just transforms these identities
into probabilities of belonging (to different classes).
• Let us see this more concretely.
 Last Layer Transformations
• These vectors will get aligned
into clusters corresponding to
semantically-similar images in
the pixel (input) space
• Illustration at right shows the
source images in each row
corresponding to feature vectors
that lie in identical clusters in the
4096-dimensional feature space
in FC7
• For example, see the elephants in
second row. There are elephants
that fill up completely different
sets of pixels in image space, but
because they are all “elephants”,
they are semantically similar. And
they are clustered similarly in
pre-last layer feature space.
 Last Layer Transformations:
Dimensionality Reduction for Visualization
• The 4096-dimensional feature space, and
corresponding clusters, cannot be
visualized
• But if they can be mapped into 2-D
space without loss of clustering
information (!), then they can be
visualized
• The t-SNE algorithm is a very significant
development in Machine Learning, that
enables this mapping without loss of
relative positional information!
• Figure at right shows clustering on the
MNIST database
Some Special types of Convolutions
 Some Special types of Convolutions

OUTLINE:

These slides present brief descriptions of the following:

• Deconvolutions or Transpose Convolutions

• Dilated Convolutions.
 Deconvolutions or Transpose Convolutions
• In a normal convolution, the data flow between two intermediate “activation maps”
is in the overall direction of input to output
• Suppose, hypothetically, between two intermediate layers, we want the data to flow
in the opposite direction
• Then, superficially, this is termed as “deconvolution”, or more aptly, as “transpose
convolution” or “fractionally strided convolution”
• Now data actually does not flow in the reverse direction; this offers us a perspective
on convolutions that take from smaller-area activation maps onto larger-area
activation maps – the opposite of what happens in normal convolutions
• Recall that in almost all typical CNN architectures one moves from larger area
activation maps to smaller areas, either by reduced zero-padding or by pooling
• However, there could be situations where one would like to increase the activation
map area in the forward direction, and that is exactly where “deconvolution” comes
into play.
 Deconvolutions or Transpose Convolutions

• The transpose of convolving a 3 x 3 filter kernel over a 4 x 4 input using unit strides
• This is equivalent to convolving a 3 x 3 filter over a 2 x 2 input padded with a 2 x 2
border of zeros using unit stride.
 Deconvolutions or Transpose Convolutions

• The transpose of convolving a 3 x 3 filter kernel over a 5 x 5 input using a stride of

2 x 2. Application of the formula gives an output size of 2.
• This is equivalent to convolving a 3 x 3 filter over a 2 x 2 input padded with a 2 x 2
border of zeros AND with one pad of zeros inserted between the pixels, using unit
stride.
• Note that two strides (of one unit each) of the filter take it from one input pixel to
another, which means the effective stride is half. That is why this is also referred to
as fractionally strided convolutions.
 Dilated Convolutions
• The working principle of dilated convolutions is to effectively increase the size of (i.e. inflate /
dilate) the filter (kernel) by inserting rows and columns of zeros between the original elements
• The dilation ‘rate’ is controlled by a factor d that is 1 normally, and d-1 is the number of
spaces introduced between the elements of a filter. So when d = 2, we have effectively
introduced 1 space
• So a filter of size 3 with d = 2 becomes a filter of size 5, and with d = 3 becomes size 7
• If k’ denotes the modified size of the filter, and k the original size, then the relation between
them is k ' = k + ( k − 1)( d − 1) ….. (1)
• The original formula relating the output and input sizes of activation maps,
( N in + 2 P − K )
N out = +1 ….. (2)
stride
( N in + 2 P − K − ( K − 1)( d − 1))
'
then becomes N out = +1 ….. (3)
stride
• Now it is obvious that N’out  Nout; implying that Dilations make the output maps shrink faster
• And the 2nd advantage is that it increases the filter size and the receptive field (zone of
dependence) of convolutions; effectively the resolution, without increasing filter memory.
 Dilated Convolutions

• Convolving a 3 x 3 filter over a 7 x 7 input with a dilation factor of 2. Note that the
output size is 3.
• Q: What would have been the output size if d == 1?
THANK YOU