DEEP LEARNING
Padding, Strides
Rajesh A
Convolution Operation – Loss of information
0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
-1 -1 -1
0 0 0 0 0 0 255 255 255 255
255 255 255 255 255 255 ∗ 0 0 0 = 255 255 255 255
1 1 1
255 255 255 255 255 255 0 0 0 0
255 255 255 255 255 255
X - Input Filter or Kernel Feature Map
6x6 3x3 4x4
▪ The size of feature map is less than size of Input image (5 x 5 to 4 x 4)
▪ If this feature map is convolved with another 3 x 3 kernel; Feature map is size 2 x 2
▪ As depth of convolution layers increase more Loss of Information
Convolution Operation – Bias towards centre of image
1 Convolution
0 0 0 0 0 0
0 0 0 0 0 0
2 Convolutions
0 0 0 0
-1 -1 -1
0 0 0 0 0 0 255 255 255 255
255 255 255 255 255 255 ∗ 0 0 0 = 255 255 255 255
1 1 1
255 255 255 255 255 255 0 0 0 0
Filter or Kernel
255 255 255 255 255 255 3x3
X - Input
3 Convolutions Feature Map
6x6
4x4
▪ Side pixels are participating in less number of convolutions compared to center pixels
▪ As consequence, center pixels are given more weightage in feature extraction. If side
information is more crucial?
Padding
▪ Padding of input image addresses these problems
▪ We add additional rows and columns to input image so that output image remains same as
input image
(n’-f+1) x (n’-f+1) = n x n (n’-f+1) = n (n’) = n + f -1 n = 6, f = 3 then n’ = 8
0 0 0 0 0 0 0 0 (n-f+1) n+2p-f+1
Padding
0 0
0 - Padding
0 0
0 0
0 0 ∗ =
0 0
0 0
0 0 0 0 0 0 0 0 3x3
Filter or Kernel (n-f+1) x (n-f+1)
(8 x 8)
X – Input (6x6) Feature Map (6 x 6)
Padding
import tensorflow
from tensorflow import keras Keras: Valid Padding (n-f+1)
from [Link] import Dense, Conv2D, Flatten
from keras import Sequential
from [Link] import mnist Model: "sequential_3"
_________________________________________________________________
(x_train, y_train), (x_test, y_test) = Layer (type) Output Shape Param #
mnist.load_data() ==============================================================
conv2d_4 (Conv2D) (None, 26, 26, 32) 320
conv2d_5 (Conv2D) (None, 24, 24, 32) 9248
model = Sequential()
conv2d_6 (Conv2D) (None, 22, 22, 32) 9248
# Convolution layers
[Link](Conv2D(32, kernel_size=(3,3), flatten_1 (Flatten) (None, 15488) 0
padding='valid', activation='relu',
input_shape=(28,28,1) )) dense_2 (Dense) (None, 128) 1982592
[Link](Conv2D(32, kernel_size=(3,3),
padding='valid', activation='relu')) dense_3 (Dense) (None, 10) 1290
[Link](Conv2D(32, kernel_size=(3,3),
padding='valid', activation='relu’)) ==============================================================
Total params: 2,002,698
# Flattening output after Conv layers Trainable params: 2,002,698
[Link](Flatten()) Non-trainable params: 0
_________________________________________________________________
#Fully connected layers
[Link](Dense(128,activation='relu'))
[Link](Dense(10,activation='softmax'))
[Link]()
Padding
import tensorflow
from tensorflow import keras Keras: Same Padding (n+2p-f+1)
from [Link] import Dense, Conv2D, Flatten
from keras import Sequential
from [Link] import mnist
Model: "sequential_4"
_________________________________________________________________
(x_train, y_train), (x_test, y_test) = Layer (type) Output Shape Param #
mnist.load_data() =================================================================
conv2d_7 (Conv2D) (None, 28, 28, 32) 320
model = Sequential() conv2d_8 (Conv2D) (None, 28, 28, 32) 9248
# Convolution layers conv2d_9 (Conv2D) (None, 28, 28, 32) 9248
[Link](Conv2D(32, kernel_size=(3,3),
padding=‘same', activation='relu', flatten_2 (Flatten) (None, 25088) 0
input_shape=(28,28,1) ))
[Link](Conv2D(32, kernel_size=(3,3), dense_4 (Dense) (None, 128) 3211392
padding=‘same', activation='relu'))
[Link](Conv2D(32, kernel_size=(3,3), dense_5 (Dense) (None, 10) 1290
padding=‘same', activation='relu’))
=================================================================
# Flattening output after Conv layers Total params: 3,231,498
[Link](Flatten()) Trainable params: 3,231,498
Non-trainable params: 0
#Fully connected layers _________________________________________________________________
[Link](Dense(128,activation='relu'))
[Link](Dense(10,activation='softmax'))
[Link]()
Strides
▪ Stride is a hyperparameter that controls the convolution operation
▪ Stride specifies number of pixels Kernel is moved over input image at a time
▪ Stride of 1 means that the kernel is moved one pixel at a time, while stride of 2 means that
kernel is moved two pixels at a time and so on
▪ A Stride of 1 used when input image is small; while larger stride may be used when the input
is large
▪ Stride has a significant impact on the size of the output feature map
▪ Stride is represented as a tuple : (1,1) means horizontal stride - 1 and vertical stride - 2
Stride – (1,1)
1
0 0 0 0 0 0 0
1
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 3 3 3 3 3
-1 -1 -1
3 3 3 3 3
1 1 1 1 1 1 1 ∗ 0 0 0 =
1 1 1 1 1 1 1 1 1 1
0 0 0 0 0
1 1 1 1 1 1 1 0 0 0 0 0
3x3
1 1 1 1 1 1 1 5x5
7x7
▪ Both horizontal and vertical strides – 1
𝒏−𝒇+𝟏
Stride – (2,2)
2
0 0 0 0 0 0 0
2
0 0 0 0 0 0 0
0 0 0 0 0 0 0 -1 -1 -1 0 0 0
1 1 1 1 1 1 1 ∗ 0 0 0 = 3 3 3
1 1 1 1 1 1 1 1 1 1 0 0 0
1 1 1 1 1 1 1 3x3 3x3
1 1 1 1 1 1 1
7x7 𝒏−𝒇
+𝟏
𝟐
▪ Both horizontal and vertical strides – 2
▪ As stride size increases, size of feature map decreases 𝒏 + 𝟐𝒑 − 𝒇
𝒏−𝒇 +𝟏
+𝟏 𝒔
𝒔
Strided Convolution – s > 1 s = Stride size
s = Stride size p = Padding size
Strided Convolution – Feature Map Size
▪ Consider a case where Input Image = 6x7, Kernel size = 3x3, Stride size =2 ;
Feature Map Size ? 𝒏−𝒇
+𝟏= 𝟑
𝒔
𝟕−𝟑
+𝟏= 𝟑
𝟐
0 0 0 0 0 0 0 𝟔−𝟑
+ 𝟏 = 𝟐. 𝟓
𝟐
0 0 0 0 0 0 0
𝒇𝒍𝒐𝒐𝒓
0 0 0 0 0 0 0 -1 -1 -1
0 0 0
1 1 1 1 1 1 1 ∗ 0 0 0 =
1 1 1 3 3 3
1 1 1 1 1 1 1
1 1 1 1 1 1 1 3x3 2x3
Advantages of strided convolution
▪ Reduces computation: Kernel is applied only subset of input pixels
▪ Reduces the memory usage: Output feature map smaller than input image
▪ Allows for larger kernel sizes:
▪ Can be used for down sampling: reducing size of image while preserving its essential
features
Strides
import tensorflow Stride size(s) = 2 𝒏−𝒇
from tensorflow import keras +𝟏
from [Link] import Dense, Conv2D, Flatten Input size (n) = 28 𝒔
from keras import Sequential Kernel size (f) = 3
from [Link] import mnist
(x_train, y_train), (x_test, y_test) =
mnist.load_data() Model: "sequential_1"
_______________________________________________________
Layer (type) Output Shape Param #
model = Sequential() =======================================================
# Convolution layers
conv2d_3 (Conv2D) (None, 13, 13, 32) 320
[Link](Conv2D(32, kernel_size=(3,3), conv2d_4 (Conv2D) (None, 6, 6, 32) 9248
padding=‘valid', strides = (2,2), activation='relu', conv2d_5 (Conv2D) (None, 2, 2, 32) 9248
input_shape=(28,28,1) )) flatten_1 (Flatten) (None, 128) 0
[Link](Conv2D(32, kernel_size=(3,3),
padding=‘valid', strides = (2,2),
dense_2 (Dense) (None, 128) 16512
activation='relu')) dense_3 (Dense) (None, 10) 1290
[Link](Conv2D(32, kernel_size=(3,3), =======================================================
padding=‘valid', strides = (2,2), Total params: 36,618
activation='relu’))
Trainable params: 36,618
# Flattening output after Conv layers Non-trainable params: 0
[Link](Flatten())
#Fully connected layers
[Link](Dense(128,activation='relu'))
[Link](Dense(10,activation='softmax'))
[Link]()