Deep Learning

for
Computer Vision
Thang Nguyen
Viet Nguyen

[Link]/@vietnh10091
 Nội dung
Deep Learning Image Classification
01 Tổng quan về Deep 04 Bài toán phân loại ảnh
Learning

Neural Network Object detection

02 Cấu trúc của Neural 05 Bài toán định vị đối
network tượng

Convolutional NN Other topics

03 06 Image segmentation,
Cấu trúc của CNN
GANs
2
Tổng quan về Deep Learning

3
Tổng quan về Deep Learning

4
Deep Learning for Computer Vision

5
Deep Learning for Computer Vision

6
Deep Learning for Computer Vision

7
Neural Network vs Deep Learning

8
Image Classification

9
Dataset

10
 Linear Classifier - score function
Matrix multiplication

Element-wise multiplication

11
Linear Classifier explanation

MNIST dataset
12
Linear Classifier - Weight visualization

13
Linear Classifier - Weight visualization

14
Linear Classifier - Weight visualization

15
Linear Classifier - Weight visualization

CIFAR dataset

16
Biological Neuron vs Artificial Neuron

Neuron = linear classifier + activation function

17
From linear classifier to neuron

18
Activation function

19
Neural Network Architecture

20
 Neural Network

Forward propagation
Feedforward neural network
Forward pass Demo
21
 Complete example of Neural Network

Demo
22
 Deep Neural Network

Demo
23
Loss function

24
 Loss function
Regression Classification

Mean Absolute Error Loss Cross-Entropy Loss

Mean Squared Error Loss

Huber Loss

25
 Loss function
Mean Absolute Error Loss

26
 Loss function
Mean Squared Error Loss

27
 Loss function
Cross-Entropy Loss

28
Gradient

29
Optimization function
Gradient Descent

30
 Optimization function
Gradient Descent in Linear Classifier vs Neural network

31
 Gradient Descent: Different versions
(Stochastic ) Mini-batch
Batch Gradient Descent Stochastic Gradient Descent
Gradient Descent

Tất cả datapoints Từng datapoint một sẽ N datapoint sẽ được đưa

được đưa vào mô hình được đưa vào mô hình để vào mô hình cùng lúc để
cùng 1 lúc để tính tính gradient tính gradient
gradient

32
How parameters are updated?

33
Challenge with Gradient Descent: Loss function

34
Challenge with Gradient Descent: Local minima

35
Challenge with Gradient Descent: Saddle points

36
Challenge with Gradient Descent: Ravines

37
Momentum

38
Momentum

39
Adaptive Gradient Descent (1)

40
Adaptive Gradient Descent (2)

41
Adaptive Gradient Descent (3)

42
Adaptive Gradient Descent (4)

43
Adaptive Gradient Descent (4)

44
Root Mean Square Propagation

45
BackPropagation

46
BackPropagation

47
Backpropagation

48
BackPropagation - example

forward

backward 49
BackPropagation - example

forward

backward 50
BackPropagation - example

forward

backward 51
BackPropagation - example

forward

backward 52
BackPropagation - example

forward

backward 53
BackPropagation - example

forward

backward 54
BackPropagation - example

forward

backward 55
BackPropagation - explanation

56
Neural network’s learning process

Demo
57
Parameters vs Hyperparameters

58
Neural Network

59
Gray image is represented in Computer Vision

60
Color image is represented in Computer Vision

61
Data preprocessing

62
 Train a (Convolutional) Neural Network in Pytorch
01 02 03 04
Step 1 Step 2 Step 3 Step 4
Define a loss Define an
Setup a dataset Define a model
function optimizer
Xây dựng 1 dataset từ Chọn/tự xây dựng 1
Định nghĩa mô hình Chọn 1 optimizer
tập ảnh hay nhiều hàm loss

05 06 07
Step 5 Step 6 Step 7
Train Validate Test
Đánh giá trong quá
Huấn luyện mô hình Đánh giá mô hình cuối
trình huấn luyện với
với training data cùng với test data
validation data

63
 Pytorch: tensor

Tensor in Pytorch is equivalent to Array in Numpy

64
Pytorch: tensor

65
Pytorch: tensor

66
Pytorch: Image to tensor

67
 Pytorch: Datasets
Built-in datasets CIFAR dataset

68
 Pytorch: Datasets
ImageFolder Butterfly dataset

69
Pytorch: Built-in datasets
Pytorch all built-in datasets

70
Pytorch: Datasets

71
Step 1: Setup a dataset

72
Step 2: Define a Neural Network

73
Step 3: Define a loss function and an optimizer

74
Step 4: Train the network

75
Step 4: Train the network

76
Convolutional Neural Network

77
 Problems in Neural Network
Variations Scalable
● Position
● Memory
● Orientation
● Calculation
● Scale

78
Convolutional Neural Network

79
Convolutional Neural Network

80
Convolutional operation

81
Convolutional operation

82
Convolutional operation on Volume

83
Convolutional operation on Volume

84
Convolutional operation with multiple Filters/Kernels

85
1x1 Convolutional operation

86
Convolutional layer

87
Convolutional layer’s advantage
Local Connectivity Parameter Sharing

88
Convolutional layer’s advantage

89
Pooling layer

90
Internal Covariate Shift

91
 Batch Normalization layer

Batch Normalization
92
Why Normalization helps?

93
Batch Normalization: Training phase

94
Batch Normalization: Inference phase

95
Normalization layer

96
Dropout layer

97
 Common CNN’s layer pattern (part 1)
Hình thức phổ biến nhất của CNN: Các layer lần lượt được sắp xếp theo thứ tự sau:
1. Một vài block của Conv->(BatchNorm)->ReLU
2. Max-pooling
3. Lặp lại bước 1 và 2 cho đến khi feature map đủ bé
4. Flatten (làm phẳng) feature map
5. Một (vài) fully connected layer

98
Receptive Field

99
Receptive Field

100
 Receptive Field
Add more Conv layers
01
How to increase Receptive field
Make the network deeper

Add pooling layers

02
sub-sampling

Use Dilated Conv

03

Use Depth-wise Conv

04

101
 Common CNN’s layer pattern (part 2)
Nhiều Conv layer với filter/kernel bé tốt hơn 1 Conv layer với filter/kernel lớn
2 3x3-kernel Conv layers 1 5x5 kernel Conv layer
Receptive field of last layer 5x5
Number of parameters 2*3*3 5*5
Number of non-linear layers 2 1

102
Effective Receptive Field

103
Use Pooling layer

104
Dilated/Atrous Convolutions

105
 Common CNN’s layer pattern (part 3)
Làm sao để thiết kế CNN architecture?
Rule 1: Don’t be a hero. Use whatever works best one ImageNet

106
 Common CNN’s layer pattern (part 3)
Làm sao để thiết kế CNN architecture?
Rule 2: If rule 1 does not work, you could consider the following question
● Mô hình nên có bao nhiêu layer?
● Mỗi layer nên có parameter như thế nào?
○ Convolutional có bao nhiêu channel? Kernel size là bao nhiêu?
○ Nên dùng activation function gì?
○ Pooling nên có kernel size là bao nhiêu?
○ Nên flatten feature map ở đâu?
○ Nên có bao nhiêu fully connected layer?
○ Có nên dùng Batchnorm, Dropout, …?
107
Common CNN architectures

108
ImageNet Large Scale Visual Recognition Challenge

109
LeNet (1998)

110
AlexNet (2012)

111
LeNet vs AlexNet

112
VGGNet (2014)

113
VGGNet (2014)

114
Universal Approximation Theorem

115
ResNet (2015)

116
ResNet (2015)

117
Problems in Neural Network: Vanishing gradients

118
Sigmoid vs Vanishing gradients

119
ReLU vs Vanishing gradients

120
Leaky ReLU vs Vanishing gradients

121
Các biến thể khác của ReLU

122
SWISH

123
 Problems in Neural Network: Exploding gradients
Ngược lại với vanishing gradient
Dấu hiệu
● Model’s parameter tăng rất nhanh (exponential growth)
● Model’s parameter có thể thành NaN trong quá trình training
● Loss trở thành NaN trong quá trình training
● Loss lớn dù train lâu
Nguyên nhân
● Improper initial weights => large weight update
● Poor initial learning rate
● Poor loss function => large loss value => large weight update
124
 Solutions for Vanishing/Exploding gradients
● Proper activation functions, like ReLU and its derivative
● Batch Normalization
● Regularization
● Proper weight initialization
● Gradient clipping (against exploding gradients)
○ By values
○ By norm

125
 Weight initialization
● Performance của CNN/NN phụ thuộc vào weight initialization
● Các lợi ích của good weight initialization:
○ Reproducible model
○ Model converge faster
○ Tránh được vanishing/exploding gradients
● Các cách phổ biến:
○ Zero/Constant/Random initialization

126
 Weight initialization: Zero/Constant initialization
● Tất cả các parameter đều được khởi tạo là 0 hoặc 1 giá trị cố định
● Tất cả các node sẽ học theo cùng 1 cách/1 kiểu => fail to break
symmetry
● Luôn dẫn đến kết quả tồi

127
 Weight initialization: Random initialization
● Giải quyết được vấn đề break the symmetry
● Có thể dẫn đến vấn đề mới: Vanishing/exploding gradients
● Random initialization from a distribution:
○ Random Normal
○ Random Uniform

128
 Weight initialization: Xavier/Glorot initialization
● Được áp dụng cho các layers (fully connected layers, Conv layers) mà
sử dụng activation function là sigmoid hay tanh
● Có 2 phiên bản:

129
 Weight initialization: He/Kaiming initialization
● Được áp dụng cho các layers (fully connected layers, Conv layers) mà
sử dụng activation function là ReLU và các biến thể của nó
● Có 2 phiên bản:

130
Gradient clipping

131
Transfer Learning

132
Transfer Learning: Use CNN as fixed feature extractor

133
Transfer Learning: Fine-tune CNN

134
Transfer Learning: How to choose approach

135
Data augmentation

136
Data augmentation: Position manipulation

Another name: Geometric transformations 137

Data augmentation: Color manipulation

Another name: Color distortion 138

Data augmentation: Example

139
Data augmentation: Example

140
Object Detection

141
Image Classification vs Object Detection

142
Object Detection

143
Object Detection - Single Object

144
Object Detection - Multiple Objects

145
Sliding Windows

146
Selective Search

147
R-CNN -> Fast R-CNN -> Faster R-CNN

148
R-CNN -> Fast R-CNN -> Faster R-CNN

149
R-CNN -> Fast R-CNN -> Faster R-CNN

150
 Bounding box regression
predict

label

p: region proposals (anchors)

g: actual bbox
ĝ: predicted bbox
151
Intersection over Union

152
Non-maximum Suppression

153
Mean Average Precision

154
R-CNN -> Fast R-CNN -> Faster R-CNN

155
RoI Pooling

156
R-CNN -> Fast R-CNN -> Faster R-CNN

157
R-CNN -> Fast R-CNN -> Faster R-CNN

158
Anchors

159
Region Proposal Network (RPN)

160
Faster R-CNN losses

161
Comparison

162
You Only Look Once

163
Faster R-CNN vs YOLO

164
YOLO

165
YOLO v1

166
YOLO v1: feature map

167
YOLO v1: Bounding box

168
YOLO v1: Confidence score

169
YOLO v1: Bounding box + Confidence score

170
YOLO v1: Class confidence score

171
YOLO v1: Put all together

172
YOLO v1: Post-processing

173
YOLO v1: Loss function

174
YOLO v1: Evaluation

175
YOLO v2: Adjustments from Yolov1

176
 YOLO v2: Adjustments from Yolov1

YOLO v1 output
177
YOLO v2: Adjustments from Yolov1

178
YOLO v2: Adjustments from Yolov1

vs

179
YOLO v2: Adjustments from Yolov1

180
 From YOLO v2 to YOLO v3

Darknet-19 Darknet-53

181
YOLO v3: Residual block and skip connection

182
YOLO v3: Feature Pyramid Network

183
YOLO v3: Different scales

184
One-stage vs two-stage object detector

185
Image Segmentation

186
Types of Image segmentation tasks

187
Annotations

188
Basic structure

189
Transfer learning

190
Fully Convolutional Network

191
FCN: From Image Classification…

192
FCN: …to Semantic Segmentation

193
Upsampling/Unpooling methods

194
Transposed Convolution (Deconvolution)

195
Transposed Convolution (Deconvolution)

196
Skip connection

197
Result

198
U-Net

199
Deeplab

200
Deeplab v1: DCNN
Dilated/Atrous Convolution

201
Deeplab v1: DCNN

202
Deeplab v1: Fully Connected Conditional Random Fields

203
Deeplab v1: Pipeline

204
Deeplab v2: Atrous Spatial Pyramid Pooling

205
Deeplab v2: Difference from v1

206
Deeplab v2: Difference from v1

207
 Deeplab v3: Difference from v2
Differences from Deeplab v2

208
Deeplab v3: Difference from v2

209
Generative adversarial networks

210
Generative model vs Discriminative model

211
Which task is harder?

212
Overview of GAN structure

213
Overview of GAN structure

214
Overview of GAN structure

215
Discriminator’s training

216
Generator’s training

217
 GAN’s training
● Discriminator được huấn luyện trong 1 hoặc 1 vài epochs. Trong giai
đoạn này generator được giữ cố định
● Generator được huấn luyện trong 1 hoặc 1 vài epochs. Trong giai đoạn
này discriminator được giữ cố định
● Lặp lại 2 bước trên

218
Recurrent Neural Network

219
Recurrent Neural Network

220
Recurrent Neural Network

221
Recurrent Neural Network

222
Example

223
Problem of RNN

The clouds are in the sky

I was born in
Vietnam… I could
speak Vietnamese
fluently
224
Standard RNN

225
Long Short Term Memory

226
Long Short Term Memory: Idea

227
LSTM: Forget gate layer

228
LSTM: Input gate layer

229
LSTM: New state

230
LSTM: Output

231
Gated Recurrent Unit: GRU

232
Optical character recognition

233
Example

234
 Challenges

Computer Vision Natural Language Processing

Optical Character Recognition

235
Challenges

236
 Datasets

Street View House Number dataset Scene Text dataset

237
OCR general pipeline

238
Step 1: Image pre-processing

239
Step 1: Image pre-processing

240
Step 1.1: Normalization

241
Step 1.2: Skew correction

242
Step 1.3: Image scaling

243
Step 1.4: Noise Removal

244
Step 1.5: Thinning and Skeletonization

245
Step 1.6: Gray scale image

246
Step 1.7: Thresholding or Binarization

247
Step 2: Text detection

248
Step 3: Text recognition

249
Convolutional-recurrent neural network

Feature column

250
Step 4: Restructuring

251