R-CNN based Object Detection

Object detection:

• So far we learn object detection with handcrafted features

such as gradient vectors and HOG….

• Today we are going to learn a set of models in the R-CNN

(“Region-based CNN”) family.

• R-CNN
• Fast R-CNN
• Faster –CNN
• YOLO
• SSD

2
R-CNN:
• R-CNN (Girshick et al., 2014) is short for “Region-based
Convolutional Neural Networks”.
• The main idea is composed of two steps. First, using selective
search, it identifies a manageable number of bounding-box
object region candidates (“region of interest” or “RoI”).
• And then it extracts CNN features from each region
independently for classification.

3
R-CNN Model Workflow:
Step-1:
Pre-train a CNN network on image classification tasks; for
example, VGG or ResNet trained on ImageNet dataset. The
classification task involves N classes.

• NOTE: You can find a pre-trained AlexNet, ResNet, VGG, and

others.

Step-2:
Propose category-independent regions of interest by selective
search (~2k candidates per image). Those regions may contain
target objects and they are of different sizes.

4
R-CNN Model Workflow:
Step-3:
Region candidates are warped to have a fixed size as required
by CNN.

Step-4:
Continue fine-tuning the CNN on warped proposal regions for K
+ 1 classes; The additional one class refers to the background
(no object of interest). In the fine-tuning stage, we should use a
much smaller learning rate and the mini-batch oversamples the
positive cases because most proposed regions are just
background.

5
R-CNN Model Workflow:
Step-5:
Given every image region, one forward propagation through the
CNN generates a feature vector. This feature vector is then
consumed by a binary SVM trained for each
class independently.

The positive samples are proposed regions with IoU

(intersection over union) overlap threshold >= 0.3, and
negative samples are irrelevant others.

Step-6:
To reduce the localization errors, a regression model is trained
to correct the predicted detection window on bounding box
correction offset using CNN features.

6
R-CNN Bounding Box Regression:

7
R-CNN Bounding Box Regression:

8
R-CNN (Common Terminologies)
Non-Max Suppression
• Likely the model is able to find multiple bounding boxes for
the same object. Non-max suppression helps avoid
repeated detection of the same instance.

• After we get a set of matched bounding boxes for the same

object category: Sort all the bounding boxes by confidence
score. Discard boxes with low confidence scores.

• While there is any remaining bounding box, repeat the

following: Greedily select the one with the highest score.
• Skip the remaining boxes with high IoU (i.e. > 0.5) with
previously selected one.

9
R-CNN (Common Terminologies)
Non-Max Suppression

10
R-CNN (Common Terminologies)
Hard Negative Mining
• We consider bounding boxes without objects as negative
examples. Not all the negative examples are equally hard
to be identified.

• For example, if it holds pure empty background, it is likely

an “easy negative”; but if the box contains weird noisy
texture or partial object, it could be hard to be recognized
and these are “hard negative”.

• The hard negative examples are easily misclassified. We

can explicitly find those false positive samples during the
training loops and include them in the training data so as
to improve the classifier.
11
Selective Search:

• Selective search is a common algorithm to provide region

proposals that potentially contain objects.

• It is built on top of the image segmentation output and

use region-based characteristics

• (NOTE: not just attributes of a single pixel) to do a

bottom-up hierarchical grouping.

12
Selective Search Workflow:

1. At the initialization stage, we apply a graph-based image

segmentation algorithm to create regions to start with.

2. Use a greedy algorithm to iteratively group regions

together:
I. First the similarities between all neighbouring
regions are calculated.
II. The two most similar regions are grouped together,
and new similarities are calculated between the
resulting region and its neighbours.

3. The process of grouping the most similar regions (Step 2)

is repeated until the whole image becomes a single
region.
13
Selective Search:

14
R-CNN Summary:

15
R-CNN Bottlenecks:
Looking through the R-CNN learning steps, you could easily
find out that training an R-CNN model is expensive and slow,
as the following steps involve a lot of work:
• Running selective search to propose 2000 region
candidates for every image;

• Generating the CNN feature vector for every image

region (N images * 2000).

• The whole process involves three models separately

without much shared computation: the convolutional
neural network for image classification and feature
extraction; the top SVM classifier for identifying target
objects; and the regression model for tightening region
bounding boxes.
16
R-CNN Bottlenecks:
• No joint learning

• Use ad hoc training objectives

• Fine tune network with softmax
classier (log loss)
• Train post-hoc linear SVMs
(hinge loss)
• Train post-hoc bounding-box
regressors (squared loss)

• Training ( 3 days) and testing (47s

per image) is slow (using VGGNet)

• Takes a lot of disk space

17
Fast R-CNN

18
Fast R-CNN:
➢ To make R-CNN faster, Girshick (2015) improved the training
procedure by unifying three independent models into one
jointly trained framework and increasing shared
computation results, named Fast R-CNN.

➢ Instead of extracting CNN feature vectors independently for

each region proposal, this model aggregates them into one
CNN forward pass over the entire image and the region
proposals share this feature matrix.

➢ Then the same feature matrix is branched out to be used for

learning the object classifier and the bounding-box regressor.

➢ In conclusion, computation sharing speeds up R-CNN.

19
Fast R-CNN:

20
CNN Network of Fast R-CNN

► Fast R-CNN is experimented with three pre-trained ImageNet

networks each with 5 max-pooling layers and 5-13 convolution
layers (such as VGG-16). There are some changes proposed in
this pre-trained network, These changes are:
► The network is modified in such a way that it two inputs the
image and list of region proposals generated on that image.
► Second, the last pooling layer (here (7*7*512)) before fully
connected layers needs to be replaced by the region of interest
(RoI) pooling layer.
► Third, the last fully connected layer and softmax layer is
replaced by twin layers of softmax classifier and K+1 category-
specific bounding box regressor with a fully connected layer.

21
 CNN Network of Fast R-CNN (VGG-16)

This CNN architecture takes the image (size = 224 x 224 x 3 for
VGG-16) and its region proposal and outputs the convolution
feature map (size = 14 x 14 x 512 for VGG-16).
22
ROI Pooling:
❖ RoI pooling is a novel thing that was introduced in the Fast
R-CNN paper. Its purpose is to produce uniform, fixed-size
feature maps from non-uniform inputs (RoIs). It takes two
values as inputs:

❖ A feature map was obtained from the previous CNN layer

(14 x 14 x 512 in VGG-16).

❖ An N x 4 matrix represents regions of interest, where N

is a number of RoIs, the first two represent the
coordinates of the upper left corner of RoI and the other
two represent the height and width of RoI denoted as (r,
c, h, w).

23
ROI pooling

► Let's consider we have 8*8 feature maps, we need to extract

an output of size 2*2. We will follow the steps below.

24
ROI pooling
► Suppose we were given RoI’s left corner coordinates
as (0, 3) and height, and width as (5, 7).

25
Fast R-CNN Model Workflow:
How Fast R-CNN works is summarized as follows; many steps
are same as in R-CNN:
1. First, pre-train a convolutional neural network on image
classification tasks.
2. Propose regions by selective search (~2k candidates per
image).
3. Alter the pre-trained CNN:
• Replace the last max pooling layer of the pre-trained CNN
with a RoI pooling layer. The RoI pooling layer outputs
fixed-length feature vectors of region proposals. Sharing
the CNN computation makes a lot of sense, as many
region proposals of the same images are highly
overlapped.
• Replace the last fully connected layer and the last
softmax layer (K classes) with a fully connected layer and
softmax over K + 1 classes. 26
Fast R-CNN Model Workflow:
How Fast R-CNN works is summarized as follows; many steps
are same as in R-CNN:

Finally the model branches into two output layers:

• A softmax estimator of K + 1 classes (same as in R-CNN,
+1 is the “background” class), outputting a discrete
probability distribution per RoI.

• A bounding-box regression model which predicts offsets

relative to the original RoI for each of K classes.

27
Loss Function: Symbols
The model is optimized for a loss combining two tasks
(classification + localization).

𝑢 : True class label, u=0, 1, 2, … , 𝐾; by convention, the catch−all

background class has u=0.

𝑝: Discrete probability distribution (per RoI) over K + 1 classes: ,

computed by a softmax over the K + 1 outputs of a fully connected
layer

𝑣: True bounding box

𝑡 𝑢 : Predicted bounding box correction

28
Loss Function: Symbols

29
Smooth L1 loss:

30
Fast R-CNN

31
SVD in Fast R-CNN:
➢ Though Fast R-CNN uses CNN layers for feature
extraction process, the final layers are made of fully
connected linear layers, which are responsible for
predicting the class and bounding boxes of the objects.

➢ The inclusion of this fully connected layer significantly

enlarges the model size.

➢ However, since the weight parameters of this fc-layers

are stored as Matrix dictionary, we could leverage the
advantage of SVD to decompose it.

32
SVD in Fast R-CNN:
➢ In a fully connected layer (FC-layer), the weight
matrix 'W' typically has dimensions n_out, n_in, representing
the number of output nodes and number of input nodes
respectively.

➢ This weight matrix can be decomposed into two matrices: a

matrix that maps from the input space to a hidden space,
and a matrix that maps from the hidden space to the output
space.

➢ Let’s call these matrices W1 and W2 , respectively.

33
► So, instead of having just one weight matrix W, we have two
weight matrices W1 and W2.

► An additional convolutional layer will be added to

accommodate these decomposed matrices .

► The dimensions of W1 would be n_hidden, n_in while the

dimension of W2 would be n_out, n_hidden.

► By introducing this additional hidden layer, we can potentially

decrease the total number of parameters required and thus
reducing the model size after training is performed.

34
➢ Fast-RCNN have adopted truncated-SVD to be applied on the
fully connected layers.
➢ Fast R-CNN introduced the Region of Interest (RoI) pooling
operation, which efficiently extracts fixed-size feature maps
from each region proposal.
➢ Though this pooling operation ensures that features extracted
from different-sized regions can be fed into subsequent layers
of the network, they have increased the number of parameters in
the network architecture.
➢ Furthermore, they typically uses deep convolutional neural
networks (CNNs), such as VGG, ResNet, or similar architectures,
which contain millions of parameters.
➢ Nevertheless, by decomposing the trained-weight matrix of the
fully connected layers, the number of parameters in the
inference model could be reduced.

35
Detection time:

36
Fast R-CNN Bottleneck:

➢ Fast R-CNN is much faster in both training and testing

time.

➢ However, the improvement is not dramatic because

the region proposals are generated separately by
another model and that is very expensive.

37
Faster R-CNN

38
Faster R-CNN:

➢ An intuitive speedup solution is to integrate the region

proposal algorithm into the CNN model.

➢ Faster R-CNN (Ren et al., 2016) is doing exactly this:

construct a single, unified model composed of RPN
(region proposal network) and fast R-CNN with
shared convolutional feature layers.

39
Faster R-CNN: Workflow

1. Pre-train a CNN network on image classification tasks.

2. Fine-tune the RPN (region proposal network) end-to-end

for the region proposal task, which is initialized by the
pre-train image classifier. Positive samples have IoU
(intersection-over-union) > 0.7, while negative samples
have IoU < 0.3.
• Slide a small n x n spatial window over the conv
feature map of the entire image.
• At the center of each sliding window, we predict
multiple regions of various scales and ratios
simultaneously. An anchor is a combination of (sliding
window center, scale, ratio). For example, 3 scales + 3
ratios => k=9 anchors at each sliding position.
40
Faster R-CNN Model Workflow:
3. Train a Fast R-CNN object detection model using the
proposals generated by the current RPN.

4. Then use the Fast R-CNN network to initialize RPN

training. While keeping the shared convolutional layers,
only fine-tune the RPN-specific layers.

5. At this stage, RPN and the detection network have shared

convolutional layers!

6. Finally fine-tune the unique layers of Fast R-CNN.

7. Step 4-5 can be repeated to train RPN and Fast R-CNN

alternatively if needed.

41
Faster R-CNN Model Workflow:

42
Faster R-CNN Loss function:

Where L_cls is the log loss function over two classes, as we can easily translate a multi-
class classification into a binary classification by predicting a sample being a target object
versus not. The second one is the smooth L1 loss.

43
Mask R-CNN:

➢ Mask R-CNN (He et al., 2017) extends Faster R-CNN to

pixel-level image segmentation.

➢ The key point is to decouple the classification and the

pixel-level mask prediction tasks.

➢ Based on the framework of Faster R-CNN, it added a

third branch for predicting an object mask in parallel
with the existing branches for classification and
localization.

➢ The mask branch is a small fully-connected network

applied to each RoI, predicting a segmentation mask in
a pixel-to-pixel manner.
44
Mask R-CNN:

45
Mask R-CNN:
Because pixel-level segmentation requires much more fine-
grained alignment than bounding boxes, mask R-CNN
improves the RoI pooling layer (named “RoIAlign layer”) so
that RoI can be better and more precisely mapped to the
regions of the original image.

46
Mask R-CNN output:

47
Mask R-CNN loss function:

48
Summary of R-CNN:

49
YOLO (You Only Look Once)

50
Faster R-CNN

51
Two stage v/s Single Stage Detectors:

52
Two stage v/s Single Stage Detectors:

53
YOLO:

➢ The R-CNN pipelines separate proposal generation and

proposal classification into two separate stages.

➢ Can we have an end-to-end architecture which does both

proposal generation and clasification simultaneously?

➢ The solution gives the YOLO (You Only Look Once)

architectures.

o I J Redmon, S Divvala, R Girshick and A Farhadi, `You

Only Look Once: Unied, Real-Time Object Detection',
CVPR 2016 - YOLO v1
o I J Redmon and A Farhadi, `YOLO9000: Better, Faster,
Stronger', CVPR 2017 - YOLO v2

54
YOLO:

55
YOLO:

56
YOLO:

57
YOLO:

58
YOLO:

59
YOLO:

60
YOLO:

61
YOLO:

62
YOLO:

63
YOLO:

64
YOLO:

65
YOLO Architecture:
➢YOLO was implemented as a convolutional neural
network and was evaluated on the PASCAL VOC
detection dataset.

➢The initial convolutional layers of the network extract

features from the image while the fully connected layers
predict the output probabilities and coordinates.

➢The network architecture is inspired by the GoogLeNet

model for image classification.

➢The network has 24 convolutional layers followed by 2

fully connected layers.

66
YOLO Architecture:

➢ Instead of the inception modules used by GoogLeNet,

authors simply use 11 reduction layers followed by 33
convolutional layers.

➢ YOLO was train a fast version of YOLO designed to push

the boundaries of fast object detection.

➢ Fast YOLO uses a neural network with fewer convolutional

layers (9 instead of 24) and fewer filters in those layers.

➢ Other than the size of the network, all training and testing
parameters are the same between YOLO and Fast YOLO.

67
YOLO Architecture:

68
YOLO Prediction:

69
YOLO Prediction:

70
YOLO Training:

71
YOLO Loss Function:

72
YOLO Loss Function:

73
YOLO Loss Function:

74
YOLO Loss Function:

75
YOLO total Loss:

76
YOLO training:

77
THANKYOU

78