Bounding boxes
DEEP LEARNING FOR IMAGES WITH PYTORCH
Michal Oleszak
Machine Learning Engineer
What is object recognition?
Object recognition identifies objects in
images:
Location of each object in an image
(bounding box)
Class label of each object
Applications: surveillance, medical diagnosis,
traffic management, sports analytics
In this video: annotation with bounding
boxes
In later videos: evaluation and models
DEEP LEARNING FOR IMAGES WITH PYTORCH
Bounding box representation
A rectangular box describing the object's
spatial location
Training data annotations & model outputs
Ground truth bounding box: precise object
location
DEEP LEARNING FOR IMAGES WITH PYTORCH
Bounding box representation
A rectangular box describing the object's
spatial location
Training data annotations & model outputs
Ground truth bounding box: precise object
location
Bounding box coordinates:
Top left and bottom right
Bounding box = (x1, y1, x2, y2)
x1 = x_min, x2 = x_max, ...
DEEP LEARNING FOR IMAGES WITH PYTORCH
Pixels and coordinates
Coordinates: x - the column number, y - the row number
Origin: (0, 0) - the top left corner
DEEP LEARNING FOR IMAGES WITH PYTORCH
Converting pixels to tensors
Transforming with ToTensor() Tranforming with PILToTensor()
Tensor type: Tensor type:
[Link] torch.uint8 (8-bit integer)
Scaled tensor range: Unscaled tensor range:
[0.0, 1.0] [0, 255]
import [Link] as transforms import [Link] as transforms
transform = [Link]([ transform = [Link]([
[Link](224), [Link](224),
[Link]() [Link]()
]) ])
image_tensor = transform(image) image_tensor = transform(image)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Drawing the bounding box
from [Link] import draw_bounding_boxes Import draw_bounding_boxes
Collect coordinates into a tensor
bbox = [Link]([x_min, y_min, x_max, y_max])
bbox = [Link](0) Unsqueeze to two dimensions
bbox_image = draw_bounding_boxes(
image_tensor, bbox, width=3, colors="red" Transform to image and plot
)
transform = [Link]([
[Link]()
])
pil_image = transform(bbox_image)
import [Link] as plt
[Link](pil_image)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Let's practice!
DEEP LEARNING FOR IMAGES WITH PYTORCH
Evaluating object
recognition models
DEEP LEARNING FOR IMAGES WITH PYTORCH
Michal Oleszak
Machine Learning Engineer
Classification and localization
Output 1: Classification (e.g., cat)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Classification and localization
Output 1: Classification (e.g., cat)
Output 2: Bounding box regression [x1, y1, x2, y2]
DEEP LEARNING FOR IMAGES WITH PYTORCH
Intersection over union (IoU)
Object of interest: object in image we want to detect (e.g., dog)
Ground truth box: the accurate bounding box around the object of interest
Intersection over Union: a metric to measure the overlap between two boxes
IoU = Area of Intersection / Area of Union
IoU = 0 no overlap, IoU = 1 perfect overlap
IoU >0.5 is a good prediction
DEEP LEARNING FOR IMAGES WITH PYTORCH
IoU in PyTorch
bbox1 = [50, 50, 150, 150] Two sets of boxes (x1, y1, x2, y2)
bbox2 = [100, 100, 200, 200]
bbox1 = [Link](bbox1).unsqueeze(0)
bbox2 = [Link](bbox2).unsqueeze(0)
from [Link] import box_iou
Convert vectors to 2-D tensors
iou = box_iou(bbox1, bbox2)
Calculate IoU
print(iou)
tensor([[0.1429]])
DEEP LEARNING FOR IMAGES WITH PYTORCH
Predicting bounding boxes
[Link]()
with torch.no_grad():
output = model(input_image)
print(output)
[{'boxes': tensor([[ 42.8553, 271.9481, 180.6003, 346.7082],
[191.6016, 80.4759, 247.8009, 387.5475], ....),
'scores': tensor([1.0000, 1.0000, 0.9998, ... ]),
'labels': tensor([18, 1, 20, 18, 18, 18 ...])
}]
boxes = output[0]["boxes"]
scores = output[0]["scores"]
DEEP LEARNING FOR IMAGES WITH PYTORCH
Non-max suppression (NMS)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Non-max suppression (NMS)
Non-max suppression: a common technique to select the most relevant bounding boxes
Non-max: discarding boxes with low confidence score to contain an object
Suppression: discarding boxes with low IoU
DEEP LEARNING FOR IMAGES WITH PYTORCH
Non-max suppression in PyTorch
from [Link] import nms
Boxes: tensors with the bounding box
coordinates of the shape [N, 4]
box_indices = nms( Scores: tensor with the confidence score for
boxes=boxes, each box of the shape [N]
scores=scores,
iou_threshold: the threshold between 0.0
iou_threshold=0.5,
and 1.0
)
Output: indices of filtered bounding boxes
print(box_indices)
tensor([ 0, 1, 2, 8])
filtered_boxes = boxes[box_indices]
DEEP LEARNING FOR IMAGES WITH PYTORCH
Let's practice!
DEEP LEARNING FOR IMAGES WITH PYTORCH
Object detection
using R-CNN
DEEP LEARNING FOR IMAGES WITH PYTORCH
Michal Oleszak
Machine Learning Engineer
Region-based CNN family: R-CNN
R-CNN family: R-CNN, Fast-CNN, Faster CNN
Module 1: generation of region proposals
1 Citation: Jason Brownlee. 2019. Deep Learning for Computer Vision.
DEEP LEARNING FOR IMAGES WITH PYTORCH
Region-based CNN family: R-CNN
R-CNN family: R-CNN, Fast-CNN, Faster CNN
Module 1: generation of region proposals
Module 2: feature extraction (convolutional layers)
1 Citation: Jason Brownlee. 2019. Deep Learning for Computer Vision.
DEEP LEARNING FOR IMAGES WITH PYTORCH
Region-based CNN family: R-CNN
R-CNN family: R-CNN, Fast-CNN, Faster CNN
Module 1: generation of region proposals
Module 2: feature extraction (convolutional layers)
Module 3: class and bounding box prediction
1 Citation: Jason Brownlee. 2019. Deep Learning for Computer Vision.
DEEP LEARNING FOR IMAGES WITH PYTORCH
R-CNN: backbone
Convolutional layers: pre-trained models
Backbone: the core CNN architecture responsible for feature extraction
Convolutional & pooling layers
Extract features for region proposals and object detection
DEEP LEARNING FOR IMAGES WITH PYTORCH
R-CNN: backbone with PyTorch
import [Link] as nn
from [Link] import vgg16,
VGG16_Weights
vgg = vgg16(weights=VGG16_Weights.DEFAULT)
DEEP LEARNING FOR IMAGES WITH PYTORCH
R-CNN: backbone with PyTorch
import [Link] as nn
from [Link] import vgg16,
VGG16_Weights
vgg = vgg16(weights=VGG16_Weights.DEFAULT)
.features : only convolutional layers
DEEP LEARNING FOR IMAGES WITH PYTORCH
R-CNN: backbone with PyTorch
import [Link] as nn
from [Link] import vgg16,
VGG16_Weights
vgg = vgg16(weights=VGG16_Weights.DEFAULT)
.features : only convolutional layers
.children() : all layers from block
DEEP LEARNING FOR IMAGES WITH PYTORCH
R-CNN: backbone with PyTorch
import [Link] as nn
from [Link] import vgg16,
VGG16_Weights
vgg = vgg16(weights=VGG16_Weights.DEFAULT)
backbone = [Link](
*list([Link]())
)
[Link](*list()) : all sub-layers
are placed into a sequential block as a list .features : only convolutional layers
* : unpacks the elements from the list
.children() : all layers from block
DEEP LEARNING FOR IMAGES WITH PYTORCH
R-CNN: classifier layer
Extract backbone's output size
input_dimension = [Link](*list(
vgg_backbone.[Link]())
)[0].in_features
Create a new classifier
classifier = [Link](
[Link](input_dimension, 512),
[Link](),
[Link](512, num_classes),
)
DEEP LEARNING FOR IMAGES WITH PYTORCH
R-CNN: box regressor layer
Sits on top of the backbone box_regressor = [Link](
4 outputs for the 4 box coordinates [Link](input_dimension, 32),
[Link](),
[Link](32, 4),
)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Putting it all together: object detection model
class ObjectDetectorCNN([Link]):
def __init__(self):
super(ObjectDetectorCNN, self).__init__()
vgg = vgg16(weights=VGG16_Weights.DEFAULT)
[Link] = [Link](*list([Link]()))
input_features = [Link](*list([Link]()))[0].in_features
[Link] = [Link](
[Link](input_features, 512),
[Link](),
[Link](512, 2),
)
self.box_regressor = [Link](
[Link](input_features, 32),
[Link](),
[Link](32, 4),
)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Putting it all together: object detection model
class ObjectDetector([Link]):
(...)
def forward(self, x):
features = [Link](x)
bboxes = [Link](features)
classes = [Link](features)
return bboxes, classes
DEEP LEARNING FOR IMAGES WITH PYTORCH
Running object recognition
1. Load and transform the image
2. unsqueeze() the image to add the batch dimension
3. Pass the image tensor to the model
4. Run Non-Max Suppression ( nms() ) over model's output
5. draw_bounding_boxes() on top of the image
DEEP LEARNING FOR IMAGES WITH PYTORCH
Let's practice!
DEEP LEARNING FOR IMAGES WITH PYTORCH
Region network
proposals with
Faster R-CNN
DEEP LEARNING FOR IMAGES WITH PYTORCH
Michal Oleszak
Machine Learning Engineer
Regions and anchor boxes
Region: a smaller area of the image that could contain objects of interest, grouped by visual
characteristics
DEEP LEARNING FOR IMAGES WITH PYTORCH
Regions and anchor boxes
Region: a smaller area of the image that could contain objects of interest, grouped by visual
characteristics
Anchor box: predefined bounding box templates of different sizes and shapes
DEEP LEARNING FOR IMAGES WITH PYTORCH
Faster R-CNN model
Faster R-CNN: an advanced version of R-CNN
Backbone (convolutional layers)
1 Edward Raff. 2022. Inside Deep Learning.
DEEP LEARNING FOR IMAGES WITH PYTORCH
Faster R-CNN model
Faster R-CNN: an advanced version of R-CNN
Backbone (convolutional layers)
Region proposal network (RPN) for bounding box proposals
1 Edward Raff. 2022. Inside Deep Learning.
DEEP LEARNING FOR IMAGES WITH PYTORCH
Faster R-CNN model
Faster R-CNN: an advanced version of R-CNN
Convolution layers (backbone): feature maps
Region proposal network (RPN): bounding box proposals
Classifier and regressor to produce predictions
1 Edward Raff. 2022. Inside Deep Learning.
DEEP LEARNING FOR IMAGES WITH PYTORCH
Region proposal network (RPN)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Region proposal network (RPN)
Anchor generator:
Generate a set of anchor boxes of different sizes and aspect ratios
DEEP LEARNING FOR IMAGES WITH PYTORCH
Region proposal network (RPN)
Anchor generator:
Generate a set of anchor boxes of different sizes and aspect ratios
Classifier and regressor:
Predict if the box contains an object and provide coordinates
DEEP LEARNING FOR IMAGES WITH PYTORCH
Region proposal network (RPN)
Anchor generator:
Generate a set of anchor boxes of different sizes and aspect ratios
Classifier and regressor:
Predict if the box contains an object and provide coordinates
Region of interest (RoI) pooling:
Resize the RPN proposal to a fixed size for fully connected layers
DEEP LEARNING FOR IMAGES WITH PYTORCH
RPN in PyTorch
from [Link] import AnchorGenerator
anchor_generator = AnchorGenerator(
sizes=((32, 64, 128),),
aspect_ratios=((0.5, 1.0, 2.0),),
)
from [Link] import MultiScaleRoIAlign
roi_pooler = MultiScaleRoIAlign(
featmap_names=["0"],
output_size=7,
sampling_ratio=2,
)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Fast R-CNN loss functions
RPN classification loss: R-CNN classification loss:
region contains object or not multiple object classes
binary cross-entropy cross-entropy
rpn_cls_criterion = rcnn_cls_criterion =
[Link]() [Link]()
RPN box regression loss: R-CNN box regression loss:
bounding box coordinates bounding box coordinates
mean squared error mean squared error
rpn_reg_criterion = [Link]() rcnn_reg_criterion = [Link]()
DEEP LEARNING FOR IMAGES WITH PYTORCH
Faster R-CNN in PyTorch
from [Link] import FasterRCNN
backbone = [Link].mobilenet_v2(weights="DEFAULT").features
backbone.out_channels = 1280
model = FasterRCNN(
backbone=backbone,
num_classes=num_classes,
rpn_anchor_generator=anchor_generator,
box_roi_pool=roi_pooler,
)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Faster R-CNN in PyTorch
Load pre-trained Faster R-CNN
from [Link].faster_rcnn import FastRCNNPredictor
model = [Link].fasterrcnn_resnet50_fpn(weights="DEFAULT")
Define number of classes and classifier input sise
num_classes = 2
in_features = model.roi_heads.box_predictor.cls_score.in_features
Replace model's classifier with a one with the desired number of classes
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Let's practice!
DEEP LEARNING FOR IMAGES WITH PYTORCH