Unit – 3

Object Detection using Machine Learning

Topic 1: Object Detection and Object Detection Methods

Object detection is a critical task in computer vision that involves identifying and
localizing objects within an image or a video. Unlike image classification, which
assigns a single label to an entire image, object detection predicts both the class labels
and the spatial locations (bounding boxes) of multiple objects in an image.

Introduction to Object Detection

1. Object detection aims to recognize and locate objects of interest within an

image.
2. It is a combination of two tasks: classification (what the object is) and
localization (where the object is).
3. Object detection algorithms output a bounding box with the object label and
a confidence score.
4. It is used in various applications, such as autonomous driving, security
systems, medical imaging, and video analysis.
5. The complexity of object detection arises due to challenges like varying
object sizes, shapes, orientations, occlusions, and background clutter.

Key Concepts in Object Detection

Bounding Boxes

• Bounding boxes are rectangular boxes drawn around the detected objects.
• A bounding box is represented by its coordinates:(𝑥min,𝑦min,𝑥max,𝑦max) or
by its center coordinates, width, and height.

Intersection over Union (IoU)

• IoU measures the overlap between the predicted bounding box and the
ground truth box.
• It is calculated as:

IoU=Area of Overlap / Area of Union

• IoU is used to evaluate the quality of predictions in object detection
models.
Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 Confidence Scores

• Each detected object is assigned a confidence score, which indicates the

model's certainty about the presence of the object in the bounding box.
• A higher confidence score represents a more reliable detection.

Non-Maximum Suppression (NMS)

• NMS is a post-processing step that eliminates overlapping bounding

boxes for the same object.
• It retains the box with the highest confidence score and suppresses the rest.

Challenges in Object Detection

• Occlusion: Objects are partially blocked or obscured.

• Scale Variations: Objects appear in varying sizes.
• Deformation: Objects can have different shapes and poses.
• Background Clutter: Complex backgrounds make detection difficult.
• Speed vs. Accuracy Trade-off: Models need to balance between fast
inference and precise detection.

Object Detection Methods

Object detection methods can be broadly categorized into traditional methods

and deep learning-based methods.

Traditional Object Detection Methods

Before deep learning became dominant, traditional object detection methods
relied on handcrafted features and machine learning models.
Feature-Based Methods
1. Haar Cascades:
• Haar Cascade detects objects in an image using Haar-like features and a
series of classifiers organized in a cascade structure.
• It uses a sliding window approach to scan the image at different scales and
positions, checking for the presence of objects.
• The algorithm is trained using positive (contains object) and negative
(does not contain object) images to classify regions.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 • Haar-like features are simple patterns that resemble edges, lines, and
rectangles.
• They calculate the difference in intensity between adjacent rectangular
regions within the detection window.
• Common Haar-like features:
o Edge Features: Detect vertical or horizontal intensity changes.
o Line Features: Detect lines in regions.
o Rectangle Features: Detect differences between adjacent
rectangular areas.
• Each feature is calculated by subtracting the sum of pixel intensities in
one region from the other.
• To compute Haar-like features efficiently, the algorithm uses an integral
image.
• The integral image allows rapid computation of the sum of pixel
intensities in a rectangular region.

• A cascade classifier is a series of stages where each stage is a weak

classifier trained to identify the object.
• The cascade structure eliminates regions that are unlikely to contain the
object early, reducing computation.
o If a region fails a stage, it is immediately discarded.
o If it passes all stages, it is classified as containing the object.
• This hierarchical approach ensures both efficiency and accuracy.

How Haar Cascade Works

Training Phase:

o Positive and negative samples are used to train the classifier.

o Haar-like features are extracted and evaluated using AdaBoost to create a
strong classifier.
o A cascade structure is built with multiple stages of classifiers.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 Detection Phase:

o The image is scanned using a sliding window approach.

o The classifier evaluates each window for the presence of the object.
o Only regions passing all stages of the cascade are classified as containing
the object.
o The process is repeated at multiple scales to detect objects of different
sizes.

Steps in Haar Cascade Detection

1. Preprocessing:
o Convert the image to grayscale (Haar Cascade works with
grayscale images).
2. Sliding Window:
o Slide a fixed-size window over the image.
3. Feature Calculation:
o Compute Haar-like features for the region in the window using
the integral image.
4. Classification:
o Pass the computed features through the cascade classifier.
o Regions failing any stage are discarded immediately.
5. Scaling:
o Repeat the process for different scales of the image to detect
objects of various sizes.
6. Output:
o Return all regions where the object is detected.

Histogram of Oriented Gradients

Histogram of Oriented Gradients (HOG) is a feature descriptor widely used in

computer vision and image processing tasks, such as object detection and recognition.
It captures edge and gradient structure, which is robust to variations in illumination and
small deformations.

• HOG is used to describe the local appearance of objects in an image by

encoding gradient directions (orientations).
• It is commonly applied in detecting objects like humans, cars, or animals in
images.

Process

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 • Divide the image into small regions called cells.
• Compute the gradient direction and magnitude for each pixel in a cell.
• Create a histogram of gradient orientations for each cell to represent its
features.

Steps in HoG

1. Gradients

2. Cells
1. A cell is a small, fixed-size block of pixels (e.g., 8×88 ).
2. Gradients are calculated for each pixel in the cell, and a histogram of
gradient orientations is created.

3. Histogram of Gradients

1. The gradient orientations are divided into bins (e.g., 9 bins for 0o−180o).
2. Each pixel in the cell votes for a histogram bin based on its gradient
orientation.
3. The vote is weighted by the pixel's gradient magnitude.
4. Blocks
1. A block is a group of neighboring cells (e.g., 2×22 cells).
2. The histograms of the cells within the block are concatenated to form a
feature vector.
Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 3. Normalization is applied to the block's feature vector to improve robustness
to lighting variations.

5. Feature Descriptor
1. The final HOG descriptor is obtained by concatenating the normalized
histograms of all blocks in the image.
2. This descriptor is used as an input for object detection models.

Region-Based Methods

R-CNN (Region-based CNN):

• Extracts ~2000 region proposals using selective search.

• Applies a CNN to each proposal for classification.
• Slow due to separate steps for region proposal and classification.

Fast R-CNN:

• Combines region proposal and classification into a single network.

• Uses region of interest (RoI) pooling to improve speed.

Faster R-CNN:

• Introduces a Region Proposal Network (RPN) for generating proposals.

• Significantly faster and more accurate than R-CNN and Fast R-CNN.

Mask R-CNN: This network extends Faster R-CNN by adding the prediction of
segmentation masks on each region of interest.

YOLO: You Only Look Once architecture. It proposes a single Neural Network to
predict bounding boxes and class probabilities from an image in a single evaluation.

SSD: Single Shot MultiBox Detector. It presents a model to predict objects in

images using a single deep Neural Network

Evaluation Metrics in Object Detection

Precision and Recall:

• Precision: Fraction of true positives among all positive predictions.

• Recall: Fraction of true positives among all actual positives.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 Average Precision (AP):

• Measures the precision-recall tradeoff for a single class.

Mean Average Precision (MAP):

• Average of AP values across all classes.

• Commonly used to evaluate object detection models.

Applications of Object Detection

1. Autonomous Driving:
o Detects vehicles, pedestrians, and traffic signs.
2. Security Systems:
o Identifies intruders and suspicious objects.

3. Healthcare:
o Detects tumors and anomalies in medical images.
4. Retail:
o Analyzes shelf inventory and customer behavior.
5. Sports Analytics:
o Tracks players and objects during games.

Object Classification Vs. Object Localization Vs. Object Detection

Feature Object Classification Object Localization Object Detection
Identify the class and
Identify the classes and
Identify the class of location of a single
Objective locations of multiple
object(s) in the image. dominant object in the
objects in the image.
image.
Labels and bounding
A single label or a set of A label and a bounding box
Output boxes for all detected
labels for the image. for the object.
objects.
"What objects are in the
What It "What is in the image, and
"What is in the image?" image, and where are
Answers where is it?"
they?"
Focus on
No Yes (single object) Yes (multiple objects)
Location
Assumes one or Detects multiple objects
Number of Assumes a single dominant
multiple objects but no with varying sizes and
Objects object in the image.
localization. locations.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 Feature Object Classification Object Localization Object Detection
- Detect cars and
Example - Is there a cat in this - Is there a car in this image,
pedestrians with their
Task image? and where is it located?
locations.
- Car: [x_min, y_min,
Output - Car: [x_min, y_min, x_max, y_max]
- Cat x_max, y_max]
Format - Person: [x_min,
y_min, x_max, y_max]
- Object tracking in single- - Autonomous driving
- Scene classification
Use Cases object videos - Surveillance
- Image search engines
- Robotics - Retail analytics
- ResNet - Faster R-CNN (single - Faster R-CNN
Example
- AlexNet object) - YOLO
Algorithms
- VGG - YOLO (single object) - SSD

Use cases of Object Detection

Deep Learning has expanded many capabilities across domains and

organizations. Object detection is a key one and is a very powerful solution which is
making huge ripples in our business and personal world. The major use cases of object
detection are

1. Object Detection is the key intelligence behind autonomous driving

technology. It allows the users to detect the cars, pedestrians, the background,
motorbikes, and so on to improve road safety.

2. We can detect objects in the hands of people, and the solution can be used for
security and monitoring purposes. Surveillance systems can be made much more
intelligent and accurate. Crowd control systems can be made more sophisticated, and
the reaction time will be reduced.

3. A solution might be used for detecting objects in a shopping basket, and it can
be used by the retailers for the automated transactions. This will speed up the overall
process with less manual intervention.

4. Object Detection is also used in testing of mechanical systems and on

manufacturing lines. We can detect objects present on the products which might be
contaminating the product quality.

5. In the medical world, the identification of diseases by analyzing the images of

a body part will help in faster treatment of the diseases.
Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Topic 2: Deep Learning Frameworks for object detection
Topics Covered are
o Sliding window approach for Object Detection
o Bounding box approach
o Intersection over Union (IoU)
o Non-max suppression
o Anchor boxes concept
Sliding window approach for Object Detection
The sliding window approach is one of the traditional techniques used for object detection.
It involves systematically scanning an image at multiple locations and scales using a fixed-
size window to detect objects.
This method was widely used before the advent of deep learning-based object detection
models like YOLO or Faster R-CNN.
How Sliding Window Works
Divide the Image into Windows
A fixed-size window (e.g., 64×64) is slid across the image horizontally and
vertically.
Each window represents a subregion of the image to be analyzed.
Feature Extraction
For each window, features are extracted using methods such as Histogram of
Oriented Gradients (HOG), Haar-like features, or other feature descriptors.
Classification
A pre-trained classifier (e.g., Support Vector Machine, Random Forest) is applied
to determine whether the window contains an object of interest.
Multi-Scale Detection
To detect objects of varying sizes, the image is resized (scaled down) multiple
times, and the sliding window is applied at each scale
Post-Processing
Overlapping windows that detect the same object are refined using Non-Maximum
Suppression (NMS) to retain the window with the highest confidence score

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Advantages
Simplicity:
Easy to implement and understand.
Applicability:
Works well for simple object detection tasks with objects of uniform sizes.
Feature Flexibility:
Compatible with a variety of feature extraction and classification techniques.
Disadvantages
Computationally Expensive:
Requires evaluating all possible windows across multiple scales, leading to high
computational cost.
Fixed Window Size:
May fail to detect objects of irregular shapes or sizes.
Redundant Computations:
Overlapping windows result in repetitive calculations.
Not Robust:
Struggles in cluttered scenes with varying object appearances and backgrounds.
Slow for Real-Time Applications:
The exhaustive nature of the sliding window makes it unsuitable for real-time
systems.
Comparison with modern techniques

Deep Learning-Based Models (e.g.,

Feature Sliding Window
YOLO)
Exhaustive and slow due to Fast and optimized using end-to-end
Computation
window scans architectures
Handcrafted (e.g., HOG, Haar
Feature Extraction Automatic feature learning (CNNs)
features)
Object Size Handles objects of all sizes in a single
Uses multi-scale approach
Handling pass
More robust due to hierarchical feature
Robustness Sensitive to noise and occlusion
learning

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Bounding box approach
The Bounding Box Approach is a fundamental technique in object detection, where objects
in an image are localized and represented using rectangular boxes.
These boxes, known as bounding boxes, are defined by their coordinates and are used to
mark the position and size of the detected object.
Purpose:
To localize objects within an image.
To provide a spatial representation for object detection models.
Use Cases:
Object detection and tracking.
Region proposals for further classification.
Localization tasks in robotics, healthcare, and autonomous vehicles.
Key Components of Bounding Box Approach
Bounding Box Representation
Bounding boxes are typically represented by:
Coordinates:
Two-Point Representation: Top-left corner:(𝑥min,𝑦min)
Bottom-right corner:(𝑥max,𝑦max)
Center and Dimensions: Center:(𝑥𝑐,𝑦𝑐)
Width (𝑤) and Height (h).
These representations allow easy computation of overlap, scaling, and adjustments.
Object Localization
• Object detection models predict bounding boxes for all detected objects in an
image.
• Each bounding box is associated with:
A class label (e.g., "Car", "Dog").
A confidence score indicating the likelihood that the box contains the object.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Types and its comparisons

Type Best Fit Complexity Applications

Axis-Aligned Bounding Rectangular or simple
Low General object detection
Box shapes
Rotated Bounding Box Tilted or rotated objects Moderate Aerial imagery, text detection
Minimum Bounding
Irregular shapes Moderate GIS, satellite data
Rectangle
Multi-scale object
Anchor Boxes Moderate YOLO, SSD, Faster R-CNN
detection
Precisely bounded Instance segmentation, medical
Tight Bounding Box Low
objects imaging
Complex or irregular Instance segmentation, map
Bounding Polygons High
shapes annotations
3D Bounding Box 3D objects High Autonomous driving, robotics

Applications of Bounding Box Approach

1. Object Detection:
o Localize and classify objects in images.
2. Object Tracking:
o Track moving objects in video sequences using bounding boxes.
3. Instance Segmentation:
o Bounding boxes are used to initialize more detailed segmentation tasks.
4. Autonomous Vehicles:
o Detect pedestrians, vehicles, and obstacles using bounding boxes.
5. Healthcare:
o Localize tumors or anomalies in medical imaging.

Advantages of Bounding Box Approach

1. Simple Representation:
o Easy to interpret and compute.
2. Efficient for Localization:
o Provides a quick and reliable way to localize objects in images.
3. Compatible with Models:
o Works well with various object detection models.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Limitations of Bounding Box Approach

1. Precision Issues:
o Bounding boxes may not perfectly align with object boundaries (e.g., irregular
shapes).
2. Overlap Ambiguities:
o When objects overlap, bounding boxes may include multiple objects.
3. Not Shape-Adaptive:
o Bounding boxes do not adapt to non-rectangular objects.

Intersection over Union (IoU)

Intersection over Union (IoU) is a crucial metric used in object detection tasks to evaluate
how well the predicted bounding box aligns with the ground truth bounding box.
It measures the degree of overlap between two bounding boxes and is widely used for
assessing the accuracy of object detection models.
Definition:
IoU quantifies the overlap between the predicted bounding box and the ground truth
bounding box. It is defined as the ratio of the area of overlap to the area of the union of the
two boxes.
Formula: IoU = Area of Overlap / Area of Union
Purpose:
To evaluate how closely a predicted box matches the ground truth box.
Higher IoU values indicate better localization accuracy
IoU Calculation Steps
Step 1: Identify the Predicted and Ground Truth Boxes
Let 𝐵𝑝 represent the predicted bounding box.
Let 𝐵𝑔 represent the ground truth bounding box.
Step 2: Compute the Area of Overlap
Determine the coordinates of the overlapping region between 𝐵𝑝 and 𝐵𝑔
Overlap’s top-left corner:(𝑥min,𝑦min)
Overlap’s bottom-right corner:(𝑥max,𝑦max)
Compute the width (𝑤) and height (ℎ) of the overlapping region:
𝑤=max(0,𝑥max−𝑥min)

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 ℎ=max(0,𝑦max−𝑦min)
Calculate the area of overlap:
Area of Overlap=𝑤×ℎ
Step 3: Compute the Area of Union
Compute the areas of the individual boxes:
Area of 𝐵𝑝=𝑤𝑝×ℎ𝑝
Area of 𝐵𝑔=𝑤𝑔×ℎ𝑔
Calculate the area of union:
Area of Union=Area of 𝐵𝑝+Area of 𝐵𝑔−Area of Overlap
Step 4: Compute IoU
Divide the area of overlap by the area of union:
IoU=Area of Overlap / Area of Union
IoU Solved Example

Solution

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Result: IoU = 0.1429 (low overlap).

Threshold Value:

• IoU thresholds determine whether a predicted bounding box is considered correct.

• Common thresholds:
o 0.5: The predicted box is considered a True Positive (TP) if IoU>0.5
o Higher thresholds (e.g., 0.7) are used for stricter evaluation.

Classification Based on IoU:

• IoU > Threshold: True Positive (TP).

• IoU ≤ Threshold: False Positive (FP).

Variants of IoU

Generalized IoU (GIoU)

Addresses cases where predicted and ground truth boxes do not overlap.

Adds a penalty for the distance between the boxes.

Distance IoU (DIoU)

Incorporates the distance between the centers of the predicted and ground truth
boxes.

Complete IoU (CIoU)

Combines GIoU and DIoU and considers the aspect ratio of bounding boxes.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Advantages of IoU

1. Simplicity:
o Easy to compute and interpret.
2. Standardized Metric:
o Widely adopted in object detection tasks for benchmarking.

Limitations of IoU

1. Insensitivity to Alignment:
o IoU may not penalize poorly aligned boxes with small overlaps sufficiently.
2. No Overlap Issue:
o When there is no overlap, IoU = 0, providing no gradient for optimization during
training.
3. Sensitivity to Size:
o IoU tends to penalize small objects more heavily compared to large objects.

Non-Maximum Suppression (NMS)

Non-Maximum Suppression (NMS) is a post-processing technique used in object detection

to eliminate redundant bounding boxes that predict the same object.

It ensures that only the most accurate bounding box is retained for each detected object,
improving detection results and reducing overlap.

Anchor boxes

Anchor boxes are predefined bounding boxes of specific sizes and aspect ratios used in
object detection models to detect objects of varying scales and shapes efficiently.

They serve as reference boxes that the model adjusts to predict the location of objects more
accurately.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Topic 3: Various Deep Learning architectures
Topics Covered
1. R-CNN (About, Salient Features, Architecture, Code, Advantages, Disadvantages and
Usecases
2. Fast R-CNN (About, Salient Features, Architecture, Code, Advantages, Disadvantages
and Usecases
3. YOLO (About, Salient Features, Architecture, Code, Advantages, Disadvantages and
Usecases
4. Comparison of all the above three architectures.
R-CNN
R-CNN, or Region-based Convolutional Neural Network, is a foundational algorithm in the field
of object detection.
It was introduced by Ross Girshick in 2014, providing a significant leap in accuracy compared to
traditional methods.
The primary innovation of R-CNN lies in combining region proposals with convolutional neural
networks (CNNs), enabling more accurate object localization and classification.
R-CNN operates by first generating approximately 2000 region proposals using algorithms like
Selective Search.
These proposals are regions in an image that are likely to contain objects.
Each region is then resized to a fixed size and passed through a pre-trained CNN to extract feature
maps.
This step leverages the CNN’s ability to learn hierarchical and spatial features, which are essential
for accurate object detection.
Once the features are extracted, a classifier, typically a Support Vector Machine (SVM), is used to
categorize the object in each proposal into predefined classes or as background (no object).
Simultaneously, a bounding box regressor adjusts the coordinates of the proposed region to better
fit the detected object.
Despite its high accuracy, R-CNN has several limitations.
Its pipeline involves multiple steps, including feature extraction, classification, and bounding box
regression, which are computationally expensive. It takes 40–50 seconds to make a prediction for an image,
and hence it becomes a problem for huge datasets
Furthermore, processing thousands of region proposals individually makes the algorithm slow,
especially for real-time applications.
The selective search algorithm is fixed, and not much improvements can be made.
Nevertheless, R-CNN laid the foundation for its successors, such as Fast R-CNN and Faster R-
CNN, which address its inefficiencies.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 The process in R-CNN
R-CNN Architecture
The architecture of R-CNN can be divided into three main components:
Region Proposal Generation:
R-CNN relies on Selective Search to generate region proposals. These proposals are
candidate regions that potentially contain objects.
Approximately 2000 proposals are generated per image.
Feature Extraction:
Each region proposal is resized to a fixed size (e.g., 224×224) to match the input
requirements of the CNN.
A pre-trained CNN processes the resized regions to extract feature maps, which serve as
input for classification and localization tasks.
Classification and Bounding Box Regression:
Features extracted from each region proposal are passed to an SVM for classification into
object categories or background.
A bounding box regressor further refines the coordinates of the bounding boxes to better
align with the objects.
This architecture operates as a multi-step pipeline, where each module is trained independently.
While it achieves high accuracy, its reliance on processing each region proposal individually makes it
computationally intensive and unsuitable for real-time applications.

Steps to Implement R-CNN

1. Input Image:
o Load and preprocess the input image (e.g., resize, normalize).
2. Region Proposal Generation:

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 o Use a region proposal algorithm like Selective Search to generate candidate regions
likely to contain objects.
3. Feature Extraction:
o Resize each region proposal to a fixed size (e.g., 224×224224 \times 224224×224).
o Pass the resized regions through a pre-trained CNN to extract feature maps.
4. Classification:
o Use the extracted features to classify each region proposal into object categories or
as background using SVMs.
5. Bounding Box Regression:
o Use a regression model to refine the bounding box coordinates for better
localization.
6. Post-Processing:
o Apply non-maximum suppression (NMS) to remove redundant bounding boxes and
keep the most confident predictions.

R-CNN Code
import cv2
import numpy as np
from [Link] import VGG16
from [Link] import SVC

# Load pre-trained CNN (e.g., VGG16)

cnn = VGG16(weights='imagenet', include_top=False)

# Step 1: Generate Region Proposals

image = [Link]('[Link]')
selective_search = [Link]()
selective_search.setBaseImage(image)
selective_search.switchToSelectiveSearchFast()
rects = selective_search.process()

# Step 2: Extract Features for Each Proposal

features = []
labels = [] # Ground truth labels for proposals
Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 for rect in rects:
x, y, w, h = rect
region = [Link](image[y:y+h, x:x+w], (224, 224))
region = np.expand_dims(region, axis=0)
feature = [Link](region)
[Link]([Link]())
# Assume labels are manually assigned or extracted
[Link](assign_label_to_region(rect))

# Step 3: Train SVM for Classification

svm = SVC(kernel='linear')
[Link](features, labels)
Advantages:
High Accuracy:
Combines region proposals and CNNs, leading to better object detection results.
Effective Localization:
Bounding box regression improves object localization.
Flexible Framework:
Can handle multiple object classes and complex scenes.
Disadvantages:
Slow:
Processing each region proposal individually makes it computationally expensive.
Storage-Intensive:
Storing extracted features for thousands of proposals requires significant disk
space.
Complex Pipeline:
Requires separate training for CNN, SVM, and bounding box regression.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Use Cases:
Object Detection:
Detect vehicles, animals, or objects in images.
Medical Imaging:
Detect tumors or anomalies in X-rays or MRI scans.
Image Segmentation:
Serve as a base for more complex tasks like instance segmentation.
Fast R-CNN
Fast R-CNN is an improved version of the original R-CNN (Region-based Convolutional
Neural Network) designed to address its inefficiencies, particularly in speed and computational
cost.
Introduced by Ross Girshick in 2015, Fast R-CNN optimizes the object detection pipeline
by integrating feature extraction, region proposal classification, and bounding box regression into
a single network.
The key innovation of Fast R-CNN is the use of a single convolutional forward pass to
extract features for the entire image, rather than processing each region proposal individually as in
R-CNN.
This significantly reduces redundancy in feature computation. It employs a Region of
Interest (RoI) Pooling layer, which extracts fixed-size feature maps for each region proposal from
a shared feature map of the entire image.
The architecture comprises a convolutional neural network (e.g., VGG or ResNet) for
feature extraction, followed by an RoI pooling layer.
The extracted RoI features are fed into fully connected layers to predict object classes and
refine bounding box coordinates.
Fast R-CNN also integrates multi-task learning, optimizing for classification and bounding
box regression simultaneously.
It reduces storage requirements by removing the need to save intermediate feature maps
and eliminates the need for external SVM classifiers.
While it offers significant speed improvements over R-CNN, its reliance on external region
proposal methods (e.g., Selective Search) still makes it slower compared to Faster R-CNN, which
incorporates region proposal networks.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Architecture of Fast R-CNN

The architecture of Fast R-CNN is designed for efficiency and consists of the following
components:

1. Input:
o An input image and a set of region proposals (e.g., from Selective Search).
2. Shared Convolutional Layers:
o A convolutional neural network (e.g., VGG16 or ResNet) processes the entire
image to generate a single shared feature map. This eliminates redundant feature
computation.
3. Region of Interest (RoI) Pooling:
o Each region proposal is mapped onto the shared feature map.
o The RoI pooling layer extracts fixed-size feature maps (e.g., 7×77 \times 77×7) for
each proposal, regardless of its size.
4. Fully Connected Layers:
o The pooled features are passed through fully connected layers.
5. Output Layers:
o Two parallel outputs:
1. Softmax Classifier: Predicts object classes (or background).
2. Bounding Box Regressor: Refines the coordinates of the bounding boxes.

Fast R-CNN optimizes these components in a single forward-backward pass, making it

faster and more efficient than R-CNN while maintaining high accuracy.

Architecture of Fast R-CNN

Steps to Implement Fast R-CNN

1. Input Preprocessing:
o Load the input image and preprocess it (resize, normalize).
Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 o Generate region proposals using Selective Search or another method.
2. Feature Extraction:
o Use a CNN to extract a shared feature map for the entire image.
3. Region Proposal Mapping:
o Map each region proposal onto the shared feature map using the RoI pooling
layer.
4. Classification and Regression:
o Pass the RoI features through fully connected layers to predict object classes and
refine bounding boxes.
5. Training:
o Train the network using multi-task loss:
▪ Cross-entropy loss for classification.
▪ Smooth L1 loss for bounding box regression.

Fast R-CNN Code

import tensorflow as tf
from [Link] import layers, Model

# Define RoI Pooling Layer

class RoIPoolingLayer([Link]):
def __init__(self, pooled_height, pooled_width):
super().__init__()
self.pooled_height = pooled_height
self.pooled_width = pooled_width

def call(self, feature_map, rois):

# RoI pooling implementation (placeholder)
# Use TensorFlow operations to pool each RoI
pooled_rois = [] # Placeholder logic
return [Link](pooled_rois)

# Define Fast R-CNN Model

def build_fast_rcnn_model(num_classes):
# Base CNN (Feature Extraction)
Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 base_model = [Link].VGG16(include_top=False, input_shape=(224,
224, 3))
feature_extractor = Model(inputs=base_model.input, outputs=base_model.output)

# RoI Pooling Layer

roi_pooling = RoIPoolingLayer(7, 7)

# Fully Connected Layers

x = [Link]()(roi_pooling.output)
x = [Link](4096, activation='relu')(x)
x = [Link](4096, activation='relu')(x)

# Output Layers
classifier = [Link](num_classes, activation='softmax')(x)
bbox_regressor = [Link](4 * num_classes)(x) # 4 coordinates per class

# Define Model
return Model(inputs=[feature_extractor.input, roi_pooling.input], outputs=[classifier,
bbox_regressor])

# Example Usage
model = build_fast_rcnn_model(num_classes=21) # 20 classes + background
[Link]()

Advantages:
Efficiency:
Processes the entire image in a single forward pass, reducing redundancy.
End-to-End Training:
Optimizes classification and bounding box regression in one network.
Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 RoI Pooling:
Handles region proposals of varying sizes effectively.
Disadvantages:
Reliance on External Proposals:
Depends on methods like Selective Search, which are slow.
Limited Real-Time Performance:
Still slower than models like YOLO and SSD.
Use Cases:
Object Detection:
Identify objects in images for surveillance or autonomous vehicles.
Medical Imaging:
Detect tumors or abnormalities.
Wildlife Monitoring:
Detect and classify animals in camera trap images.

You Only Look Once (YOLO) (Features, Loss functions and Architecture)
YOLO, or You Only Look Once, is a state-of-the-art real-time object detection system
introduced by Joseph Redmon in 2016.
Unlike traditional object detection methods that use a two-stage process (region proposal
followed by classification), YOLO reframes object detection as a single regression problem.
It predicts both bounding box coordinates and object classes directly from the input image
in one evaluation, making it extremely fast and efficient.
YOLO divides the input image into an 𝑆×𝑆 grid, where each grid cell predicts bounding
boxes, confidence scores, and class probabilities.
A bounding box consists of its center coordinates, width, height, and a confidence score
that reflects the likelihood of the box containing an object.
Each grid cell predicts multiple bounding boxes, but only those with high confidence scores
are retained.
YOLO uses convolutional neural networks (CNNs) to extract features from the input
image.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 The unified architecture ensures that the entire image is processed in a single forward pass,
which is why it’s called "You Only Look Once" YOLO is not only faster than earlier models like
R-CNN and Fast R-CNN, but it also improves contextual understanding by considering the entire
image during detection.
While YOLO is highly efficient, it struggles with detecting small objects in complex or
crowded scenes due to its fixed grid structure.
Despite this, YOLO remains a popular choice for real-time applications like autonomous
vehicles, surveillance systems, and robotics, where speed and efficiency are critical.

YOLO Architecture

The YOLO architecture consists of the following components:

1. Input Layer:
o Takes an image of fixed size, e.g., 448×448.
2. Feature Extraction:
o Uses a CNN (e.g., Darknet) to extract spatial and semantic features.
3. Grid Division:
o Divides the image into an S×S grid (e.g., 7×7).
4. Bounding Box Prediction:
o Each grid cell predicts:
▪ BBB bounding boxes (e.g., B=2).
▪ A confidence score for each box.
▪ Class probabilities for the object in the grid cell.
5. Output Layer:
o Produces a tensor of size [S,S,B×(5+C)]], where:
▪ 5: Center coordinates, width, height, and confidence score.
▪ C: Number of object classes.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Steps to Implement YOLO

1. Preprocessing:
o Resize the input image to the required size (e.g., 448×448448 \times 448448×448).
o Normalize pixel values.
2. Feature Extraction:
o Pass the image through a CNN to extract spatial features.
3. Grid Division:
o Divide the image into an S×SS \times SS×S grid.
4. Bounding Box Prediction:
o Each grid cell predicts multiple bounding boxes with confidence scores and class
probabilities.
5. Non-Maximum Suppression (NMS):
o Remove overlapping bounding boxes with lower confidence scores.
6. Post-Processing:
o Convert predictions to final object detections with bounding boxes and labels.

Salient Features of YOLO

1. Single-Pass Detection:
o Processes the image in one pass, making it extremely fast.
2. Global Context:
o Considers the entire image for detection, improving contextual understanding.
3. Real-Time Performance:
o Capable of detecting objects at up to 45 frames per second (FPS).
4. Unified Model:
o Combines object detection and classification in one network.

Loss Functions in YOLO

YOLO uses a multi-part loss function comprising:
Localization Loss:
Penalizes errors in predicted bounding box coordinates (𝑥,𝑦,𝑤,ℎ).
Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 Confidence Loss:
Measures the difference between predicted and actual confidence scores for
bounding boxes.
Classification Loss:
Penalizes incorrect class predictions for objects in the grid cell.
No-Object Loss:
Reduces the impact of cells predicting no objects to avoid class imbalance.
The total loss is a weighted sum of these components:
Loss=𝜆coord⋅Localization Loss+Confidence Loss+Classification Loss+𝜆noobj⋅No-Object Loss

Basic YOLO-Code
import torch
import [Link] as nn

class YOLO([Link]):
def __init__(self, grid_size=7, num_boxes=2, num_classes=20):
super(YOLO, self).__init__()
self.grid_size = grid_size
self.num_boxes = num_boxes
self.num_classes = num_classes

# Backbone: Feature Extraction

[Link] = [Link](
nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3),
[Link](),
nn.MaxPool2d(kernel_size=2, stride=2),
# Add more layers as per Darknet architecture
)

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
 # Fully Connected Layers
[Link] = [Link](
[Link](1024 * grid_size * grid_size, 4096),
[Link](),
[Link](4096, grid_size * grid_size * (num_boxes * 5 + num_classes))
)

def forward(self, x):

x = [Link](x)
x = [Link]([Link](0), -1) # Flatten
x = [Link](x)
return [Link](-1, self.grid_size, self.grid_size, self.num_boxes * 5 +
self.num_classes)

# Example usage
model = YOLO(grid_size=7, num_boxes=2, num_classes=20)
print(model)

Advantages:

1. Speed:
o Real-time object detection at high FPS.
2. Simplicity:
o End-to-end training and prediction.
3. Global Context:
o Considers the entire image for better predictions.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Disadvantages:

1. Small Object Detection:

o Struggles with detecting small objects due to the grid-based structure.
2. Overlapping Objects:
o Struggles with heavily overlapping objects.

Use Cases:

1. Autonomous Vehicles:
o Real-time detection of pedestrians, vehicles, and traffic signs.
2. Surveillance Systems:
o Detect suspicious activities or intrusions.
3. Robotics:
o Object detection for grasping and navigation.

YOLO remains one of the most efficient and widely used object detection models in various
real-world applications.

Comparisons of all the architectures

Aspect R-CNN Fast R-CNN YOLO

Single-stage detection:
Two-stage detection: Two-stage detection
combines proposal,
1. Approach region proposal + but more efficient
classification, and
classification. with shared features.
localization.
Separate modules for
region proposal, feature Unified model with Unified end-to-end model
2. Architecture
extraction, and RoI pooling. without separate stages.
classification.
3. Feature Extracts features for each Shared feature map Extracts features globally
Extraction region individually. for the entire image. for the entire image.
Faster than R-CNN
Very slow (processes Extremely fast, real-time
4. Speed due to shared
proposals one by one). detection at high FPS.
computation.
5. Computational Highly efficient due to
High computational cost. Moderately efficient.
Efficiency single-pass processing.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Aspect R-CNN Fast R-CNN YOLO
Grid-based: divides the
6. Region Proposal Uses Selective Search for Also uses Selective
image into fixed cells for
Method region proposals. Search.
predictions.
High redundancy: Redundancy reduced No redundancy;
7. Redundancy in
processes overlapping with shared feature predictions are made
Computation
regions separately. map. globally.
Variable-sized inputs
8. Input Image resized to fixed Fixed-size input
Fixed-size input images.
Size dimensions for each images.
region.
Multi-stage training
End-to-end training
9. Training (CNN, SVM, and
for classification and Fully end-to-end training.
Complexity regressor trained
regression.
separately).
Region proposals, class Class labels, bounding
10. Detection Class labels and
labels, and bounding boxes, and confidence
Output bounding boxes.
boxes. scores.
Introduced RoI
11. Use of RoI
Not used. pooling for fixed Not used.
Pooling
feature sizes.
12. Localization High accuracy but slower High accuracy with Moderate accuracy due to
Accuracy bounding box refinement. faster refinement. fixed grid structure.
13. Real-Time Not suitable due to slow Highly suitable for real-
Partially suitable.
Suitability speed. time applications.
14. Small Object Weak performance due to
Moderate performance. Better than R-CNN.
Detection grid-based predictions.
15. Overlapping Handles overlapping Handles overlapping Struggles with heavily
Object Detection objects well. objects well. overlapping objects.
Relies on external region
16. Dependency on Relies on external No dependency on
proposal methods like
External Methods region proposals. external region proposals.
Selective Search.
Multiple steps: proposal,
17. Inference Single network with
feature extraction, Single-step detection.
Pipeline shared computation.
classification.
Flexible for multiple Flexible for multiple Flexible for multiple
18. Flexibility
object classes. object classes. object classes.
Post-processing with
19. IoU Post-processing with Post-processing with
Non-Maximum
Thresholding NMS. NMS.
Suppression (NMS).
Large due to separate Smaller size, optimized
20. Model Size Moderate size.
modules. for deployment.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech
Aspect R-CNN Fast R-CNN YOLO
Difficult to deploy due to Easier to deploy than Very easy to deploy due to
21. Deployment
complexity. R-CNN. single architecture.
Struggles in complex
22. Handling Better handling with
Performs well but slow. scenes with many small
Complex Scenes faster results.
objects.
23. Use of Integrates confidence Confidence scores directly
Not explicitly integrated.
Confidence Scores scores. influence predictions.
Multi-task loss
Requires separate loss Unified loss function for
combining
24. Loss Function functions for SVM and localization, classification,
classification and
regression. and confidence.
regression.
Widely used in real-time
Used in semi-real-
25. Popular Use Historical significance in applications like
time applications
Cases object detection research. surveillance and
needing accuracy.
autonomous driving.

Prepared By
Mr. N. Adhithyaa, AP([Link].), PSGiTech