Deep Learning

End-Term Examination Study Notes

Computer Vision: Object Detection & Segmentation

Topics Covered
1. Fully Convolutional Networks (FCN)
2. U-Net Architecture
3. Region Proposal Networks (RPN)
4. R-CNN (Region-based CNN)
5. R-FCN (Region-based FCN)
6. YOLO (You Only Look Once)
1. Fully Convolutional Network (FCN)
1.1 What is FCN?
A Fully Convolutional Network (FCN) is a type of Convolutional Neural Network (CNN) that makes a
prediction for every single pixel in an image, rather than one prediction for the whole image. Think
of it this way — a regular CNN tells you "this image is a cat." An FCN tells you "these specific pixels
are the cat, and those pixels are the background."

This makes FCN perfect for tasks like semantic segmentation, where you need to colour-code
every pixel of an image by its category (e.g., road, car, person, sky).

Key Idea: FCN was built by taking the VGG16 image classification network and converting it
into a pixel-wise predictor by replacing the fully connected layers with convolutional layers.

1.2 Architecture of FCN

FCN has three main components:

Part 1: Encoder (Downsampling Path)

The encoder is similar to the convolutional layers of a regular CNN. Its job is to extract features
from the image.
• Each convolution + pooling operation reduces the spatial resolution (makes the image
smaller).
• But at the same time, it increases the richness of features (captures more complex patterns).
• Example in VGG-16: The image goes from 224x224x3 (original) all the way down to 7x7x512
(tiny feature map with lots of info).

Simple Analogy: The encoder is like squeezing all the important information about an image
into a tiny summary. You lose the exact pixel positions, but you gain understanding of WHAT is
in the image.

Part 2: Decoder (Upsampling Path)

The decoder takes the tiny, information-rich feature map from the encoder and blows it back up to
the original image size.
• It uses deconvolution (also called transposed convolution) or upsampling + convolution.
• Goal: The network learns WHERE each object is located in the image.
Part 3: Skip Connections
Here is the clever part. When we compress the image in the encoder, we lose fine spatial detail
(exact pixel locations, edges, boundaries). Skip connections fix this by taking feature maps from
earlier (higher-resolution) encoder layers and combining them with the decoder.
• They bridge the encoder and decoder to pass spatial details forward.
• The decoder combines coarse (semantic) information from deep layers with fine (spatial)
information from shallow layers.
• Result: Much sharper and more accurate segmentation maps.

1.3 FCN Variants

There are three main variants of FCN, each using more skip connections than the last:

Variant Skip Connections Upsampling Output Quality

Used Factor
FCN-32s None (only deepest 32x Very coarse / blurry
layer)
FCN-16s Pool4 layer added 16x Better, more detail
FCN-8s Pool3 + Pool4 layers 8x Best, finest detail
added

FCN-8s is the best variant because it uses the most skip connections, recovering the most spatial
detail during upsampling.

1.4 Why Skip Connections Matter

Without skip connections (FCN-32s), the network upsamples a very small 7x7 feature map all the
way to 224x224 in one go. This is like trying to draw a detailed city map from a tiny blurry thumbnail
— the result is rough and imprecise.

With skip connections (FCN-8s), the decoder at each upsampling step also receives higher-
resolution feature maps from the encoder. It element-wise sums these maps together, combining
semantic context with spatial precision.

Exam Tip: Remember FCN-8s uses Pool3 + Pool4 skip connections, FCN-16s uses only Pool4,
and FCN-32s uses none. More skip connections = finer output.
2. U-Net
2.1 What is U-Net?
U-Net is a Fully Convolutional Network with a U-shaped architecture. It was originally designed for
biomedical image segmentation (e.g., identifying cells or tumours in medical scans), but it is now
widely used in many image segmentation tasks.

The name "U-Net" comes from its shape — if you draw the architecture, it looks like the letter U.
The left side compresses the image (encoder), and the right side expands it back (decoder), with
connections bridging both sides.

2.2 U-Net Architecture Components

1. Contracting Path (Encoder)
This is the left side of the U. It progressively reduces the size of the image while extracting features.
• Uses convolution operations followed by ReLU activation to add non-linearity (helps the
model learn complex patterns).
• Uses Max Pooling to reduce spatial size (downsampling).
• As spatial size decreases, the number of feature channels increases.
• Objective: Extract rich feature information while reducing spatial dimensions.

ReLU Reminder: ReLU (Rectified Linear Unit) is an activation function that outputs the input if
it's positive, and 0 otherwise. It introduces non-linearity so the network can learn complex, non-
linear patterns.

2. Bottleneck
This is the very bottom of the U — the narrowest point. At this stage, the image is at its smallest
spatial size. However, it has the highest number of feature channels and captures the most
abstract, meaningful features of the image. It is like a compressed summary of everything important
in the input.

3. Expansive Path (Decoder)

This is the right side of the U. It progressively upsamples the feature maps back to the original
image size.
• Uses Up-Convolution (transposed convolution) to increase spatial size.
• At each step, it concatenates (not just adds) the upsampled features with the corresponding
feature map from the encoder via skip connections.
• This concatenation is a key difference from FCN — U-Net concatenates, while FCN does
element-wise summation.
4. Skip Connections in U-Net
U-Net's skip connections directly connect each encoder layer to its mirror decoder layer. These
connections:
• Preserve spatial information that gets lost during downsampling.
• Help the decoder precisely locate boundaries and edges.
• Are especially useful for medical image segmentation where precise boundaries matter
enormously.

5. Final Output Layer

A 1x1 convolution at the very end converts the feature maps into the final segmentation map, where
each pixel is classified into a specific class (e.g., foreground or background, tumour or healthy
tissue). The output has the same spatial resolution as the input image.

2.3 How U-Net Works — Step by Step

1. Input Image: A grayscale or colour image is fed into the network.
2. Feature Extraction (Encoder): Convolutions + pooling progressively extract deeper features
and reduce spatial size.
3. Bottleneck: The smallest, most abstract feature representation is computed.
4. Reconstruction (Decoder): Up-convolutions + skip connections reconstruct the full spatial
resolution step by step.
5. Final Prediction: A 1x1 convolution classifies each pixel. Output = same size as input.

2.4 FCN vs U-Net — Key Differences

Feature FCN U-Net
Purpose General semantic Biomedical / precise
segmentation segmentation
Skip Connection Element-wise addition Concatenation
Type
Architecture shape Encoder + Decoder U-shaped (symmetric)
Base Network Modified VGG16 Custom symmetric
design
Output Pixel-wise class map Pixel-wise class map
3. Region Proposal Network (RPN)
3.1 What is RPN?
A Region Proposal Network (RPN) is a fully convolutional network that looks at a feature map (from
a CNN backbone) and proposes regions in the image that are likely to contain objects. It does NOT
classify what the object is — it only says: "Hey, there is something here (foreground) or there isn't
(background)."

Key Point: RPN answers WHERE (foreground vs background), not WHAT (object class). The
object classification is done later by a network like Fast R-CNN.

3.2 How RPN Works — Step by Step

Step 1: Feature Map from CNN Backbone
The input image is first passed through a CNN backbone (e.g., VGG, ResNet). The output is a rich
feature map. The RPN operates on this feature map, not on the raw image directly.

Step 2: Sliding Window Over the Feature Map

A small 3x3 sliding window moves across every position on the feature map. At each position, it
looks at the features and makes predictions about whether there is an object there and where it
might be located.

Step 3: Anchors — Multiple Boxes per Location

At each sliding window position, the RPN doesn't just predict one box. Instead, it uses anchors — a
set of fixed-size, pre-defined reference boxes with different scales and aspect ratios.
• Think of anchors as "template guesses" for object shapes at each location.
• Different scales: small objects, medium objects, large objects.
• Different aspect ratios: tall (portrait), wide (landscape), square.
• Typical setup: 3 scales x 3 aspect ratios = 9 anchors per location.

Analogy: Imagine placing 9 different-sized rectangles at every spot on the image. For each
rectangle, you ask: "Does this rectangle contain an object? And if so, how should I adjust it to fit
better?"

Step 4: Two Outputs per Anchor

For each anchor, the RPN predicts:
 • Objectness Score: The probability that this anchor contains an object (vs. background). This
is a 2-class classification (object / not object).
• Bounding Box Regression: Four adjustment values (delta x, delta y, delta width, delta height)
to fine-tune the anchor's position and size to better match the actual object.

Step 5: Filtering with NMS

The RPN can generate thousands of proposals (many anchors x many locations). Most of these
overlap or are irrelevant. Two steps filter them down:
• Non-Maximum Suppression (NMS): Removes duplicate/overlapping boxes, keeping only the
best one in each cluster.
• Top-N selection: Keeps only the top N proposals (e.g., top 2000) with the highest objectness
scores.

Step 6: Output — Region of Interest (RoIs)

The final output of the RPN is a set of Region of Interest (RoI) bounding boxes. These are the
proposed object locations that get passed to the next stage (like Fast R-CNN) for actual
classification.

3.3 Key Terms in RPN

Anchor: A fixed-size reference bounding box placed at each location on the feature map.
Objectness Score: A score (0 to 1) indicating how likely a region contains an object.
Bounding Box Regression: Fine-tuning the anchor coordinates to better fit the object.
NMS: Non-Maximum Suppression — removes overlapping boxes, keeps the best one.
RoI: Region of Interest — a proposed bounding box likely containing an object.

4. R-CNN (Region-based Convolutional Neural Network)

4.1 What is R-CNN?
R-CNN combines region proposals (possible object locations) with CNNs to detect and classify
objects in images. It performs two tasks: localization (where is the object?) and classification (what
is the object?).

R-CNN was one of the first deep learning-based object detection methods, and although it was
slow, it set the foundation for much faster methods like Fast R-CNN and Faster R-CNN.
4.2 R-CNN Pipeline — Step by Step
Step 1: Input Image + Region Proposals
R-CNN starts with a single input image. It then uses Selective Search (a traditional computer vision
technique) to generate around 2,000 region proposals — bounding boxes that might contain
objects. Most will be background, but some will contain actual objects.

Step 2: Feature Extraction (CNN runs 2000 times!)

Each of the 2,000 proposed regions is cropped from the image and resized to a fixed size (e.g.,
227x227 pixels). Each resized region is then passed through a CNN (like AlexNet or VGG) to
extract a feature vector.

Performance Issue: Because the CNN runs once for each of the 2,000 regions, R-CNN is very
slow. This is its biggest weakness and why later versions (Fast R-CNN, Faster R-CNN) were
developed.

Step 3: Classification using SVM

The CNN feature vector for each region is fed into a Support Vector Machine (SVM) classifier. Each
SVM determines which object class the region belongs to (car, dog, person, background, etc.). A
separate SVM is trained for each class.

Step 4: Bounding Box Regression

For each classified region, a separate linear regression model is trained to refine the bounding box
coordinates. This step adjusts the initially proposed bounding box to better fit the actual boundaries
of the detected object.

Step 5: Non-Maximum Suppression (NMS)

After classifying all regions, many will overlap (multiple boxes detecting the same object). NMS is
applied to remove duplicates and keep only the most confident, non-overlapping bounding boxes.

Step 6: Output
The final output for each detected object is: the class label (from SVM), the refined bounding box
coordinates (from regressor), and the confidence score.
4.3 R-CNN Summary Table
Step Method Used Purpose
Region Selective Search Find ~2000 candidate
Proposals object regions
Feature CNN (AlexNet/VGG) Extract features from each
Extraction region
Classification SVM Classify each region into a
class
Box Refinement Linear Regression Refine bounding box
coordinates
Deduplication NMS Remove overlapping
boxes

4.4 Limitations of R-CNN

• Very slow: CNN runs 2000 separate times per image.
• Not end-to-end: Training has multiple separate stages (CNN, SVM, regressor).
• High storage: Features from all 2000 regions need to be saved to disk.

5. R-FCN (Region-based Fully Convolutional Network)

5.1 What is R-FCN and Why Was it Needed?
R-FCN combines region-based detection (like Faster R-CNN) with the efficiency of Fully
Convolutional Networks. Even though Faster R-CNN was much faster than R-CNN, it still had a
bottleneck: for every Region of Interest (RoI), it ran a heavy set of fully connected layers separately.

R-FCN solves this by making almost ALL computation shared across the entire image — meaning
computation is done once for the whole image, not repeated for each region. This makes it
significantly faster.

Core Idea: Instead of running a separate classifier for each RoI, R-FCN computes "position-
sensitive score maps" for the whole image ONCE, and then just reads off scores for each
proposed region.
5.2 R-FCN Architecture — Step by Step
Step 1: Feature Extraction
A backbone CNN (e.g., ResNet) processes the entire image once to produce a shared feature map.

Step 2: Region Proposal Network (RPN)

An RPN generates Region of Interest (RoI) bounding boxes — candidate object locations. This is
the same RPN used in Faster R-CNN.

Step 3: Position-Sensitive Score Maps

This is the key innovation of R-FCN. Instead of running a classifier for each RoI separately, the
network computes k x k position-sensitive score maps for the WHOLE image.
• k is typically set to 3, giving a 3x3 grid = 9 score maps per object class.
• Each score map encodes the response for a specific part/position of an object. For example,
for the class "cat", there's one score map for the top-left part of a cat, one for the top-center,
one for the top-right, and so on.
• For C classes + background, the network produces k x k x (C+1) score maps in total.

Step 4: Position-Sensitive RoI Pooling

For each proposed RoI:
• The RoI is divided into a k x k grid of bins (matching the k x k score maps).
• For each bin, the corresponding position-sensitive score map is looked up and the average
score is collected.
• All k x k pooled scores are then averaged to produce the final class score for that RoI.

Why this is clever: The score maps are computed once for the whole image. For each new
RoI, you're just doing a simple pooling lookup — not a full neural network forward pass. This is
why R-FCN is much faster.

Step 5: Classification and Bounding Box Regression

Based on the pooled scores, the object class is predicted. A bounding box regressor refines the
coordinates. Because all heavy computation is shared across the image, inference is much faster
than Faster R-CNN.
5.3 R-FCN vs Faster R-CNN Comparison
Feature Faster R-CNN R-FCN
Per-RoI computation Heavy FC layers for each Simple score map lookup
RoI
Shared computation Feature map only Feature map + score maps
Speed Slower Faster
Score maps Not used k*k position-sensitive maps
per class
Key innovation Shared convolutional Position-sensitive score
features + RPN maps

6. YOLO — You Only Look Once

6.1 What is YOLO?
YOLO is a real-time object detection algorithm that can identify and locate multiple objects in an
image in a single forward pass through the network. This is its revolutionary feature — all previous
methods like R-CNN required multiple passes and stages, but YOLO does everything in one go.

YOLO performs: object classification (what is the object?) and localization (where is the object? —
via bounding boxes). The input image is resized to 448x448 before processing.

Key Insight: YOLO treats object detection as a single regression problem. From the image
pixels, it directly predicts bounding box coordinates and class probabilities in one neural network
evaluation.

6.2 How YOLO Works — The 4 Core Steps

Step 1: Residual Blocks — Grid Division
The input image is divided into an N x N grid of equal-sized cells (N = 4 in simple examples, but
typically N = 7 or larger). Each grid cell is responsible for detecting any object whose CENTER falls
within that cell.

Each cell predicts: whether an object exists, what class it belongs to, and where the bounding box
is.
Step 2: Bounding Box Regression
For each grid cell, YOLO predicts bounding boxes as a vector Y:
• pc — confidence score (probability that an object exists in this cell)
• bx, by — x and y coordinates of the centre of the bounding box
• bw, bh — width and height of the bounding box
• c1, c2, ... cn — class probabilities (one for each class, e.g., car, person, dog)

So each cell can predict multiple bounding boxes, each with its own confidence score and class
probabilities.

Step 3: Intersection Over Union (IoU)

A single object can fall in multiple grid cells, generating multiple bounding box candidates. IoU is
used to determine which boxes are genuinely good.

IoU = Area of Intersection / Area of Union

• The user defines a threshold (e.g., 0.5).
• If IoU <= threshold: this box is ignored (bad match).
• If IoU > threshold: this box is kept (good match).

Simple Explanation: IoU measures how much a predicted box overlaps with the actual (ground
truth) box. A perfect prediction gives IoU = 1. No overlap gives IoU = 0. A threshold of 0.5
means at least 50% overlap is required to count as a detection.

Step 4: Non-Maximum Suppression (NMS)

Even after IoU filtering, a single object might still have multiple boxes passing the threshold. NMS
handles this:
• Sort all remaining boxes by their confidence score (highest first).
• Keep the highest-scoring box.
• Remove all other boxes that overlap heavily with the kept box (IoU above a threshold).
• Repeat for the next highest-scoring box.
• Result: Only one box per object — the most confident one.

6.3 YOLO Network Architecture Details

• Input image is resized to 448x448.
• Uses 1x1 convolutions to reduce feature channels, followed by 3x3 convolutions to generate
the output.
 • Activation function: ReLU everywhere, EXCEPT the final output layer which uses a linear
activation.
• Batch normalization and dropout are used to regularize the model and prevent overfitting.

6.4 Advantages and Disadvantages of YOLO

Advantages Disadvantages
Very fast — real-time detection Less accurate on small objects
Single forward pass — simple pipeline Struggles with objects that are very close
together
Sees whole image at once — global context Earlier versions less accurate than region-
based methods
Good for video applications Grid cells may miss objects between
boundaries

7. Quick Revision Summary

Model Type Key Idea Speed
FCN Segmentation CNN → pixel-wise predictions Moderate
with skip connections
U-Net Segmentation U-shaped FCN with encoder- Moderate
decoder + skip concat
RPN Proposal Sliding window + anchors → RoI Fast (shared
proposals features)
R-CNN Detection Selective Search + CNN per Very Slow
region + SVM
R-FCN Detection Position-sensitive score maps Fast
shared across image
YOLO Detection Grid division + single-pass Real-time
regression + NMS

7.1 Key Terms to Remember

Semantic Segmentation: Classifying every pixel in an image into a category.
Skip Connection: A shortcut that passes high-resolution features from encoder to decoder to
preserve spatial detail.
 Transposed Convolution (Deconvolution): An operation that increases spatial resolution
(upsamples feature maps).
Anchor: A pre-defined bounding box template used in RPN/YOLO for initial box predictions.
RoI (Region of Interest): A region in an image proposed as likely containing an object.
NMS (Non-Maximum Suppression): A post-processing step that removes duplicate
overlapping bounding boxes, keeping the best one.
IoU (Intersection over Union): A metric for how much two bounding boxes overlap. Range: 0
(no overlap) to 1 (perfect match).
Bounding Box Regression: The process of predicting/refining the coordinates of a bounding
box around an object.
Objectness Score: A score indicating whether a proposed region contains any object
(foreground) or not (background).
Bottleneck (U-Net): The middle part of U-Net with the smallest spatial size but richest feature
representation.

7.2 Common Exam Questions

6. Why does FCN-8s produce better results than FCN-32s?
FCN-8s uses skip connections from Pool3 and Pool4 layers, which carry higher-resolution spatial
information. This allows the decoder to recover fine-grained details lost during downsampling,
resulting in sharper segmentation. FCN-32s upsamples only from the deepest layer, producing
coarse outputs.

7. What does an anchor represent in RPN/YOLO?

An anchor is a pre-defined bounding box template of a specific size and aspect ratio placed at each
location. It serves as an initial "guess" for an object's position and shape, which is then refined
through bounding box regression.

8. Why is R-CNN slow? How does R-FCN address this?

R-CNN is slow because it runs the CNN separately for each of the ~2000 region proposals. R-FCN
addresses this by computing position-sensitive score maps for the entire image ONCE, and then
simply reading off scores per region using RoI pooling — avoiding repeated heavy computation.

9. What is the role of NMS in object detection?

NMS (Non-Maximum Suppression) removes duplicate bounding boxes that detect the same object.
It keeps only the box with the highest confidence score and removes all others that overlap
significantly (above an IoU threshold). This ensures each object is detected exactly once.

10. How does U-Net differ from FCN?

Both are encoder-decoder segmentation networks. The key differences are: U-Net uses
concatenation for skip connections while FCN uses addition; U-Net has a symmetric architecture;
and U-Net was designed specifically for biomedical segmentation where precise boundary
detection is critical.

11. What are the two outputs of the RPN for each anchor?
For each anchor, RPN produces: (1) an objectness score — a 2-class prediction for object vs
background, and (2) bounding box regression offsets (delta x, delta y, delta width, delta height) to
refine the anchor to better fit the actual object.