Superior University

IMAGE SEGMENTATION IN DEEP LEARNING

A Study of Architectures, Techniques, and a Medical Imaging Case Study

Course: Deep Learning / Computer Vision

SUBMITTED TO: MISS AYESHA MUMTAZ

DEPARTMENT OF COMPUTER SCIENCE

MUHAMMAD AWAIS
SU72-MSCSW-F25-007 /
MSCS / 2nd
Department Of Computer Science
01-AUGUST-2026

Department of Computer Science Superior University Sargodha Campus

Superior University

What is Image Segmentation?

Image Segmentation is a computer vision technique used to divide an image into multiple
segments or regions, making it easier to analyze and understand specific parts of the image.
It helps identify objects, boundaries and relevant features within an image for further
processing.

Image Segmentation

This technique is widely used in applications such as medical imaging, object detection,
autonomous driving and image editing. By classifying each pixel into meaningful categories,
image segmentation forms the foundation for many visual understanding tasks in AI and
machine learning.
Formally, image segmentation is the task of partitioning an image into multiple segments or
regions, such that pixels within a segment share common characteristics (such as belonging to
the same object or object class). The output is typically a segmentation map or mask of the
same spatial dimensions as the input image, where each pixel is assigned a class label or an
instance identifier.

Types of Segmentation
Various types of image segmentation techniques are:
1. Semantic Segmentation:
Semantic segmentation is a process in computer vision that focuses on assigning a class
label to every pixel in an image. This process transforms simple images into meaningful data
maps, enabling machines to understand and interpret complex visual scenes as humans do. It
is an end-to-end image analysis process that divides a digital image into multiple segments
and classifies the information contained in each region. Every pixel is classified into a

Department of Computer Science Superior University Sargodha Campus

Superior University

predefined category (e.g sheep, dog), but individual object instances of the same class are not
distinguished from one another.

2. Instance Segmentation:
In addition to classifying each pixel, distinct instances of the same object class are separately identified
and delineated (e.g., two overlapping sheeps are segmented as two separate objects).

3. Panoptic Segmentation:
Panoptic Segmentation is an image segmentation technique that combines Semantic
Segmentation and Instance Segmentation to provide a complete understanding of a scene. It
classifies every pixel in an image while also distinguishing between individual objects of the
same class.
 Assigns a class label to every pixel in an image.
 Provides information about both objects and background regions.
 Delivers a more complete scene understanding than semantic or instance segmentation
alone.
 Useful for detailed scene analysis and object localization.

Department of Computer Science Superior University Sargodha Campus

Superior University

Evolution of Deep Learning Architectures for Segmentation

The transition from classical to deep learning-based segmentation was marked by a series of
influential architectures, each addressing limitations of its predecessors.

1. Fully Convolutional Networks (FCN)

Fully Convolutional Networks were the first architecture to adapt standard classification CNNs
(such as VGG) for dense, pixel-wise prediction. FCNs replace fully connected layers with
convolutional layers, allowing the network to accept images of arbitrary size and produce
correspondingly sized output segmentation maps. A key contribution was the use of skip
connections that fused coarse, high-level semantic information with fine, low-level spatial
detail, improving the precision of segment boundaries.

Fully Connected (FC) layers, also called dense layers, are neural network layers where each
neuron is connected to every neuron in the previous layer.
 Every neuron is fully connected to neurons in adjacent layers
 Commonly used in deep learning and neural networks
 In CNNs, FC layers follow convolution and pooling layers
 Convert extracted feature maps into final predictions or output classes
 Serve as core building blocks in feedforward neural networks
Structure of Fully Connected Layers
The structure of a fully connected layer is based on complete connectivity, where every neuron
in one layer is connected to every neuron in the next layer.
 Each neuron connects to all neurons in the subsequent layer
 Enables information flow across the entire network
 Uses weights and biases to learn patterns from data
 Helps transform extracted features into predictions

Department of Computer Science Superior University Sargodha Campus

Superior University

Key Components of Fully Connected Layers

Fully connected layers consist of several important components that help the network learn
and make predictions.
 Neurons: Receive inputs from all neurons in the previous layer and send outputs forward
 Weights: Represent the strength of connections between neurons
 Biases: Adjust the weighted sum to improve learning flexibility
 Activation Functions: Functions like ReLU, Sigmoid or Tanh introduce non linearity for
learning complex patterns
Importance of Fully Connected Layers in Neural Networks
Fully connected (FC) layers play a major role in combining learned features and generating
final predictions in neural networks.
1. Feature Integration and Abstraction: Combines features extracted by earlier layers to
capture complex patterns and relationships.
2. Decision Making and Output Generation: Converts learned features into final outputs
or class probabilities, often using Softmax for classification.
3. Introduction of Non-Linearity: Uses activation functions like ReLU, Sigmoid or Tanh
to learn complex non-linear patterns.
4. Universal Approximation: With enough neurons, FC layers can approximate complex
continuous functions.

2. U-Net
U-Net is a kind of neural network mainly used for image segmentation which means dividing
an image into different parts to identify specific objects for example separating a tumor from
healthy tissue in a medical scan. The name “U-Net” comes from the shape of its architecture
which looks like the letter “U” when drawn. It is widely used in medical imaging because it
performs well even with a small amount of labeled data.

U-Net Architecture

Department of Computer Science Superior University Sargodha Campus

Superior University

The architecture is symmetric and has three key parts:

1. Contracting Path (Encoder):
 Uses small filters (3×3 pixels) to scan the image and find features.
 Apply an activation function called ReLU to add non-linearity help the model to learn
better.
 Uses max pooling (2×2 filters) to shrink the image size while keeping important
information. This helps the network focus on bigger features.
2. Bottleneck:
The middle of the “U” where the most compressed and abstract information is stored. It links
the encoder and decoder.
3. Expansive Path (Decoder):
 Uses upsampling i.e increasing image size to get back the original image size.
 Combines information from the encoder using “skip connections.” These connections
help the decoder get spatial details that might have been lost when shrinking the image.
 Uses convolution layers again to clean up and refine the output.
The above image shows U-Net turning a 572×572 image into a smaller 388×388 segmented
map. It shrinks the image to capture features then upsamples to restore size using skip
connections to keep details. The output labels each pixel as object or background.

How U-Net Works

After understanding the architecture, it’s important to see how U-Net actually processes data
to perform segmentation:
1. Input Image: The process starts by feeding a medical or other input image typically
grayscale into the network.
2. Feature Extraction (Encoder): The encoder extracts increasingly abstract features by
applying convolutions and downsampling. At each level the spatial size decreases while
the number of feature channels increases and allow the model to capture higher-level
patterns.
3. Bottleneck Processing: This is the middle part of the network where the image is reduced
the most. It holds a small but very meaningful version of the image that captures the main
features.
4. Reconstruction and Localization (Decoder): The decoder begins to reconstruct the
original image size through upsampling. At each level it combines decoder features with
corresponding encoder features using skip connections to retain fine-grained spatial details.
5. Skip Connections for Precision: Skip connections help preserve spatial accuracy by
bringing forward detailed features from earlier layers. These are especially useful when the
model needs to distinguish boundaries in segmentation tasks.
6. Final Prediction: A 1×1 convolution at the end converts the refined feature maps into the
final segmentation map where each pixel is classified into a specific class like foreground
or background. This output has the same spatial resolution as the input image.
Implementation of U-Net
The implementation consists of three main parts:
1. Encoder Block: The contraction path block containing two 3x3 convolutional layers with
ReLU activations, followed by a 2x2 max pooling layer.
2. Decoder Block: The expansive path block which upsamples the input, concatenates it with
the corresponding encoder features and applies two 3x3 convolutional layers with ReLU
activations.

Department of Computer Science Superior University Sargodha Campus

Superior University

3. U-Net Model: Combining the encoder and decoder blocks to define the complete U-Net
architecture.

3. SegNet
SegNet employs a similar encoder-decoder structure but distinguishes itself through its use of
max-pooling indices, recorded during the encoding stage and reused during decoding for non-
linear upsampling. This design choice reduces the number of trainable parameters compared to
alternatives that use learned deconvolution, making SegNet more memory-efficient.

SegNet is a deep learning architecture designed for semantic segmentation, where the goal is
to classify each pixel in an image into a predefined category. It is an encoder-decoder neural
network tailored for pixel-wise image segmentation, making it highly effective for tasks that
require detailed and precise segmentation of images.
 SegNet works by learning to label each pixel in an image based on its corresponding
category, providing a comprehensive understanding of the image’s content.
 SegNet particularly useful in applications such as autonomous driving, medical image
analysis, and urban scene understanding, where accurate segmentation is important.

Encoder Network
The encoder network in SegNet is composed of 13 convolutional layers, mirroring the first 13
convolutional layers of the VGG16 network, which was originally designed for object
classification.
Key points of the Encoder Network:
1. Pre-trained Weights: VGG16's pre-trained weights allows efficient initialization and
faster convergence during training.
2. Convolutional Layers: perform convolution operations to extract features from the input
image.
3. Batch Normalization: each convolutional layer is followed by batch normalization to
stabilize and accelerate the training process.
4. ReLU Activation: applied element-wise to introduce non-linearity.
5. Max-Pooling: Max-pooling with a 2×2 window and a stride of 2 is used to downsample
the feature maps, reducing their spatial resolution by half. This step helps in achieving
translation invariance over small spatial shifts.

Department of Computer Science Superior University Sargodha Campus

Superior University

Decoder Network

The decoder network consists of 13 layers, each corresponding to an encoder layer. The
decoding process is designed to upsample the feature maps back to the original image
resolution.
Key Features of the Decoder Network:
1. Upsampling Using Max-Pooling Indices: The stored max-pooling indices are used to
upsample the feature maps, creating sparse feature maps. This technique ensures that the
spatial locations of features are preserved.
2. Convolution with Trainable Filters: The sparse feature maps are convolved with
trainable decoder filters to produce dense feature maps. This step helps in refining the
feature maps and improving segmentation accuracy.
3. Batch Normalization: Similar to the encoder, batch normalization is applied to each layer
in the decoder network.
4. Soft-Max Classifier: The final output of the decoder network is passed through a multi-
class soft-max classifier, which assigns class probabilities to each pixel. The predicted
segmentation is obtained by taking the class with the highest probability for each pixel.
Decoder Variants
To evaluate the effectiveness of different decoding techniques, several variants of the SegNet
and FCN architectures were tested. These variants include:
1. SegNet-Basic: A simplified version of SegNet with 4 encoders and 4 decoders. This variant
uses max-pooling indices for upsampling without learning.
2. SegNet-Basic-EncoderAddition: Adds the encoder feature maps to the corresponding
decoder feature maps after upsampling.
3. SegNet-Basic-SingleChannelDecoder: Uses single-channel decoder filters, which
significantly reduce the number of trainable parameters and inference time.
4. FCN-Basic: A simplified version of FCN with 4 encoders and 4 decoders, using
deconvolution for upsampling.
5. FCN-Basic-NoAddition: Discards the step of adding encoder feature maps to the decoder
feature maps.
6. FCN-Basic-NoDimReduction: Does not perform dimensionality reduction on encoder
feature maps, retaining the full resolution.

Department of Computer Science Superior University Sargodha Campus

Superior University

4. DeepLab Family (v1–v3+)

The DeepLab series introduced atrous (dilated) convolutions, which expand the receptive field
of convolutional filters without increasing the number of parameters or reducing spatial
resolution. DeepLabv2 introduced Atrous Spatial Pyramid Pooling (ASPP) to capture multi-
scale contextual information, while DeepLabv3+ added a decoder module to refine
segmentation boundaries, achieving state-of-the-art results on benchmarks such as PASCAL
VOC and Cityscapes.

Architecture of DeepLab Models

The DeepLab models share a common architecture with variations in specific components to
enhance performance.
Atrous Convolution: Atrous convolution is the cornerstone of the DeepLab series. By
inserting zeros between filter elements, it allows the convolution operation to cover a larger
receptive field without increasing the number of parameters. This technique helps capture more
context from the image, which is crucial for accurate segmentation.
Atrous Spatial Pyramid Pooling (ASPP): The ASPP module applies atrous convolution with
different dilation rates in parallel, capturing information at multiple scales. By doing so, it can
effectively handle objects of varying sizes and shapes, which is essential for accurate semantic
segmentation.
Encoder-Decoder Structure: Introduced in DeepLabv3+, the encoder-decoder structure
enhances segmentation accuracy by combining high-level contextual information from the
encoder with fine-grained details from the decoder. This design helps produce sharper and
more precise segmentation maps.
Applications of DeepLab
The DeepLab series has been widely adopted in various applications due to its robust
performance and flexibility.
 Autonomous Driving: In autonomous driving, accurate scene understanding is crucial for
safe navigation. DeepLab models are used to segment road scenes into different categories
such as roads, vehicles, pedestrians, and obstacles, enabling autonomous vehicles to make
informed decisions.
 Medical Imaging: In medical imaging, semantic segmentation helps identify and delineate
anatomical structures and pathological regions. DeepLab models are employed to segment

Department of Computer Science Superior University Sargodha Campus

Superior University

organs, tumors, and other critical structures from medical scans, aiding in diagnosis and
treatment planning.
 Augmented Reality: For augmented reality applications, accurate segmentation of objects
from the background is essential for seamless integration of virtual and real-world
elements. DeepLab models provide the precision needed to achieve realistic and immersive
AR experiences.

5. Mask R-CNN
Mask R-CNN extends the Faster R-CNN object detection framework by adding a parallel
branch that predicts a segmentation mask for each detected region of interest. This architecture
is the most widely adopted approach for instance segmentation, as it simultaneously performs
object classification, bounding-box regression, and pixel-wise mask prediction.

Mask R-CNN is an advanced deep learning model for object detection and instance
segmentation that extends Faster R-CNN by adding a parallel branch for pixel-level mask
prediction. It not only detects objects and draws bounding boxes but also generates precise
segmentation masks for each object.
 Extends Faster R-CNN by adding a mask prediction branch for each Region of Interest
(RoI).
 Performs object detection and instance segmentation simultaneously with pixel-level
accuracy.
Architecture
Mask R-CNN extends Faster R-CNN by adding a parallel branch for predicting segmentation
masks along with object detection outputs.

Mask R-CNN Architecture

Department of Computer Science Superior University Sargodha Campus

Superior University

Backbone Network

The backbone network extracts feature maps from the input image using deep CNN
architectures such as ResNet-C4 and ResNet-FPN.
 Uses deep convolutional networks for feature extraction
 Feature Pyramid Network (FPN) improves multi-scale detection
 Produces feature maps such as P2, P3, P4, P5, and P6

Mask R-CNN backbone architecture

Region Proposal Network

The RPN generates candidate object regions from convolutional feature maps.
 Uses 3×3 convolution layers to generate proposals
 Predicts objectness scores and bounding box coordinates
 Uses anchor boxes of different scales and aspect ratios
 Identifies potential object locations efficiently

Anchor Generation Mask R-CNN

Mask Representation

The mask branch predicts segmentation masks for each Region of Interest (RoI).
 Uses a Fully Convolutional Network (FCN) for pixel-level prediction
 Preserves spatial structure of features
 Generates an m×m mask for each object class
 Uses RoI Align for accurate mask generation

Department of Computer Science Superior University Sargodha Campus

Superior University

RoI Align

RoI Align is used to extract fixed-size feature maps from region proposals while preserving
exact spatial alignment. It improves RoI Pooling by removing quantization and ensuring pixel-
accurate feature mapping, which is important for mask prediction.

 Takes the feature map from the previous convolution layer and divides it into an M × N
grid without rounding or integer approximation.
 Uses bilinear interpolation to compute exact feature values at sampled locations.
 Produces fixed-size feature maps for each Region of Interest, improving segmentation
accuracy.

Working
Mask R-CNN extends Faster R-CNN by adding a parallel mask prediction branch, enabling
both object detection and instance segmentation in a single unified pipeline.
 Uses a Region Proposal Network (RPN) to generate candidate object regions.
 Extracts region features using RoI Align for precise spatial alignment.
 Performs object classification using a softmax classifier to assign class labels.
 Applies bounding box regression to refine object localization.
 Generates pixel-level segmentation masks through a dedicated mask branch and outputs
final predictions.

Department of Computer Science Superior University Sargodha Campus

Superior University

Applications
 Medical Imaging: Used for tumor detection, organ segmentation, and anomaly
identification in scans like MRI and CT.
 Autonomous Vehicles: Helps detect and segment pedestrians, vehicles, and road objects
for safe driving.
 Surveillance Systems: Supports object tracking and activity monitoring in security
footage.
 Image Editing & AR: Enables object removal, background editing, and augmented reality
effects.
 Aerial Imaging: Used in drones and satellite images for mapping and object detection.

Advantages
 Reduces computational cost compared to exhaustive search methods
 Flexible architecture that supports different backbone networks
 Achieves state-of-the-art performance in instance segmentation tasks

Transformer-Based Approaches

More recently, Vision Transformer-based models such as SegFormer and Mask2Former have
demonstrated that self-attention mechanisms can capture long-range dependencies more
effectively than convolutional receptive fields, achieving strong results across semantic,
instance, and panoptic segmentation tasks with unified architectures.

Evaluation Metrics
Quantitative evaluation of segmentation models relies on a set of standard metrics that compare
predicted masks against ground truth annotations.

Metric Description
Pixel Accuracy Proportion of correctly classified pixels out of the total number of pixels.
Intersection over Union (IoU) Ratio of the overlap area to the union area between predicted and ground truth
masks; the most widely used segmentation metric.
Mean IoU (mIoU) Average IoU computed across all object classes, providing a balanced
measure across categories.
Dice Coefficient (F1 Score) Harmonic measure of overlap, commonly used in medical image
segmentation; closely related to IoU.
Boundary F1 Score Measures the accuracy of predicted segment boundaries against ground truth
contours.

Case Study: Brain Tumor Segmentation Using U-Net

Background and Motivation

Accurate delineation of brain tumors from Magnetic Resonance Imaging (MRI) scans is a
critical step in diagnosis, treatment planning, and monitoring disease progression. Manual
segmentation by radiologists is time-consuming, labor-intensive, and subject to inter-observer
variability. Deep learning-based automated segmentation offers a consistent, scalable, and

Department of Computer Science Superior University Sargodha Campus

Superior University

rapid alternative, and U-Net has become the benchmark architecture for this task due to its
strong performance on limited, high-resolution medical datasets.

Dataset
A representative study uses the BraTS (Brain Tumor Segmentation) dataset, a publicly
available benchmark comprising multi-modal MRI scans (T1, T1-contrast enhanced, T2, and
FLAIR sequences) from glioma patients, with expert-annotated ground truth masks identifying
tumor sub-regions: the necrotic core, peritumoral edema, and enhancing tumor.

Methodology

• Preprocessing: MRI volumes are normalized, skull-stripped, and resampled to a

uniform voxel spacing; 3D volumes are often processed as stacked 2D slices or directly
with 3D U-Net variants.
• Architecture: A U-Net encoder-decoder with four downsampling and four upsampling
stages, skip connections at each resolution level, and a final 1x1 convolution with
softmax activation to produce per-class probability maps.
• Loss Function: A combination of Dice Loss and weighted Cross-Entropy Loss is
typically used to address the severe class imbalance between tumor and healthy tissue
regions.
• Training: The model is trained using the Adam optimizer with data augmentation
(rotation, flipping, elastic deformation) to improve generalization given the limited size
of medical datasets.
• Post-processing: Connected-component analysis is applied to remove small, spurious
false-positive predictions and refine the final segmentation mask.
Results and Discussion
Studies applying U-Net and its 3D variants to the BraTS dataset consistently report Dice
coefficients in the range of 0.85-0.91 for whole tumor segmentation, with somewhat lower
performance (0.70-0.80) for more challenging sub-regions such as the enhancing tumor core,
which occupies a smaller and more heterogeneous area. These results demonstrate that U-Net-
based architectures can approach expert-level segmentation accuracy while reducing analysis
time from hours to seconds per scan.

Despite these successes, challenges remain. Class imbalance between tumor and background
tissue can bias models toward under-segmentation of small lesions. Domain shift, arising from
differences in MRI scanner hardware and acquisition protocols across institutions, can
significantly degrade performance when a model trained on one dataset is deployed on another.
Addressing these issues is an active area of research, involving techniques such as domain
adaptation, transfer learning, and uncertainty-aware segmentation.

Department of Computer Science Superior University Sargodha Campus

Superior University

Challenges and Future Directions

• Data Scarcity and Annotation Cost: Pixel-level ground truth labeling is expensive and
time-consuming, particularly in specialized domains such as medical imaging,
motivating research into semi-supervised and weakly supervised segmentation.
• Class Imbalance: Rare classes or small objects are often under-represented, requiring
specialized loss functions such as Dice Loss or Focal Loss.
• Real-Time Performance: Applications such as autonomous driving demand low-latency
inference, driving research into lightweight architectures such as ENet and Fast-SCNN.
• Generalization and Domain Shift: Models trained on one dataset or imaging domain
often perform poorly when applied to data from a different distribution.
• Uncertainty Quantification: Especially critical in safety-sensitive domains such as
healthcare, where knowing the confidence of a segmentation prediction can inform
clinical decision-making.
Conclusion
Deep learning has fundamentally transformed image segmentation, moving the field from
hand-engineered heuristics to powerful, end-to-end learned representations. Architectures such
as FCN, U-Net, DeepLab, and Mask R-CNN each contributed key innovations, from skip
connections to atrous convolutions to instance-aware mask prediction, that collectively
advanced the state of the art. The case study of brain tumor segmentation illustrates the
tangible, real-world impact of these techniques, demonstrating how U-Net-based models can
approach expert-level accuracy while dramatically reducing analysis time. As research
continues to address challenges related to data efficiency, generalization, and real-time
performance, segmentation models are poised to play an increasingly central role in fields as
diverse as healthcare, autonomous systems, and environmental monitoring.

Department of Computer Science Superior University Sargodha Campus