MODULE 5: Deep Learning for Computer Vision

Introduction to Computer Vision using Deep Learning

Computer vision, a field of artificial intelligence, enables computers to "see"
and interpret visual data like images and videos. Deep learning, particularly
convolutional neural networks (CNNs), has revolutionized computer vision by
allowing machines to learn complex patterns and features directly from data,
achieving high accuracy in tasks like image classification, object detection, and
image generation.

Deep Learning's Role:

Deep learning, a subset of machine learning, has become a dominant force in
computer vision. Deep learning models, especially CNNs, excel at automatically
learning hierarchical features from raw image data. This means they can
automatically extract relevant features like edges, textures, and shapes, which
are then used to make accurate predictions for various computer vision tasks
Advantages of Deep Learning in Computer Vision:
Higher Accuracy:
Deep learning models have achieved human-level or even superhuman
accuracy on many computer vision tasks.
Automation:
Deep learning automates feature extraction, which was previously a manual
and time-consuming process.
Adaptability:

By Khan
Deep learning models can be trained on large datasets and adapt to different
types of images and tasks.
Real-world Applications:
Deep learning-powered computer vision is transforming various industries,
including healthcare, transportation, security, and entertainment.
Introduction to Deep Learning
Deep Learning is transforming the way machines understand, learn and interact
with complex data. Deep learning mimics neural networks of the human brain,
it enables computers to autonomously uncover patterns and make informed
decisions from vast amounts of unstructured data.
How Deep Learning Works?
Neural network consists of layers of interconnected nodes or neurons that
collaborate to process input data. In a fully connected deep neural
network data flows through multiple layers where each neuron performs
nonlinear transformations, allowing the model to learn intricate representations
of the data.
In a deep neural network the input layer receives data which passes
through hidden layers that transform the data using nonlinear functions. The
final output layer generates the model’s prediction.

By Khan
Neural Network
Neural networks are machine learning models that mimic the complex functions
of the human brain. These models consist of interconnected nodes or neurons
that process data, learn patterns and enable tasks such as pattern recognition
and decision-making.

Understanding Neural Networks in Deep Learning

Neural networks are capable of learning and identifying patterns directly from
data without pre-defined rules. These networks are built from several key
components:
1.​Neurons: The basic units that receive inputs, each neuron is governed by a
threshold and an activation function.
2.​Connections: Links between neurons that carry information, regulated by
weights and biases.
3.​Weights and Biases: These parameters determine the strength and influence
of connections.
By Khan
4.​Propagation Functions: Mechanisms that help process and transfer data
across layers of neurons.
5.​Learning Rule: The method that adjusts weights and biases over time to
improve accuracy.

Layers in Neural Network Architecture

1.​Input Layer: This is where the network receives its input data. Each input
neuron in the layer corresponds to a feature in the input data.
2.​Hidden Layers: These layers perform most of the computational heavy lifting.
A neural network can have one or multiple hidden layers. Each layer consists
of units (neurons) that transform the inputs into something that the output
layer can use.
3.​Output Layer: The final layer produces the output of the model. The format of
these outputs varies depending on the specific task like classification,
regression.

Working of Neural Networks

1. Forward Propagation
By Khan
 When data is input into the network, it passes through the network in the forward
direction, from the input layer through the hidden layers to the output layer. This
process is known as forward propagation. Here’s what happens during this
phase:
1. Linear Transformation: Each neuron in a layer receives inputs which are
multiplied by the weights associated with the connections. These products are
summed together and a bias is added to the sum. This can be represented
mathematically as:
z=w1​x1​+w2​x2​+…+wn​xn​+b
where
●​ ww represents the weights
●​ xx represents the inputs
●​ bb is the bias
2. Activation: The result of the linear transformation (denoted as zz) is then
passed through an activation function. The activation function is crucial because
it introduces non-linearity into the system, enabling the network to learn more
complex patterns. Popular activation functions include ReLU, sigmoid and tanh.
2. Backpropagation
After forward propagation, the network evaluates its performance using a loss
function which measures the difference between the actual output and the
predicted output. The goal of training is to minimize this loss. This is where
backpropagation comes into play:
1.​Loss Calculation: The network calculates the loss which provides a measure of
error in the predictions. The loss function could vary; common choices are
mean squared error for regression tasks or cross-entropy loss for
classification.
2.​Gradient Calculation: The network computes the gradients of the loss
function with respect to each weight and bias in the network. This involves
applying the chain rule of calculus to find out how much each part of the
output error can be attributed to each weight and bias.
3.​Weight Update: Once the gradients are calculated, the weights and biases are
updated using an optimization algorithm like stochastic gradient descent
(SGD). The weights are adjusted in the opposite direction of the gradient to
minimize the loss. The size of the step taken in each update is determined by
the learning rate.

By Khan
3. Iteration
This process of forward propagation, loss calculation, backpropagation and
weight update is repeated for many iterations over the dataset. Over time, this
iterative process reduces the loss and the network's predictions become more
accurate.
Introduction to Convolution Neural Network
Convolutional Neural Network (CNN) is an advanced version
of artificial neural networks (ANNs), primarily designed to
extract features from grid-like matrix datasets. This is
particularly useful for visual datasets such as images or videos,
where data patterns play a crucial role. CNNs are widely used
in computer vision applications due to their effectiveness in
processing visual data.

How
Convolutional Layers Works?
Convolution Neural Networks are neural networks that share their parameters.
Imagine you have an image. It can be represented as a cuboid having its length,
width (dimension of the image), and height (i.e the channel as images generally
have red, green, and blue channels).

Now imagine taking a small patch of this image and running a small neural
network, called a filter or kernel on it, with say, K outputs and representing them
vertically.
By Khan
Now slide that neural network across the whole image, as a result, we will get
another image with different widths, heights, and depths. Instead of just R, G,
and B channels now we have more channels but lesser width and height. This
operation is called Convolution. If the patch size is the same as that of the image
it will be a regular neural network. Because of this small patch, we have fewer
weights.

Layers Used to Build ConvNets

A complete Convolution Neural Networks architecture is also known as covnets.
A covnets is a sequence of layers, and every layer transforms one volume to
another through a differentiable function.
Let’s take an example by running a covnets on of image of dimension 32 x 32 x
3.
●​ Input Layers: It’s the layer in which we give input to our model. In CNN,
Generally, the input will be an image or a sequence of images. This layer holds
the raw input of the image with width 32, height 32, and depth 3.
●​ Convolutional Layers: This is the layer, which is used to extract the feature
from the input dataset. It applies a set of learnable filters known as the
kernels to the input images. The filters/kernels are smaller matrices usually
2x2, 3x3, or 5x5 shape. it slides over the input image data and computes the
dot product between kernel weight and the corresponding input image patch.
The output of this layer is referred as feature maps. Suppose we use a total of
12 filters for this layer we’ll get an output volume of dimension 32 x 32 x 12.
●​ Activation Layer: By adding an activation function to the output of the
preceding layer, activation layers add nonlinearity to the network. it will apply
an element-wise activation function to the output of the convolution layer.
Some common activation functions are RELU: max(0, x), Tanh, Leaky RELU,
etc. The volume remains unchanged hence output volume will have
dimensions 32 x 32 x 12.

By Khan
●​ Pooling layer: This layer is periodically inserted in the covnets and its main
function is to reduce the size of volume which makes the computation fast
reduces memory and also prevents overfitting. Two common types of pooling
layers are max pooling and average pooling. If we use a max pool with 2 x 2
filters and stride 2, the resultant volume will be of dimension 16x16x12.

●​ Flattening: The resulting feature maps are flattened into a one-dimensional

vector after the convolution and pooling layers so they can be passed into a
completely linked layer for categorization or regression.
●​ Fully Connected Layers: It takes the input from the previous layer and
computes the final classification or regression task.

●​ Output Layer: The output from the fully connected layers is then fed into a
logistic function for classification tasks like sigmoid or softmax which converts
the output of each class into the probability score of each class.
Advantages of CNNs
1.​Good at detecting patterns and features in images, videos, and audio signals.

By Khan
2.​Robust to translation, rotation, and scaling invariance.
3.​End-to-end training, no need for manual feature extraction.
4.​Can handle large amounts of data and achieve high accuracy.
Disadvantages of CNNs
1.​Computationally expensive to train and require a lot of memory.
2.​Can be prone to overfitting if not enough data or proper regularization is
used.
3.​Requires large amounts of labeled data.
4.​Interpretability is limited, it's hard to understand what the network has
learned.

Applications of Deep Learning in Computer Vision

1. Image Classification
Image classification is one of the most fundamental tasks in computer vision,
where the goal is to assign a label to an image from a predefined set of
categories. Deep learning, particularly convolutional neural networks (CNNs),
has significantly improved the accuracy and efficiency of image classification
tasks.
●​ Applications:
o​Medical Diagnosis: CNNs are used to classify medical images, such as
X-rays and MRIs, to detect diseases like pneumonia, tumors, and other
conditions.
o​Autonomous Vehicles: In self-driving cars, image classification helps in
identifying road signs, pedestrians, and other vehicles.
o​Retail: Retailers use image classification to organize and categorize
product images, enhancing search functionality and customer experience.

2. Object Detection
Object detection goes beyond image classification by not only identifying
objects within an image but also locating them using bounding boxes. Deep
learning models such as Faster R-CNN, YOLO (You Only Look Once), and SSD
(Single Shot MultiBox Detector) are widely used for this purpose.
●​ Applications:
o​Surveillance: Object detection is used in security systems to detect and
track people, vehicles, and suspicious activities in real-time.

By Khan
 o​Healthcare: In medical imaging, object detection helps in identifying and
localizing abnormalities, such as tumors, in radiological images.
o​Manufacturing: In automated inspection systems, object detection
ensures quality control by identifying defects in products on production
lines.

3. Image Segmentation
Image segmentation involves partitioning an image into multiple segments or
regions to locate objects and boundaries accurately. Semantic segmentation
assigns a class label to each pixel, while instance segmentation distinguishes
between different objects of the same class.
●​ Applications:
o​Medical Imaging: Image segmentation is crucial for delineating
anatomical structures and abnormalities in medical scans, aiding in
precise diagnosis and treatment planning.
o​Autonomous Driving: Segmentation helps self-driving cars understand
their environment by identifying lanes, road signs, and obstacles.
o​Augmented Reality: Image segmentation enhances augmented reality
applications by accurately overlaying virtual objects onto real-world
scenes.

4. Facial Recognition
Facial recognition systems identify and verify individuals based on their facial
features. Deep learning models, particularly CNNs, have significantly improved
the accuracy and robustness of facial recognition technologies.
●​ Applications:
o​Security and Surveillance: Facial recognition is widely used in security
systems for identifying individuals in public places, access control, and
monitoring.
o​Smartphones: Many modern smartphones use facial recognition for user
authentication and unlocking devices.
o​Social Media: Platforms like Facebook use facial recognition to
automatically tag individuals in photos, enhancing user experience and
engagement.
Advanced Deep Learning Techniques

By Khan
1. Object Recognition/Classification

Classify the object in the image.

[Link] Detection

Are there any object that we want to detect in the image? If yes, draw the
bounding box around the image

3. Object Localization

Are there any object that we want to detect in the image? If yes, draw the
bounding box around the image and show the coordinates of the bounding box.

By Khan
The (x1, y1) would be the top left corner and the (x2, y2) the bottom right.

And finally … the latest one :

4. Object Segmentation

By accommodating mask rcnn, we can get the exact pixel position for each
object. This kind of development is very important for robotic vision.

Convolutional Neural Network

“A convolutional neural network is a class of deep neural networks, most
commonly applied to analyzing visual imagery. CNN is an improved version of
multilayer perceptron”. It’s a class of deep neural network inspired by human’s
visual cortex.

Basically CNN works by collecting matrix of features then predicting whether

this image contains a class or another class based on these features using
softmax probabilities.
By Khan
Convolutional Neural Network Architecture
Commonly, a convolutional neural network architecture consists of these layers :

[Link] Layer

The core idea between convolutional operation is for feature extractions or we

can say filtering. Later, the network will be trying every possible matching
features from the input image compared to the class’s image (class is an object
name that we want to recognize, e.g : a car).

In order to get enlightenment of how convolutional layer operate, have a look at

above image. Based from the above picture we have an input image of 6×6 px,
and we have a 3×3 px 2d convolution kernel. The kernel will do 1 stride from
top left pixel of the image until the bottom of the image. The kernel is a 3×3
matrix of weight (each component of the matrix is a weight). This convolutional
operation is used to extract features from image. The most frequently used
kernel for convolutional is 2d convolution kernel.

Suppose we have a 6×6 pixel image with RGB color channel.

By Khan
The Linearity

Algebraically, a convolutional operation is a linear combination. We need to

introduce non linearity hence an activation function is needed. Right after the
convolutional operation, in order to introduce non linearity, the “ReLU”
activation function is used. If we keep it linear, we do not need to use deep
learning since it’s just a simple linear functions.

Mathematically, ReLU can be defined as

After ReLU, all negative pixel value from the previous convolved feature map
matrix with negative pixel value will be replaced by 0.

Why Non Linearity is Needed ?

In Math and statistic, a non linearity is commonly used to solve complex

problem, meanwhile a linear equation is simple, if we define the input of a linear
equation, the output can be found by simple algebra. Before we use activation
function such as ReLU, basically the convolutional operation is only a linear
function.

Consider an example of a simple this linear equation :

Y = a.x

By Khan
No matter how many layers, the final activation function will always yield the
exact same predicted output. In this condition we do not need to use deep
learning with many layers, a simple one layer neural network is enough.

By using ReLU activation function right after a convolutional operation, we can

introduce the non linearity hence the system can learn how to solve more
complex problem.

2. Pooling Layer

Right after the ReLU, the next layer is a pooling layer. The pooling layer
basically is used to reduce the spatial size of the input hence reducing the
number of parameters and computational complexity.

Commonly used pooling method is max pooling.

3. Fully Connected Layer

The Fully Connected layer holds composite and aggregate information from
previous layers. Before given as input of fully connected layers, those previous
multi dimensional inputs will be flatten into a single dimensional inputs.

And finally, the prediction (voting) will be accomplished using the activation
function, e.g : softmax.

By Khan
Image Segmentation with Convolutional Neural Networks
Image segmentation using Convolutional Neural Networks (CNNs) involves
dividing an image into multiple segments or regions, where each segment
represents a different object or part of an object. CNNs are powerful tools for
this task due to their ability to learn hierarchical features from images.

By Khan
What is a Convolutional Neural Network?
CNN is a term for Convolutional Neural Network. It is a type of deep learning
algorithm frequently used for various computer vision applications, such as
image classification, object recognition, and image segmentation.

CNNs were created specifically to process data with a grid-like structure, such as
images. U-Net was originally invented and first used for biomedical Image
segmentation.

CNN Architectures for Segmentation:

U-Net:
A popular encoder-decoder architecture with skip connections, effective for
capturing both global and local context for precise tumor delineation.

Fully Convolutional Networks (FCNs):

FCNs adapt classification networks for segmentation by replacing fully
connected layers with convolutional layers, allowing for dense prediction.

DeepLab:
Employs atrous convolution (also known as dilated convolution) to control the
field of view of filters, enabling segmentation at multiple scales.

Mask R-CNN:
An extension of Faster R-CNN that adds a branch for generating segmentation
masks, achieving instance segmentation.

What is U-Net?
U-Net gets its name from its U-shaped architecture. U-Net was developed and
first used for biomedical Image segmentation. Its architecture is similar to an
encoder network followed by a decoder network. Unlike classification, where
the result of the deep network is all that matters, semantic segmentation involves
not just discrimination at the pixel level but also a technique to project the
discriminative features learned at different stages of the encoder onto the pixel
space.
By Khan
Why U-Net?
●​ U-Net finds a balance between accuracy and computational efficiency,
making it suitable for both training and real-time inference, contributing to its
wide use.
●​ Pre-trained models and open-source implementations have helped U-Net
become more popular. Variants are continually improved by researchers to
perform particular tasks, enhancing their capabilities.
●​ It is a customized convolutional neural network (CNN) architecture created
especially for image segmentation tasks, making it extremely good at
removing specific information from images.
●​ It uses skip connections between the contracting and expansive paths to
preserve fine-grained spatial information, resulting in accurate segmentation
results.

By Khan
●​ It has proven useful in various applications, including medical image
processing, autonomous driving, remote sensing, and more.

Example Image

Components of U-Net
1. Contracting Path (Encoder):
The encoder, also known as the contracting path, has the role of extracting
relevant spatial and feature information from the incoming data. This is how it
works:

Convolutional Layers: A set of convolutional layers is often the first thing the
encoder uses. These layers use filters to identify elements like edges, textures,
and patterns in the input data. Each convolutional layer builds on the knowledge
from the previous layer to learn increasingly complicated characteristics.

Activation Function (ReLU): After each convolutional layer, an activation

function, commonly ReLU (Rectified Linear Unit), is applied. ReLU introduces
non-linearity into the network, enabling it to understand complex data
connections.

By Khan
Pooling Layers: To frequently lower the spatial dimensions of the feature maps
while preserving critical information, pooling layers (typically max-pooling) are
utilized. Pooling lowers the possibility of overfitting and increases the network’s
computational efficiency.

Feature Compression: As the encoder develops, the number of channels (depth)

rises while the spatial dimensions of the feature maps contract. The network can
now collect hierarchical and abstract representations of the input data through
this feature compression.

2. Expansive Path (Decoder):

The decoder, also known as the expanding path, works with the encoder to
reassemble the segmented output. It is in the role of gradually boosting the
feature maps’ spatial resolution. This is how it works:

Transposed Convolutional Layers (Up-convolution): To Upsample the feature

maps and increase their spatial dimensions, the decoder employs transposed
convolutional layers, also known as up-convolution or deconvolution. These
layers aid the network in regaining the size of the initial input image.

Skip Connections: The utilization of skip connections is a crucial component of

U-Net and a few other CNN designs. Layers in the encoder and decoder are
linked together by skip connections. These links give the decoder access to the
encoder’s high-resolution feature maps, a vital spatial data source. Skip
connections are essential for maintaining fine details and making sure
segmentation is precise.

Output Layer: The decoder normally creates the final segmentation mask as its
final step. Depending on the job, this layer may employ a SoftMax or sigmoid
activation function to generate a probability for individual pixels to correspond
to various classes or segments.

U-Net application

By Khan
Importance of Generative Model
Artificial intelligence (AI) is rapidly transforming our world, and its importance is
undeniable. Generative models are a cornerstone of modern artificial
intelligence (AI), providing essential capabilities that drive innovation and
expand the boundaries of what AI systems can achieve. Their importance in AI
stems from their ability to model complex data distributions, generate new
data, and enable a wide range of applications that would otherwise be
challenging or impossible. So without thinking further about, what are the
generative model. Let's dive into this topic.
Examples of Generative Models
Generative models are a class of models in machine learning that aim to model
the underlying distribution of data in order to generate new samples from that
distribution. Here are some common types of generative models:
1. Probabilistic Models:
Probabilistic models use probability distributions to represent the data. They aim
to estimate the joint probability of the observed data and the latent variables.
Two notable examples of probabilistic models are Bayesian Networks and
Hidden Markov Models.
●​ Bayesian Network: Bayesian Network is a graphical representation of the
various possible relationships between a given set of random variables. is a
classifier that has no dependence on features i.e. it is condition-independent
By Khan
 They are used to model uncertainty in complex domains by encoding
probabilistic relationships among variables. These networks are particularly
effective in scenarios where we need to understand causal relationships and
make decisions under uncertainty.
●​ Hidden Markov Models (HMM): Hidden Markov Models are statistical models
that represent systems that transition between states in a Markov process,
where the system's state is partially observable through a set of observations.
It is predicated on the idea that there is an underlying process with hidden
states, each of which has a known outcome. The probabilities of switching
between hidden states and emitting observable symbols are defined by the
model.

2. Neural Network-Based Models

Neural network-based generative models leverage the power of deep learning to
capture intricate patterns in data. These are The two popular types in this
category are Generative Adversarial Network and Variational Autoencoders.
●​ Generative Adversarial Networks (GAN): This model is based on machine
learning and deep neural networks. Two stochastic neural networks – a
generator and a discriminator of the models compete against each other to
provide more accurate predictions and realistic information. Basically, A GAN
is an unsupervised learning technique that makes it possible to automatically
find and learn different patterns in input data. GANs are also used to create
incredibly lifelike renderings of various objects, people, and scenes that are
challenging for even a human brain to recognize as replicas.
●​ Variational Autoencoders (VAEs): VAEs are probabilistic models that learn to
encode and decode data. They learn a latent representation of the data and
can generate new samples by sampling from the learned latent space. Like
GAN, VAE is a model is also based on a neural network autoencoder, which
has a combination of 2 neural networks – encoder and decoder.

3. Some Other type Model

Apart from the mainstream generative models, there are also some other types of
models available that provide unique approaches to data generation.
●​ Flow-Based Models: Flow-based models utilize invertible neural networks to
learn the exact likelihood of data. These models, such as RealNVP and Glow,
allow for both efficient sampling and exact log-likelihood computation.

By Khan
●​ Energy-Based Models: Energy-based models define a scalar energy function
that assigns low energy to data points that resemble the training data and
high energy to unlikely data points. These models, including Boltzmann
Machines and their variants, focus on learning the energy landscape of the
data distribution.
Applications of Generative models
Generative models have a wide range of applications across various fields.
Here's a breakdown of some of the key areas :
1. Image Generation
●​ Generative Adversarial Networks (GANs), have revolutionized image
generation. GANs can produce highly realistic images that are often
indistinguishable from real photographs. Basically, it takes a Prompt from the
User to analyze the meaning of the query and pass it forward to creating an
image. These models are used in various applications, including creating
synthetic faces, landscapes, and objects for media, entertainment, and virtual
reality environments.
●​ Image-to-image translation involves converting images from one domain to
another, such as transforming sketches into photorealistic images or
converting daytime photos into nighttime scenes.
Examples models: Stable Diffusion, Midjourney,OpenAI DALL-E
2. Text Generation
●​ Generative models play a crucial role in natural language processing (NLP),
enabling machines to understand and generate human-like text. There are
some hugging face models such as Mistral 7b and LLama 2 7b that are used in
Natural Language Processing.
●​ Text Completion and Summarization: Text generation models are adept at
completing sentences or paragraphs based on a given prompt. They are also
used for text summarization, where they condense long documents into
concise summaries while preserving the original meaning.
Examples models: Google PaLM, Meta LLaMA,OpenAI GPT-4, Mistral,
Zephyr
3. Audio and Music Generation

By Khan
●​ Generative models have significantly improved speech synthesis, allowing the
creation of highly natural and expressive synthetic speech. Models like
WaveNet and Tacotron produce high-quality speech that is used in virtual
assistants, audiobooks, and voice-overs.
●​ In the music industry, generative models can compose original pieces by
learning from existing music datasets. Models like Musenet and Jukedeck
create music across genres and styles, help musicians compose, and provide
background scores for multimedia content.
Examples models: BachBot, WaveNet
4. Data Augmentation
●​ Enhancing Training Dataset: Generative models are used to augment training
datasets, especially in scenarios where obtaining large amounts of labeled
data is challenging. By generating synthetic data that resembles real-world
samples, these models improve the performance of machine learning
algorithm.
●​ Creating synthetic data: Model can create entirely synthetic datasets that
simulate real-world distributions. This capability is critical for tasks that
require privacy-preserving data, such as in healthcare and finance, where
synthetic data can be used to train models without revealing sensitive
information.
Examples models: StyleGAN
5. Healthcare Applications
●​ Generative models are transforming medical image analysis by enhancing and
generating medical images. It can be used to analyze medical images like
X-rays or MRIs, potentially aiding in early disease detection or treatment
planning. These applications assist in disease diagnosis, treatment planning,
and medical research.
●​ In drug discovery, models are used to design new molecules with desired
properties. Models like Variational Autoencoders (VAEs) and GANs help in
exploring the vast chemical space, predicting molecular properties, and
generating novel compounds.

Challenges and Limitations of the Generative Model

By Khan
 While generative models offer a wide range of capabilities, they also face several
challenges:
Computational complexity of training: Generative models, especially
sophisticated models such as GANs, require significant computational resources
and time. Training them requires powerful hardware and can be
resource-intensive.
●​ Quality of output generated: Output generated from generative models may
not always be accurate or error-free. This can be due to a number of factors,
including insufficient data, insufficient training, or overly complex models
●​ Security: Generative AI systems can be used to fake information or
propaganda by creating realistic and believable fake videos, images and text
●​ Trustworthy Concern The ability of generative models to generate realistic
content raises ethical issues, especially in the creation of deep imitation or
fake content. Ensuring responsible use is paramount to prevent abuse or
fraud.
●​ Data dependencies: The quality of the output generated depends heavily on
the quality of the training data. If the training data is biased or
unrepresentative, the model output will reflect those biases.
Transfer Learning for Computer Vision
Transfer learning is a powerful technique in the field of computer vision, where
a pre-trained model on a large dataset is fine-tuned for a different but related
task. This approach leverages the knowledge gained from the initial training to
improve performance and reduce training time for the new task.
Key Concepts in Transfer Learning
1.​Pre-trained Models: Models that have been previously trained on large
datasets, such as VGG, ResNet, Inception, and DenseNet, have learned rich
feature representations.
2.​Feature Extraction: Using the pre-trained model as a fixed feature extractor.
The model's earlier layers, which capture general features, are retained, while
the final layers are replaced with new ones suitable for the target task.
3.​Fine-Tuning: Adjusting the weights of the pre-trained model's layers along
with the new layers. Fine-tuning can be done selectively, where only certain
layers are updated to adapt the model to the new task.
Steps in Transfer Learning for Computer Vision
1.​Select a Pre-trained Model: Choose a model pre-trained on a large dataset.
Common choices include ResNet, VGG, and Inception due to their proven
performance and availability in popular deep-learning libraries.
By Khan
2.​Modify the Model: Replace the final classification layer of the pre-trained
model with one that matches the number of classes in the target task. This
often involves adding new fully connected layers followed by a softmax or
sigmoid activation function.
3.​Freeze Layers: Optionally freeze the weights of the earlier layers to retain
their learned features. This helps in leveraging the general patterns and
structures learned from the large dataset.
4.​Train the Model: Train the modified model on the target dataset. This involves
fine-tuning the new layers and possibly the later layers of the pre-trained
model. Fine-tuning is typically done with a lower learning rate to avoid drastic
changes to the pre-trained weights.
Advantages of Transfer Learning in Computer Vision
1.​Reduced Training Time: By leveraging pre-trained models, transfer learning
significantly reduces the time required to train a model for a new task.
2.​Improved Performance: Pre-trained models provide a strong starting point,
often leading to better performance on the target task compared to training
from scratch.
3.​Lower Data Requirements: Transfer learning is particularly beneficial when
the target dataset is small, as the pre-trained model's general features
mitigate the need for large amounts of labeled data.
Limitations of Transfer Learning in Computer Vision
1.​Domain Mismatch: Transfer learning assumes that the features learned from
the source domain (e.g., ImageNet) are applicable to the target domain.
However, if there is a significant difference between the source and target
domains, the pre-trained model may not perform well, and the transferred
features might not be as useful.
2.​Overfitting on Small Datasets: While transfer learning can help when the
target dataset is small, there is still a risk of overfitting if the target dataset is
too small to fine-tune the model properly. The model may memorize the
training data instead of learning generalizable features.
3.​Model Complexity and Size: Pre-trained models, especially those based on
deep neural networks, are often large and complex. This can lead to increased
computational and memory requirements, making it challenging to deploy
these models on devices with limited resources.
4.​Limited Adaptability: Pre-trained models are typically fine-tuned for specific
tasks. Adapting them to tasks that are significantly different from the original

By Khan
 training objective might require extensive modifications and fine-tuning,
which can be computationally expensive and time-consuming.
Applications of Transfer Learning in Computer Vision
1.​Image Classification: Transfer learning can be used to adapt pre-trained
models for classifying images into different categories specific to a new
dataset.
2.​Object Detection: Models like Faster R-CNN and YOLO, pre-trained on
datasets like COCO, can be fine-tuned for detecting objects in specific
domains.
3.​Semantic Segmentation: Pre-trained models can be adapted for segmenting
images into meaningful regions, useful in medical imaging and autonomous
driving.
4.​Style Transfer: Transfer learning techniques can be employed to apply artistic
styles from one image to another, leveraging features learned from diverse
datasets.

By Khan