Deep Learning
CS F425
BITS Pilani Prof. Pratik Narang
Department of CSIS
Pilani Campus
BITS Pilani
Pilani Campus
Convolutional Neural Networks
How do we perceive images?
Source: [Link] BITS Pilani, Pilani Campus
CNN
• CNNs are a special kind of neural network for processing data that as a
known, grid-like topology.
• Work extremely well for Image data (2D grid of pixels) and also for time-
series data (1D grid taking samples are regular intervals)
• The network employs a mathematical operation, referred as “convolution”.
• In simple terms, CNNs are neural networks that use convolution in place of
general matrix multiplication (in place of at least one of their layers).
BITS Pilani, Pilani Campus
Example – LeNet-5
Courtesy: Yann LeCun
BITS Pilani, Pilani Campus
Components and layers in CNN
Convolutional neural networks use three basic ideas:
local receptive fields, shared weights, and pooling.
Types of layers:
Convolution layer
Pooling layer
Fully connected layer
BITS Pilani, Pilani Campus
Input image
Source: [Link]
BITS Pilani, Pilani Campus
Input image
• Consider the input to the CNN as a 28×28
square of neurons, whose values correspond
to the 28×28 pixel intensities of input image
BITS Pilani, Pilani Campus
Local receptive fields
As usual, we will connect the input pixels to a layer of hidden neurons.
But we won't connect every input pixel to every hidden neuron.
Instead, we only make connections in small, localized regions of the input image.
BITS Pilani, Pilani Campus
Local receptive fields
• That region in the input image is called the local receptive field for the hidden
neuron. It's a little window on the input pixels.
• Each connection learns a weight. The hidden neuron learns an overall bias as
well.
• We can think of that particular hidden neuron as learning to analyze its particular
local receptive field.
BITS Pilani, Pilani Campus
Local receptive fields
We then slide the local receptive field across the entire input image.
For each local receptive field, there is a different hidden neuron in the first
hidden layer.
BITS Pilani, Pilani Campus
BITS Pilani
Pilani Campus
Shared weights (and biases)
Locally connected layers
• CNNs use the notion of proximity (local features) through locally
connected layers.
• Each unit in the (first) hidden layer detects patterns in a small
portion of the input image, rather than the entire image.
Weight sharing
• In a neural network, "knowing how to detect a local feature" implies
tuning those weights and biases appropriately which connect the
input neurons to the hidden neuron in the next layer.
• If we learn how to detect a local feature in one region of the image
(i.e., the tuned weights and biases), then we know how to detect that
feature in all other regions of the image.
• If we detect a feature (say, a horizontal edge) at the top right of the
image, we can use the same detector on the bottom-left corner too!
• We can therefore reuse the same weights everywhere else in the
image!
• All the neurons in the first hidden layer detect exactly the same
feature! (just at different locations in the input image)
• We often refer the map from the input layer to the hidden layer
a feature map.
• The shared weights and bias are often said to define a kernel or filter.
Simple example
Convolution operation
BITS Pilani, Pilani Campus
Convolution operation
BITS Pilani, Pilani Campus
Convolution operation
BITS Pilani, Pilani Campus
Convolution operation
BITS Pilani, Pilani Campus
Convolution operation
BITS Pilani, Pilani Campus
BITS Pilani
Pilani Campus
CNNs as edge detectors
Vertical edge detector
BITS Pilani, Pilani Campus
Horizontal edge detector
BITS Pilani, Pilani Campus
Sample image
BITS Pilani, Pilani Campus
Edge detectors
BITS Pilani, Pilani Campus
Intuitively…
Source: [Link]
BITS Pilani, Pilani Campus
Source: [Link] BITS Pilani, Pilani Campus
Source: [Link]
BITS Pilani, Pilani Campus
BITS Pilani
Pilani Campus
Padding
Convolution operation – issues
BITS Pilani, Pilani Campus
Padding
BITS Pilani, Pilani Campus
Convolution with padding
BITS Pilani, Pilani Campus
Padding
BITS Pilani, Pilani Campus
Padding
BITS Pilani, Pilani Campus
BITS Pilani
Pilani Campus
Striding
Striding
BITS Pilani, Pilani Campus
BITS Pilani, Pilani Campus
BITS Pilani, Pilani Campus
BITS Pilani, Pilani Campus
Parameter size
• A big advantage of sharing weights and biases is that it greatly reduces the
number of parameters involved.
• For each feature map, we need 25=5×5 shared weights, plus a single shared
bias. So each feature map requires 26 parameters.
• For 20 feature maps -> 20×26=520 parameters for the convolutional layer.
• By comparison: suppose a fully connected first layer with 784=28×28 input
neurons, and a relatively modest 30 hidden neurons.
• That's 784×30 weights + 30 biases, total 23,550 parameters.
• The fully-connected layer has more than 40 times as many parameters as
the convolutional layer.
BITS Pilani, Pilani Campus
BITS Pilani
Pilani Campus
Pooling layer
Pooling layer
Pooling layer is used for subsampling.
It reduces the size of the representation.
It also speeds up the computation.
Pooling is applied on each of the channels
It is a fixed computation
No use of gradient descent or activation function.
Padding is not used
The output dimensions are:
𝑛−𝑓 𝑛−𝑓
+1 x +1
𝑠 𝑠
BITS Pilani, Pilani Campus
Pooling
Broadly two types:
Max pooling
Average pooling
BITS Pilani, Pilani Campus
Max pooling
BITS Pilani, Pilani Campus
Average pooling
BITS Pilani, Pilani Campus
Flattening
• Converting the data into a 1-dimensional array for inputting it to the next layer.
• We flatten the output of the convolutional layers to create a single long feature
vector.
• This is connected to the final classification model, which is a fully-
connected layer.
BITS Pilani, Pilani Campus
Fully connected layer
Stack of neurons in one layer connected to every other neuron in the next layer
BITS Pilani, Pilani Campus
Putting the pieces together
Source: [Link]
BITS Pilani, Pilani Campus
BITS Pilani
Pilani Campus
Why CNNs?
Why CNNs?
• Feature learning!
• Shared weights!
• CNN take advantage of local spatial coherence of images, dramatically
reducing the number of operation needed to process an image.
• CNNs are more efficient in terms of memory and complexity.
BITS Pilani, Pilani Campus
BITS Pilani
Pilani Campus
CNN Architectures, Applications and Case
studies
BITS Pilani
Pilani Campus
CNN architectures
LeNet-5
BITS Pilani, Pilani Campus
AlexNet
BITS Pilani, Pilani Campus
VGG 16
BITS Pilani, Pilani Campus
BITS Pilani
Pilani Campus
Residual Networks
Training deep neural networks
Training very deep networks is hard
Vanishing gradients – as the gradient is back-propagated to earlier layers,
repeated multiplication may make the gradient infinitively small.
As the network goes deeper, its performance gets saturated or even starts
degrading rapidly.
Source: [Link]
BITS Pilani, Pilani Campus
Residual Network or ResNet
ResNets use Residual blocks as a shortcut
BITS Pilani, Pilani Campus
BITS Pilani
Pilani Campus
Specialized CNN Architecture Patterns
Specialized CNN Architecture Patterns
Motivation
• Reducing computational cost without sacrificing accuracy
• Extracting richer features using novel convolutional operation
• Targeting specific tasks (e.g., real-time inference on mobile, or dense
prediction in segmentation).
Key Methods
• Depthwise Separable Convolutions (MobileNet)
• Group Convolutions (ResNeXt)
• Dilated/Atrous Convolutions (DeepLab)
BITS Pilani, Pilani Campus
Depthwise Separable Convolutions
(MobileNet)
Break a standard convolution into two steps:
Depthwise Convolution: Apply a separate 3×3 filter to each input channel.
Pointwise Convolution: Combine outputs across channels using 1×1 filters.
BITS Pilani, Pilani Campus
Depthwise Separable Convolutions
(MobileNet)
Standard Convolution
In a normal convolutional layer:
You have an input with shape Hin × Win × C_in (height, width, input channels).
You apply C_out filters (kernels), each of size K × K × C_in.
Each filter mixes all input channels and outputs one channel.
Computational cost =
𝐻 × 𝑊 × 𝐶𝑜𝑢𝑡 × 𝐾 × 𝐾 × 𝐶𝑖𝑛
Standard convolution is expensive because every filter looks at all channels at
once.
BITS Pilani, Pilani Campus
Depthwise Separable Convolutions
(MobileNet)
Instead of one big convolution, break it into two simpler operations:
(a) Depthwise Convolution
Each input channel is convolved separately with its own filter (of size K × K × 1).
So, if input has C_in channels, we apply C_in filters independently.
Output shape remains H × W × C_in.
Cost =
𝐻 × 𝑊 × 𝐾 × 𝐾 × 𝐶𝑖𝑛
This step extracts spatial features per channel, but does not combine
information across channels.
BITS Pilani, Pilani Campus
Depthwise Separable Convolutions
(MobileNet)
(b) Pointwise Convolution (1×1 conv)
Now, to combine information across channels, we apply a 1×1 convolution.
This means each pixel location (across all channels) is linearly combined.
We use C_out 1×1 filters of size 1 × 1 × C_in.
Cost =
𝐻 × 𝑊 × 𝐶𝑖𝑛 × 𝐶𝑜𝑢𝑡
This step mixes channel information.
BITS Pilani, Pilani Campus
Depthwise Separable Convolutions
(MobileNet)
Total Cost (Efficiency Gain)
Standard conv cost:
𝐻 × 𝑊 × 𝐾 2 × 𝐶𝑖𝑛 × 𝐶𝑜𝑢𝑡
Depthwise separable conv cost:
𝐻 × 𝑊 × (𝐾 2 × 𝐶𝑖𝑛 + 𝐶𝑖𝑛 × 𝐶𝑜𝑢𝑡 )
Reduction factor:
𝐾 2 ⋅ 𝐶𝑖𝑛 + 𝐶𝑖𝑛 ⋅ 𝐶𝑜𝑢𝑡
𝐾 2 ⋅ 𝐶𝑖𝑛 ⋅ 𝐶𝑜𝑢𝑡
BITS Pilani, Pilani Campus
Group Convolutions (ResNeXt)
Group Convolution Basics
• Instead of convolving across all input channels, the channels are split into
groups; each group is convolved separately, and their outputs are then
concatenated.
• Reduces the parameter count while still enabling hierarchical feature extraction.
ResNeXt Architecture
• Extends the ResNet “bottleneck” block by introducing grouped convolutions in
the 3×3 layer.
• Each group learns different feature representations in parallel (increased
“cardinality”).
• Improves accuracy vs. standard ResNet with a similar number of parameters.
BITS Pilani, Pilani Campus
Dilated (Atrous) Convolutions (DeepLab)
What is Dilation / Atrous Convolution?
• A convolution where the kernel is “spread out” by inserting gaps
(dilation rate > 1).
• Increases the receptive field without increasing the number of
parameters or kernel size.
BITS Pilani, Pilani Campus
BITS Pilani
Pilani Campus
Transfer learning
Transfer learning
• Over the course of the last years, we have obtained the ability to train more
and more accurate models.
• The latest residual networks on ImageNet achieve superhuman performance
at recognizing objects.
• Google's Smart Reply automatically handles 10% of all mobile responses
• Baidu can generate realistic sounding speech in real-time.
• However, these successful models are immensely data-hungry and rely on
huge amounts of labeled data to achieve their performance.
• Large amounts of labeled data are usually proprietary or expensive to obtain.
BITS Pilani, Pilani Campus
Transfer learning
Transfer learning and domain adaptation refer to the situation where what has
been learned in one setting … is exploited to improve generalization in
another setting
BITS Pilani, Pilani Campus
Transfer learning
Why Transfer Learning?
– Traditional deep learning models are extremely data-hungry.
– In many practical scenarios, labeled data is scarce or expensive.
Key Idea
– Leverage knowledge (features, weights) learned from large labeled “source” data to
improve performance on a smaller “target” dataset.
Real-World Impact
– Medical imaging (MRI, X-ray classification) where each label is costly to obtain.
– NLP tasks such as sentiment analysis or chatbots (fine-tuning massive language
models).
BITS Pilani, Pilani Campus
Transfer learning
Source:
BITS Pilani, Pilani Campus
Types of Transfer Learning
Inductive, Transductive, and Unsupervised
• Inductive: Source and target tasks differ, but there is plenty of labeled data
in source domain. Model “induces” general knowledge used in the target
task.
• Transductive: Source and target tasks are the same, but target data has no
labels; the model leverages structure in the unlabeled target domain.
• Unsupervised: No labeled data in either source or target, often focusing on
representation learning.
Domain Adaptation
When source and target distributions differ (e.g., photos → sketches), we adapt
learned representations to the new domain.
BITS Pilani, Pilani Campus
Practical Approaches & Implementation
Feature Extraction vs. Fine-Tuning
• Feature Extraction: Freeze early layers and only train the classifier
head on the target domain.
• Fine-Tuning: Retrain some or all of the network’s layers on the target
domain.
BITS Pilani, Pilani Campus
Practical Approaches & Implementation
Implementation Tips
• Layer Freezing Strategy: Start by freezing most layers, fine-tune top layers.
Gradually unfreeze if performance stalls.
• Learning Rate: Use lower learning rates for pretrained layers to avoid
“destroying” useful features.
BITS Pilani, Pilani Campus
BITS Pilani
Pilani Campus
Thank you!