ResNet Architectures
Session No.:2
Course NSession No.:24
Course Name: Deep Learning
Course Code: R1UC604C
Instructor Name: Dr. JAYAPRAKASH C
Review of the key concepts of session no. 23
• VGGNet showed the power of deep CNNs for
image recognition
• VGG16 & VGG19 are powerful pre-trained
models for transfer learning
• Simple, uniform design—computationally heavy
but conceptually clear
• Provide a strong baseline for comparing with
modern architectures
Ask Questions
How is Resnet ? Why Resnet needed
then AlexNet ?
At the end of this session students will be able to
Learning Outcome 1:
Describe ResNet and
Architecure
Learning Outcome 2:
To understand advantages
and key features of ResNet.
Session Outline
1 Residual Networks (ResNet) - Deep Learning
2 Introduction to ResNet
3 Key features
4 Activity 1
5 ResNet-34 Architecture
6 Activity 2
7 Conclusion
Residual Networks (ResNet) - Deep
Learning
• ResNet introduced by Microsoft Research in 2015
• Addressed vanishing/exploding gradient issues in deep networks
• Enabled very deep CNNs (up to 1000 layers) to train effectively
• ResNet (residual network) is a type of neural network
architecture that enables the training of very deep
networks by using "skip connections" to bypass
layers. This allows the network to learn residual
functions—the difference between the input and
output—making it easier to train and solving the
vanishing gradient problem common in deep
networks. Introduced in 2015, ResNet won the
ImageNet challenge and has been widely used for
tasks like image classification and object detection
Background: Increasing Network Depth
• After AlexNet (ImageNet 2012), deeper
networks were used to reduce error rates
• However, very deep networks suffered from
vanishing/exploding gradients
• Result: Increasing layers beyond a limit led to
higher training and testing errors
Problem: Vanishing/Exploding Gradient
• In deep neural networks, gradients can
become very small (vanish) or too large
(explode)
• This leads to unstable training and poor
convergence
• Example: 56-layer CNN performed worse than
a 20-layer CNN on ImageNet
Introduction to ResNet
• Proposed in 2015 by Kaiming He et al.
(Microsoft Research)
• Introduced a new architecture called Residual
Networks
• Key innovation: Residual Blocks using skip
(shortcut) connections
• Allows deeper networks to train effectively
without gradient issues
Key features
• Residual blocks: The core of ResNet is the residual block, which takes an
input, passes it through a few layers, and then adds the original input to
the output of those layers.
• Skip connections: These are the "shortcut" paths that allow the input to
be directly added to the output of a block. This enables gradients to flow
back to earlier layers more easily, preventing the vanishing gradient
problem.
• Learning the residual: Instead of learning the entire mapping from input
to output (\(Y=F(X)\)), ResNet learns the residual or the difference (\
(Y=F(X)+X\)). This is more efficient because much of the information from
the input is often already present in the output.
• Training very deep networks: By mitigating the vanishing gradient
problem, ResNet allows for the creation of networks with hundreds or
even thousands of layers, which was previously difficult or impossible.
Residual Blocks: Core Idea
• Instead of learning direct mapping H(x),
network learns residual mapping F(x):
• F(x) = H(x) - x (residual function)
• Final output: H(x) = F(x) + x
• This is achieved using skip connections that
bypass one or more layers
Skip (Shortcut) Connections
• Skip connections link earlier activations
directly to later layers
• If a layer harms performance, the skip
connection bypasses it
• This improves gradient flow and stabilizes
deep network training
• Allows training of networks with 100–1000
layers without vanishing gradients
Learning Activity 1:
Activity 1:
(pen and paper)
• Think–Pair–Share Activity
• Step 1 (Think):
Students individually write down one advantage and one disadvantage of Dropout
and Early Stopping each.
• Step 2 (Pair):
Share your answers with a partner. Discuss which regularization method seems
more suitable for a small dataset and why.
• Step 3 (Share):
A few pairs present their conclusions to the class.
Advantages of Skip Connections
• Improves gradient flow during
backpropagation
• Prevents degradation in very deep networks
• Faster convergence and better accuracy
• Supports flexible network depth (can scale
easily)
Related Architecture: Highway Networks
• Similar to ResNet but includes parametric
gates in skip paths
• Inspired by LSTMs – gates control how much
information passes through
• Highway Networks did not achieve better
accuracy than ResNet
ResNet-34 Architecture
• Based on a 34-layer plain network (inspired by
VGG-19)
• Shortcut connections convert this into a
residual network
• Each block consists of convolution, batch
normalization, ReLU, and skip addition
• Used as the baseline ResNet model
Schematic of ResNet-34. (a) The overall structure of
ResNet-34. It consists of a convolutional unit (Conv1) and
four residual blocks (Conv2_x, Conv3_x, Conv4_x and
Conv5_x), which generate the final probability vectors
and output the category labels by average pooling layer
and fully connected layer;
(b) Internal structure of the four residual blocks. Dashed
and solid lines between layers represented two different
short-circuit connection mechanisms of the residual units;
(c) The short-circuit connection mechanism (taking
Conv3_x block as an example) indicates that the number
of input and output channels are different, corresponding
with dashed lines in (b);
(d) The short-circuit connection mechanism (taking
Conv3_x block as an example) indicates that the number
of input and output channels are the same, corresponding
with solid lines in (b).
Implementation using TensorFlow & Keras
• Dataset: CIFAR-10 (60,000 color images, 32×32
pixels, 10 classes)
• Keras datasets API used to load CIFAR-10
• ResNet implemented by stacking multiple
residual blocks
• Can easily extend to deeper versions like
ResNet-50, ResNet-101, etc.
Impact and applications
• Image recognition: ResNet won the ImageNet Large
Scale Visual Recognition Challenge in 2015 and has
become a fundamental part of many computer vision
systems.
• Benchmark performance: It set a new standard for
performance on benchmark problems like ImageNet,
with models achieving substantial improvements over
previous methods.
• Versatility: Beyond image classification, ResNet is used in
other applications such as image super-resolution and
modeling physical systems.
Learning Activity 2:
Activity 2: (pen and paper)
• Think–Pair–Share Activity
• Step 1 (Think):
Students individually write down one advantage and one disadvantage of Dropout
and Early Stopping each.
• Step 2 (Pair):
Share your answers with a partner. Discuss which regularization method seems
more suitable for a small dataset and why.
• Step 3 (Share):
A few pairs present their conclusions to the class.
Summary and Conclusion
• ResNet revolutionized deep learning by
addressing vanishing gradients
• Residual connections make it easier to train
very deep models
• ResNet models (ResNet-34, 50, 101, 152)
remain foundational in vision tasks
• Inspired later architectures such as DenseNet,
EfficientNet, and Transformers
At the end of this session students will be able to
Learning Outcome 1:
Describe ResNet and
Architecure
Learning Outcome 2:
To understand advantages
and key features of ResNet.
• Post session activities
• Information to next topic of the
course
session no. 25
Transfer Learning Basics and Fine-
Tuning Strategies
Review and Reflection
from students
Galgotias University 30