UNIT – II

Convolutional Neural Networks (CNN),

Recurrent Neural Networks (RNN), Advanced
RNN Architectures
Complete Exam-Oriented Notes

TABLE OF CONTENTS
1. Introduction to Deep Learning
2. Convolutional Neural Networks (CNN)
3. Basics of CNN
4. Building Blocks of CNN
5. Convolution Operation
6. Padding and Stride
7. Pooling Layers
8. Activation Functions
9. Backpropagation in CNN
10. Dropout Layers
11. Optimizers
◦ Momentum
◦ RMSProp
◦ Adam
12. CNN Architectures
◦ LeNet
◦ AlexNet
◦ VGG16
◦ ResNet
13. Transfer Learning
14. R-CNN and Types
15. Skip Connection Networks
16. Fully Connected CNNs
17. Recurrent Neural Networks (RNN)
18. Bidirectional RNN
19. Encoder–Decoder Models
20. Backpropagation Through Time (BPTT)
21. Applications of RNN
22. Advanced RNN Architectures
◦ LSTM
◦ GRU
◦ Seq2Seq
◦ Attention Mechanism
23. Important Formulae

1
 24. Frequently Asked Questions
25. Long Answers for 20-Mark Questions
26. Unit Summary
27. Previous Year Question Style Answers

1. INTRODUCTION TO DEEP LEARNING

Definition
Deep Learning is a subset of Machine Learning that uses multi-layered neural networks to
automatically learn features from data.

It is inspired by the working of the human brain.

Characteristics of Deep Learning

• Uses artificial neural networks
• Learns automatically from data
• Requires large datasets
• Performs well on image, speech, and text data
• Uses multiple hidden layers

Applications

Area Application

Computer Vision Face Recognition

NLP Chatbots

Healthcare Disease Detection

Autonomous Cars Object Detection

Finance Fraud Detection

2. CONVOLUTIONAL NEURAL NETWORKS (CNN)

Definition
A Convolutional Neural Network (CNN) is a type of deep neural network specially designed for image
processing and computer vision tasks.

2
CNN automatically extracts important features from images.

Why CNN is Needed?

Traditional neural networks:

• Require huge parameters

• Cannot efficiently process images
• Ignore spatial relationships

CNN solves these problems using:

• Convolution layers
• Weight sharing
• Local connectivity

Key Features of CNN

• Automatic feature extraction
• Translation invariance
• Parameter sharing
• High accuracy in image tasks

Applications of CNN
Application Description

Image Classification Identify objects in image

Face Recognition Detect human faces

Medical Imaging Tumor detection

Self-driving Cars Lane and object detection

OCR Handwritten character recognition

CNN ARCHITECTURE OVERVIEW

Input Image
↓
Convolution Layer
↓

3
 Activation Function
↓
Pooling Layer
↓
Fully Connected Layer
↓
Output Layer

3. BASICS OF CNN
Input Image
Images are represented as matrices of pixel values.

Example:

• Grayscale image → 2D matrix

• RGB image → 3D matrix

Example:

28 × 28 × 1 → Grayscale
224 × 224 × 3 → RGB

Feature Maps
Feature maps are outputs generated after applying filters.

These maps detect:

• Edges
• Shapes
• Textures
• Patterns

Filters/Kernels
A filter is a small matrix used to scan an image.

Example:

4
 3 × 3 filter

Purpose:

• Edge detection
• Blur detection
• Pattern extraction

4. BUILDING BLOCKS OF CNN

Major Components
1. Convolution Layer
2. Activation Function
3. Pooling Layer
4. Fully Connected Layer
5. Output Layer

5. CONVOLUTION OPERATION

Definition
Convolution is the mathematical operation where a filter slides over an input image to produce feature
maps.

Mathematical Formula
The convolution operation is:

genui{"math_block_widget_always_prefetch_v2":{"content":"(I*K)(i,j)=\sum_m\sum_n I(i-m,j-
n)K(m,n)"}}

Where:

• I = Input image
• K = Kernel/filter
• i,j = Pixel positions

5
Step-by-Step Convolution Example
Input:

1 1 1 0 0
0 1 1 1 0
0 0 1 1 1
0 0 1 1 0
0 1 1 0 0

Kernel:

1 0 1
0 1 0
1 0 1

Steps: 1. Place kernel on image 2. Multiply corresponding elements 3. Add results 4. Move kernel 5.
Repeat

Advantages of Convolution
• Reduces parameters
• Preserves spatial information
• Efficient feature extraction

6. PADDING AND STRIDE

Padding
Padding means adding extra pixels around image borders.

Types of Padding

Type Description

Valid Padding No padding

Same Padding Output size same as input

6
Why Padding is Needed?
• Prevents shrinking of image
• Preserves edge information

Stride
Stride is the number of steps the filter moves.

Example:

• Stride = 1 → Move one pixel

• Stride = 2 → Move two pixels

Output Size Formula

genui{"math_block_widget_always_prefetch_v2":{"content":"Output=\frac{(N-F+2P)}{S}+1"}}

Where:

• N = Input size
• F = Filter size
• P = Padding
• S = Stride

Solved Numerical Example

Input size = 32 Filter size = 3 Padding = 1 Stride = 1

Output:

(32 − 3 + 2×1)/1 + 1
= 32

7. POOLING LAYERS

Definition
Pooling reduces feature map size while preserving important information.

7
Types of Pooling
Pooling Type Description

Max Pooling Select maximum value

Average Pooling Select average value

Max Pooling Example

Input:

1 3 2 4
5 6 1 2
7 2 8 1
0 1 2 3

2×2 Max Pooling Output:

6 4
7 8

Advantages of Pooling
• Reduces computation
• Prevents overfitting
• Improves generalization

Disadvantages
• Information loss
• Reduced localization accuracy

8
8. ACTIVATION FUNCTIONS

Definition
Activation functions introduce non-linearity into neural networks.

Types of Activation Functions

1. Sigmoid Function
genui{"math_block_widget_always_prefetch_v2":{"content":"\sigma(x)=\frac{1}{1+e^{-x}}"}}

Advantages

• Smooth output
• Used in binary classification

Disadvantages

• Vanishing gradient problem

2. Tanh Function
genui{"math_block_widget_always_prefetch_v2":{"content":"tanh(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}"}}


Range:

−1 to +1

3. ReLU Function
genui{"math_block_widget_always_prefetch_v2":{"content":"f(x)=max(0,x)"}}

Advantages

• Fast computation
• Removes vanishing gradient

Disadvantages

• Dying ReLU problem

9
4. Softmax Function
Used in multi-class classification.

genui{"math_block_widget_always_prefetch_v2":{"content":"Softmax(x_i)=\frac{e^{x_i}}{\sum_j
e^{x_j}}"}}

Comparison Table
Function Range Use

Sigmoid 0 to 1 Binary classification

Tanh -1 to 1 Hidden layers

ReLU 0 to ∞ Deep networks

Softmax Probabilities Multi-class output

9. BACKPROPAGATION IN CNN

Definition
Backpropagation is the process of updating weights using gradients to minimize error.

Steps in Backpropagation
1. Forward propagation
2. Calculate loss
3. Compute gradients
4. Update weights
5. Repeat

Loss Function
Common loss function:

genui{"math_block_widget_always_prefetch_v2":{"content":"Loss=\frac{1}{n}\sum (y-\hat{y})^2"}}

10
Gradient Descent Weight Update
genui{"math_block_widget_always_prefetch_v2":{"content":"w_{new}=w_{old}-\eta \frac{\partial L}
{\partial w}"}}

Where:

• η = Learning rate
• L = Loss function

Advantages
• Reduces error
• Improves accuracy

Limitations
• Slow training
• Vanishing gradients

10. DROPOUT LAYERS

Definition
Dropout is a regularization technique where some neurons are randomly deactivated during training.

Purpose
• Prevent overfitting
• Improve generalization

Working
Example:

Dropout rate = 0.5

11
Means 50% neurons are ignored during training.

Advantages
• Reduces overfitting
• Better performance

Disadvantages
• Slower convergence

11. OPTIMIZERS

Definition
Optimizers update neural network weights to minimize loss.

11.1 MOMENTUM OPTIMIZER

Definition
Momentum accelerates gradient descent using past gradients.

Formula
genui{"math_block_widget_always_prefetch_v2":{"content":"v_t=\gamma v_{t-1}+\eta \nabla
J(\theta)"}}

Advantages
• Faster convergence
• Reduces oscillation

12
Disadvantages
• Requires tuning

11.2 RMSProp

Definition
RMSProp adjusts learning rate adaptively.

Formula
genui{"math_block_widget_always_prefetch_v2":{"content":"E[g^2]t=\beta E[g^2]}+(1-\beta)g_t^2"}

Advantages
• Good for non-stationary problems
• Faster learning

11.3 ADAM OPTIMIZER

Definition
Adam combines Momentum and RMSProp.

Formula
genui{"math_block_widget_always_prefetch_v2":{"content":"\theta_t=\theta_{t-1}-\eta \frac{m_t}
{\sqrt{v_t}+\epsilon}"}}

Advantages
• Fast convergence
• Adaptive learning
• Most widely used

13
Comparison of Optimizers
Optimizer Speed Memory Performance

SGD Slow Low Moderate

Momentum Faster Medium Good

RMSProp Fast Medium Better

Adam Very Fast Higher Excellent

12. CNN ARCHITECTURES

12.1 LENET

Introduction
LeNet was developed by Yann LeCun for handwritten digit recognition.

Architecture

Input → Conv → Pool → Conv → Pool → FC → Output

Features
• Simple CNN
• Used for MNIST dataset

Advantages
• Lightweight
• Easy to implement

14
Limitations
• Not suitable for large datasets

12.2 ALEXNET

Introduction
AlexNet won the ImageNet competition in 2012.

Features
• Deep CNN
• Uses ReLU
• Uses Dropout
• Uses GPU training

Architecture Overview

Input → Conv → Pool → Conv → Pool → FC → Softmax

Advantages
• High accuracy
• Reduced training time

Limitations
• Large computational cost

15
12.3 VGG16

Introduction
VGG16 uses very small filters (3×3).

Features
• 16 layers
• Deep architecture
• Uniform design

Advantages
• High feature extraction capability

Disadvantages
• Very large parameters
• High memory usage

12.4 RESNET

Introduction
ResNet introduced skip connections.

Main Idea
Instead of learning direct mapping:

H(x)

ResNet learns:

16
 F(x) = H(x) − x

Skip Connection Diagram

Input x
↓
Conv Layer
↓
Conv Layer
↓
Add x
↓
Output

Advantages
• Solves vanishing gradient problem
• Allows very deep networks

Comparison of Architectures
Architecture Year Key Feature

LeNet 1998 Basic CNN

AlexNet 2012 Deep CNN + ReLU

VGG16 2014 Small filters

ResNet 2015 Skip connections

13. TRANSFER LEARNING

Definition
Transfer learning uses a pre-trained model on a new related task.

17
Why Transfer Learning?
• Saves training time
• Requires less data
• Improves performance

Techniques
1. Feature Extraction
Use pre-trained CNN as feature extractor.

2. Fine Tuning
Train some layers on new dataset.

Workflow

Pre-trained Model
↓
Remove Output Layer
↓
Add New Layer
↓
Train on New Data

Use Cases with Image Data

Use Case Example

Medical Imaging Tumor classification

Agriculture Plant disease detection

Security Face recognition

Retail Product classification

18
Advantages
• Less computational cost
• Better accuracy

Disadvantages
• Domain mismatch issues

14. R-CNN AND TYPES

Definition
R-CNN stands for Region-based Convolutional Neural Network.

It is used for object detection.

Working of R-CNN
1. Generate region proposals
2. Extract features using CNN
3. Classify regions

Types of R-CNN
1. R-CNN
• Slow training
• Separate stages

2. Fast R-CNN
• Single CNN for image
• Faster than R-CNN

19
3. Faster R-CNN
• Uses Region Proposal Network (RPN)
• Very fast

4. Mask R-CNN
• Adds image segmentation
• Produces masks

Comparison Table
Model Speed Accuracy

R-CNN Slow Good

Fast R-CNN Faster Better

Faster R-CNN Very Fast High

Mask R-CNN Moderate Very High

Applications
• Object detection
• Medical image segmentation
• Autonomous vehicles

15. SKIP CONNECTION NETWORKS

Definition
Skip connections allow information to bypass layers.

Advantages
• Reduces vanishing gradients
• Faster training
• Better accuracy

20
Example
ResNet

16. FULLY CONNECTED CNNs

Definition
Fully Convolutional Networks (FCNs) replace fully connected layers with convolution layers.

Applications
• Semantic segmentation
• Pixel-level classification

Advantages
• Accept variable image sizes
• Efficient computation

17. RECURRENT NEURAL NETWORKS (RNN)

Definition
RNN is a neural network designed for sequential data.

It remembers previous information using hidden states.

Need for RNN

Traditional neural networks cannot handle sequence dependencies.

Examples:

• Language
• Speech
• Time series

21
RNN Architecture

x1 → h1 → y1
↓
x2 → h2 → y2
↓
x3 → h3 → y3

Mathematical Representation
genui{"math_block_widget_always_prefetch_v2":{"content":"h_t=f(Wx_t+Uh_{t-1})"}}

Advantages
• Handles sequence data
• Maintains memory

Disadvantages
• Vanishing gradients
• Difficult long-term learning

Applications
Application Example

NLP Language translation

Speech Speech recognition

Finance Stock prediction

Healthcare ECG analysis

22
18. BIDIRECTIONAL RNN

Definition
Bidirectional RNN processes sequence in both directions.

Working
• Forward RNN
• Backward RNN
• Combined output

Advantages
• Better context understanding
• Higher accuracy

Applications
• NLP
• Speech recognition

19. ENCODER–DECODER MODELS

Definition
Encoder-decoder architecture converts one sequence into another sequence.

Working
Encoder
Converts input into context vector.

Decoder
Generates output sequence.

23
Diagram

Input Sentence
↓
Encoder
↓
Context Vector
↓
Decoder
↓
Output Sentence

Applications
• Machine translation
• Chatbots
• Summarization

20. BACKPROPAGATION THROUGH TIME (BPTT)

Definition
BPTT is the training algorithm for RNNs.

Steps
1. Unroll RNN
2. Forward propagation
3. Compute loss
4. Backward propagation through time
5. Update weights

Problems in BPTT
1. Vanishing Gradient
Gradients become very small.

24
2. Exploding Gradient
Gradients become very large.

Solutions
• Gradient clipping
• LSTM
• GRU

21. APPLICATIONS OF RNN

Field Application

NLP Translation

Chatbots Response generation

Speech Voice assistants

Finance Stock forecasting

Healthcare Disease prediction

22. ADVANCED RNN ARCHITECTURES

22.1 LONG SHORT-TERM MEMORY (LSTM)

Definition
LSTM is a special RNN designed to solve long-term dependency problems.

Components of LSTM
1. Forget Gate
2. Input Gate
3. Output Gate
4. Cell State

25
LSTM Structure

Input → Forget Gate → Input Gate → Cell State → Output Gate

Forget Gate Formula

genui{"math_block_widget_always_prefetch_v2":{"content":"f_t=\sigma(W_f[h_{t-1},x_t]+b_f)"}}

Input Gate Formula

genui{"math_block_widget_always_prefetch_v2":{"content":"i_t=\sigma(W_i[h_{t-1},x_t]+b_i)"}}

Output Gate Formula

genui{"math_block_widget_always_prefetch_v2":{"content":"o_t=\sigma(W_o[h_{t-1},x_t]+b_o)"}}

Advantages
• Learns long-term dependencies
• Solves vanishing gradient problem

Disadvantages
• Complex architecture
• High computation

Applications
• Speech recognition
• Translation
• Time-series forecasting

26
22.2 GRU (GATED RECURRENT UNIT)

Definition
GRU is a simplified version of LSTM.

Gates in GRU
1. Reset Gate
2. Update Gate

Advantages
• Faster than LSTM
• Fewer parameters

Disadvantages
• Slightly less expressive

Comparison: LSTM vs GRU

Feature LSTM GRU

Gates 3 2

Complexity High Low

Speed Slower Faster

Parameters More Less

22.3 SEQ2SEQ MODELS

Definition
Seq2Seq models convert one sequence into another.

27
Components
1. Encoder
2. Decoder

Applications
• Translation
• Text summarization
• Chatbots

22.4 ATTENTION MECHANISM

Definition
Attention allows the model to focus on important parts of input.

Why Attention is Needed?

Context vector in Seq2Seq cannot store all information.

Attention solves this issue.

Working
1. Compute attention scores
2. Assign weights
3. Generate weighted output

Advantages
• Better translation
• Handles long sequences

28
Applications
• Transformers
• NLP
• Image captioning

MEMORY TRICKS
CNN Memory Trick

C-P-F-O

Meaning:

Convolution → Pooling → Fully Connected → Output

LSTM Gates Memory Trick

FIO

Meaning:

Forget → Input → Output

IMPORTANT DIFFERENCES

CNN vs RNN
Feature CNN RNN

Data Type Images Sequential Data

Memory No Yes

Main Use Vision NLP

Architecture Feedforward Recurrent

29
Transfer Learning vs Training from Scratch
Feature Transfer Learning Training from Scratch

Data Required Less More

Training Time Low High

Accuracy High Depends

IMPORTANT FORMULAE
CNN Formulae
1. Output size:

(N − F + 2P)/S + 1

1. ReLU:

f(x)=max(0,x)

1. Sigmoid:

1/(1+e−x)

RNN Formulae
1. Hidden state:

ht=f(Wxt+Uht−1)

1. Softmax:

e^xi / Σe^xj

30
FREQUENTLY ASKED EXAM QUESTIONS
Short Questions
1. Define CNN.
2. What is pooling?
3. Explain dropout.
4. What is transfer learning?
5. Define RNN.
6. What is BPTT?
7. Difference between LSTM and GRU.
8. Explain attention mechanism.

Long Questions
1. Explain CNN architecture with diagram.
2. Discuss AlexNet, VGG16, and ResNet.
3. Explain transfer learning techniques.
4. Explain R-CNN and its types.
5. Explain RNN architecture and applications.
6. Explain LSTM with gates and equations.
7. Explain Seq2Seq and attention mechanism.

20-MARK EXAM ORIENTED ANSWERS

Question 1
Explain CNN architecture with suitable diagram.

Answer Structure

Introduction

CNN is a deep learning architecture designed for image processing.

Main Components

1. Input Layer
2. Convolution Layer
3. Activation Layer
4. Pooling Layer
5. Fully Connected Layer
6. Output Layer

31
Diagram

Input → Conv → ReLU → Pool → FC → Output

Advantages

• Automatic feature extraction

• High image accuracy

Applications

• Face recognition
• Medical imaging

Conclusion

CNN is widely used in computer vision because of its efficiency and accuracy.

Question 2
Explain LSTM architecture with gates.

Introduction

LSTM solves long-term dependency issues in RNN.

Gates

1. Forget Gate
2. Input Gate
3. Output Gate

Working

• Forget irrelevant information

• Store useful information
• Produce output

Advantages

• Handles long sequences

• Better memory

Applications

• Translation
• Speech recognition

32
Conclusion

LSTM is highly effective for sequence learning.

PREVIOUS YEAR QUESTION STYLE ANSWERS

PYQ 1
What is Transfer Learning?

Answer

Transfer learning is a deep learning technique in which a model trained on one task is reused for
another related task.

Advantages

• Less data required

• Faster training
• Better accuracy

Example

Using ResNet trained on ImageNet for medical image classification.

PYQ 2
Explain vanishing gradient problem.

Answer

During backpropagation in deep networks, gradients become very small. Due to this, weights stop
updating effectively.

Effects

• Slow learning
• Poor performance

Solutions

• ReLU
• LSTM
• Residual Networks

33
UNIT SUMMARY
CNN Summary
• CNN is used for image processing.
• Main layers are convolution, pooling, and fully connected layers.
• Popular architectures include LeNet, AlexNet, VGG16, and ResNet.

RNN Summary
• RNN handles sequential data.
• Uses hidden states for memory.
• LSTM and GRU solve long-term dependency issues.

Attention Summary
• Attention improves Seq2Seq models.
• Focuses on important information.
• Basis of Transformers.

MOST IMPORTANT EXAM QUESTIONS

1. Explain convolution operation with example.
2. Discuss CNN building blocks.
3. Explain dropout and optimizers.
4. Compare AlexNet, VGG16, and ResNet.
5. Explain transfer learning.
6. Explain R-CNN and its variants.
7. Explain RNN architecture.
8. Explain BPTT.
9. Explain LSTM with diagram.
10. Compare LSTM and GRU.
11. Explain Seq2Seq model.
12. Explain attention mechanism.

QUICK REVISION NOTES

CNN

CNN = Image Processing

34
RNN

RNN = Sequential Data

LSTM

LSTM = Long-Term Memory

Attention

Attention = Focus on Important Inputs

FINAL CONCLUSION
Convolutional Neural Networks and Recurrent Neural Networks are the backbone of modern deep
learning applications.

CNNs are mainly used for image processing tasks, while RNNs and their advanced variants such as
LSTM and GRU are designed for sequential data.

Transfer learning, attention mechanisms, and advanced architectures like ResNet have significantly
improved the performance of deep learning systems in real-world applications.

These concepts are highly important for university examinations, research, and industrial applications.

35