2.

1 Introduction: Basics of Supervised Deep Learning

Deep Learning
 Deep learning is a type of artificial intelligence (AI) where computers learn from lots
of data.
 It lets computers solve tough problems that were previously very hard or impossible.

2. Why Is Deep Learning Becoming So Popular?

 Better Computers: Modern high-performance computers are much faster, so they can
process more data.
 More Data: We now have huge amounts of data available, some labeled (already
identified as, say, “cat” or “not cat”) and some unlabeled (not identified).
 Helpful Tools: There are open-source (free to use) software libraries that make deep
learning easier for everyone.

3. Types of Deep Learning

 There are two main types:
 Supervised Deep Learning: The computer learns from “labeled” data, where
the correct answer is given.
 Unsupervised Deep Learning: The computer learns by trying to find patterns
in “unlabeled” data (no correct answers provided).

4. What is Supervised Deep Learning? (Main Focus)

 In this approach, you teach a computer by showing it examples that are paired with
correct answers.
 Example: Show lots of labelled pictures of animals ("cat", "dog", etc.) so the
model learns to tell them apart.
 The computer “learns” which features in the data are important to make these
decisions.

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 5. Real-World Applications
 Supervised deep learning is used in:
 Recognizing objects in images (for things like self-driving cars)
 Speech recognition (like Siri or Alexa)
 Natural language processing (understanding human language)
 Medical image analysis (helping doctors find diseases)

6. Why Is Supervised Deep Learning So Powerful?

 It can solve complex problems and learn features that humans might not even notice.
 As computers, data, and tools get better, supervised deep learning is becoming more
common and successful.

7. A Key Model: Convolutional Neural Networks (CNNs/ConvNets)

 An especially popular model for supervised deep learning is the Convolutional Neural
Network (CNN or ConvNet).

2.2: Convolutional Neural Network (ConvNet/CNN)

1. What Is a Convolutional Neural Network (CNN)?
 A Convolutional Neural Network (CNN), also called ConvNet, is a type of deep
learning model made of several layers.
 CNNs are inspired by how the visual cortex in animal brains work: certain brain cells
(neurons) react to different patterns, like lines or edges.
 These models are especially powerful for tasks where recognizing patterns in data is
important, such as:
 Image classification (figuring out what’s in a picture)
 Object detection
 Speech and language recognition
 Medical image analysis

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 2. How Does a CNN Work?
Concept: CNNs are designed to automatically find and learn patterns (called “features”) in
the input (often an image), which get combined to identify higher-level concepts.
 Imagine squinting your eyes and seeing blurry shapes—those are like the first patterns
CNNs see.
 As you “open your eyes” (go deeper in the network), you see more detail: edges,
colors, textures, and finally the whole object (like a cat or a car).

3. Layers in a CNN
 Input Layer: Takes in the raw data (such as an image with pixels).
 Convolutional Layers: Use filters to scan across the image and pick up small features,
such as edges or colors.
 Pooling Layers: Shrink the image size by summarizing information, making the
network faster and less sensitive to slight changes.
 Fully Connected Layers: Combine the features detected to make predictions, like
deciding what’s in the image.

4. Why Are CNNs Powerful for Images?

 Each layer of a CNN learns more complex patterns:
 Early layers: notice simple things (edges, corners)
 Deeper layers: notice complex things (shapes, textures, objects)
 This step-by-step learning mimics how humans recognize objects: from simple shapes
to complex forms.

5. Why Do CNNs Need Lots of Computing Power?

 Because CNNs have many layers and millions of calculations, they are often trained
on special hardware called GPUs (Graphics Processing Units), which are much faster
than regular CPUs.
 Training a CNN on a large amount of data can take several days.

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 6. Example: Recognizing Handwritten Digits
Let’s say we want a computer to recognize handwritten numbers (like those on a postal
envelope). Here’s how a CNN helps:
1. Input: The computer receives a small black-and-white image of a single digit (for
example, the number “7”).
2. First Convolutional Layer: The CNN looks for simple patterns, like short straight
lines or curves.
3. Pooling Layer: It simplifies the picture, keeping only the most important features.
4. Deeper Layers: The CNN starts recognizing combinations of lines and shapes that
might mean “7” or “9.”
5. Output Layer: It decides which digit the picture most likely represents.

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 2.3: Evolution of Convolutional Neural Network Models
1. LeNet — The Beginning
 One of the first practical CNNs, made for reading handwritten numbers (like postal
codes on envelopes).
 Why it mattered: First to use backpropagation (a way of learning) to train the
network.
 Problems: Could not handle big/complex data due to small training sets, slow
computers, and a poor choice of activation function (sigmoid/tanh), which caused
problems with learning in deep networks ("vanishing gradients").
Example:
Imagine teaching a child to recognize handwritten numbers, but you only have a few
examples and the child is also learning very slowly. They will struggle on more complex or
varied handwriting.

2. AlexNet — The Breakthrough (2012)

 Achieved a massive leap in performance on a big image competition (ImageNet) by
reducing the error rate from 26% to 15%.
 Why it worked:
 Used a huge labeled dataset (ImageNet: 15 million images, 22,000 categories).
 Trained on much-faster GPUs, over 5–6 days.
 Used the ReLU activation function, which is faster and fixes the vanishing
gradient issue.
 Structure: 5 convolutional layers, 3 pooling layers, 3 fully connected layers, and a
1000-class output.
Example:
Now imagine giving the child millions of number pictures and a calculator to help speed up
learning—they do much better!

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 3. ZFNet — Tuning the Details (2013)
 Improved AlexNet by:
 Making the first layer's filter size smaller (from 11×11 to 7×7).
 Switching the stride to 2 (from 4), to get more local, detailed features.
 Impact: Won the 2013 image recognition challenge.

4. VGGNet — Going Deeper (2014)

 Increased network depth (up to 19 layers) mainly by adding more 3x3 convolutional
layers.
 Used 2x2 max pooling.
 Result: Gained better accuracy (7.32% error) due to its depth, though at the cost of
more computation.

5. GoogLeNet — Inception of Efficiency (2015)

 Went deeper (22 layers) but cleverly used a new structure called the “inception
module” to reduce the number of parameters.
 Won the 2015 image recognition contest with a 6.7% error rate.
 Key idea: Instead of just stacking layers, use several types of convolution and pooling
in parallel within an "inception module" and combine the results.

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 Fig 2.1: Inception Module in GoogLeNet
 This figure shows an inception module structure:
 Several convolutional operations happen at the same time: 1x1, 3x3, 5x5, and
pooling.
 All results (feature maps) are stacked together (concatenated), forming a
diverse set of features for the next stage.
 Why it matters: Makes the network both deep and efficient, learning complex features
without needing as many parameters.

6. ResNet — Residual Learning (2015)

 Proposed a super-deep network (152 layers!) but solved deep-learning training
problems with “residual connections.”
 Residual connections: The layer’s output is added to its input before passing to the
next layer, helping the flow of learning signals (gradients).
 Performance: Won the 2015 competition with a very low 3.6% error rate—better than
human performance!

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 Fig 2.2: Residual Connection in ResNet
 Shows how a "shortcut" or “skip connection” jumps over some layers. Instead of only
passing data forward layer to layer, it "adds" the input to the output after some
operations, which helps information and gradients flow better during training.

7. Other Innovations
 Inception-ResNet: Combines Inception modules with Residual connections for faster
and even more accurate training.
 Xception: Builds on Inception by using “depthwise separable convolutions” with
residuals.
 SqueezeNet: Focuses on reducing the model’s size (less memory) but keeps
performance high by using many small filters.
 ShuffleNet: For mobile devices; saves computational power using "pointwise group
convolution" and "channel shuffle".

8. Model Performance Comparison

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 Fig 2.3: ILSVRC Top-5 Error on ImageNet since 2010
 Fig 2.3 is a chart showing how the top-5 error rate (how often the correct answer
wasn’t in the model's top 5 guesses) has improved each year since 2010.
 Insight: As new architectures (AlexNet, VGG, GoogLeNet, ResNet, Xception, etc.)
were introduced, the error rate steadily dropped—showing big progress in how well
computers understand images.

2.4: Convolution Operation

1. What Is Convolution?
 Convolution is a mathematical operation used in many areas: image processing, signal
processing, statistics, probability, and deep learning.
 In deep learning, especially in CNNs, convolution is what allows computers to "look"
at parts of images and find patterns (like edges or textures).
 Think of convolution as sliding a small window over an image, doing simple math at
each position, and making a new, smaller "image" called a feature map.

2. Understanding the Convolution Formula

1D Convolution:

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

  f(m): The first function or sequence (e.g., image).
 g(n−m): The second function, but "shifted" by n.
 The ∑: Add up results for all possible mm.
Example (1D):
Suppose f= and g=[1,0,−1]:
Calculate at n=1:
(f∗g)(1)=f(0)⋅g(1)+f(1)⋅g(0)+f(2)⋅g(−1)
Assume missing values are 0 (common for images on edges).
So:
 f(0)⋅g(1)=2⋅0=0
 f(1)⋅g(0)=3⋅1=3
 f(2)⋅g(−1)=1⋅0=0 f(2)⋅g(−1)=1⋅0=0
Add them:
(f∗g)(1)=0+3+0=3

2D Convolution (Images):
For images, the formula is:

 A(m,n): Value of the input image at row mm, column nn.

 K(i−m,j−n): Value of the kernel (filter), shifted for the current position (i,j).

What does this do?

This slides the filter (kernel) over the image, multiplies the overlapping numbers, and adds
them up for each position. The result goes into a new matrix (feature map).

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 3. Fig 2.4: Convolution Operation
Let’s explain the process visually:
 You have:
 An image (input) represented as a grid of numbers (pixels).
 A small filter or kernel (for example, 3x3 grid) with its own set of numbers.

4. Cross-Correlation vs. Convolution

 Real CNNs often use cross-correlation rather than true mathematical convolution,
because it's more convenient in code.
 In cross-correlation, the kernel is not flipped. The formula looks like:

It's just a way to match patterns without flipping, but the effect for deep learning practical
purposes is similar.

5. Why Use Convolution in CNNs?

 Filters (kernels) can detect features in an image:
 Edges (if numbers in kernel are positive on one side, negative on the other)
 Textures
 CornersBy applying many different kernels, CNNs find many patterns to understand
what’s in the image.

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 2.5: Architecture of CNN
What is the CNN Architecture?
 Convolutional Neural Networks (CNNs) are special types of neural networks designed
to process data with a grid-like topology such as images, audio, and videos.
 The main idea: Instead of all neurons being connected to each other (fully connected),
CNNs use small filters to look at local parts of the data, finding patterns like edges,
corners, etc.
 CNNs have layers stacked in a sequence, each with a specific role.

Key Features of CNNs:

 Local Receptive Field: Each neuron in a convolutional layer connects only to a small
region of the input, called its "receptive field." This helps find simple patterns like lines
or edges in a local part of the image.
 Weight Sharing: The same filter (set of weights) is reused for the whole input. This
means the network searches for the same pattern everywhere in the image, saving
memory and computation.
 Subsampling (Pooling): The pooling layer reduces the size of the data, discarding less
important details but keeping the most important features. It helps in making the
network faster and prevents overfitting.

Important Layers in a CNN

1. Convolutional layer,
2. Activation function layer (ReLU),
3. Pooling layer,
4. Fully connected layer and
5. Dropout layer.

[Link] layer:
 Role: Extracts features from the input by sliding (convolving) small filters across the
image/inputs.
 Process: Each filter creates a feature map by multiplying and adding the filter values
to regions in the input. Each filter can learn to detect a specific feature (like a horizontal
edge).

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

  Hyperparameters:
 Filter Size: Typical sizes are 3×3, 5×5, up to 11×11.
 Number of Filters: Each filter detects a different feature. More filters mean
more features extracted.
 Stride: Steps size for moving the filter across the input (higher stride = smaller
output size).
 Zero Padding: Adds zeros to the input borders to control the output size or
keep edge information.
 Output calculation: For input size A, filter size K, padding P, and stride S:

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 2. Activation Function (ReLu):
Each activation function layer in a CNN applies a simple mathematical rule to the
outputs of the convolutional layer, transforming these values into the "activation map" that
continues through the network. The main purpose of an activation function is to introduce non-
linearity, allowing neural networks to solve complex tasks—not just simple linear mappings.
What Is an Activation Function?
 The activation function processes the "feature map" (the output of the convolution) and
creates an "activation map" as its output.
 It transforms the raw numbers from the feature map into a more manageable range,
usually between 0 and 1 or between −1 and 1 (this is called the squashing effect).
 The output specifies the neuron’s response to the input, deciding which features should
go forward in the network.
Why Is the Activation Function Important?
 Introduces non-linearity: Without activation functions, the neural network would
behave like a simple linear system, unable to model real-world complexities.
 Squashing effect: Keeps outputs in a manageable range, preventing numbers from
growing too large or too small for computers to handle.
 Enables gradient-based learning: As CNNs use backpropagation and gradients for
training, the activation function should generally be differentiable and continuous so
gradients can be calculated at every point.

There are many of activation functions in use with Artificial Neural Networks (ANNs) and
some of the commonly used activation functions are as follows:

Common Activation Functions

 Logistic/Sigmoid: Maps values to. Used in early neural networks but less common
now due to vanishing gradients.

 Tanh: Maps values to [−1, 1]. Similar to sigmoid but covers negative values too.

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

  ReLU (Rectified Linear Unit):
ReLU Activation Function: Rectified Linear Unit (ReLU) has gained some
importance in recent years and currently is the most popular activation function for deep
neural networks. Neural networks with ReLU train much faster than other. Most
common in deep learning. Returns 00 if the input is less than 00, otherwise returns the
input itself (making it simple, fast, and effective.
It is mathematically given as

 Softmax: Used in the last layer for classification, outputs a probability distribution.

 Swish: A newer function, combines properties of sigmoid and linear, helps training in
some advanced models.

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 3. Pooling:
 The pooling layer follows the convolution and activation function layers in a CNN and
is used mainly for down-sampling—reducing the spatial size (width, height) of the
feature maps.
 By shrinking the data, pooling reduces the number of parameters and computations,
helping the system run faster and preventing overfitting.
 The pooling layer works by summarizing a region of neurons in the convolution layer.

Main Pooling Methods:

 Max-Pooling: The most common technique; it takes the maximum value from each
small region (like 2×2) in the input, keeping only the strongest feature in that region.
 Average Pooling and L2-norm Pooling: Other less common methods, which compute
the average or the root mean square in each region.

For example, if input region is of size 2 × 2, the max-pooling unit will output the maximum
of the four values as shown in Fig. 2.10.

4. Fully Connected Layer:

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 Convolutional Neural Networks (CNNs) have a two-part structure: feature extraction
and classification. The first part, made of convolution and pooling layers, pulls out important
features from the input (like shapes and textures in images). The second part, made of fully
connected layers and a classifier, uses these features to make the final decision about which
class the input belongs to.
CNN Two-Stage Architecture
Feature Extraction Stage
 Convolution and pooling layers work together to identify patterns and details at
different levels.
 This stage reduces input complexity and creates feature maps—simplified
representations capturing essential information from the input (e.g., edges,
corners, motifs).
 The process continues as features are detected at increasing depth.
Classification Stage
 Fully connected layers appear after the feature extraction stage. Here, every
neuron in one layer connects to every neuron in the next layer, allowing the
network to mix and combine different detected features.
 One or more fully connected layers can be stacked to learn complex
relationships between features.
Final Classification Layer
 The output from the last fully connected layer is passed to a classifier which
assigns a class score.
 Softmax classifier: Produces probabilities for each class, ensuring all
probabilities add up to 1. Used for multiclass problems.
 Support Vector Machine (SVM) classifier: Assigns a score to each class, and
the class with the highest score is selected.
 This structure allows CNNs to accurately predict which class the input best
matches.

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C

 [Link]:

Deep neural networks have many hidden layers that enable them to learn complex features.
After these layers, fully connected layers perform the decision-making or classification
task. However, fully connected layers connect every neuron to all features, which makes
them prone to overfitting—when the model learns the training data too well but does not
perform well on new, unseen data.
 To combat overfitting, a dropout layer can be used during training. In dropout:
 Some neurons and their connections are randomly dropped out (disabled) in each
training iteration.
 This creates a reduced network that is trained for that iteration.
 Incoming and outgoing edges to the dropped neurons are also removed temporarily.
 After training, the dropped neurons are reinserted with their original weights.
 This process forces the network to learn more robust features that do not rely on
any specific neuron, thereby significantly reducing overfitting and improving the
model's generalization to new data

Prepared By: Harish Kumar M Approved By: Dr Kantharaju H C