DEEP LEARNING(MSDS202)

MODULE-3
Optimization for Training Deep Models: How Learning Differs from Pure
Optimization, Challenges in Neural Network Optimization, Basic
Algorithms. Parameter Initialization Strategies, Algorithms with Adaptive
Learning Rates. Convolutional Networks: The Convolution Operation,
Motivation, Pooling, Convolution and Pooling as an Infinitely Strong Prior,
Variants of the Basic Convolution Function, Structured Outputs, Data
Types, Efficient Convolution Algorithms, Random or Unsupervised
Features.
UNIT 1: Optimization for Training Deep Models
1. Introduction
Training a Deep Neural Network (DNN) means finding the optimal values of weights and
biases that minimize the error (loss function).
Optimization is the process of adjusting the parameters using optimization algorithms to
reduce the loss and improve prediction accuracy.

Definition
Optimization is the process of minimizing or maximizing an objective (loss) function by
updating the model parameters iteratively.
Mathematically,
[
\theta^*=\arg\min_{\theta}L(\theta)
]
Where
 θ = Model parameters (weights & biases)
 L(θ) = Loss function

Objectives of Optimization
 Minimize training error
 Improve prediction accuracy
 Faster convergence
 Avoid overfitting
 Achieve global minimum
 Reduce computational cost

Diagram
Input Data
│
Forward Propagation
│
Loss Function
 │
Backpropagation
│
Optimization Algorithm
│
Update Weights
│
Repeat until convergence

2. How Learning Differs from Pure Optimization

Definition
Pure optimization focuses only on minimizing the objective function.
Machine learning focuses on minimizing training loss and improving performance on unseen
data (generalization).

Comparison Table
Pure Optimization Machine Learning
Goal is minimum cost Goal is good prediction
Uses complete dataset Uses training samples
Ignores generalization Generalization is important
Only optimization Optimization + Learning
Global minimum preferred Better generalization preferred
Mathematical objective Statistical objective

Key Differences
Pure Optimization
 Mathematical problem
 No training/testing concept
 Only objective function matters
Deep Learning
 Learns patterns
 Works with unseen data
 Generalization is important
 Uses regularization techniques

Why Learning is Different?

Because neural networks must
 Learn hidden representations
 Avoid overfitting
 Generalize well
 Handle noisy data
 Deal with high-dimensional parameter space
Advantages
 Better prediction
 Learns automatically
 Handles complex data

Disadvantages
 Computationally expensive
 Requires large datasets
 Difficult optimization

VTU Important Points

Definition
Comparison
Need for learning
Applications

3. Challenges in Neural Network Optimization

Neural networks contain millions of parameters, making optimization difficult.

Major Challenges
1. Non-Convex Optimization
Neural network loss functions are highly non-convex.
Loss

/\__
/ \___
__/ \___

Global Minimum
Problems
 Multiple minima
 Difficult convergence

2. Local Minima
Optimizer may stop at a local minimum instead of the global minimum.

3. Saddle Points
Gradient becomes zero but is not minimum.
/
----O-----
\
Very common in deep learning.

4. Vanishing Gradient
During backpropagation,
Gradient becomes very small.
Effects
 Slow learning
 Early layers stop learning
Occurs in
 Sigmoid
 Tanh

5. Exploding Gradient
Gradient becomes extremely large.
Effects
 Weight overflow
 Training instability

6. Ill-Conditioning
Different parameters learn at different speeds.
Leads to
 Zig-zag optimization
 Slow convergence

7. Overfitting
Model memorizes training data.
Solution
 Dropout
 Regularization
 Early stopping

8. High Computational Cost

Deep models require
 GPU
 Large memory
 Long training time

9. Hyperparameter Selection
Choosing
 Learning rate
 Batch size
 Epochs
 Optimizer
is difficult.

Summary Diagram
Optimization Challenges
 |
------------------------------------
| | | | |
Local Saddle Vanish Explode Overfit
Minimum Point Gradient Gradient

Advantages of Good Optimization

 Faster convergence
 Better accuracy
 Stable learning
 Better generalization

4. Basic Optimization Algorithms

Optimization algorithms update weights.
General Formula
[
W=W-\eta\frac{\partial L}{\partial W}
]
Where
η = Learning Rate

A. Batch Gradient Descent

Uses the entire dataset.
Algorithm
1. Initialize weights
2. Compute gradient
3. Update weights
4. Repeat
Advantages
 Stable convergence
Disadvantages
 Slow
 High memory

B. Stochastic Gradient Descent (SGD)

Uses one sample at a time.
Advantages
 Fast
 Less memory
Disadvantages
 Noisy updates

C. Mini-Batch Gradient Descent

Uses a small batch.
Example
Batch size = 32
Advantages
 Fast
 Stable
 Most widely used

Comparison
Batch SGD Mini Batch
Whole dataset One sample Small batch
Slow Fast Moderate
Stable Noisy Stable
High memory Low memory Medium

Flowchart
Training Data
│
Compute Gradient
│
Update Weight
│
Repeat

5. Parameter Initialization Strategies

Definition
Parameter initialization is assigning initial values to weights and biases before training.
Proper initialization improves convergence.

Importance
 Faster training
 Prevent vanishing gradient
 Prevent exploding gradient
 Stable optimization

Types
1. Zero Initialization
All weights = 0
Disadvantage
All neurons learn the same thing.
Not recommended.

2. Random Initialization
Small random numbers.
Advantages
Breaks symmetry.

3. Xavier Initialization
Suitable for
 Sigmoid
 Tanh
Formula
[
W\sim\sqrt{\frac{1}{n}}
]
Advantages
Maintains variance.

4. He Initialization
Used with ReLU.
Formula
[
W\sim\sqrt{\frac{2}{n}}
]
Advantages
Prevents vanishing gradients.

5. Orthogonal Initialization
Uses orthogonal matrices.
Suitable for
Deep networks
RNN

Comparison
Initialization Activation
Zero Not recommended
Random General
Xavier Sigmoid/Tanh
He ReLU
Orthogonal Deep Networks

Diagram
Initialize Parameters
│
Random / Xavier / He
│
Start Training

6. Algorithms with Adaptive Learning Rates

Definition
Adaptive learning rate algorithms automatically adjust the learning rate during training.
Advantages
 Faster convergence
 Better optimization
 Less manual tuning

A. AdaGrad
Working
Learning rate decreases over time.
Formula
[
\theta=\theta-\frac{\eta}{\sqrt{G+\epsilon}}g
]
Advantages
Good for sparse data.
Disadvantages
Learning rate becomes too small.

B. RMSProp
Improves AdaGrad.
Uses moving average of gradients.
Advantages
Fast convergence.
Suitable for
Deep neural networks.

C. AdaDelta
Improves RMSProp.
No manual learning rate needed.
Advantages
Stable updates.

D. Adam (Adaptive Moment Estimation)

Most popular optimizer.
Combines
 Momentum
 RMSProp
Formula
Uses
First Moment
Second Moment
Advantages
 Fast
 Accurate
 Stable
 Works for large datasets
Disadvantages
Requires more memory.

E. AdamW
Improved Adam with proper weight decay.
Advantages
 Better regularization
 Better generalization

Comparison Table
Optimizer Learning Rate Speed Memory Performance
SGD Fixed Medium Low Good
AdaGrad Adaptive Slow later Low Sparse Data
RMSProp Adaptive Fast Medium Excellent
Adam Adaptive Very Fast Medium Best Overall
AdamW Adaptive Very Fast Medium Better Regularization

Flowchart
Initialize Parameters
│
Forward Pass
│
Loss Calculation
│
Backpropagation
│
Adaptive Optimizer
│
Update Weights
│
Repeat

Advantages of Adaptive Learning Rate Algorithms

 Automatic learning rate adjustment
 Faster convergence
 Better optimization
 Suitable for deep learning
  Less manual tuning

Disadvantages
 Higher memory requirement
 More computation
 Hyperparameter tuning may still be needed
 Some optimizers may generalize worse than SGD in certain tasks

Applications
 Image Classification
 Natural Language Processing (NLP)
 Speech Recognition
 Recommendation Systems
 Medical Image Analysis
 Autonomous Vehicles
 Financial Forecasting
 VTU [Link] (Data Science / AI & ML) Notes
 Module: Convolutional Networks (CNN)
 Prepared as per VTU [Link] CBCS Syllabus (10 & 15 Marks Exam Answers)


 MODULE: CONVOLUTIONAL NETWORKS

 Introduction
 Convolutional Neural Networks (CNNs) are a class of Deep Learning models
specifically designed to process grid-like data, such as images, videos, and audio
spectrograms. CNNs automatically learn spatial features such as edges, textures,
shapes, and objects without manual feature extraction.
 CNNs are widely used in computer vision, medical imaging, autonomous vehicles,
facial recognition, and natural language processing (NLP).


 Architecture of CNN
 Input Image
 │
 Convolution Layer
 │
 Activation (ReLU)
 │
 Pooling Layer
 │
 Convolution Layer
 │
 Pooling Layer
 │
 Flatten Layer
 │
 Fully Connected Layer
 │
 Output Layer


 Advantages of CNN
 Automatic feature extraction
 High classification accuracy
 Parameter sharing reduces complexity
 Translation invariance
 Less preprocessing required
 Suitable for large-scale image datasets


 Disadvantages
 Requires large training datasets
 Computationally expensive
 Needs GPUs for efficient training
 Difficult to interpret learned features


 1. The Convolution Operation

 Definition
 The convolution operation is the fundamental building block of CNNs. It applies a
small filter (kernel) over the input image to extract local features such as edges,
corners, and textures.


 Mathematical Formula
 For an input image (I) and kernel (K),
 [
S(i,j)=\sum_m\sum_n I(i+m,j+n)\times K(m,n)
]
 Where:
 (I) = Input image
 (K) = Kernel (Filter)
 (S) = Feature map


 Working of Convolution
 Input image is given.
 A filter slides over the image.
 Element-wise multiplication is performed.
 Values are summed.
 A feature map is generated.


 Example
 Input Image (4 × 4)
 1231
 4562
 7893
 1234
 Kernel (2 × 2)
 10
 01
 The filter moves across the image and produces the output feature map.


 Diagram
 Input Image
 +-------------+
 | |
 | Kernel |
 | ↓ |
 +-------------+
 │
 Convolution
 │
 Feature Map


 Advantages
 Feature extraction
 Parameter sharing
 Sparse connectivity
 Efficient computation


 Applications
 Image recognition
 Face detection
 Object detection
 Medical diagnosis

 2. Motivation Behind CNN
 Why CNN?
 Traditional Neural Networks suffer from:
 Huge number of parameters
 High memory consumption
 Overfitting
 Poor scalability for images
 CNN overcomes these issues by using:
 Local receptive fields
 Shared weights
 Pooling operations


 Motivation
 Reduce parameters
 Learn local image features
 Translation invariance
 Automatic feature extraction
 Efficient training


 Comparison
 Fully Connected NN  CNN
 Large parameters  Fewer parameters
 Manual feature extraction  Automatic feature learning
 High computation  Efficient computation
 Prone to overfitting  Better generalization

 3. Pooling
 Definition
 Pooling is a downsampling operation that reduces the spatial dimensions of feature
maps while preserving important information.


 Objectives
 Reduce computation
 Reduce memory
 Improve robustness
 Prevent overfitting


 Types of Pooling
 A. Max Pooling
 Selects the maximum value.
 Example
 Input

 15
 28

 Output = 8


 B. Average Pooling
 Calculates average value.
 Example
 Input

 24
 68

 Output

 (2+4+6+8)/4 = 5


 C. Global Average Pooling

 Average of the complete feature map.


 Diagram
 Feature Map

 48
 29

 Max Pool

 9


 Advantages
 Reduces feature size
 Improves speed
 Prevents overfitting

 Disadvantages
 Information loss
 May remove fine details


 4. Convolution and Pooling as an Infinitely Strong Prior

 Definition
 A prior is an assumption made by the model before learning.
 CNN assumes:
 Nearby pixels are related.
 Same feature may appear anywhere in the image.
 This assumption acts as a very strong prior.


 Why is it called an Infinitely Strong Prior?

 Because CNN architecture permanently enforces:
 Local connectivity
 Weight sharing
 Translation invariance
 The network cannot ignore these assumptions.


 Benefits
 Better generalization
 Faster learning
 Reduced parameters
 Improved performance


 Limitations
 Less suitable for non-grid data
 Fixed assumptions may not fit every problem


 5. Variants of the Basic Convolution Function

 Different convolution operations improve efficiency and accuracy.


 A. Standard Convolution
 Uses full convolution filters.


 B. Strided Convolution
 Moves filter by more than one pixel.
 Advantages
 Faster
 Downsampling


 C. Dilated Convolution
 Adds gaps between filter elements.
 Advantages
 Larger receptive field
 Better context


 D. Transposed Convolution
 Used for image upsampling.
 Applications
 Image segmentation
 Super-resolution


 E. Depthwise Convolution
 Each channel has a separate filter.
 Advantages
 Faster
 Less computation


 F. Pointwise Convolution
 Uses 1×1 filters.
 Advantages
 Reduces channels
 Efficient computation


 Comparison
 Variant  Purpose
 Standard  Feature extraction
 Strided  Downsampling
 Dilated  Larger receptive field
 Transposed  Upsampling
 Depthwise  Efficient computation
 Pointwise  Channel reduction

 6. Structured Outputs
 Definition
 Structured outputs refer to outputs where multiple dependent variables are predicted
simultaneously.
 Unlike classification, outputs have relationships.


 Examples
 Image segmentation
 Object detection
 Pose estimation
 Machine translation


 Diagram
 Input Image
 │
 CNN
 │
 Pixel-wise Output


 Advantages
 Rich output
 Better prediction
 Suitable for complex tasks


 Applications
 Medical image segmentation
 Self-driving cars
 OCR
 Video analysis


 7. Data Types Processed by CNN

 CNN handles different types of data.


 A. Images
 Grayscale
 RGB


 B. Videos
 Sequence of images.


 C. Audio
 Spectrogram representation.


 D. Text
 Word embeddings.


 E. Time Series
 Sensor signals.


 Table
 Data Type  Example
 Image  Face recognition
 Video  Surveillance
 Audio  Speech recognition
 Text  Sentiment analysis
 Time Series  ECG analysis

 8. Efficient Convolution Algorithms
 Efficient algorithms reduce computation.


 A. FFT Convolution
 Uses Fast Fourier Transform.
 Advantages
 Faster for large filters


 B. Winograd Algorithm
 Reduces multiplication operations.
 Advantages
 Faster CNN training


 C. GEMM-Based Convolution
 Converts convolution into matrix multiplication.
 Advantages
 GPU optimization


 D. Depthwise Separable Convolution

 Used in lightweight CNNs.
 Advantages
 Faster
 Lower memory


 Comparison
 Algorithm  Benefit
 FFT  Large kernels
 Winograd  Fewer multiplications
 GEMM  GPU acceleration
 Depthwise  Mobile devices

 Advantages
 Faster execution
 Less computation
 Lower memory usage


 9. Random or Unsupervised Features

 Definition
 Instead of learning filters using labeled data, features can be generated randomly or
learned without supervision.


 Random Features
 Filters initialized randomly.
 Advantages
 Simple
 Fast
 Disadvantages
 Lower accuracy


 Unsupervised Features
 Learned from unlabeled data.
 Methods
 Autoencoders
 Sparse Coding
 Self-Supervised Learning


 Advantages
 No labeled data required
 Learns useful representations
 Better feature extraction


 Applications
 Image retrieval
 Clustering
 Medical diagnosis
 Feature learning


 Comparison Table
 Random Features  Unsupervised Features
 Random filters  Learned filters
 No training  Learns from data
 Fast  More accurate
 Lower performance  Better performance

 Applications of CNN
 Image Classification
 Face Recognition
 Medical Image Analysis
 Object Detection
 Video Analytics
 Self-Driving Cars
 OCR (Optical Character Recognition)
 Satellite Image Processing
 Industrial Quality Inspection


 Comparison: CNN vs Traditional Neural Network

 Feature  Traditional ANN  CNN
 Input  1D vector  2D/3D grid
 Feature Extraction  Manual  Automatic
 Parameters  Large  Fewer (weight sharing)
 Computation  High  Lower
 Feature  Traditional ANN  CNN
 Best For  General data  Images and spatial data

 Advantages of CNN
 Automatic feature learning
 High accuracy
 Translation invariance
 Reduced parameters
 Efficient for image processing
 Better generalization


 Limitations
 Requires large datasets
 High computational cost
 Long training time
 Sensitive to hyperparameters
 Difficult to interpret