DEEP LEARNING

CSA3007

Module 4 : OPTIMIZATION ON DEEP MODELS:

OPTIMIZATION ON DEEP MODELS
 Introduction
• Optimization in deep learning refers to:
Finding model parameters (weights) that minimize the loss function.
min 𝐽 𝜃
𝜃

Where:𝜃= model parameters

𝐽(𝜃) = loss function

In Deep Learning

• Models have millions of parameters

• Optimization is done using: Gradient-based methods

Iterative updates
OPTIMIZATION ON DEEP MODELS
However, due to:
• High-dimensional parameter space
• Non-convex loss functions
• Large datasets
basic optimization methods are not sufficient.
• Therefore, optimization strategies and meta-algorithms are used to:
• Improve convergence
• Enhance generalization
• Stabilize training
 OPTIMIZATION ON DEEP MODELS
 Challenges in Neural Network Optimization
 Non-Convex Loss Function
• Deep networks have many local minima
• Optimization is difficult
 Overfitting
• Model memorizes training data
• Poor generalization
 Vanishing Gradient Problem
 Exploding Gradient Problem
 Saddle points
 OPTIMIZATION ON DEEP MODELS
 Vanishing Gradient Problem
• Gradients become very small (close to zero) as they are propagated backward
through layers.
• Cause
• Activation functions like sigmoid/tanh
• Repeated multiplication of small derivatives
• Effect
• Early layers learn very slowly or stop learning
• Model fails to capture long-term dependencies
 OPTIMIZATION ON DEEP MODELS
• Impact on Training
• Poor convergence
• Reduced accuracy
• Difficulty in training deep networks
• Example
• In deep networks, gradient may reduce like:
0.510 = 0.00098
• Solutions
• ReLU activation
• Batch normalization
• LSTM (for sequence models)
 OPTIMIZATION ON DEEP MODELS
 Exploding Gradient Problem
• Gradients become very large during backpropagation.
• Cause
• Large weight values
• Repeated multiplication of gradients.
• Effect
• Weight updates become unstable
• Loss function fluctuates or diverge.
• Impact on Training
• Training becomes unstable
• Model may fail to converge
 OPTIMIZATION ON DEEP MODELS
• Solutions
• Gradient clipping
• Proper weight initialization
• Normalization techniques
 Saddle Points
• A saddle point is a point where:
• Gradient = 0
• But it is not a minimum
• Characteristics
• Flat region
• No clear direction for descent
 OPTIMIZATION ON DEEP MODELS
• Effect
• Optimization slows down significantly
• Model gets stuck temporarily
• Impact on Training
• Slow convergence
• Inefficient learning
• Solutions
• Momentum
• Adam optimizer
• Random initialization
OPTIMIZATION ON DEEP MODELS

Challenge Effect on Training

Vanishing gradient Slow or no learning in early layers

Exploding gradient Unstable updates

Saddle points Slow convergence

 Optimization Strategies and Meta-Algorithms
 Optimization Strategies and Meta-Algorithms
• These are techniques applied on top of optimization algorithms (like SGD,
Adam) to improve training efficiency.
 Learning Rate Scheduling
• The learning rate (η) controls how much weights are updated.
• Learning rate scheduling involves changing the learning rate during
training.
• Why Needed?
• Large learning rate → fast but unstable
• Small learning rate → slow but precise
 Optimization Strategies and Meta-Algorithms
• Types of Learning Rate Scheduling
(a) Step Decay
• Reduce learning rate after fixed epochs.

• Simple and widely used

(b) Exponential Decay

• Smooth decay
(c) Cosine Annealing
• Learning rate gradually decreases in a cosine curve
• Helps escape local minima
 Optimization Strategies and Meta-Algorithms
• Advantages
• Faster convergence
• Avoids overshooting
• Improves final accuracy
• Importance in Deep Learning
• Faster initial learning
• Fine-tuning near optimum
• Avoid oscillations
 Optimization Strategies and Meta-Algorithms
 Early Stopping
• During training: Training loss decreases continuously and Validation loss
decreases initially but later increases. This indicates overfitting.
• Early stopping is a technique where:
Training is stopped when validation error starts increasing
Working
• Monitor validation loss
• Stop training when:
Validation Loss ↑ while Training Loss ↓
• Stop training before the model starts memorizing data
 Optimization Strategies and Meta-Algorithms
• Purpose
• Prevent overfitting
• Improve generalization
• Advantages
• Simple and effective
• Saves computation time
• Reduces unnecessary computation
 Optimization Strategies and Meta-Algorithms
 Regularization
• Regularization techniques are used toReduce overfitting by adding
constraints to the model
• Regularization adds a penalty term to the loss function:

• Types of Regularization
(a) L1 Regularization

• Produces sparse weights

• Feature selection
 Optimization Strategies and Meta-Algorithms
(b) L2 Regularization (Weight Decay)

• Penalizes large weights

• Smooth learning
(c) Dropout
• Randomly removes neurons during training.
• Effect: Prevents co-adaptation
Improves robustness
 Optimization Strategies and Meta-Algorithms
• Without regularization: Model memorizes training data
• With regularization: Model learns general patterns
• Regularization balances bias and variance
• Advantages of Regularization
• Reduces overfitting
• Improves generalization
• Makes model stable
 Optimization Strategies and Meta-Algorithms
 Batch Normalization
• Problem: Internal Covariate Shift
• During training:
• Distribution of inputs to layers keeps changing
• Makes training unstable
• Batch normalization normalizes inputs to each layer:

• BatchNorm stabilizes gradient flow in deep networks

 Optimization Strategies and Meta-Algorithms
Steps
• Compute mean (𝜇)
• Compute variance (𝜎2)
• Normalize inputs
• Scale & shiftand variance
• Advantages
• Faster convergence
• Allows higher learning rate
• Reduces vanishing gradient
• Acts as regularizer
 Optimization Strategies and Meta-Algorithms
 Role of Meta-Algorithms
• Meta-algorithms help in:
• Improving optimization efficiency
• Handling instability
• Enhancing generalization

Technique Role in Optimization

Learning Rate Scheduling Controls update speed

Early Stopping Prevents overfitting

Regularization Controls complexity

Batch Normalization Stabilizes training

 Convolutional Networks:

 Motivation for Convolutional Networks:

Problem with Traditional Neural Networks
• In standard Artificial Neural Networks:
• Fully connected layers → too many parameters
• Loss of spatial information
• Not efficient for image data
Example:
• Image (224×224×3) → ~150,000 inputs
• Fully connected → millions of weights
 Convolutional Networks:
 Key Motivations for CNN
(a) Spatial Structure in Images
• Nearby pixels are related
• Patterns are local (edges, textures)
(b) Local Receptive Fields
• Neurons connect to small regions
• Capture local features
(c) Weight Sharing
• Same filter applied across image
• Reduces number of parameters
 Convolutional Networks:
(d) Translation Invariance
• Object can appear anywhere
• CNN detects features regardless of position

 Structured Output in CNN

• Structured output means: Output is not a single label but a structured entity
• Examples
• Image segmentation → pixel-wise labels
• Object detection → bounding boxes + class
• Pose estimation → keypoints
 Convolutional Networks:
 CNN for Structured Output
• CNNs preserve spatial structure, so they are suitable for:
(a) Fully Convolutional Networks (FCN)
• No fully connected layers
• Output is image-sized
(b) U-Net
• Encoder + decoder
• Used in medical imaging
• Importance
• Maintains spatial relationships
• Produces dense predictions
• Used in real-world vision tasks
Convolutional Networks:
 Unsupervised Feature Learning in CNN
• Unsupervised feature learning means:
Learning useful features without labeled data

• Automatic feature learning directly from raw data without manual feature
engineering.

• The process by which a model learns useful patterns and representations from
input data without using labeled outputs.
Why Needed?
• Labeling data is expensive and time-consuming
• Large amount of unlabeled data is available
• Improves generalization
 Convolutional Networks:
• In this approach, the model tries to capture:
• important structures in the data,
• repeated patterns,
• relationships among pixels or regions,
• meaningful latent representations.
• The goal is not initially to classify the data, but to build internal feature
representations that can later be used for tasks such as: classification,
clustering, retrieval, segmentation, and anomaly detection.
 Convolutional Networks:
How CNNs Perform Unsupervised Feature Learning
 Through Convolution Layers
• CNNs use:
• Filters (kernels)
• Feature maps
• These filters are:
• Learned automatically
• Updated during training
 Learning Process
1. Input image is passed through convolution layers
2. Filters detect patterns
 Convolutional Networks:
3. Output feature maps represent learned features
4. Features improve with deeper layers
 Unsupervised Techniques in CNN
• In CNNs, unsupervised feature learning usually takes place through methods
such as:
1. Autoencoders
• A convolutional autoencoder consists of:
• an encoder that compresses the input image into a lower-dimensional
feature representation,
• a decoder that reconstructs the original image from that representation.
 Convolutional Networks:
The network is trained to minimize reconstruction error:

where:𝑥 is the original image,

𝑥 is the reconstructed image
• By trying to reconstruct the input, the CNN learns important features such as
edges, textures, and structures.
2. Self-supervised learning
• In self-supervised learning, artificial tasks are created from unlabeled data. For
example:
• predicting missing image patches,
• predicting image rotation,
 Convolutional Networks:
• contrastive learning between augmented views of the same image.
• In this way, CNNs learn useful internal representations without external labels.
3. Unsupervised pretraining
• Earlier deep learning methods used unsupervised pretraining, where
convolutional layers were first trained to learn representations from unlabeled
data, and then fine-tuned using labeled data.
 Convolutional Networks:
 Hierarchical Feature Learning in CNNs
• One of the most important strengths of CNNs is their ability to learn hierarchical
feature representations.
• This means that different layers of the CNN learn different levels of abstraction.
• Basic idea:
• Lower layers learn simple features.
• Middle layers learn more complex patterns.
• Higher layers learn semantic and task-specific features.
• This hierarchical learning happens because each convolutional layer processes
the output feature maps of the previous layer.
 Convolutional Networks:
 Layer-wise Hierarchical Feature Learning
1. First Convolutional Layers – Low-Level Features
• The first layers of a CNN learn very basic features such as:
• edges,
• lines,
• corners,
• simple color contrasts.
• These are small local patterns that appear frequently in images.
• For example:
• a horizontal filter may detect horizontal edges,
• a vertical filter may detect vertical edges.
Convolutional Networks:
2. Middle Layers – Mid-Level Features
• The middle layers combine low-level features to form:
• textures,
• curves,
• repeated patterns,
• shapes,
• object parts.
• For example:
• edges combine to form contours,
• contours combine to form corners and shapes,
• textures capture surface patterns.
• These representations are more informative than raw pixels.
Convolutional Networks:
3. Deep Layers – High-Level Features

• The deeper layers combine mid-level patterns into:

• object components,
• semantic parts,
• whole object representations..

• For example:
• eyes, nose, and mouth may combine into a face representation,
• wheels, windows, and body shape may combine into a car representation..
• These deep features are highly abstract and task-specific..
Convolutional Networks:
Example of Hierarchical Feature Learning
• Consider a CNN trained on face images:
• Layer 1 learns edges and gradients.
• Layer 2 learns facial contours and simple textures.
• Layer 3 learns eyes, nose, lips, and ears.
• Layer 4 learns full face patterns.
• Final layer uses these learned features for recognition or classification.
• Thus, the CNN builds knowledge gradually, from simple to complex.
Convolutional Networks:
Advantages of Unsupervised Feature Learning in CNNs
• Unsupervised feature learning offers several advantages:
• It uses unlabeled data effectively.
• It reduces the dependence on annotated datasets.
• It learns useful and generalizable features.
• It improves transfer learning.
• It helps in domains where labels are scarce.
• It can reveal hidden structures in data.
• Applications • Medical image analysis
• Image classification • Face recognition
• Object detection • Autonomous driving