MODULE -II
• COURSE NAME: ESSENTIALS OF DEEP
LEARNING COURSE CODE : 23CSE116
•
•
•
TOPICS TO BE COVERED
Neural Weight Initialisation
Regularisation for Neural Networks
Different activation functions
Loss Function for Neural Networks
Regression Loss Functions vs Classification Loss
Functions
Cross-Entropy Loss Function
Softmax Function
Dropconnect, Batch Normalisation, and Greedy
Layerwise Training
NEURAL NETWORK WEIGHT
INITIALIZATION
• Weight initialisation is the process of assigning starting values to the
parameters (weights and biases) of a neural network before training.
• Proper initialisation helps the model converge faster during training.
• If weights are not initialised correctly, the network may suffer from
problems like vanishing gradients, exploding gradients, or slow
convergence.
• Random small values are usually preferred over large or fixed ones.
• The goal is to ensure stable gradients during backpropagation
A SIMPLE NEURAL NETWORK SHOWING
INITIALIZED WEIGHTS CONNECTING
NEURONS
• Weight initialisation is a crucial step in training neural
networks.
• It sets the starting point for optimisation.
• Poor initialisation may lead to vanishing/exploding
gradients.
• In this presentation, we explore:
• 1. Zero Initialization
• 2. Random Initialization
• 3. Xavier Initialization
ZERO INITIALIZATION
• In zero initialization, all weights are set to zero at the beginning.
• This method makes all neurons learn the same features.
• Gradients become identical across neurons, preventing effective learning.
• Works only for very simple models, not for deep networks.
• Biases can be initialized to zero safely, but not weights
ZERO INITIALIZATION -
EXAMPLE
• Example:
• In a neural network with hidden layers, if all
weights = 0,gradients will be identical for all
neurons.
RANDOM INITIALIZATION
Weights are initialised with small random values,
often drawn from a uniform or normal
distribution.
This breaks symmetry, allowing neurons to learn
different features.
However, choosing values too small may cause
vanishing gradients, and too large may cause
exploding gradients.
• It helps the optimization process start with
variation.
• Common method: Uniform random distribution
between [-0.5, 0.5].
• W∼U(−ϵ,ϵ)
MATRIX OF ORTHOGONAL
WEIGHTS SHOWN IN GRID
FORM
XAVIER INITIALISATION
• Proposed by Xavier Glorot.
• Balances the variance of inputs and outputs across layers.
• Weights drawn from distribution: Var(w) = 2 / (n_in +
n_out).
• Prevents vanishing and exploding gradients.
• Faster convergence during training.
• Works well with sigmoid and tanh activation functions.
COMPARISON OF
INITIALIZATION TECHNIQUES
• Zero Initialization:
• • Simple, but fails due to symmetry.
• Random Initialization:
• • Works, but may cause instability.
• Xavier Initialization:
• • Balances variance, widely used in practice.
REGULARIZATION
• Regularization is a technique used in machine learning and
deep learning to prevent overfitting by adding extra
constraints or penalties to the model.
• Overfitting happens when a model learns the training data
too well, including noise, and fails to generalize to unseen
data.
• Regularization addresses this by discouraging overly
complex models, keeping the learned parameters (weights)
smaller or sparser
INTRODUCTION – WHY
REGULARIZATION IS NEEDED
• Neural networks often suffer from overfitting.
• Overfitting: Model performs well on training data but
poorly on unseen data.
• Regularization introduces constraints to improve
generalization.
• Goal: Reduce variance without increasing bias too
much.
L1 REGULARIZATION (LASSO)
• L1 regularization adds the absolute value of weights to the
loss function.
• It forces some weights to become exactly zero.
• This makes the model sparse and selects only important
features.
• Good for feature selection when there are many irrelevant
inputs
• It can lead to simpler and more interpretable models.
• Adds a penalty proportional to absolute value of weights.
• Cost Function: Loss + λ Σ |wi|
• Encourages sparsity – pushes some weights to zero.
• Helps with feature selection.
This image illustrates how L1 regularization works geometrically in a 2D
weight space (with weights w1and w2).
•The blue diamond represents the constraint region imposed by the L1
penalty.
•The red arrow shows the direction of the unregularized loss minimization
(i.e., where the model would go without regularization).
•The origin (center) is where both weights are zero.
L2 REGULARIZATION (RIDGE)
• L2 regularization adds the squared value of
weights to the loss function.
• Unlike L1, it does not shrink weights to zero but
makes them small.
• Prevents overfitting by discouraging large
weights.
• Useful when all features are important but
should be controlled
• cost Function: Loss + λ Σ wi²
• Prevents very large weights.
• Helps distribute weights more evenly.
• Produces smoother models compared to L1.
L2 Regularization (Ridge) – Circular Constraint
•The circle has smooth, curved edges — no sharp corners.
•The optimal solution will typically touch the circle at a point where
both w1and w2are non-zero, just smaller in magnitude.
•Even if one weight is very small, it's never exactly zero.
•→ All features are retained, but their influence is reduced.
DROPOUT REGULARIZATION
•Dropout randomly “drops” neurons during training with
probability p.
•Prevents co-adaptation of neurons.
•Each training iteration uses a different sub-network.
•It reduces overfitting by forcing robustness.
•At inference, all neurons are used with scaled weights.
•Helps in large deep networks like CNNs and RNNs.
•Improves generalization significantly
COMPARISON OF TECHNIQUES
• L1: Sparsity, feature selection.
• L2: Prevents large weights, smooth solutions.
• Dropout: Random neuron deactivation, robust
learning.
ACTIVATION FUNCTION
• Activation functions are essential components
of deep learning models. They introduce non-
linearity, enabling networks to learn complex
relationships beyond simple linear mapping.
• An activation function determines how the
weighted sum of inputs is transformed into an
output from a neuron. Different activation
functions are used depending on the task, such
as regression, binary classification, or multi-
class classification.
IMPORTANCE
• Without activation functions, deep neural
networks would behave like linear models. Non-
linearity allows networks to capture intricate
patterns in data, making them effective in fields
like image recognition, speech processing, and
natural language understanding.
LINEAR ACTIVATION FUNCTION
• The linear activation function outputs the input
directly. While simple, it is rarely used in hidden
layers since it cannot handle non-linear data.
However, it is useful in the output layer of
regression models.
f(x)=ax
• The range of the output spans from (−∞ to +∞)
(−∞ to +∞).
• Linear activation function is used at just one
place i.e. output layer.
• Using linear activation across all layers makes
the network's ability to learn complex patterns
limited
• Binary Step Function
• The binary step function outputs either 0 or 1
based on a threshold. Although it played a role
in early perceptrons, it is not used in modern
deep learning because it is not differentiable
and unsuitable for gradient descent
optimization
• Sigmoid Function
• The sigmoid function squashes input values
into the range of 0 to 1, making it useful for
binary classification tasks. It provides smooth
gradients but suffers from vanishing gradient
issues for large input values.
• It allows neural networks to handle and model
complex patterns that linear equations cannot.
• The output ranges between 0 and 1, hence
useful for binary classification.
• The function exhibits a steep gradient when x
values are between -2 and 2. This sensitivity
means that small changes in input x can cause
significant changes in output y which is critical
during the training process
• Hyperbolic Tangent (tanh) Function
• Tanh maps values between -1 and +1. Being
zero-centered, it often performs better than
sigmoid for hidden layers, but it still faces
vanishing gradient problems for extreme
values.
• The hyperbolic tangent function is a shifted version
of the sigmoid, allowing it to stretch across the y-
axis.
• Value Range: Outputs values from -1 to +1.
• Non-linear: Enables modelling of complex data
patterns.
• Use in Hidden Layers: Commonly used in hidden
layers due to its zero-centred output, facilitating
easier learning for subsequent layers.
• ReLU (Rectified Linear Unit)
• ReLU is the most widely used activation
function. It outputs the input directly if positive,
otherwise zero. It is computationally efficient,
speeds up training, and is effective for deep
networks.
• A(x)=max(0,x), this means that if the input x is
positive, ReLU returns x, if the input is negative,
it returns 0.
• Value Range: [0,∞)[0,∞), meaning the function
only outputs non-negative values.
• Nature: It is a non-linear activation function,
allowing neural networks to learn complex
patterns and making backpropagation more
efficient.
• Advantage over other Activation: ReLU is
less computationally expensive than tanh and
sigmoid because it involves simpler
mathematical operations. At a time only a few
neurons are activated making the network
sparse making it efficient and easy for
computation
• Comparison of Activation Functions
• Each activation function has strengths and
weaknesses. ReLU and its variants are popular
for hidden layers, sigmoid and tanh are useful
in smaller networks, and softmax is essential
for classification outputs.
LOSS FUNCTIONS FOR NEURAL
NETWORKS
• Loss functions measure the difference between
predicted output and actual target values. They
guide the optimization process by telling the
model how far prediction takes place.
Importance of Loss Functions
• Provide feedback for weight updates.
• Different tasks (classification, regression)
require different loss functions.
• Categories of Loss Functions
• Regression Losses – for predicting continuous
values.
• Classification Losses – for predicting
categories.
• Ranking / Custom Losses – for specialized
tasks
REGRESSION LOSS
• Mean Squared Error (MSE)
• Used for regression tasks.
• Formula:
• Penalizes large errors more heavily.
• Mean Absolute Error (MAE)
• Formula:
• Less sensitive to outliers compared to MSE.
• Provides linear error measurement.
• Huber Loss
• Combines advantages of MSE and MAE.
• Quadratic for small errors, linear for large
errors.
• Useful when dealing with noisy data.
CLASSIFICATION LOSS
• Cross-Entropy Loss (Log Loss)
• Widely used for classification tasks.
• Measures the dissimilarity between predicted
probability distribution and actual labels.
• Works well with Softmax or Sigmoid outputs.
• Binary Cross-Entropy
• For binary classification problems.
• Formula:
• Categorical Cross-Entropy
• For multi-class classification.
• Compares predicted probability distribution
across multiple classes.
• Used with Softmax activation.
• Kullback–Leibler (KL) Divergence
• Measures how one probability distribution
diverges from another.
• Often used in Variational Autoencoders and
probabilistic models.
• Custom Loss Functions
• Neural networks allow designing task-specific
loss functions.
• Examples: Weighted losses for imbalanced
datasets, perceptual loss for image tasks,
adversarial loss for GANs.
• Comparison of Loss Functions
• MSE/MAE: Good for regression.
• Cross-Entropy: Best for classification.
• Huber: Robust with noisy data.
• KL Divergence: For probabilistic models.
• Custom Losses: For domain-specific tasks.
THE SOFTMAX FUNCTION
• Softmax is an activation function used in
classification problems.
• Converts raw output values (logits) into a
probability distribution.
• Ensures that all output probabilities sum to 1.
• Where:
• zi= raw score for class i
K = total number of classes
• Output = probability of class i.
• Used in the output layer of neural networks
for classification.
• Helps in decision-making by selecting the class
with highest probability.
• Works together with Cross-Entropy Loss
OPTIMIZATIONS FOR NEURAL
NETWORKS
• Enhancing training speed, convergence, and
performance.
• Covers gradient descent, adaptive methods,
and advanced techniques.
• Essential for deep learning success.
• Focus on reducing loss and improving accuracy.
• A key component in building intelligent
systems.
• Optimization is the process of minimizing the
loss function.
• Determines how weights and biases are
updated.
• Poor optimization leads to slow or failed
training.
• Various algorithms improve convergence
efficiency.
GRADIENT DESCENT
• The most fundamental optimization method.
• Updates weights in the opposite direction of
gradients.
• Learning rate controls the step size in updates.
• Too high learning rate → overshooting, too low
→ slow convergenc
• Basis for many advanced optimization methods.
• Gradient Descent is an optimization
algorithm used to minimize the cost/loss
function in machine learning.
• The algorithm updates model parameters θ
iteratively to reach the lowest error.
• The update rule is:
• θ=θ−α⋅∇J(θ)
• The gradient represents the slope of the cost
function with respect to parameters.
• If the slope is positive, parameters move left; if
negative, they move right.
• Learning rate (α) controls the step size—too
small = slow, too large = overshoot.
• The process continues until the algorithm
converges to a minimum value of the cost
function.
• Variants of Gradient Descent
• Batch Gradient Descent: uses the whole dataset,
stable but slow.
• Stochastic Gradient Descent (SGD): updates per
sample, faster but noisy.
• Mini-Batch Gradient Descent: compromise,
balances stability and speed.
• Mini-batch is most widely used in deep learning.
• Provides efficiency with GPU parallelization.
MOMENTUM OPTIMIZATION
• Accelerates gradient descent by considering
past updates.
• Helps escape local minima by adding "inertia".
• Reduces oscillations in steep directions
• Update rule includes velocity term.
• Leads to faster and smoother convergence.
RMSPROP OPTIMIZATION
• Root Mean Square Propagation adapts learning
rate.
• Maintains separate learning rates per
parameter.
• Normalizes gradients by their recent
magnitudes
• Prevents issues with vanishing/exploding
gradients.
ADAM OPTIMIZATION
• Combines Momentum + RMSProp.
• Maintains both first and second moment
estimates of gradients.
• Widely used due to its efficiency and
robustness.
• Adaptive learning rates make it versatile across
problems.
DROPCONNECT IN NEURAL
NETWORKS
• Dropconnect is a regularization technique.
• Instead of dropping neurons, it drops weights.
• Helps reduce overfitting in deep models.
• Generalizes better than Dropout in some cases.
• Used in fully connected layers.
• Random subset of weights are set to zero.
• Forward pass uses the reduced weight matrix.
• Each training step has a different sparse
network.
• At test time, weights are scaled appropriately.
• Helps in model robustness.
• Advantages & Limitations:
• Reduces overfitting.
• Increases generalization.
• Higher computational cost than dropout.
• Slower training in some models.
BATCH NORMALIZATION
• Training deep networks is difficult.
• Batch Normalization (BN) normalizes layer
inputs.
Internal covariate shift slows convergence.
Makes training faster and more stable.
Reduces sensitivity to initialization.
• Compute mean & variance for each batch.
• Normalize activations using these statistics.
• Apply learnable scale (γ) and shift (β).
• Keeps output distribution consistent.
• Acts as a mild regularizer.
• Faster convergence.
• Allows higher learning rates.
• Reduces overfitting (slightly).
• Improves gradient flow in deep nets.
• Widely used in CNNs & Transformers.
GREEDY LAYERWISE TRAINING
• Early deep networks faced training difficulty.
• Greedy Layerwise Training (GLT) was a solution.
• Trains one layer at a time in unsupervised
manner.
• Then fine-tunes with supervised learning.
• Used in Deep Belief Nets and Autoencoders.
• Used in Deep Belief Nets and Autoencoders.
• Freeze it, then train next layer on its outputs.
• Repeat until all layers are trained.
• Finally, fine-tune whole network with backprop.
• Greedy approach ensures stability.
• Greedy layer-wise pre-training is used to initialize the
parameters of deep neural networks layer by layer,
beginning with the first layer and working through
each one that follows.
• A layer is trained as if it were a stand-alone model at
each step, using input from the layer before it and
output to go to the layer.
STAGES IN GREEDY LAYER
• Initialization: The neural network's first layer is trained
on its own using autoencoders and other unsupervised
learning strategies.
• Extracting Feature: The activations of the first layer
are utilized as features to train the subsequent layer after
it has been trained. Each layer learns to represent the
traits discovered by the layer before it in a higher-level
abstraction when this process is repeated repeatedly.
• Fine-Tuning: The network is adjusted as
a whole using supervised learning
methods once every layer has been
pretrained in this way. To maximize
performance on a particular job, this
entails simultaneously modifying all of the
network's parameters using a [Link]
dataset
ADVANTAGES
• Solves vanishing gradient problem
(historically).
• Helps initialize deep networks.
• Better convergence than random initialization.
• Reduced computational burden in early
research.
• Reduced computational burden in early
research.
CONCLUSION
• Proper weight initialization is crucial for stable
and efficient training of deep neural networks.
• Regularization techniques (L1, L2, Dropout,
Early Stopping, Data Augmentation) help prevent
overfitting and improve generalization.
• Different activation functions (Sigmoid, Tanh,
ReLU, Leaky ReLU, Softmax) play key roles in
learning non-linear relationships.
• Loss functions (MSE, Cross-Entropy, Hinge Loss,
etc.) guide the optimization process by measuring
prediction errors.
• Optimization algorithms (Gradient Descent
variants, Adam, RMSProp, etc.) accelerate
convergence and improve accuracy.
• Advanced methods like DropConnect, Batch
Normalization, and Greedy Layerwise Training
further stabilize and enhance deep learning
performance.
• Thank you