Unit I

Basics of artificial neural networks (ANN): Artificial neurons, Computational models

of neurons, Structure of neural networks, Functional units of ANN for pattern
recognition tasks Feedforward neural networks: Pattern classification using perceptron,
Multilayer feedforward neural networks (MLFFNNs), Backpropagation learning,
Empirical risk minimization, Regularization, Autoencoders
Unit II
Deep neural networks (DNNs): Difficulty of training DNNs, Greedy layer wise training,
Optimization for training DNNs, Newer optimization methods for neural networks
(AdaGrad, RMSProp, Adam), Second order methods for training, Regularization
methods (dropout, drop connect, batch normalization)
Unit III
Convolution neural networks (CNNs): Introduction to CNNs – convolution, pooling,
Deep CNNs, Different deep CNN architectures – LeNet, AlexNet, VGG, PlacesNet,
training a CNNs: weights initialization, batch normalization, hyperparameter
optimization, Understanding and visualizing CNNs.
Unit IV
Recurrent neural networks (RNNs): Sequence modeling using RNNs, Backpropagation
through time, Long Short Term Memory (LSTM), Bidirectional LSTMs, Bidirectional
RNNs, Gated RNN Architecture - Generative models: Restricted Boltzmann Machines
(RBMs), Stacking RBMs, Belief nets.
Unit V
Learning sigmoid belief nets, Deep belief nets Under complete - Auto encoder,
Regularized Auto encoder, stochastic Encoders and Decoders, Contractive Encoders.
Applications: Applications in vision, speech and natural language processing.
Recommended Texts:
1. S. Haykin, Neural Networks and Learning Machines , Prentice Hall of India, 2016
2. Ian Goodfellow, Yoshua Bengio and Aaron Courville, “ Deep Learning”, MIT Press,
2017
Reference Books:
1. Satish Kumar, Neural Networks - A ClassRoom
2. B. Yegnanarayana, Artificial Neural Networks, Prentice- Hall of India, 1999
3. Giancarlo Zaccone, Md. RezaulKarim, Ahmed Menshawy "Deep Learning with
TensorFlow: Explore neural networks with Python", Packt Publisher, 2017.
4. Antonio Gulli, Sujit Pal "Deep Learning with Keras", Packt Publishers, 2017.
5. Francois Chollet "Deep Learning with Python", Manning Publications, 2017.

1
 Unit I
Basics of artificial neural networks (ANN): Artificial neurons, Computational models
of neurons, Structure of neural networks, Functional units of ANN for pattern
recognition tasks Feed forward neural networks: Pattern classification using perceptron,
Multilayer feed forward neural networks (MLFFNNs), Back propagation learning,
Empirical risk minimization, Regularization, Auto encoders
BASICS OF ARTIFICIAL NEURAL NETWORKS (ANN)
Neural Networks
 Neural networks are parallel computing devices, which is basically an attempt to
make a computer model of the brain.
 The main objective is to develop a system to perform various computational
tasks faster than the traditional systems.
 These tasks include pattern recognition and classification, approximation,
optimization, and data clustering.
Artificial Neural Networks (ANN)
 Artificial Neural Networks contain artificial neurons, which are called units.
 These units are arranged in a series of layers that together constitute the whole
Artificial Neural Network in a system.
 A layer can have only a dozen units or millions of units, as this depends on how
the complex neural networks will be required to learn the hidden patterns in the
dataset.
 Commonly, an Artificial Neural Network has an input layer, an output layer, as
well as hidden layers.
 The input layer receives data from the outside world, which the neural network
needs to analyze.
 Then, this data passes through one or multiple hidden layers that transform the
input into data that is valuable for the output layer.
 Finally, the output layer provides an output in the form of a response of the
Artificial Neural Networks to the input data provided.

2
  The structures and operations of human neurons serve as the basis for artificial
neural networks.
 It is also known as neural networks or neural nets.
 The input layer of an artificial neural network is the first layer, and it receives
input from external sources and releases it to the hidden layer, which is the
second layer.
 In the hidden layer, each neuron receives input from the previous layer neurons,
computes the weighted sum, and sends it to the neurons in the next layer.
 These connections are weighted means effects of the inputs from the previous
layer are optimized more or less by assigning different weights to each input.
 It is adjusted during the training process by optimizing these weights for
improved model performance.
Artificial Neurons
 An artificial neuron is a connection point in an artificial neural network.
 Artificial neural networks (ANNs), like the human body's biological neural
network, have a layered architecture and each network node, or connection point,
can process input and forward output to other nodes in the network.

Artificial Neuron
Human Brain Neuron

3
  It receives input signals, processes them using a mathematical function, and
produces an output signal.
Structure of an Artificial Neuron:
1. Inputs (x₁, x₂, ..., xₙ):
The values fed into the neuron from either raw data or previous neurons.
2. Weights (w₁, w₂, ..., wₙ):
Each input is multiplied by a corresponding weight that signifies its importance.
3. Summation Function:
Calculates the weighted sum:
z=w1x1+w2x2+...+wnxn+b
where b is the bias term (adds flexibility to the model).
4. Activation Function (f):
Applies a non-linear function to the weighted sum to produce the output:
output=f(z)
COMPUTATIONAL MODELS OF NEURONS
 In deep learning, computational models of neurons are simplified mathematical
abstractions of biological neurons.
 These models are the core building blocks of artificial neural networks (ANNs)
and serve as the processing units that take inputs, perform a computation, and
produce an output.
Perceptron (Single-Layer Neuron Model)
 The Perceptron is the simplest type of artificial neuron, introduced by Frank
Rosenblatt in 1958.
 It is a binary classifier that decides whether an input belongs to one class or
another by learning a linear decision boundary.
𝑛

𝑦 = 𝑓 (∑ 𝑤𝑖 𝑥𝑖 + 𝑏)
𝑖=1

Where:
 xi: input features
 wi: weights
 b: bias
4
  f: activation function (e.g., step function)
 y: output
Sigmoid Neuron
 The Sigmoid Neuron is an enhanced version of the perceptron that uses a smooth,
differentiable activation function.
 This makes it suitable for gradient-based optimization techniques like
backpropagation, which are essential for training deep neural networks.
1
𝑦=
1 + 𝑒 −(∑ 𝑤𝑖 𝑥𝑖 +𝑏)
ReLU Neuron (Rectified Linear Unit)
 The ReLU neuron is the most widely used activation model in modern deep
learning.
 It introduces non-linearity into the model while being computationally simple
and highly effective.

𝑦 = 𝑚𝑎𝑥(0, ∑ 𝑤𝑖 𝑥𝑖 + 𝑏)

Softmax Neuron
 The Softmax Neuron is used in the output layer of neural networks for multi-
class classification tasks.
 It transforms a vector of raw scores (logits) into probabilities that sum up to 1
making it ideal for classification where each input belongs to exactly one class.

5
 𝑒 𝑧𝑖
𝑦𝑖 = 𝑛
∑𝑗=1 𝑒 𝑧𝑗
Where 𝑧𝑖 =∑ 𝑤𝑖 𝑥𝑖 + 𝑏
LSTM and GRU Neurons
 LSTM and GRU are advanced types of neurons used in Recurrent Neural
Networks (RNNs) to handle sequential data, such as text, speech, and time series.
LSTM Neuron (Long Short-Term Memory)
 LSTM (Long Short-Term Memory) is a recurrent neural network (RNN)
architecture widely used in Deep Learning.
 It excels at capturing long-term dependencies, making it ideal for sequence
prediction tasks.

GRU Neuron (Gated Recurrent Unit)

 The core idea behind GRUs is to use gating mechanisms to selectively update
the hidden state at each time step allowing them to remember important
information while discarding irrelevant details.
 GRUs aim to simplify the LSTM architecture by merging some of its
components and focusing on just two main gates: the update gate and the reset
gate.

6
 Feature LSTM GRU

Gates 3 (input, forget, output) 2 (update, reset)

Cell state Separate from hidden state Combined with hidden state

Parameters More Fewer

Complexity Higher Lower

Performance Better on complex tasks Faster, good on small data

Training Time Slower Faster

STRUCTURE OF NEURAL NETWORKS

 A neural network is a collection of interconnected layers of artificial neurons
(also called nodes or units) designed to mimic the human brain’s learning
process.
 Each layer transforms the data and passes it forward, enabling the network to
learn complex patterns.

7
Input Layer (i)
 The first vertical section on the left.
 Each circle (node) in this layer represents an input feature (e.g., pixel value, word
embedding).
 Example: For image data, these might be raw pixel values; for tabular data, each
input might be a different column.
 Inputs are fed into the network and passed forward.

Hidden Layers (h₁, h₂, ..., hₙ)

 The middle sections labeled as h₁, h₂, ..., hₙ are the hidden layers.
 These are where the real computation happens, they extract and learn complex
patterns from the input.
 Each node in a hidden layer: Takes input from every node in the previous layer
and applies a weighted sum followed by an activation function (e.g., ReLU,
sigmoid).
 These layers are called "hidden" because they are not directly observable from
the input or output.
Output Layer (o)
 It contains output neurons which produce the final predictions of the network.
 The number of output neurons depends on the task:
 Binary classification: 1 neuron (with sigmoid)
 Multi-class classification: 1 neuron per class (with softmax)
 Regression: 1 neuron (with linear activation)
PATTERN CLASSIFICATION USING PERCEPTRON
 Pattern classification is the task of assigning a label or category to an input
pattern based on its features.
 The perceptron is a foundational model in machine learning used for binary
pattern classification.

8
Perceptron
A perceptron is a type of artificial neuron that takes multiple input features,
applies weights to them, adds a bias, and passes the result through an activation function
(typically a step function) to make a decision.
 Input Pattern: A set of feature values (e.g., pixel intensities in an image, exam
scores, etc.)
 Weighted Sum: Each input is multiplied by a corresponding weight, and all are
summed with a bias.
 Activation Function: The result of the sum is passed through an activation
function (usually a threshold step function).
 Output: The perceptron outputs 1 if the sum is above the threshold, or 0 if below
thus classifying the input into one of two categories.

 Inputs (x₁, x₂, ..., xm): Features of the input pattern.

 Weights (w₁, w₂, ..., wm): Learnable parameters applied to each input.
 Bias (b): A constant added to shift the decision boundary.
 Summation (Σ): Calculates the weighted sum:
𝑛

𝑧 = ∑ 𝑤𝑖 𝑥𝑖 + 𝑏
𝑖=1

 Activation Function: A step function that outputs:

1, 𝑖𝑓𝑧 ≥ 0
𝑦={
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
 Output: The class label (0 or 1) assigned to the input pattern.

9
 The image illustrates the Perceptron Model used in pattern classification. Below is
a detailed explanation of each component shown in the diagram:
1. Inputs
 x1,x2,...,xm are the input features of the pattern (e.g., pixel values, sensor data).
 1 is a constant input used to incorporate the bias term via w0.
2. Weights
 Each input xi is associated with a weight wi.
 The weights determine the importance of each input feature in the classification
decision.
3. Net Input Function (Σ)
 All weighted inputs are summed:
𝑧 = 𝑤0 . 1 + 𝑤1 𝑥1 + 𝑤2 𝑥2 + ⋯ … … … … … . . +𝑤𝑚 𝑥𝑚
 This sum is the net input to the neuron.
4. Activation Function
 The net input is passed through an activation function (usually a step function)
to decide the output class:
1, 𝑖𝑓𝑧 ≥ 0
𝑦={
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
5. Output
 The final binary output (0 or 1) represents the predicted class of the input
pattern.
Error Calculation and Feedback Loop
 The actual output is compared to the target label.
 If the prediction is wrong, an error is calculated:
Error=Target−Output
This error is used to adjust the weights using the perceptron learning rule:
𝑤𝑖 = 𝑤𝑖 + 𝜂. 𝐸𝑟𝑟𝑜𝑟. 𝑥𝑖
where η is the learning rate.

10
MULTILAYER FEEDFORWARD NEURAL NETWORKS (MLFFNNS)
 Deep-learning feed-forward neural networks are used in a variety of applications,
including computer assistants, search engines, and machine translation.
 They serve as the foundation for several significant neural networks used today,
including recurrent neural networks, which are widely used in natural language
processing and sequence learning, and convolutional neural networks, which are
widely used in computer vision applications.
Feed Forward Neural Network components
Neurons

 The fundamental component of a neural network is an artificial neuron.

 The following is a schematic illustration of a neuron.

 It operates in two parts, as can be seen above: first, it computes the weighted sum
of its inputs, and then it uses an activation function to normalize the total.
 There are both linear and nonlinear activation functions.
 Additionally, each input to a neuron has a corresponding weight.
 The network must learn these parameters during the training phase.
Multi-layer Feed Forward Neural Network
 An entrance point into sophisticated neural networks, where incoming data is
routed through several layers of artificial neurons.

11
  Every node is linked to every neuron in the following layer, resulting in a fully
connected neural network.
 There are input and output layers, as well as several hidden levels, for a total of
at least three or more layers.
 It possesses bidirectional propagation, which means it can propagate both
forward and backward.

 Inputs are multiplied by weights and supplied into the activation function, where
they are adjusted to minimize loss during backpropagation.
 Weights are just machine-learned values from Neural Networks.
 They modify themselves based on the gap between projected and training
outcomes.
 Softmax is used as an output layer activation function after nonlinear activation
functions.
Input layer
 This layer's neurons take in information and send it to the network's other levels.
 The number of neurons in the input layer must equal the number of features or
attributes in the dataset.

12
Output Layer
 This layer is the one that provides the predictions.
 For various issues, a distinct activation function should be used in this layer.
 We want the result to be either 0 or 1 for a binary classification task.
 As a result, the sigmoid activation function is employed.
 A Softmax (think of it as a sigmoid applied to several classes) is used for
multiclass classification problems.
 We can utilize a linear unit to solve regression problems where the result does
not fall into a predetermined category.
Hidden layer
 Layers concealed between the input and output are used to segregate them.
 The number of hidden layers will depend on the type of model.
 In order to actually transfer the input to the next layer, numerous neurons in
hidden layers first modify it.
 For the purpose of improving predictability, this network's weights are adjusted
continuously.
BACKPROPAGATION LEARNING
 Back Propagation is also known as "Backward Propagation of Errors" is a
method used to train neural network.
 Its goal is to reduce the difference between the model’s predicted output and the
actual output by adjusting the weights and biases in the network.
 It works iteratively to adjust weights and bias to minimize the cost function.
 In each epoch the model adapts these parameters by reducing loss by following
the error gradient.
 It often uses optimization algorithms like gradient descent or stochastic gradient
descent.
 The algorithm computes the gradient using the chain rule from calculus allowing
it to effectively navigate complex layers in the neural network to minimize the
cost function.

13
Back Propagation plays a critical role in how neural networks improve over time.
Efficient Weight Update:
It computes the gradient of the loss function with respect to each weight using
the chain rule making it possible to update weights efficiently.
Scalability:
The Back Propagation algorithm scales well to networks with multiple layers
and complex architectures making deep learning feasible.
Automated Learning:
With Back Propagation the learning process becomes automated and the model
can adjust itself to optimize its performance.
1. Forward Pass Work
 In forward pass the input data is fed into the input layer.
 These inputs combined with their respective weights are passed to hidden layers.
 For example in a network with two hidden layers (h1 and h2) the output from h1
serves as the input to h2.
 Before applying an activation function, a bias is added to the weighted inputs.
 Each hidden layer computes the weighted sum (`a`) of the inputs then applies an
activation function like ReLU (Rectified Linear Unit) to obtain the output (`o`).
14
  The output is passed to the next layer where an activation function such as
softmax converts the weighted outputs into probabilities for classification.

2. Backward Pass
 In the backward pass the error (the difference between the predicted and actual
output) is propagated back through the network to adjust the weights and biases.
 One common method for error calculation is the Mean Squared Error (MSE)
given by:
MSE = (Predicted Output−Actual Output)2
 Once the error is calculated the network adjusts weights using gradients which
are computed with the chain rule.
 These gradients indicate how much each weight and bias should be adjusted to
minimize the error in the next iteration.
 The backward pass continues layer by layer ensuring that the network learns and
improves its performance.
 The activation function through its derivative plays a crucial role in computing
these gradients during Back Propagation.

15
Advantages
✅ Efficient for training deep networks
✅ Handles large-scale problems
✅ Enables automatic feature learning
✅ Used in all modern deep learning frameworks (TensorFlow, PyTorch)
EMPIRICAL RISK MINIMIZATION
 Empirical Risk Minimization is a fundamental principle in statistical learning
theory used to find a predictive model by minimizing the error or loss on a given
training dataset.
 The objective is to choose a hypothesis from a hypothesis set that minimizes the
empirical risk, which is the average loss on the training data.
Hypothesis Set: A set of possible function (models) from which we aim to choose the
best one.
Loss Function: Measure the error between the predicted and actual values. Common
examples include mean squared error for regression and cross-entropy for classification.
Empirical Risk: The average loss over the training sample. If the training set consists
of N samples and the loss function is L, the empirical risk is given by:
1
𝑅𝑒𝑚𝑝 (ℎ) = ( ∑ 𝐿(ℎ(𝑥𝑖 )𝑦𝑖 ))
𝑁
where ℎ( 𝑥𝑖 ) is the prediction of the model ℎ for the input , and is the actual output.
Practical: Provide a concrete criterion to select the best model based on observed data.
Theoretical Foundation: Forms the basis for many machine learning algorithm,
ensuring they generalize well to unseen data.
Flexibility: Can be applied to various types of loss functions and hypothesis sets.
Advantages:
 Foundation of Supervised Learning:
ERM is a core principle in supervised learning, providing a systematic
approach to model training and evaluation.

16
  Flexibility:
ERM can be applied to various types of models and loss function, making it
versatile across different problem domains.
 Optimization Framework:
Provides a clear objective for optimization, minimizing the empirical risk
(average loss on training data)
 Generalization:
When combined with techniques like cross-validation and regularization, ERM
helps models generalize well to unseen data.
 Adaptability:
Can be adapted to different learning algorithms, from linear regression to deep
neural networks, by choosing appropriate loss functions and hypothesis sets.
Disadvantages:
 Focusing soley on minimizing empirical risk may lead to overfitting, where the
model performs well on training data but poorly on new data.
 Dependence on Training data Quality
 Computational Complexity
 Choice of loss function
REGULARIZATION
 Regularization is an important technique in machine learning that helps to
improve model accuracy by preventing overfitting which happens when a model
learns the training data too well including noise and outliers and perform poor
on new data.
 By adding a penalty for complexity it helps simpler models to perform better on
new data.
 It will see main types of regularization i.e Lasso, Ridge and Elastic Net and see
how they help to build more reliable models.
Types of Regularization
1. Lasso Regression

17
  A regression model which uses the L1 Regularization technique is called LASSO
(Least Absolute Shrinkage and Selection Operator) regression.
 It adds the absolute value of magnitude of the coefficient as a penalty term to the
loss function (L).
 This penalty can shrink some coefficients to zero which helps in selecting only
the important features and ignoring the less important ones.

where
m - Number of Features
n- Number of Examples
yi- Actual Target Value
𝑦̂-
𝑖 Predicted Target Value

2. Ridge Regression
 A regression model that uses the L2 regularization technique is called Ridge
regression.
 It adds the squared magnitude of the coefficient as a penalty term to the loss
function(L).

m - Number of features i.e predictor variables

n- Number of examples or data points
yi- Actual target value for the ith example
th
𝑦̂-
𝑖 Predicted Target Value for the i example

wi Coefficients of the features

-

λ - Regularization parameter that controls the strength of regularization

3. Elastic Net Regression
 Elastic Net Regression is a combination of both L1 as well as L2 regularization.
 That shows that we add the absolute norm of the weights as well as the squared
measure of the weights.

18
  With the help of an extra hyper parameter that controls the ratio of the L1 and
L2 regularization.

m - Number of features i.e predictor variables

n- Number of examples or data points
yi- Actual target value for the ith example
th
𝑦̂-
𝑖 Predicted Target Value for the i example

wi Coefficients of the features

-

λ - Regularization parameter that controls the strength of regularization

α = Mixing parameter where 0 ≤ α≤ 1 and α= 1 corresponds to Lasso (L1) regularization,
α= 0 corresponds to Ridge (L2) regularization and Values between 0 and 1 provide a
balance of both L1 and L2 regularization.
Benefits of Regularization
 Prevents Overfitting
 Improves Interpretability
 Enhances Performance
 Stabilizes Models
 Prevents Complexity
 Handles Multicollinearity
 Allows Fine-Tuning
 Promotes Consistency
AUTOENCODERS
 Data encodings are unsupervised learned using an artificial neural network called
an autoencoder.
 An autoencoder learns a lower-dimensional form (encoding) for a higher-
dimensional data to learn a higher-dimensional data in a lower-dimensional
form, frequently for dimensionality reduction.

19
  In an autoencoder, there are two parts, an encoder, and a decoder.
 First, the encoder takes the input and encodes it.
 For example, let the input data be x.
 Then, we can define the encoded function as f(x).
 Between the encoder and the decoder, there is also an internal hidden layer.
 Let’s call this hidden layer h.
 This hidden layer learns the coding of the input that is defined by the encoder.
 So, basically after the encoding, we get h = f(x).
 Finally, the decoder function tries to reconstruct the input data from the hidden
layer coding.

Types of Autoencoders
 An unsupervised neural network operating completely under autoencoders can
be used to compress the input data.
 It is important to take an input image and try to predict the same image as an
output to reconstruct the image from its compressed bottleneck region.

20
Sparse Autoencoders
 To control sparse autoencoders, one can alter the number of nodes at every
hidden layer.
 Since it is challenging to construct a neural network with a customizable number
of nodes in its hidden levels, sparse autoencoders work by suppressing the
activity of certain neurons in those layers.
Contractive Autoencoders
 Prior to rebuilding the input in the decoder, a contractive autoencoder funnels it
through a bottleneck.
 The bottleneck function is being used to learn an image representation of the
image while it is being processed.
 The contractive autoencoder additionally has a regularization term to prevent the
network from figuring out the identity function and converting input to output.
Denoising Autoencoders
 Denoising autoencoders perform similarly to traditional autoencoders in that
they accept an input and output it.
 But they differ from one another in that they don't accept the input image as the
absolute truth. Instead, they use a louder version.
Variational Autoencoders
 Variational autoencoders (VAEs) are models created to address a specific
problem with conventional autoencoders.
 An autoencoder learns to solely represent the input in the so called latent space
or bottleneck during training.
 The post-training latent space is not necessarily continuous, which makes
interpolation challenging.

21
 Unit II
Deep neural networks (DNNs): Difficulty of training DNNs, Greedy layer wise training,
Optimization for training DNNs, Newer optimization methods for neural networks
(AdaGrad, RMSProp, Adam), Second order methods for training, Regularization
methods (dropout, drop connect, batch normalization)
Deep neural networks (DNNs)
 A deep neural network (DNN) is an ANN with multiple hidden layers between
the input and output layers.
 Similar to shallow ANNs, DNNs can model complex non-linear relationships.
 The main purpose of a neural network is to receive a set of inputs, perform
progressively complex calculations on them, and give output to solve real world
problems like classification.
 We restrict ourselves to feed forward neural networks.
 We have an input, an output, and a flow of sequential data in a deep network.

 Neural networks are widely used in supervised learning and reinforcement

learning problems.
 These networks are based on a set of layers connected to each other.
 In deep learning, the number of hidden layers, mostly non-linear, can be large;
say about 1000 layers.

22
  DL models produce much better results than normal ML networks.
 We mostly use the gradient descent method for optimizing the network and
minimising the loss function.
DIFFICULTY OF TRAINING DNNS
 Deep learning offers immense potential, but several challenges can hinder its
effective implementation.
 Addressing these challenges is crucial for developing reliable and efficient
models. Here are the main challenges faced in deep learning:
1. Overfitting and Underfitting
 Balancing model complexity to ensure it generalizes well to new data is
challenging.
 Overfitting occurs when a model is too complex and captures noise in the
training data.
 Underfitting happens when a model is too simple and fails to capture the
underlying patterns.
2. Data Quality and Quantity
 Deep learning models require large, high-quality datasets for training.
 Insufficient or poor-quality data can lead to inaccurate predictions and model
failures.
 Acquiring and annotating large datasets is often time-consuming and
expensive.
3. Computational Resources
 Training deep learning models demands significant computational power and
resources.
 This can be expensive and inaccessible for many organizations.
 High-performance hardware like GPUs and TPUs are often necessary to
handle the intensive computations.
4. Interpretability
 Deep learning models often function as "black boxes," making it difficult to
understand how they make decisions.

23
  This lack of transparency can be problematic, especially in critical
applications.
 Understanding the decision-making process is crucial for trust and
accountability.
5. Hyperparameter Tuning
 Finding the optimal settings for a model’s hyperparameters requires
expertise.
 This process can be time-consuming and computationally intensive.
 Hyperparameters significantly impact the model’s performance, and tuning
them effectively is essential for achieving high accuracy.
6. Scalability
 Scaling deep learning models to handle large datasets and complex tasks
efficiently is a major challenge.
 Ensuring models perform well in real-world applications often requires
significant adjustments.
 This involves optimizing both algorithms and infrastructure to manage
increased loads.
7. Ethical and Bias Issues
 Deep learning models can inadvertently learn and perpetuate biases present
in the training data.
 This can lead to unfair outcomes and ethical concerns.
 Addressing bias and ensuring fairness in models is critical for their
acceptance and trustworthiness.
8. Hardware Limitations
 Training deep learning models requires substantial computational resources,
including high-performance GPUs or TPUs.
 Access to such hardware can be a bottleneck for researchers and practitioners.

24
 9. Adversarial Attacks
 Deep learning models are susceptible to adversarial attacks, where subtle
perturbations to input data can cause misclassification.
 Robustness against such attacks remains a significant concern in safety-
critical applications.
GREEDY LAYER WISE TRAINING
 Greedy layer-wise training is a strategy for training deep neural networks one
layer at a time, instead of training the entire network all at once.
 Each layer is trained independently (greedily), with the output of one layer
serving as the input to the next.
 This method was introduced to overcome the difficulties of training deep
networks, such as:
 Vanishing gradients
 Difficulty in optimizing deep models
 Poor initialization

25
Steps
Train the first layer:
 Use the raw input data to train the first layer (often an autoencoder or Restricted
Boltzmann Machine (RBM)).
 The layer learns to extract low-level features.
Freeze the first layer:
 Keep the weights fixed after training.
Train the second layer:
 Use the output of the first layer as input.
 Again, use unsupervised training (autoencoder or RBM).
Repeat the process:
 Stack and train each subsequent layer one at a time.
 Continue this process to build a deep network.
Fine-tuning
 After stacking all layers, perform supervised fine-tuning using back propagation
on the whole network.
Advantages
 Avoids vanishing gradient problem in early layers
 Provides better weight initialization
 Requires less labeled data (unsupervised pretraining)
Limitations
 Slower overall training (layer-by-layer)
 May require extra steps for fine-tuning
 Largely replaced by better techniques like residual connections, batch norm, and
advanced optimizers
OPTIMIZATION FOR TRAINING DNNS
 Deep Learning is an iterative way of training the machine.
 Like any iterative or cyclic process, DL involves three main components namely,
formulate- training the NN, test the model, and evaluate the model.
 In other words, iteration in DL indicates the number of times the
hyperparameters are update upon.

26
 Hyperparameters are the core entities in any DL models.
 The best combination of various permutations of the hyperparameters must be
set up to ensure accurate results.

 Training deep learning models takes a long time.

 In order to achieve the best training efficiency, the performance of the
optimization algorithm becomes an important factor.
 Deep learning algorithms require optimization in different circumstances but
training the neural network is said to be the most difficult task for the following
reasons:
o Time-consuming: In real-time, training a single neural network instance
on several machines will take days to months in real-time scenarios.
o Expensive: Training the NN is expensive
 A loss function is in use for neural network models to optimize the parameter
values.
 Loss function can be classified into two broad categories as shown.

27
  A loss function evaluates a model's effectiveness by computing the difference
between expected and actual outputs.
 Common loss functions include log loss, hinge loss, and mean square loss.
 An optimizer improves the model by adjusting its parameters (weights and
biases) to minimize the loss function value.
 Examples include RMSProp, ADAM, and SGD.
Regression Losses
Used in problems where the output is a continuous value (e.g., predicting house
prices).
 Mean Absolute Error (MAE) / L1 Loss:
o Calculates the average of absolute differences between predicted and
actual values.
o Less sensitive to outliers.
 Mean Squared Error (MSE) / Quadratic Loss / L2 Loss:
o Computes the average of the squares of errors.
o Penalizes larger errors more than smaller ones.
o Commonly used in regression problems.
 Mean Bias Error (MBE):
o Measures the average bias in the predictions.
o Indicates whether predictions are systematically high or low.
Classification Losses
Used when the output is a class label (e.g., cat vs. dog).
 Cross-Entropy Loss / Negative Log Likelihood:
o Measures the difference between two probability distributions (predicted
vs. actual).
o Commonly used in classification tasks, especially with softmax outputs.
 Hinge Loss / Multiclass SVM Loss:
o Used in Support Vector Machines (SVMs).
o Encourages the correct class to have a score higher than incorrect classes
by a margin.

28
NEWER OPTIMIZATION METHODS FOR NEURAL NETWORKS
(ADAGRAD, RMSPROP, ADAM)
 Modern deep learning requires efficient optimizers to handle large data, deep
architectures, and complex loss landscapes.
 Traditional Stochastic Gradient Descent (SGD) has limitations like slow
convergence and sensitivity to learning rate.
 To address these, advanced optimizers such as AdaGrad, RMSProp, and Adam
were developed.
AdaGrad
 AdaGrad adapts the learning rate for each parameter based on the historical
gradient information.
 The learning rate decreases over time, making AdaGrad effective for sparse
features.

Where:
Gt -is the sum of squared gradients.
Ε - is a small constant to avoid division by zero.
Advantages: Adapts the learning rate, improving training efficiency.
Disadvantages: Learning rate decays too quickly, causing slow convergence.
RMSProp
 RMSProp improves upon AdaGrad by introducing a decay factor to prevent the
learning rate from decreasing too rapidly.

29
Where:
γ - is the decay rate.
E[g2]t - is the exponentially moving average of squared gradients.
Advantages: Prevents excessive decay of learning rates.
Disadvantages: Computationally expensive due to the additional parameter.
Adam (Adaptive Moment Estimation)
 Adam combines the advantages of Momentum and RMSProp.
 It uses both the first moment (mean) and second moment (variance) of gradients
to adapt the learning rate for each parameter.
Update the first moment:

Update the second moment:

Bias correction:

Update parameters:

Where:
β1 and β2 - are the decay rates for the first and second moments.
ε - is a small constant to prevent division by zero.
Advantages: Fast convergence.
Disadvantages: Requires significant memory due to the need to store first and
second moment estimates.

30
SECOND ORDER METHODS FOR TRAINING
 Second-order optimization methods are advanced techniques used in training
neural networks that rely not only on the gradient (first derivative) of the loss
function but also on the curvature information provided by the second
derivative, known as the Hessian matrix.
 The gradient indicates the direction of the steepest descent, while the Hessian
matrix captures how the loss function curves in different directions.
 By incorporating both, second-order methods can make more informed and
accurate updates to the model parameters.
 One of the classic second-order techniques is Newton’s Method, where the
update rule involves multiplying the inverse of the Hessian matrix with the
gradient to determine the next set of weights.
 This allows for faster convergence near the optimum and better handling of
complex error surfaces, including saddle points and flat regions.
 However, second-order methods come with significant computational costs,
as calculating and inverting the Hessian matrix becomes infeasible for large-
scale deep learning models.
 To address this, approximate methods like BFGS and L-BFGS are used,
which estimate the Hessian using limited memory and past gradients.
 Another technique, Hessian-Free Optimization, avoids computing the
Hessian explicitly and instead uses matrix-vector products and conjugate
gradient methods.
 Despite their accuracy and faster convergence, second-order methods are
rarely used in practice for large networks due to their high complexity and
memory demands.
 Nevertheless, they remain important in research settings and for specific
applications where precise optimization is crucial.

31
REGULARIZATION METHODS (DROPOUT, DROP CONNECT, BATCH
NORMALIZATION)
 Regularization refers to a set of techniques used in deep learning to prevent
overfitting by improving the generalization ability of a neural network.
 Overfitting occurs when a model learns the training data too well, including its
noise and outliers, and fails to perform effectively on unseen data.
 Several powerful regularization methods have been introduced to address this
issue, among which Dropout, DropConnect, and Batch Normalization are widely
used.
Dropout
 Dropout is a stochastic regularization technique that randomly deactivates (or
"drops") a fraction of neurons in the network during each training iteration.
 This prevents units from co-adapting too strongly and forces the network to learn
more robust features that generalize better.
 During testing, all neurons are used, but their outputs are scaled by the dropout
rate to maintain consistency.
 Dropout effectively acts as an ensemble of many smaller networks, reducing the
chance of overfitting.
DropConnect
 DropConnect is a variant of dropout that applies regularization at the level of the
weights rather than neurons.
 In DropConnect, individual weights are randomly set to zero during training,
meaning connections between neurons are probabilistically removed.
 This creates a sparse weight matrix and encourages the network to be less reliant
on specific connections, thus enhancing generalization.
 While similar in spirit to dropout, DropConnect can be even more effective in
some architectures but is computationally more expensive.
Batch Normalization
 Batch Normalization, unlike dropout and dropconnect, is not primarily a
regularization method but also serves that function.

32
  It normalizes the input of each layer across the mini-batch so that they have
zero mean and unit variance.
 This helps stabilize and accelerate training by reducing internal covariate
shift, the change in the distribution of inputs to a layer during training.
 Batch normalization also introduces slight noise into the training process
(because it depends on the current mini-batch), which has a regularization
effect similar to dropout.
 Moreover, it allows the use of higher learning rates and reduces sensitivity to
initialization.
 Dropout, DropConnect, and Batch Normalization are crucial techniques for
improving the performance of deep neural networks by reducing overfitting and
speeding up training.
 These methods work in different ways by randomly deactivating neurons,
dropping connections, or normalizing activations but all contribute to building
more stable and generalizable models.

33
 Unit III
Convolution neural networks (CNNs): Introduction to CNNs – convolution, pooling,
Deep CNNs, Different deep CNN architectures – LeNet, AlexNet, VGG, PlacesNet,
training a CNNs: weights initialization, batch normalization, hyperparameter
optimization, Understanding and visualizing CNNs.
Convolution neural networks (CNNs)
 A CNN is composed of an input layer, an output layer, and many hidden layers
in between.
 These layers perform operations that alter the data with the intent of learning
features specific to the data.
Introduction to CNNs
 A Convolutional Neural Network (CNN) is a type of deep learning model
designed specifically to process data that has a grid-like topology, such as
images.
 CNNs are particularly well-suited for image-related tasks because they are
capable of automatically and efficiently learning spatial hierarchies of features
from input data.
 Instead of manually extracting features like edges, corners, or textures, CNNs
learn to detect them through multiple layers of convolution and pooling
operations.
 This makes CNNs extremely powerful for tasks such as image classification,
object detection, and facial recognition.
 The basic building blocks of a CNN include convolutional layers, activation
functions, pooling layers, and fully connected layers.
 The convolutional layers apply a set of filters to the input image, producing
feature maps that highlight important patterns.
 These are followed by activation functions like ReLU (Rectified Linear Unit),
which introduce non-linearity into the model.
 Pooling layers, such as max pooling, reduce the spatial size of the feature maps,
which helps decrease computational cost and prevent overfitting.

34
  After several rounds of convolution and pooling, the output is flattened and
passed through one or more fully connected layers, which perform high-level
reasoning and make the final classification.
 CNNs are preferred over traditional fully connected neural networks for visual
tasks because they use fewer parameters and exploit the spatial structure of
images.
 This not only makes them more computationally efficient but also more effective
in recognizing visual patterns.
 CNNs have found widespread application in many domains, including medical
image diagnosis, autonomous driving, security systems, and even language
processing tasks involving visual data.
 With their ability to learn from raw pixels and generalize across different inputs,
CNNs have become one of the foundational models in modern deep learning.
CNN Architecture

 The fundamental principle of a convolutional neural network (CNN) lies in its

utilization of sliding windows that scan various parts of an image, extracting
valuable features such as color, shape, and contours.
 This process generates a collection of feature maps, each capturing different
aspects of the input image.
 These feature maps are then utilized for tasks such as classification or latent
representation of the image, enabling the network to learn and discern intricate
patterns and structures within the data.

35
CNN Layers
Now let's consider the most important layers of CNN architecture.
Input Layer
 Takes an image as input, usually represented as a 3D array (height × width ×
channels).
Convolutional Layers
Feature Extraction:
 Convolutional layers apply filters (also called kernels) to the input data.
 These filters slide over the input data, performing element-wise
multiplication between the filter weights and the corresponding pixels in the
input image.
 This operation generates feature maps that highlight important patterns and
structures in the data;
Parameter Sharing:
 One key characteristic of convolutional layers is parameter sharing.
 Instead of learning separate parameters for every location in the input image,
the same set of weights is used across the entire image.
 This reduces the number of parameters in the model, making it more efficient
and capable of learning spatial hierarchies of features.
Pooling Layers
Downsampling:
 Pooling layers downsample the feature maps generated by convolutional
layers.
 They do this by reducing the spatial dimensions of the feature maps while
retaining the most important information.
 This helps in reducing computational complexity and preventing overfitting;
Translation Invariance:
 Pooling layers introduce translation invariance, meaning that the network
becomes less sensitive to small translations or shifts in the input data.
 This property makes the network more robust to variations in input images.

36
Dense Layer
 The final dense layer is utilized to flatten the outputs of preceding layers and
extract the final feature vector;
 This layer serves to aggregate the learned representations from earlier layers into
a concise feature vector,
 Enabling the network to make predictions or generate data based on these
extracted features.
Output Layer
 Produces the final prediction (e.g., a label for image classification).
Deep CNNs
 Deep Convolutional Neural Networks (Deep CNNs) are advanced architectures
of CNNs that consist of multiple convolutional, pooling, and fully connected
layers stacked one after another.
 While traditional CNNs may have only a few layers, deep CNNs go much deeper
sometimes containing dozens or even hundreds of layers allowing the network
to learn highly abstract and hierarchical representations of input data.
 As the network goes deeper, the layers capture increasingly complex features:
the initial layers may detect simple edges or colors, while the deeper layers can
recognize shapes, textures, and eventually entire objects or scenes.

37
  The primary advantage of deep CNNs is their ability to automatically extract
deep feature representations from raw data without manual intervention.
 This makes them extremely effective for challenging tasks such as fine-grained
image classification, facial recognition, object detection, and image
segmentation.
 Deep CNNs leverage large datasets and advanced training techniques like batch
normalization, dropout, and data augmentation to avoid overfitting and improve
generalization.
 Examples of deep CNN architectures include AlexNet, VGGNet, GoogLeNet
(Inception), ResNet (Residual Networks), and DenseNet.
 These models have achieved state-of-the-art performance in benchmarks like
ImageNet, demonstrating the power of depth in neural networks.
 However, training deep CNNs requires high computational resources, and
careful architecture design is essential to avoid issues like vanishing gradients.
 To address these challenges, innovations such as residual connections and skip
connections have been introduced to allow gradient flow through very deep
networks.
Different Deep CNN Architectures
 Over the years, several deep Convolutional Neural Network (CNN) architectures
have been developed to improve performance, scalability, and accuracy in
computer vision tasks.
LeNet
 LeNet is one of the earliest and most influential Convolutional Neural Network
(CNN) architectures, developed by Yann LeCun in the late 1980s and early
1990s.
 It was originally designed for handwritten digit recognition, specifically for the
MNIST dataset, which contains images of digits from 0 to 9.
 The LeNet architecture demonstrated the feasibility and power of CNNs in
extracting visual features and performing classification tasks directly from raw
pixel data.

38
  The LeNet architecture consists of several layers that progressively extract and
condense information from input images.
 Here, is it the description of each layer of the LeNet architecture:
 Input Layer: Accepts 32x32 pixel images, often zero-padded if original images
are smaller.
 First Convolutional Layer (C1): Consists of six 5x5 filters, producing six
feature maps of 28x28 each.
 First Pooling Layer (S2): Applies 2x2 average pooling, reducing feature maps'
size to 14x14.
 Second Convolutional Layer (C3): Uses sixteen 5x5 filters, but with sparse
connections, outputting sixteen 10x10 feature maps.
 Second Pooling Layer (S4): Further reduces feature maps to 5x5 using 2x2
average pooling.
Fully Connected Layers:
 First Fully Connected Layer (C5): Fully connected with 120 nodes.
 Second Fully Connected Layer (F6): Comprises 84 nodes.
 Output Layer: Softmax or Gaussian activation that outputs probabilities across
10 classes (digits 0-9).
Applications of LeNet
 Handwritten Digit Recognition
 Bank Cheque Processing
 Postal Code Recognition
 Digitized Document Analysis

39
  Educational Use
 Image Classification
 Prototype and Testing
AlexNet
 AlexNet is a landmark deep learning architecture that significantly advanced the
field of computer vision.
 Developed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, AlexNet
won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in
2012, reducing the top-5 error rate by nearly 10% compared to previous models.
 It demonstrated the true potential of deep convolutional neural networks (CNNs)
trained on large datasets using powerful GPUs.
AlexNet Architecture
 AlexNet consists of 8 layers, including 5 convolutional layers and 3 fully
connected layers.
 It uses traditional stacked convolutional layers with max-pooling in between.
 Its deep network structure allows for the extraction of complex features from
images.
 The architecture employs overlapping pooling layers to reduce spatial
dimensions while retaining the spatial relationships among neighbouring
features.
 Activation function: AlexNet uses the ReLU activation function and dropout
regularization, which enhance the model's ability to capture non-linear
relationships within the data.

 AlexNet was created to be more computationally efficient than earlier CNN

topologies.
 It introduced parallel computing by utilising two GPUs during training.

40
  AlexNet is a relatively shallow network compared to GoogleNet.
 It has eight layers, which makes it simpler to train and less prone to overfitting
on smaller datasets.
 In 2012, AlexNet produced ground-breaking results in the ImageNet Large Scale
Visual Recognition Challenge (ILSVRC).
 It outperformed prior CNN architectures greatly and set the path for the rebirth
of deep learning in computer vision.
 Several architectural improvements were introduced by AlexNet, including the
use of rectified linear units (ReLU) as activation functions, overlapping pooling,
and dropout regularisation.
 These strategies aided in the improvement of performance and generalisation.
Applications
 Image Classification
 Object Detection and Recognition
 Face Recognition
 Medical Image Analysis
 Video Frame Classification
 Scene and Landmark Recognition
 Agricultural and Environmental Monitoring
VGG
 The Visual Geometry Group (VGG) models, particularly VGG-16 and VGG-19,
have significantly influenced the field of computer vision since their inception.
 These models, introduced by the Visual Geometry Group from the University of
Oxford, stood out in the 2014 ImageNet Large Scale Visual Recognition
Challenge (ILSVRC) for their deep convolutional neural networks (CNNs) with
a uniform architecture.
 VGG-19, the deeper variant of the VGG models, has garnered considerable
attention due to its simplicity and effectiveness.
 Before the advent of VGG models, CNN architectures like LeNet-5 and AlexNet
laid the groundwork for deep learning in computer vision.

41
  LeNet-5, introduced in the 1990s, was one of the first successful applications of
CNNs in recognizing handwritten digits.
 AlexNet, which won the ILSVRC in 2012, marked a significant breakthrough by
leveraging deeper architectures and GPU acceleration.
 The VGG models were introduced by Karen Simonyan and Andrew Zisserman
in their 2014 paper titled "Very Deep Convolutional Networks for Large-Scale
Image Recognition."
 The primary objective was to investigate the effect of increasing the depth of
CNNs on large-scale image recognition tasks.
 VGG-16 and VGG-19, with 16 and 19 weight layers respectively, were among
the most notable models presented in the paper.
 Their design was characterized by using small 3x3 convolution filters
consistently across all layers, which simplified the network structure and
improved performance.

 VGG-19 is a deep convolutional neural network with 19 weight layers,

comprising 16 convolutional layers and 3 fully connected layers.
 The architecture follows a straightforward and repetitive pattern, making it easier
to understand and implement.
The key components of the VGG-19 architecture are:

42
 Convolutional Layers: 3x3 filters with a stride of 1 and padding of 1 to preserve
spatial resolution.
Activation Function: ReLU (Rectified Linear Unit) applied after each
convolutional layer to introduce non-linearity.
Pooling Layers: Max pooling with a 2x2 filter and a stride of 2 to reduce the
spatial dimensions.
Fully Connected Layers: Three fully connected layers at the end of the network
for classification.
Softmax Layer: Final layer for outputting class probabilities.
Uniform Convolution Filters: Consistently using 3x3 convolution filters
simplifies the architecture and helps maintain uniformity.
Deep Architecture: Increasing the depth of the network enables learning more
complex features.
ReLU Activation: Introducing non-linearity helps in learning complex patterns.
Max Pooling: Reduces the spatial dimensions while preserving important
features.
Fully Connected Layers: Combines the learned features for classification.
PlacesNet
 PlacesNet is a Convolutional Neural Network (CNN) model specifically
designed for scene recognition, as opposed to object recognition like ImageNet-
trained models (e.g., AlexNet, VGG).
 It was developed as part of the Places Database project, led by MIT's Computer
Science and Artificial Intelligence Laboratory (CSAIL).
 The Places CNNs are trained on the Places dataset, which contains millions of
images labeled with over 400 scene categories such as “kitchen,” “forest,”
“office,” “airport,” and “stadium.”
 This makes PlacesNet exceptionally good at understanding contextual
environments rather than focusing solely on individual objects.
 The architecture of PlacesNet is similar to popular CNNs like AlexNet, VGG, or
ResNet, but the key difference lies in the training data: it’s trained on scenes, not
objects.

43
  This distinction allows it to perform well on tasks like scene classification,
robotic navigation, autonomous driving, and context-aware computing.
 For example, while ImageNet-trained models might recognize a "microwave" or
"refrigerator," a PlacesNet model would identify the broader environment as a
"kitchen."
 Because of its specialization, PlacesNet is often used in combination with object
detection models in hybrid systems for tasks requiring a deeper understanding of
surroundings.
 It also serves as a strong feature extractor in research involving context-based
image retrieval, scene parsing, and environment recognition in videos and still
images.
Training a CNNs
 The image data from 32 pixels × 32 pixels are presented to the network and
passed through the network layers
Weights Initialization
 Weight initialization is a critical step in training a Convolutional Neural Network
(CNN), as it can significantly impact the convergence speed, training stability,
and final performance of the model.
 In a neural network, weights determine how input data is transformed as it passes
through the layers.
 If weights are poorly initialized, it can lead to problems such as vanishing
gradients, exploding gradients, or the network getting stuck in local minima early
in training.

44
Zero Initialization
 All the weights are assigned zero as the initial value is zero initialization.
 This kind of initialization is highly ineffective as neurons learn the same feature
during each iteration.
 Rather, during any kind of constant initialization, the same issue happens to
occur.
 Thus, constant initializations are not preferred.
Random Initialization
 In an attempt to overcome the shortcomings of Zero or Constant Initialization,
random initialization assigns random values except for zeros as weights to
neuron paths.
 However, assigning values randomly to the weights, problems such as
Overfitting, Vanishing Gradient Problem, Exploding Gradient Problem might
occur.
Uniform Distribution:

Normal Distribution:

Where:
 nin= number of input units to the layer
 nout= number of output units from the layer
He Normal Initialization
 In He Normal weight initialization, the weights are assigned from values of a
normal distribution as follows:

45
 Normal Distribution:

Uniform Distribution:

nin = number of input units to the layer

Batch Normalization
 Batch Normalization is a technique used to improve the training and performance
of neural networks, particularly CNNs.
 The article aims to provide an overview of batch normalization in CNNs along
with the implementation in PyTorch and TensorFlow.
Need for Batch Normalization
 Batch Normalization in CNN addresses several challenges encountered during
training.
 There are following reasons highlight the need for batch normalization in CNN:
 Addressing Internal Covariate Shift:
 Internal covariate shift occurs when the distribution of network
activations changes as parameters are updated during training.
 Batch normalization addresses this by normalizing the activations in each
layer, maintaining consistent mean and variance across inputs throughout
training.
 This stabilizes training and speeds up convergence.

46
  Improving Gradient Flow:
 Batch normalization contributes to stabilizing the gradient flow during
backpropagation by reducing the reliance of gradients on parameter
scales.
 As a result, training becomes faster and more stable, enabling effective
training of deeper networks without facing issues like vanishing or
exploding gradients.
 Regularization Effect:
 During training, batch normalization introduces noise to the network
activations, serving as a regularization technique.
 This noise aids in averting overfitting by injecting randomness and
decreasing the network's sensitivity to minor fluctuations in the input
data.

Compute Batch Mean:

Compute Batch Variance:

47
Normalize:

Scale and Shift

Where:
 ε : Small constant to avoid division by zero
 γ, β: Learnable parameters that allow the network to undo normalization if
needed
 Reduces internal covariate shift (change in input distribution of layers)
 Accelerates training by allowing higher learning rates
 Improves generalization and reduces overfitting
 Makes deep networks more stable

Hyperparameter Optimization

 Hyperparameter Optimization is the process of selecting the best combination of

hyperparameters (not learned from data) to improve the performance of a neural
network.
 These include learning rate, batch size, number of layers, optimizer type, dropout
rate, etc.
 It can affect both the speed and quality of the model's performance.
 A high learning rate can cause the model to converge too quickly possibly
skipping over the optimal solution.
 A low learning rate might lead to slower convergence and require more time
and computational resources.

48
Techniques for Hyperparameter
 Models can have many hyperparameters and finding the best combination of
parameters can be treated as a search problem.
 The two best strategies for Hyperparameter tuning are:
GridSearchCV
 GridSearchCV is a brute-force technique for hyperparameter tuning.
 It trains the model using all possible combinations of specified hyperparameter
values to find the best-performing setup.
 It is slow and uses a lot of computer power which makes it hard to use with big
datasets or many settings.
 It works using below steps:
 Create a grid of potential values for each hyperparameter.
 Train the model for every combination in the grid.
 Evaluate each model using cross-validation.
 Select the combination that gives the highest score.

RandomizedSearchCV
 As the name suggests RandomizedSearchCV picks random combinations of
hyperparameters from the given ranges instead of checking every single
combination like GridSearchCV.
 In each iteration it tries a new random combination of hyperparameter values.
 It records the model’s performance for each combination.
 After several attempts it selects the best-performing set.

49
Advantages
 Improved Model Performance
 Reduced Overfitting and Underfitting
 Enhanced Model Generalizability
 Optimized Resource Utilization
 Improved Model Interpretability
Understanding and visualizing CNNs
 Understanding and visualizing Convolutional Neural Networks (CNNs) is
essential to interpret how these deep learning models process visual data and
make decisions.
 CNNs operate in a hierarchical manner, where the initial layers detect simple
patterns like edges and textures, while deeper layers learn complex features such
as object parts and overall shapes.
 However, due to their complexity, CNNs are often seen as “black boxes.”
 Visualization techniques help to open up this black box by showing what the
model learns at different stages, making the model’s decisions more transparent
and interpretable.
 One commonly used technique is feature map visualization, where we observe
the output of convolutional layers in response to an input image.
 This helps to identify which features the filters are extracting.

50
 Filter (kernel) visualization shows the weights learned by convolutional layers,
giving insights into what kind of patterns each filter is tuned to detect.
 Activation maps and saliency maps highlight the regions of an image that
strongly influence the model’s output.
 Advanced methods like Grad-CAM (Gradient-weighted Class Activation
Mapping) generate heatmaps over input images, indicating which areas
contributed most to the prediction.
 This is especially useful in applications like medical imaging, where it's
important to know why a model predicted a certain diagnosis.
 Other methods, such as deconvolutional networks, attempt to reverse the
convolutional process and project activations back to the input space to show
what kind of input activated a particular neuron.
 Dimensionality reduction techniques like t-SNE and PCA are also used to
visualize how features are clustered in high-dimensional space after being
processed by the CNN.
 These visualization methods are not only helpful for gaining intuition but also
for debugging models, checking for overfitting, and ensuring the network is
learning meaningful and generalizable patterns.
 Tools like TensorBoard, Netron, Captum (for PyTorch), and tf-explain (for
TensorFlow) are commonly used for implementing these visualizations.

51
 Unit IV
Recurrent neural networks (RNNs): Sequence modeling using RNNs, Backpropagation
through time, Long Short Term Memory (LSTM), Bidirectional LSTMs, Bidirectional
RNNs, Gated RNN Architecture - Generative models: Restricted Boltzmann Machines
(RBMs), Stacking RBMs, Belief nets.
Recurrent neural networks (RNNs)
 A recurrent neural network (RNN) is a deep learning model that is trained to
process and convert a sequential data input into a specific sequential data output.

 RNNs share similarities in input and output structures with other deep learning
architectures but differ significantly in how information flows from input to
output.
 Deep neural networks where each dense layer has distinct weight matrices.
 RNNs use shared weights across time steps, allowing them to remember
information over sequences.
 At each time step RNNs process units with a fixed activation function.
 These units have an internal hidden state that acts as memory that retains
information from previous time steps.
 This memory allows the network to store past knowledge and adapt based on
new inputs.

52
Sequence Modeling Using RNNs
 Sequence modeling refers to the task of learning patterns or dependencies in data
where the order of elements matters.
 Examples include time-series forecasting, natural language processing (NLP),
and audio processing.
 Traditional neural networks process fixed-size inputs and ignore the sequential
nature of data.
 RNNs are specially designed to model sequences by maintaining a hidden state
that carries information across time steps.
Types of Recurrent Neural Networks
 There are four types of RNNs based on the number of inputs and outputs in the
network:
1. One-to-One RNN
 This is the simplest type of neural network architecture where there is a single
input and a single output.
 It is used for straightforward classification tasks such as binary classification
where no sequential data is involved.

2. One-to-Many RNN
 In a One-to-Many RNN the network processes a single input to produce multiple
outputs over time.
 This is useful in tasks where one input triggers a sequence of predictions
(outputs).

53
  For example in image captioning a single image can be used as input to generate
a sequence of words as a caption.

3. Many-to-One RNN
 The Many-to-One RNN receives a sequence of inputs and generates a single
output.
 This type is useful when the overall context of the input sequence is needed to
make one prediction.
 In sentiment analysis the model receives a sequence of words (like a sentence)
and produces a single output like positive, negative or neutral.

4. Many-to-Many RNN
 The Many-to-Many RNN type processes a sequence of inputs and generates a
sequence of outputs.
 In language translation task a sequence of words in one language is given as
input and a corresponding sequence in another language is generated as output.

54
Challenges in Sequence Modeling with RNNs:
 Vanishing/Exploding gradients: Makes learning long-term dependencies
difficult.
 Limited memory: Standard RNNs struggle with long sequences.

Backpropagation through Time

 Backpropagation through time (BPTT) is a method used in recurrent neural
networks (RNNs) to train the network by backpropagating errors through time.
 In a traditional feedforward neural network, the data flows through the network
in one direction, from the input layer through the hidden layers to the output layer.
 However, in RNNs, there are connections between nodes in different time steps,
which means that the output of the network at one time step depends on the input
at that time step as well as the previous time steps.

55
  BPTT works by unfolding the RNN over time, creating a series of interconnected
feedforward networks.
 Each time step corresponds to one layer in this unfolded network, and the weights
between layers are shared across time steps.
 The unfolded network can be thought of as a very deep feedforward network,
where the weights are shared across layers.
 During training, the error is backpropagated through the unfolded network, and
the weights are updated using gradient descent.
 This allows the network to learn to predict the output at each time step based on
the input at that time step as well as the previous time steps.
Uses of BPTT:
 BPTT is a widely used technique for training recurrent neural networks (RNNs)
that can be used for various applications such as speech recognition, language
modeling, and time series prediction.
 Here are some specific use cases for BPTT:
 Speech recognition:
 BPTT can be used to train RNNs for speech recognition tasks, where the
network takes in a sequence of audio samples and predicts the
corresponding text.
 BPTT allows the network to learn the temporal dependencies in the audio
signal and use them to make accurate predictions.
 Language modeling:
 BPTT can also be used to train RNNs for language modeling tasks, where
the network predicts the probability distribution of the next word in a
sequence given the previous words.
 This can be useful for applications such as text generation and machine
translation.

56
  Time series prediction:
 BPTT can be used to train RNNs for time series prediction tasks, where
the network takes in a sequence of data points and predicts the next value
in the sequence.
 BPTT allows the network to learn the temporal dependencies in the data
and use them to make accurate predictions.
Limitation of BPTT:
 Vanishing gradients
 Exploding gradients
 Memory limitations
 Difficulty in parallelization
Long Short Term Memory (LSTM) / Gated RNN Architecture
 Long Short-Term Memory (LSTM) is an enhanced version of the Recurrent
Neural Network (RNN).
 LSTMs can capture long-term dependencies in sequential data making them
ideal for tasks like language translation, speech recognition and time series
forecasting.
LSTM Architecture
LSTM architectures involves the memory cell which is controlled by three gates:
o Input gate: Controls what information is added to the memory cell.
o Forget gate: Determines what information is removed from the memory
cell.
o Output gate: Controls what information is output from the memory cell.
 LSTM architecture has a chain structure that contains four neural networks and
different memory blocks called cells.

57
1. Forget Gate
 The information that is no longer useful in the cell state is removed with the
forget gate.
 Two inputs xt (input at the particular time) and ht−1 (previous cell output) are fed
to the gate and multiplied with weight matrices followed by the addition of bias.
 The resultant is passed through an activation function which gives a binary
output.
 If for a particular cell state the output is 0, the piece of information is forgotten
and for output 1, the information is retained for future use.

Where:
Wf - represents the weight matrix associated with the forget gate.
[ht−1,xt] - denotes the concatenation of the current input and the previous hidden state.
bf - is the bias with the forget gate.
σ - is the sigmoid activation function.
2. Input gate
 The addition of useful information to the cell state is done by the input gate.
 First the information is regulated using the sigmoid function and filter the values
to be remembered similar to the forget gate using inputs ht−1 and xt .

58
  Then, a vector is created using tanh function that gives an output from -1 to +1
which contains all the possible values from ht−1 and xt.
 At last the values of the vector and the regulated values are multiplied to obtain
the useful information.
 The equation for the input gate is:

3 Output gate
 The task of extracting useful information from the current cell state to be
presented as output is done by the output gate.
 First, a vector is generated by applying tanh function on the cell.
 Then, the information is regulated using the sigmoid function and filter by the
values to be remembered using inputs ht−1 and xt.
 At last the values of the vector and the regulated values are multiplied to be sent
as an output and input to the next cell.
 The equation for the output gate is:

Applications of LSTM
 Language Modeling
 Speech Recognition
 Time Series Forecasting
 Anomaly Detection
 Recommender Systems
 Video Analysis

59
Bidirectional LSTMs (Bi-LSTM)
 Bidirectional LSTM networks function by presenting each training sequence
forward and backward to two independent LSTM networks, both of which are
coupled to the same output layer.
 This means that the Bi-LSTM contains comprehensive, sequential information
about all points before and after each point in a particular sequence.
 Rather than encoding the sequence in the forward direction only,
 We encode it in the backward direction as well and concatenate the results from
both forward and backward LSTM at each time step.
 The encoded representation of each word now understands the words before and
after the specific word.

Example: Consider the sentence “I will swim today”. The below image represents the
encoded representation of the sentence in the Bi-LSTM network.

60
  So when forward LSTM occurs, “I” will be passed into the LSTM network at
time t = 0, “will” at t = 1, “swim” at t = 2, and “today” at t = 3.
 In backward LSTM “today” will be passed into the network at time t = 0, “swim”
at t = 1, “will” at t = 2, and “I” at t = 3.
 In this way, results from both forward and backward LSTM at each time step are
calculated.
Bidirectional RNNs
 A Bidirectional Recurrent Neural Network (BRNN) is an extension of the
traditional RNN that processes sequential data in both forward and backward
directions.
 This allows the network to utilize both past and future context when making
predictions providing a more comprehensive understanding of the sequence.
 A BRNN moves forward through the sequence, updating the hidden state based
on the current input and the prior hidden state at each time step.
 The key difference is that a BRNN also has a backward hidden layer which
processes the sequence in reverse,
 Updating the hidden state based on the current input and the hidden state of the
next time step.

61
Working of Bidirectional Recurrent Neural Networks (BRNNs)
1. Inputting a Sequence: A sequence of data points each represented as a vector with
the same dimensionality is fed into the BRNN. The sequence may have varying lengths.
2. Dual Processing: BRNNs process data in two directions:
 Forward direction: The hidden state at each time step is determined by the
current input and the previous hidden state.
 Backward direction: The hidden state at each time step is influenced by the
current input and the next hidden state.
3. Computing the Hidden State: A non-linear activation function is applied to the
weighted sum of the input and the previous hidden state creating a memory mechanism
that allows the network to retain information from earlier steps.
4. Determining the Output: A non-linear activation function is applied to the weighted
sum of the hidden state and output weights to compute the output at each step. This
output can either be:
 The final output of the network.
 An input to another layer for further processing.
Restricted Boltzmann Machines (RBMs)
 A Restricted Boltzmann Machine (RBM) is a type of generative stochastic neural
network that learns the underlying probability distribution of input data through
unsupervised learning.
 It consists of two layers: a visible layer that represents the observed input data
and a hidden layer that captures complex features and patterns.
 The key characteristic of RBMs is that there are no connections between nodes
in the same layer, making them "restricted" and allowing for more efficient
learning and inference.
 RBMs operate based on an energy function, which defines the joint configuration
of the visible and hidden units.
 The model learns by minimizing this energy, thereby maximizing the probability
of input data. The energy function is defined as:

62
  where vi and hj represent the visible and hidden units, ai and bj are their respective
biases, and wij are the weights connecting them.
 Training an RBM involves a technique called Contrastive Divergence (CD).
 In this process, the model first performs a forward pass to compute the activation
probabilities of the hidden units given the visible units.
 Then, it performs a reconstruction step, generating the visible units again from
the hidden layer.
 The model updates its weights based on the difference between the product of
the original input and hidden activations and the reconstructed values.
 This iterative learning gradually improves the model's ability to represent the
data distribution.
 RBMs are widely used in applications such as feature extraction, dimensionality
reduction, and collaborative filtering.
 They also serve as the fundamental building blocks for Deep Belief Networks
(DBNs) and were historically significant in the development of deep learning
architectures.
 Compared to autoencoders, RBMs are probabilistic, generative, and typically use
sigmoid or binary activation functions,
 Whereas autoencoders use deterministic functions and are trained using
backpropagation.
Stacking RBMs
 Stacking Restricted Boltzmann Machines (RBMs) is a technique used to form a
Deep Belief Network (DBN), which is a deep architecture composed of multiple
layers of RBMs trained sequentially.
 Each RBM in the stack learns to capture higher-level representations of the data
from the previous layer, allowing the network to model increasingly abstract
features.

63
  The idea behind stacking is to leverage the unsupervised learning power of
individual RBMs and build a hierarchical model that can be fine-tuned for
various tasks such as classification or regression.

 The process of stacking begins by training the first RBM with the raw input data
in an unsupervised manner using Contrastive Divergence.
 Once trained, the activations of the hidden layer from the first RBM are treated
as the "visible" data for the second RBM, which is then trained similarly.
 This process is repeated, stacking multiple RBMs on top of each other.
 As a result, each successive RBM learns a representation of the data that is more
abstract and structured than the layer before it.
 After the unsupervised pretraining of all RBMs, the entire stack forms a Deep
Belief Network (DBN).
 To use this network for supervised tasks like classification, a softmax or logistic
regression layer is typically added on top,
 The full network is fine-tuned using backpropagation on labeled data.
 This two-stage training process helps in better weight initialization and improves
the convergence and generalization of deep networks.
Belief nets
 Belief Networks, also known as Bayesian Belief Networks (BBNs) or simply
Bayesian Networks, are a type of probabilistic graphical model that represents a
set of variables and their conditional dependencies via a directed acyclic graph
(DAG).

64
 In these networks, each node represents a random variable, and each directed
edge indicates a conditional dependency between variables.
 The strength of these dependencies is quantified using conditional probability
distributions (CPDs).
 Belief nets provide a structured way to model uncertainty in complex systems
and allow for reasoning and inference under uncertainty.
 Each variable in a belief net is conditionally independent of its non-descendants,
given its parents in the graph.
 This factorization allows the joint probability distribution of all variables in the
network to be represented as a product of smaller, local probability distributions:

 This efficient representation makes belief networks useful for modeling domains
where knowledge is incomplete or probabilistic in nature, such as medical
diagnosis, risk analysis, and decision support systems.
 Belief nets can be used for both inference and learning.
 Inference involves computing the probability of certain variables given observed
evidence,
 While learning can involve either parameter learning (estimating CPDs) or
structure learning (learning the graph itself) from data.
 Techniques like Bayesian inference, expectation-maximization (EM), and
Markov Chain Monte Carlo (MCMC) are commonly used in this context.

65
 Unit V
Learning sigmoid belief nets, Deep belief nets Under complete - Auto encoder,
Regularized Auto encoder, stochastic Encoders and Decoders, Contractive Encoders.
Applications: Applications in vision, speech and natural language processing.
Learning sigmoid belief nets
 A Sigmoid Belief Net (SBN) is a type of directed probabilistic graphical model
composed of binary stochastic units, where the activation of each node is
governed by a sigmoid function.
 Restricted Boltzmann Machines (RBMs), which are undirected models, SBNs
are directed acyclic graphs where information flows in one direction, typically
from input to output.
 In an SBN, each unit represents a binary variable that is conditionally dependent
on its parent units in the graph.

 The activation probability of a unit in an SBN is determined by the sigmoid

function applied to a weighted sum of its parent nodes.
 The probability that a unit hj is activated (i.e., takes the value 1) is given by:

 Learning in Sigmoid Belief Nets involves adjusting the weights and biases to
maximize the likelihood of observed data.

66
  This is challenging because computing the exact posterior distribution over
hidden variables is generally intractable due to the complex dependencies
introduced by the directed structure.
 To address this, approximate inference methods such as variational inference,
mean-field approximation are used during training.
 Training is typically performed using the Expectation-Maximization (EM)
algorithm or gradient-based methods.
 In deep SBNs, which consist of multiple layers of stochastic hidden units, layer-
wise pretraining may be used to initialize the network,
 Fine-tuning using backpropagation through stochastic nodes with techniques like
stochastic gradient descent and reparameterization tricks.
 SBNs can model complex distributions and are useful for generative tasks, such
as data generation, missing data imputation, and unsupervised feature learning.
 Though less popular today compared to deep neural networks and VAEs,
sigmoid belief nets have historical significance in the development of deep
generative models and probabilistic learning frameworks.
Deep belief nets Under complete
 Deep Belief Networks (DBNs) are a type of deep learning architecture
combining unsupervised learning principles and neural networks.
 They are composed of layers of Restricted Boltzmann Machines (RBMs), which
are trained one at a time in an unsupervised manner.
 The output of one RBM is used as the input to the next RBM, and the final output
is used for supervised learning tasks such as classification or regression.
 DBNs have been used in various applications, including image recognition,
speech recognition, and natural language processing.
 They have been shown to achieve state-ofthe-art results in many tasks and are
one of the most powerful deep learning architectures currently available.

Architecture of DBN
 The basic structure of a DBN is composed of several layers of RBMs.

67
 A probability distribution is learned over the input data by each RBM, which is
a generative model.
 While the successive layers of the DBN learn higher-level features, the initial
layer of the DBN learns the fundamental structure of the data.
 For supervised learning tasks like classification or regression, the DBN's last
layer is used.

 Each RBM in a DBN is trained independently using contrastive divergence,

which is an unsupervised learning method.
 The gradient of the log-likelihood of the data for the RBM's parameters can be
approximated using this method.
 The output of one trained RBM is then used as the input for the subsequent RBM,
which is done by stacking the trained RBMs on top of one another.
 After the DBN has been trained, supervised learning tasks can be performed on
it by adjusting the weights of the final layer using a supervised learning technique
like backpropagation.
 This fine-tuning process can improve the DBN's performance on the specific task
it was trained for.

68
Auto encoder
 Autoencoders are very useful in the field of unsupervised machine learning.
 They can be used to reduce the data's size and compress it.
 Principle Component Analysis (PCA), which finds the directions along which
data can be extrapolated with the least amount of variance, and autoencoders,
which reconstruct our original input from a compressed version of it, differ from
one another.
 If necessary, the original data can be recovered using an autoencoder using the
compressed data.
Autoencoder architecture
 An autoencoder is composed of three parts:
 Encoder,
 Bottleneck or code
 Decoder
 These components work together to capture the key features of the input data
and use them to generate accurate reconstructions.
 Autoencoders optimize their output by adjusting the weights of both the encoder
and decoder, aiming to produce a compressed representation of the input that
preserves critical features.
 This optimization minimizes reconstruction error, which represents the
difference between the input and the output data.

69
Encoder
 First, the encoder compresses the input data into a more efficient representation.
 Encoders generally consist of multiple layers with fewer nodes in each layer.
 As the data is processed through each layer, the reduced number of nodes forces
the network to learn the most important features of the data to create a
representation that can be stored in each layer.
 This process, known as dimensionality reduction, transforms the input into a
compact summary of the key characteristics of the data.
 Key hyperparameters in the encoder include the number of layers and neurons
per layer, which determine the depth and granularity of the compression,
 The activation function, which dictates how data features are represented and
transformed at each layer.
Bottleneck
 The bottleneck, also known as the latent space or code, is where the compressed
representation of the input data is stored during processing.
 The bottleneck has a small number of nodes; this limits the amount of data that
can be stored and determines the level of compression.
 The number of nodes in the bottleneck is a tunable hyperparameter, allowing
users to control the trade-off between compression and data retention.
 If the bottleneck is too small, the autoencoder may reconstruct the data
incorrectly due to the loss of important details.
 On the other hand, if the bottleneck is too large, the autoencoder may simply
copy the input data instead of learning a meaningful, general representation.
Decoder
 In this final step, the decoder re-creates the original data from the compressed
form using the key features learned during the encoding process.
 The quality of this decompression is quantified using the reconstruction error,
which is essentially a measure of how different the reconstructed data is from the
input.
 Reconstruction error is generally calculated using mean squared error (MSE).

70
  Because MSE measures the squared difference between the original and
reconstructed data,
 It provides a mathematically straightforward way to penalize larger
reconstruction errors more heavily.
Regularized Auto encoder
 A Regularized Autoencoder is an enhanced version of the standard autoencoder
that includes a regularization term in the loss function to improve the quality of
learned representations and prevent overfitting.
 While a basic autoencoder aims to reconstruct its input through a bottleneck,
regularized autoencoders introduce constraints or penalties that force the model
to learn more robust, generalizable, and meaningful features.
Types of autoencoders

There are several types of specialized autoencoders, each optimized for specific
applications, similar to other neural networks.
Denoising autoencoders:
 Denoising autoencoders are designed to reconstruct clean data from noisy or
corrupted input.
 During training, noise is intentionally added to input data, enabling the model
to learn features that remain consistent despite the noise.
 Outputs are then compared to the original clean inputs.
 This process makes denoising autoencoders highly effective in image- and
audio-noise reduction tasks, including removing background noise in video
conferences.
Sparse autoencoders:
 Sparse autoencoders restrict the number of active neurons at any given time,
encouraging the network to learn more efficient data representations compared
to standard autoencoders.
 This sparsity constraint is enforced through a penalty that discourages activating
more neurons than a specified threshold.

71
  Sparse autoencoders simplify high-dimensional data while preserving essential
features, making them valuable for tasks such as extraction of interpretable
features and visualization of complex datasets.
Variational autoencoders (VAEs):
 VAEs generate new data by encoding features from training data into a
probability distribution, rather than a fixed point.
 By sampling from this distribution, VAEs can generate diverse new data, instead
of reconstructing the original data from the input.
 This capability makes VAEs useful for generative tasks, including synthetic data
generation.
 For example, in image generation, a VAE trained on a dataset of handwritten
numbers can create new, realistic-looking digits based on the training set that are
not exact replicas.
Contractive autoencoders:
 Contractive autoencoders introduce an additional penalty term during the
calculation of reconstruction error, encouraging the model to learn feature
representations that are robust to noise.
 This penalty helps prevent overfitting by promoting feature learning that is
invariant to small variations in input data.
 As a result, contractive autoencoders are more robust to noise than standard
autoencoders.
Convolutional autoencoders (CAEs):
 CAEs utilize convolutional layers to capture spatial hierarchies and patterns
within high-dimensional data.
 The use of convolutional layers makes CAEs particularly well suited for
processing image data.
 CAEs are commonly used in tasks like image compression and anomaly
detection in images.

72
Stochastic Encoders and Decoders
 Stochastic encoders and decoders are components of probabilistic models, such
as Variational Autoencoders (VAEs), where randomness is intentionally
introduced into the encoding and decoding processes.
 Unlike deterministic autoencoders, which map each input to a single point in the
latent space, stochastic models map each input to a probability distribution over
latent variables.
 This allows the network to capture uncertainty, variation, and richer
representations in the data.
 In a stochastic encoder, instead of directly outputting a fixed latent vector, the
encoder learns to output parameters of a distribution (usually a Gaussian), such
as the mean (μ) and standard deviation (σ) for each input.
 A latent vector z is then sampled from this distribution:

 The stochastic decoder then takes this sampled latent vector z and maps it back
to a distribution over the output space, instead of a single output point.
 For example, it might predict the parameters of a Gaussian or Bernoulli
distribution for each pixel in the case of image reconstruction, allowing it to
generate diverse outputs from the same latent representation.
 Stochastic encoders and decoders are at the heart of generative models like
VAEs, where the aim is to learn a latent space from which realistic new samples
can be drawn.
 These models are especially useful for data generation, uncertainty estimation,
anomaly detection, and semi-supervised learning.
Contractive Encoders
 A Contractive Encoder, primarily used in Contractive Autoencoders (CAEs), is
a type of encoder designed to learn robust and invariant features by penalizing
sensitivity to small input variations.

73
  Unlike standard autoencoders that focus solely on minimizing reconstruction
error,
 Contractive encoders include a regularization term in their loss function that
forces the encoder’s output to change very little in response to small changes in
the input.
 This makes the learned representations more stable and useful for tasks like
classification, clustering, or denoising.
 The key idea behind contractive encoding is to minimize the Jacobian of the
encoder's activations with respect to the input.
 Mathematically, the loss function includes a term:

 Here, x is the input, x^ is the reconstructed input, h(x) is the encoded

representation, ∇xh(x) is the Jacobian matrix of partial derivatives, and λ is the
regularization strength.
 This term penalizes large gradients, encouraging the encoder to be insensitive to
input perturbations.
 Contractive encoders help the model focus on essential, underlying features of
the data rather than memorizing surface-level noise.
 This makes them particularly effective for tasks where robust feature learning is
critical, such as in noisy environments, semi-supervised learning, and low-data
regimes.
Applications in vision
Image and Video Recognition
Deep learning has made it possible for machines to understand visual
information in ways similar to humans.
 Self-driving cars use deep learning with cameras to detect pedestrians, traffic
signs and other vehicles to navigate safely.
 Facial recognition systems match people’s facial features for security, phone
unlocking or crowd identification.

74
  Apps use image classification to recognize plants, animals and products making
it useful in education and e-commerce.
Natural Language Processing (NLP)
NLP allows systems to read, understand and write human language with context
and clarity.
 Virtual assistants like Siri and Alexa use NLP to interpret spoken commands and
respond naturally.
 Chatbots use NLP to interact with users and answer queries in customer support.
 Text summarization helps create short summaries from long documents, saving
time.
Speech Recognition
Deep learning has made voice interaction with machines more practical and
accurate. It converts speech into text and understands spoken language.
 Voice typing and dictation tools let users speak instead of typing.
 Automated customer support systems respond to voice commands and help users
navigate services.
 It is used in virtual meetings and live events for real-time transcription.
 Many smart devices now come with voice control features powered by deep
learning.
Recommendation Systems
Recommendation engines use deep learning to personalize content and product
suggestions. These systems learn from user behavior and improve experiences across
platforms.
 Netflix and YouTube suggest videos based on your watch history and
preferences.
 E-commerce platforms like Amazon recommend products based on browsing
and purchase patterns.
 Music apps suggest playlists and songs that match your taste.

75
Healthcare and Drug Discovery
Deep learning in healthcare helps by speeding up diagnosis and drug
development. It assists doctors and researchers in making medical decisions with higher
confidence.
 Medical imaging tools detect diseases like cancer from scans such as X-rays and
MRIs.
 AI models can predict drug effectiveness by simulating molecular behavior.
 Researchers use it to find potential drug targets faster from biological datasets.
 Deep learning reduces trial-and-error in medicine.
Cybersecurity and Scientific Research
Deep learning plays a key role in both securing digital systems and driving
scientific discovery. It can detect threats and support faster breakthroughs in research.
 Cybersecurity systems use it to detect unusual activity and prevent hacking or
malware attacks.
 Fraud detection models flag suspicious transactions in real time, reducing
financial losses.
 It processes massive datasets in fields like physics and material science.
Speech and Natural Language Processing
 Speech and Natural Language Processing (NLP) are two crucial areas of artificial
intelligence that deal with the interaction between humans and machines using
natural language.
1. Speech Processing
Speech processing focuses on enabling machines to understand, interpret, and
generate human speech. It includes:
 Speech Recognition: Converting spoken language into text (e.g., Google Voice
Typing, Siri).
 Speech Synthesis (Text-to-Speech): Generating spoken output from text (e.g.,
screen readers).
 Speaker Identification: Identifying or verifying who is speaking based on voice
characteristics.

76
  Speech Emotion Recognition: Understanding emotions through voice tone and
pitch.
Speech processing requires techniques like:
 Signal processing
 Hidden Markov Models (HMM)
 Recurrent Neural Networks (RNNs) and LSTMs
 Transformer-based models
2. Natural Language Processing (NLP)
NLP is the field that helps computers understand, interpret, and generate human
languages. It involves several tasks:
 Text Preprocessing: Tokenization, stemming, lemmatization, stop-word
removal.
 Syntactic Analysis: Parsing and part-of-speech tagging.
 Semantic Analysis: Understanding the meaning of words and sentences.
 Sentiment Analysis: Determining whether text expresses positive, negative, or
neutral sentiment.
 Machine Translation: Translating text between languages (e.g., Google
Translate).
 Question Answering & Chatbots: Developing systems that can answer
questions (e.g., ChatGPT).
 Information Retrieval & Extraction: Finding relevant data or facts from large
text sources.
Advanced NLP uses deep learning models like:
 RNNs/LSTMs/GRUs: For sequence modeling.
 Transformers (e.g., BERT, GPT): For attention-based understanding and
generation.
 Word Embeddings (e.g., Word2Vec, GloVe): For capturing word meanings in
vector form.
Applications in Real Life
 Virtual Assistants: Alexa, Siri, Google Assistant
 Language Translation Services

77
 Speech-to-text for transcription
 Smart reply and email filtering
 Customer support bots
 Accessibility tools for the visually impaired

 The diagram illustrates the working of a speech and natural language processing
system, such as a voice assistant.
 The process begins with a human user speaking a query or command.
 This spoken input is captured and passed through a speech-to-text conversion
module, which transcribes the audio waveform into written text.
 The transcribed text is then processed using Natural Language Processing (NLP)
techniques to understand the intent and meaning behind the user's query.
 Once the query is understood, it is passed as input to a knowledge base through
an API (Application Programming Interface).
 The knowledge base analyzes the request and retrieves or generates an
appropriate response.
 This response is then converted from text back into speech through text-to-
speech conversion.
 Finally, the voice assistant communicates the answer or response to the user in
spoken form, completing the interaction loop.

78