Module-II

Optimization and Neural Networks

 Perceptron - Stochastic Gradient Descent - Gradient Descent Solution for Perceptron - Multilayer
Perceptron.
 Activation Functions - Sigmoid- Tanh – Softmax – ReLU - Leaky ReLU.
 Architecture Design - Chain Rule - Back Propagation.
 Gradient Based Learning - Gradient Based Optimization.
 Linear Least Squares Using a Suitable Platform Building ML Algorithms and Challenges

40
Perceptron

 Perceptron is also understood as an Artificial Neuron or neural network unit that

helps to detect certain input data computations in business intelligence.
 It is a single-layer neural network with four main parameters, i.e., input values,
weights and Bias, net sum, and an activation function.


41
Parameters
Input Nodes or Input Layer:
 This is the primary component of Perceptron which accepts the initial data into the
system for further processing. Each input node contains a real numerical value.
Weight and Bias:
 Weight parameter represents the strength of the connection between units. This is
another most important parameter of Perceptron components. Weight is directly
proportional to the strength of the associated input neuron in deciding the output.
Further, Bias can be considered as the line of intercept in a linear equation.
Activation Function:
 These are the final and important components that help to determine whether the
neuron will fire or not. Activation Function can be considered primarily as a step
function.

42
Types of Activation functions:
• Sign function
• Step function, and
• Sigmoid function

43
Working

 Perceptron model works in two important steps as follows:

Step-1
 In the first step first, multiply all input values with corresponding weight values and then
add them to determine the weighted sum. Mathematically, we can calculate the weighted
sum as follows:
 ∑wi*xi = x1*w1 + x2*w2 +…wn*xn
 Add a special term called bias 'b' to this weighted sum to improve the model's
performance.
 ∑wi*xi + b
Step-2
 In the second step, an activation function is applied with the above-mentioned weighted
sum, which gives us output either in binary form or a continuous value as follows:
 Y = f(∑wi*xi + b)
Presentation title 44
 Mathematically, the output of a perceptron can be represented as follows

Presentation title 45
Stochastic Gradient Descent (SGD):

• Stochastic Gradient Descent (SGD) is a variant of the gradient descent optimization

algorithm commonly used in machine learning.
• In SGD, the model parameters (weights) are updated after computing the gradient of
the loss function with respect to a single training example or a small batch of
examples, rather than the entire dataset.
• SGD is computationally more efficient than batch gradient descent, especially for large
datasets, as it processes one example at a time and updates the weights incrementally.

46
Presentation title 47
Gradient Descent Solution for Perceptron

• To apply gradient descent to train a perceptron, we need to define a loss function that
measures the error between the perceptron's output and the target output.
• One common loss function for perceptrons is the hinge loss, which is used in support vector
machines (SVMs) and is defined as:

• where w is the weight vector, x is the input vector, b is the bias term, and y is the target
output (-1 or 1).
 The weights are updated using the gradient of the loss function with respect to the weights:

 where η is the learning rate and ∇wL(w) is the gradient of the loss function.
48
Multilayer Perceptron
 Single Layer Perceptron is one of the oldest and first introduced neural networks.
 Perceptron is also known as an artificial neural network. Perceptron is mainly used
to compute the logical gate like AND, OR, and NOR which has binary input and
binary output.

Presentation title 49
1. Sigmoid Function:
1. Formula:
2. Range: (0, 1)
3. Properties: Smooth, differentiable
4. Output interpretation: Used in binary classification problems where the output needs to be
between 0 and 1, representing probabilities.
2. Tanh (Hyperbolic Tangent) Function:
1. Formula:
2. Range: (-1, 1)
3. Properties: Similar to sigmoid but centered at 0, which helps in dealing with negative inputs.
4. Output interpretation: Often used in hidden layers of neural networks.

Presentation title 50
1. Softmax Function:
1. Formula:
2. Range: (0, 1) for each element in the output vector, with the sum of all elements equal to 1.
3. Properties: Converts logits (raw scores) into probabilities.
4. Output interpretation: Commonly used in the output layer of a neural network for multi-class
classification problems.
2. ReLU (Rectified Linear Unit):
1. Formula:
2. Range: [0, ∞)
3. Properties: Simple and computationally efficient, but not differentiable at x = 0 (subgradients are
used in practice).
4. Output interpretation: Often used in hidden layers of neural networks for introducing non-linearity.

Presentation title 51
1. Leaky ReLU Function:
1. Formula:
2. Range: (-∞, ∞)
3. Properties: Similar to ReLU but allows a small, non-zero gradient when x < 0, which can help with the "dying ReLU"
problem.
4. Output interpretation: An improvement over ReLU to address the vanishing gradient issue.
 These activation functions are crucial in neural networks as they introduce non-linearity, allowing the network
to learn complex patterns in the data.

 Watch
 [Link]
plane&learningRate=0.3&regularizationRate=0&noise=0&networkShape=4,2&seed=0.22755&showTestData
=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false
&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&
hideText=false

52
Architecture Design

 Design Considerations for ML Architectures

• Problem Complexity: Consider the complexity of the problem being solved and choose
an architecture that can handle it.
• Data Availability: Availability and size of the dataset can influence the choice of
architecture.
• Computational Resources: Consider the computational resources available for training
and inference.
• Interpretability vs. Performance: Balance between model interpretability and
performance based on the application requirements.

Presentation title 53
Convolutional Neural Networks (CNNs):

• Designed for processing structured grid-like data, such as images.

• Consist of convolutional layers, pooling layers, and fully connected layers.
• Used for image classification, object detection, and image segmentation.

54
Recurrent Neural Networks (RNNs):
• Designed for sequential data processing.
• Have loops in their architecture to persist information.
• Used for natural language processing, speech recognition, and time series analysis.

55
Chain Rule

Presentation title 56
Chain Rule

 The chain rule states that if we have a composite function f(g(x)), then the derivative of
this function with respect to x is given by:


 In the context of deep learning, f represents the final output of the neural
network, g represents the intermediate activations at each layer, and x represents the
input data.

Presentation title 57
Back Propagation-The Learning Algorithm

 Backpropagation: Backpropagation is the cornerstone of training neural networks.

 It involves the iterative process of computing gradients and updating the network’s
parameters to minimize a chosen loss function.
 At the core of backpropagation is the application of the chain rule to compute these
gradients efficiently.

Presentation title 58
Steps
 [Link] Pass:
• The input data is passed through the network layer by layer.
• Each layer performs a weighted sum and applies an activation function to produce the
output.
• These outputs are stored for later use in the backward pass.
Consider a single-layer neural network with a linear transformation followed by a sigmoid
activation function.
In Forward Pass:
• The input data x is transformed through a linear transformation: z=Wx+b, where W is
the weight matrix and b is the bias.
• The output y is obtained by applying the sigmoid activation function: y=σ(z).

Presentation title 59
 [Link] Pass (Backpropagation):
• Starting from the output layer, the gradient of the loss function with respect to the output is computed.
• The chain rule is applied iteratively to compute the gradient of the loss function with respect to each
layer’s parameters and inputs.
• Gradients are propagated backward through the network.
Previous example,
 a. Compute the gradient of the loss (L) with respect to the output (y): dy/ dL
 b. Apply the chain rule to find the gradient of the loss with respect to the input z:

 Compute the gradient of the loss with respect to the parameters (W and b):

60
 [Link] Update:
• The computed gradients are used to update the network’s parameters (weights and biases)
through optimization algorithms like gradient descent.
• This process is repeated for a predefined number of iterations or until convergence.

This example illustrates how the chain rule is used to efficiently compute the gradients
necessary for parameter updates. The process is analogous for deep neural networks, with
the chain rule applied layer by layer during the backpropagation process.

Presentation title 61
Gradient Based Learning - Gradient
Based Optimization.
 To apply gradient-based learning we must choose a cost function, and we must
choose how to represent the output of the model.
 Largest difference between simple ML Models and neural networks are nonlinearity of
a neural network causes most interesting loss functions to become non-convex.
 Gradient-based learning is the backbone of many deep learning algorithms.
 This approach involves iteratively adjusting model parameters to minimize the loss
function, which measures the difference between the actual and predicted outputs.
 At its core, Gradient-based learning leverages the gradient of the loss function to
navigate the complex landscape of parameters

Presentation title 62
Cost Functions: The Mathematical Backbone

 Learning Conditional Distributions with Max Likelihood

 Maximizing likelihood is finding parameter values that make the observed

data most probable. This is often expressed as

Presentation title 63
Learning Conditional Statistics

 This involves understanding the relationships between variables and focusing on the
conditional expectation. The goal is to minimize the difference between predicted and
actual values, often using mean squared error (MSE) as a cost function.


Presentation title 64
Output Units: Adapting to Data Types
 Linear Units for Gaussian Output Distributions
 The linear unit is used for outputs resembling a Gaussian distribution. The output is a linear
combination of inputs:\

 Sigmoid Units for Bernoulli Output Distributions

 These units are used for binary outcomes, modeled as

 This function maps any input to a value between 0 and 1, ideal for binary
classification.
 Softmax Units for Multinoulli Output Distributions
 For multi-class classification, the softmax function, which generalizes the sigmoid function for
multiple classes, is used.


65
Role of an Optimizer in Deep Learning

66
 Optimizers are algorithms designed to minimize the cost function. They play a critical
role in Gradient-based learning by updating the weights and biases of the network
based on the calculated gradients.
 Intuition Behind Optimizers with an Example
 Consider a hiker trying to find the lowest point in a valley. They take steps
proportional to the steepness of the slope. In deep learning, the optimizer works
similarly, taking steps in the parameter space proportional to the gradient of the loss
function.

Presentation title 67
Batch Gradient Descent (GD)

 This optimizer calculates the gradient using the entire dataset, ensuring a smooth
descent but at a computational cost. The update rule is


Presentation title 68
Stochastic Gradient Descent (SGD)

 SGD updates parameters for each training example, leading to faster but less stable
convergence. The update rule is:

 Mini-batch Gradient Descent (MB-GD)

 MB-GD strikes a balance between GD and SGD by using mini-batches of the dataset. It
combines efficiency with a smoother convergence than SGD.

Presentation title 69
Challenges with All Types of Gradient-Based
Optimizers
 The journey of mastering Gradient-based learning in deep learning has its challenges. Each optimizer in
deep learning faces unique hurdles that can impact the learning process:
• Learning Rate Dilemmas: One of the foremost challenges in Gradient-based learning is selecting the
optimal learning rate. A rate too high can cause the model to oscillate or even diverge, missing the
minimum—conversely, a rate too low leads to painfully slow convergence, increasing computational
costs.
• Local Minima and Saddle Points: These are areas in the cost function where the gradient is zero, but
they are not the global minimum. In high-dimensional spaces, common in deep learning, these points
become more prevalent and problematic. This issue is particularly challenging for certain types of
optimizers in deep learning, as some may get stuck in these points, hindering effective learning.
• Vanishing and Exploding Gradients: A notorious problem in deeper networks. With vanishing
gradients, as the error is back-propagated to earlier layers, the gradient can become so small that it has
virtually no effect, stopping the network from learning further. Exploding gradients occur when large
error gradients accumulate, causing large updates to the network weights, leading to an unstable
network. These issues are a significant concern for Gradient-based learning.
• Plateaus: A plateau is a flat region of the cost function. When using Gradient-based learning, the
learning process can slow down significantly on plateaus, making it difficult to reach the minimum.
Presentation title 70
• Choosing the Right Optimizer: With various types of optimizers in deep learning, such as Batch Gradient
Descent, Stochastic Gradient Descent (SGD), and Mini-batch Gradient Descent (MB-GD), selecting the
right one for a specific problem can be challenging. Each optimizer has its strengths and weaknesses,
and the choice can significantly impact the efficiency and effectiveness of the learning process.
• Hyperparameter Tuning: In Gradient-based learning, hyperparameters like learning rate, batch size, and
the number of epochs need careful tuning. This process can be time-consuming and requires both
experience and experimentation.
• Computational Constraints: Deep learning models can be computationally intensive, particularly those
that leverage complex Gradient-based learning techniques. This challenge becomes more pronounced
when dealing with large datasets or real-time data processing.
• Adapting to New Data Types and Structures: As deep learning evolves, new data types and structures
emerge, requiring adaptability and innovation in Gradient-based learning methods.
 These challenges highlight the complexity and dynamic nature of Gradient-based learning in deep
learning.

Presentation title 71
Linear Least Squares Using a Suitable
Platform Building ML Algorithms and
Challenges

72
References

 [Link]
 [Link]
253703569
 [Link]
 [Link]
deep-learning-backpropagation-
9d35affc05e7#:~:text=The%20chain%20rule%20states%20that,x%20represents%20th
e%20input%20data.
 [Link]
deeplearning/#:~:text=Gradient%2Dbased%20learning%20is%20the,the%20actual%2
0and%20predicted%20outputs.
 [Link]
based%[Link](Note)
 [Link]

73
 Illustrate a feed forward network diagrammatically.
 How does an MLP differ from a single-layer perceptron?
 What are the outputs of the SoftMax and Sigmoid activation functions when applied to the input [5, 3, 4]?
 An artificial neuron receives n inputs 𝑥1 , 𝑥2 ,𝑥3 , …. 𝑥𝑛 with weights 𝑤1 , 𝑤2 , 𝑤3 , … . . 𝑤𝑛 attached to the input links. How the weighted sum can be
calculated and given to an activation function to produce an output?
 Suppose a fully-connected neural network has a single hidden layer with 15 nodes. The input is represented by a 5D feature vector and the number of classes is 3.
Calculate the number of parameters of the network. Consider there are NO bias nodes in the network?
 What is the chain rule in the context of neural networks. How is the chain rule used in backpropagation to calculate gradients for updating weights?
 Give any three differences between stochastic gradient descent and batch gradient descent algorithm.
 Describe the concept of a gradient in the context of machine learning. What does the gradient represent?
 What is the formula for a linear least square fit? Illustrate it diagramatically.
 How do we describe the situations where a model either memorizes the training data too well , struggles to capture its complexities , or requires tuning to find
the right balance?
 Analyze the concepts of bias and variance in machine learning, and discuss their effects on model interpretability and robustness.
 Explain the concept of a perceptron and how it differs from a multilayer perceptron. Give the realisation of AND gate.
 Can XOR gate be implemented using perceptron algorithm. Justify and prove the case with the help of a 2 input model.
 How can we model the behaviour of a neuron in a network such that it activates or 'fires' based on the input it receives?
 Compare and contrast the characteristics of different activation functions.
 Describe the importance of architecture design in neural networks and how the chain rule is used in the backpropagation algorithm.
 Explain the role of backpropagation in training a neural network and how it helps in adjusting the network's weights.
 Differentiate between batch gradient descent , stochastic gradient descent and mini -batch gradient descent algorithm.?
 What strategy can we employ to navigate a complex, multidimensional landscape in search of the optimal solution?
 Consider the following data points (x,y)= ((1,1),(-2,-1) and (3,2)). Find the best fit line using linear least square regression algorithm. Illustrate it diagramatically.
 Discuss the various steps involved in Linear least square approximation algorithm to find the best fit line with help of an example.

Presentation title 74