MODULE 2

DEPT OF EI,SJCET,PALAI 1
MODULE - 2 (ARTIFICIAL NEURAL NETWORKS)

• Introduction to artificial neural networks -Single layer perceptrons, Multi-Layer

Perceptrons (MLPs),Representation Power of MLPs, Activation functions -
Sigmoid, Tanh, ReLU, Softmax. Riskminimization, Loss function, Training
MLPs with backpropagation, Practical issues in neuralnetwork training - The
Problem of Overfitting, Vanishing and exploding gradient problems,Difficulties
in convergence, Local and spurious Optima, Computational Challenges.
Applications of neural networks for classification and regression tasks.

• Reference:.Neural Networks and Deep Learning, Aggarwal, Charu C., Springer International
Publishing AG, part of Springer Nature 2018

DEPT OF EI,SJCET,PALAI 2
ARTIFICIAL NEURAL NETWORK
• An ANN is an information processing system that mimic the working of human brain.
• It consists of collection of artificial neurons(mathematical model of neurons)arranged in layers

DEPT OF EI,SJCET,PALAI 3
• The human nervous system contains cells, which are referred to as neurons. The
neurons are connected to one another with the use of axons and dendrites, and
the connecting regions between axons and dendrites are referred to as synapses.
• The strengths of synaptic connections often change in response to external
stimuli. This change is how learning takes place in living organisms.
• This biological mechanism is simulated in artificial neural networks, which
contain computation units that are referred to as neurons.
• Each input to a neuron is scaled with a weight, which affects the function
computed at that unit.
• An artificial neural network computes a function of the inputs by propagating the
computed values from the input neurons to the output neuron(s) and using the
weights as intermediate parameters. Learning occurs by changing the weights
connecting the neurons. DEPT OF EI,SJCET,PALAI 4
• Dendrites − They are tree-like branches, responsible for receiving the information
from other neurons it is connected to. In other sense, we can say that they are like the
ears of neuron.
• Soma − It is the cell body of the neuron and is responsible for processing of
information, they have received from dendrites.
• Axon − It is just like a cable through which neurons send the information.
• Synapses − It is the connection between the axon and other neuron dendrites.

DEPT OF EI,SJCET,PALAI 5
DEPT OF EI,SJCET,PALAI 6
SINGLE LAYER NEURAL NETWORK OR
PERCEPTRON
• In the singlelayer network, a set of inputs is directly mapped to an output by using a generalized
variation of a linear function.
• The simplest neural network is referred to as the perceptron. This neural network contains a single input
layer and an output node

DEPT OF EI,SJCET,PALAI 7
DEPT OF EI,SJCET,PALAI 8
PERCEPTRON NODE – THRESHOLD
LOGIC UNIT
x1 w1

x2 w2 𝜃 z

xn wn
n
1 if åx w ³q
i =1
i i

• Learn weights such that an objective z= n

åx w <q
function is maximized.
0 if i i
i =1

CS 472 - Perceptron 9
DEPT OF EI,SJCET,PALAI 10
DEPT OF EI,SJCET,PALAI 11
DEPT OF EI,SJCET,PALAI 12
DEPT OF EI,SJCET,PALAI 13
DEPT OF EI,SJCET,PALAI 14
DEPT OF EI,SJCET,PALAI 15
MULTILAYER PERCEPTRON
• A perceptron that has a single layer of weights can only approximate
linear functions of the input and cannot solve problems like the XOR,
where the discriminant to be estimated is nonlinear.
• Similarly, a perceptron cannot be used for nonlinear regression. This
limitation does not apply hidden layers to feedforward networks with
intermediate or hidden layers between the multilayer input and the
output layers. If used for classification, such multilayer perceptrons
perceptrons (MLP) can implement nonlinear discriminants and, if used
for regression, can approximate nonlinear functions of the input.

DEPT OF EI,SJCET,PALAI 16
DEPT OF EI,SJCET,PALAI 17
DEPT OF EI,SJCET,PALAI 18
REPRESENTATION OF POWER OF MLP-
XOR PROBLEM

DEPT OF EI,SJCET,PALAI 19
DEPT OF EI,SJCET,PALAI 20
DEPT OF EI,SJCET,PALAI 21
DEPT OF EI,SJCET,PALAI 22
DEPT OF EI,SJCET,PALAI 23
DEPT OF EI,SJCET,PALAI 24
ACTIVATION FUNCTIONS
[Link]
• Activation functions transform the weighted
sum of inputs that goes into the artificial
neurons.
• These functions should be non-linear to
encode complex patterns of the data. The
most popular activation functions
are Softmax, Sigmoid, Tanh and ReLU.
• ReLU is the most popular activation function
in deep neural networks.

DEPT OF EI,SJCET,PALAI 25
• Sigmoid takes a real value as input and outputs another value between 0 and 1. It’s easy
to work with and has all the nice properties of activation functions: it’s non-linear,
continuously differentiable, monotonic, and has a fixed output range.
• The logistic sigmoid function can cause a neural network to get stuck at the training time.

DEPT OF EI,SJCET,PALAI 26
• Advantages
• 1. Easy to understand and apply
• 2. Easy to train on small dataset
• 3. Smooth gradient, preventing “jumps” in output values.
• 4. Output values bound between 0 and 1, normalizing the output of each neuron.
• Disadvantages:
• Vanishing gradient—for very high or very low values of X, there is almost no change to the
prediction, causing a vanishing gradient problem. This can result in the network refusing to
learn further, or being too slow to reach an accurate prediction.
• Outputs not zero centered.
• Computationally expensive

DEPT OF EI,SJCET,PALAI 27
TANH
• Tanh neurons use a similar kind of S-shaped nonlinearity, but instead of ranging from 0 to 1, the output
of tanh neurons range from −1 to 1. When S-shaped nonlinearities are used, the tanh neuron is
• often preferred over the sigmoid neuron because it is zero-centered.

DEPT OF EI,SJCET,PALAI 28
• The advantage is that the negative inputs will be mapped strongly negative and the zero inputs
will be mapped near zero in the tanh graph.
• The function is differentiable.
• The function is monotonic while its derivative is not monotonic.
• The tanh function is mainly used classification between two classes.
• Both tanh and logistic sigmoid activation functions are used in feed-forward nets.
• Advantages Zero centered—making it easier to model inputs that have strongly negative,
neutral, and strongly positive values. Disadvantages Like the Sigmoid function is also suffers
from vanishing gradient problem hard to train on small datasets

DEPT OF EI,SJCET,PALAI 29
RELU (RECTIFIED LINEAR UNIT)
ACTIVATION FUNCTION
• The ReLU is the most used activation function in the world right [Link], it is used in almost
all the convolutional neural networks or deep learning.
• ReLU is half rectified (from bottom). f(z) is zero when z is less than zero and f(z) is equal to z
when z is above or equal to zero.
• Range: [ 0 to infinity)
• The function and its derivative both are monotonic.
• But the issue is that all the negative values become zero immediately which decreases the
ability of the model to fit or train from the data properly. That means any negative input given
to the ReLU activation function turns the value into zero immediately in the graph, which in
turns affects the resulting graph by not mapping the negative values appropriately.

DEPT OF EI,SJCET,PALAI 30
DEPT OF EI,SJCET,PALAI 31
• Advantages Avoids vanishing gradient problem. Computationally efficient—allows the
network to converge very quickly Non-linear—although it looks like a linear function, ReLU
has a derivative function and allows for backpropagation
• Disadvantages Can only be used with a hidden layer hard to train on small datasets and
need much data for learning nonlinear behavior. The Dying ReLU problem—when inputs
approach zero, or are negative, the gradient of the function becomes zero, the network cannot
perform backpropagation and cannot learn.
• The function and its derivative both are monotonic. All the negative values are converted into
zero, and this conversion rate is so fast that neither it can map nor fit into data properly which
creates a problem.

DEPT OF EI,SJCET,PALAI 32
SOFTMAX
• Softmax: Softmax can handle multiple cases. Softmax function squeeze the output for each
class between 0 and 1 with sum of them is 1. It is ideally used in the final output layer of the
classifier, where we are actually trying to attain the probabilities. Softmax produces multiple
outputs for an input array. For this reason, we can build neural network models that can
classify more than 2 classes instead of binary class solution.

DEPT OF EI,SJCET,PALAI 33
• Advantages Able to handle multiple classes only one class in other activation functions—
normalizes the outputs for each class between 0 and 1with the sum of the probabilities been
equal to 1, and divides by their sum, giving the probability of the input value being in a specific
class. Useful for output neurons—typically Softmax is used only for the output layer, for neural
networks that need to classify inputs into multiple categories.

DEPT OF EI,SJCET,PALAI 34
DEPT OF EI,SJCET,PALAI 35
DEPT OF EI,SJCET,PALAI 36
FULLY CONNECTED NETWORKS

DEPT OF EI,SJCET,PALAI 37
DEPT OF EI,SJCET,PALAI 38
DEPT OF EI,SJCET,PALAI 39
DEPT OF EI,SJCET,PALAI 40
TRAINING MLP WITH
BACKPROPAGATION

DEPT OF EI,SJCET,PALAI 41
DEPT OF EI,SJCET,PALAI 42
DEPT OF EI,SJCET,PALAI 43
DEPT OF EI,SJCET,PALAI 44
DEPT OF EI,SJCET,PALAI 45
DEPT OF EI,SJCET,PALAI 46
DEPT OF EI,SJCET,PALAI 47
DEPT OF EI,SJCET,PALAI 48
GRADIENT DESCENT ALGORITHM.

• The weights are adjusted using a process called gradient descent.

• Gradient descent is an optimization algorithm that is used to find the weights that
minimize the cost function. Minimizing the cost function means getting to the minimum
point of the cost function. So, gradient descent aims to find a weight corresponding to
the cost function’s minimum point.
• To find this weight, we must navigate down the cost function until we find its minimum
point.

DEPT OF EI,SJCET,PALAI 49
• But first, to navigate the cost function, we need two things: the direction in which to
navigate and the size of the steps for navigating.
• The Direction
• The direction for navigating the cost function is found using the gradient.
• The Gradient
• To know in which direction to navigate, gradient descent uses backpropagation. More
specifically, it uses the gradients calculated through backpropagation. These gradients
are used for determining the direction to navigate to find the minimum point.
Specifically, we aim to find the negative gradient. This is because a negative gradient
indicates a decreasing slope. A decreasing slope means that moving downward will lead
us to the minimum point. For example:

DEPT OF EI,SJCET,PALAI 50
DEPT OF EI,SJCET,PALAI 51
• Learning Rate
• The learning rate is a tuning parameter that determines the step size at each iteration of
gradient descent. It determines the speed at which we move down the slope.
• The step size plays an important part in ensuring a balance between optimization time
and accuracy. The step size is measured by a parameter alpha (α). A small α means a
small step size, and a large α means a large step size. If the step sizes are too large, we
could miss the minimum point completely. This can yield inaccurate results. If the step
size is too small, the optimization process could take too much time. This will lead to a
waste of computational power.

DEPT OF EI,SJCET,PALAI 52
DEPT OF EI,SJCET,PALAI 53
• The step size is evaluated and updated according to the behavior of the cost function.
The higher the gradient of the cost function, the steeper the slope and the faster a
model can learn (high learning rate). A high learning rate results in a higher step value,
and a lower learning rate results in a lower step value. If the gradient of the cost
function is zero, the model stops learning.

DEPT OF EI,SJCET,PALAI 54
DEPT OF EI,SJCET,PALAI 55
• Adjusting the weights consists of multiple iterations. We take a new step down for each
iteration and calculate a new weight. Using the initial weight and the gradient and
learning rate, we can determine the subsequent weights.

DEPT OF EI,SJCET,PALAI 56
• From the graph of the cost function, we can see that:
[Link] start descending the cost function, we first initialize a random weight.
[Link], we take a step down and obtain a new weight using the gradient and learning
rate. With the gradient, we can know which direction to navigate. We can know the
step size for navigating the cost function using the learning rate.
[Link] are then able to obtain a new weight using the gradient descent formula.
[Link] repeat this process until we reach the minimum point of the cost function.
[Link] we’ve reached the minimum point, we find the weights that correspond to the
minimum of the cost function.

DEPT OF EI,SJCET,PALAI 57
DEPT OF EI,SJCET,PALAI 58
P R A C T I C A L I S S U E S I N A RT I F I C I A L N E U R A L N E T W O R K T R A I N I N G
[Link] PROBLEM OF OVERFITTING

• The problem of overfitting in machine learning, which occurs when a model is fit to a specific
training dataset and does not perform well on new, unseen data. This is particularly common
when the dataset is small, and the model is complex.

DEPT OF EI,SJCET,PALAI 59
• The correct parameter vector in this case is W = [2, 0, 0, 0, 0] based on the known relationship
between the first feature and target. The training data also provides zero error with this
solution.
• The problem is that the number of training points is fewer than the number of parameters and
it is possible to find an infinite number of solutions with zero error.
• For example, the parameter set [0, 2, 4, 6, 8] also provides zero error on the training data.
However, if we used this solution on unseen test data, it is likely to provide very poor
performance because the learned parameters are spuriously inferred and are unlikely to
generalize well to new points in which the target is twice the first attribute.
• A good rule of thumb is that the total number of training data points should be at least 2 to 3
times larger than the number of parameters in the neural network.
• In general, models with a larger number of parameters are said to have high capacity, and
they require a larger amount of data in order to gain generalization power to unseen test data.

DEPT OF EI,SJCET,PALAI 60
D E S I G N M E T H O D S U S E D TO M I T I G AT E T H E I M PA C T O F OV E R F I T T I N G .
1 . R EG U L A R I Z AT I O N

• Regularization is a technique used in machine learning to prevent overfitting by constraining

the model to use fewer non-zero parameters.
• This is done by adding a penalty term, λ||W||p, to the loss function, where p is typically set to 2,
which leads to Tikhonov regularization.
• The squared value of each parameter (multiplied with the regularization parameter λ > 0) is
added to the objective function, and a quantity proportional to λwi is subtracted from the
update of the parameter wi.
• This penalization is seen as a kind of weight decay during the updates.
• Regularization is particularly important when the amount of available data is limited. It is also
advisable to use more complex models with regularization rather than simpler models without
regularization. A biological interpretation of regularization is that it corresponds to gradual
forgetting, as a result of which "less important" (i.e., noisy) patterns are removed.

DEPT OF EI,SJCET,PALAI 61
2 . N E U R A L A R C H I T EC T U R E A N D PA R A M E T E R S H A R I N G

• The most effective way of building a neural network is by tailoring the architecture of the
network to the underlying data domain.
• This involves leveraging insights about the relationships between data points in the domain to
create specialized architectures with fewer parameters.
• For example, text data often involves successive words that are related to one another,
whereas image data often involves nearby pixels that are related.
• Recent advancements in neural network architectures, such as recurrent neural networks and
convolutional neural networks, take advantage of these insights and use shared parameters to
learn features in a more efficient manner.

DEPT OF EI,SJCET,PALAI 62
3 . EA R LY S TO P P I N G

• Another common form of regularization is early stopping, in which the gradient descent is
ended after only a few iterations.
• One way to decide the stopping point is by holding out a part of the training data, and then
testing the error of the model on the held-out set.
• The gradient-descent approach is terminated when the error on the held-out set begins to
rise.
• Early stopping essentially reduces the size of the parameter space to a smaller neighborhood
within the initial values of the parameters.
• From this point of view, early stopping acts as a regularizer because it effectively restricts the
parameter space

DEPT OF EI,SJCET,PALAI 63
-

• The model with more number of parameters is more prone to over-fitting

DEPT OF EI,SJCET,PALAI 64
• The model tries to chase the loss function crazily on the training data, by tuning the
parameters.
• Now, we keep another set of data as the validation set and as we go on training, we keep a
record of the loss function on the validation data, and when we see that there is no
improvement on the validation set, we stop, rather than going all the epochs.
• This strategy of stopping early based on the validation set performance is called Early Stopping.

DEPT OF EI,SJCET,PALAI 65
DEPT OF EI,SJCET,PALAI 66
[Link] METHODS

• These methods can be combined with many neural network architectures to obtain an additional
accuracy improvement of about 2% in many real settings. However, the precise improvement depends to
the type of data and the nature of the underlying training.
• EXAMPLE- Baggging Bootstrap Aggregation
• The idea behind bagging is combining the results of multiple models (for instance, all
decision trees) to get a generalized result.

DEPT OF EI,SJCET,PALAI 67
DEPT OF EI,SJCET,PALAI 68
DEPT OF EI,SJCET,PALAI 69
• What is Bootstrapping?
• Bagging is composed of two parts: aggregation and bootstrapping. Bootstrapping
is a sampling method, where a sample is chosen out of a set, using the
replacement method. The learning algorithm is then run on the samples selected.
• The bootstrapping technique uses sampling with replacements to make the
selection procedure completely random. When a sample is selected without
replacement, the subsequent selections of variables are always dependent on the
previous selections, making the criteria non-random.
• What is Aggregation?
• Model predictions undergo aggregation to combine them for the final prediction
to consider all the possible outcomes. The aggregation can be done based on the
total number of outcomes or the probability of predictions derived from the
bootstrapping of every model in the procedure.

DEPT OF EI,SJCET,PALAI 70
[Link] AND EXPLODING
GRADIENT PROBLEMS
• While increasing depth often reduces the number of parameters of the network, it leads to
different types of practical issues.
• When training a deep neural network with gradient based learning and backpropagation, we
find the partial derivatives by traversing the network from the the final layer (y_hat) to the
initial layer.
• Using the chain rule, layers that are deeper into the network go through continuous matrix
multiplications in order to compute their derivatives.
• In a network of n hidden layers, n derivatives will be multiplied together. If the derivatives are
large then the gradient will increase exponentially as we propagate down the model until they
eventually explode, and this is what we call the problem of exploding gradient.
• Alternatively, if the derivatives are small then the gradient will decrease exponentially as we
propagate through the model until it eventually vanishes, and this is the vanishing gradient
problem. DEPT OF EI,SJCET,PALAI 71
• In the case of exploding gradients, the accumulation of large derivatives results
in the model being very unstable and incapable of effective learning,
• The large changes in the models weights creates a very unstable network,
which at extreme values the weights become so large that is causes overflow
resulting in NaN weight values of which can no longer be updated.
• On the other hand, the accumulation of small gradients results in a model that
is incapable of learning meaningful insights since the weights and biases of the
initial layers, which tends to learn the core features from the input data (X),
will not be updated effectively.
• In the worst case scenario the gradient will be 0 which in turn will stop the
network further training.
DEPT OF EI,SJCET,PALAI 72
• There are many approaches to addressing exploding and vanishing gradients
• Reducing the amount of Layers
• This is the solution could be used in both, scenarios (exploding and vanishing gradient).
However, by reducing the amount of layers in our network, we give up some of our models
complexity, since having more layers makes the networks more capable of representing
complex mappings.
• Gradient Clipping (Exploding Gradients)
• Checking for and limiting the size of the gradients.

DEPT OF EI,SJCET,PALAI 73
[Link] IN CONVERGENCE

DEPT OF EI,SJCET,PALAI 74
• The errors are decremental as the count of iteration is increasing but one different thing is we can not
tell from which point the error is varying within a smaller range.

DEPT OF EI,SJCET,PALAI 75
• Most of the neural network fails to converge because of an error in the modelling. Let us say
the data is required to transform within the network and the nodes we have provided in the
networks are way smaller in number. In such a situation how can we expect the network to
work properly? So in the majority of the cases when the network fails to converge, it comes
into the picture because of inaccurate modelling. Some of the reasons behind this thing are as
follows:

• Implementation of not enough nodes may be a reason behind this issue because models with
fewer nodes need to change their architecture drastically to model the data better and fail to
converge.
• The amount of the training data is low or the data we are pushing on the model is corrupted
or not collected with the data integrity.
• The activation function we are using with the network often leads to good results from the
model but if complexity is higher then the model can fail to converge.

DEPT OF EI,SJCET,PALAI 76
• Inappropriate weight application in the network can also cause a failure in convergence. The
weights we are applying to the network should be well calculated according to the activation
function.
• The learning rate parameter we have given in the network should be moderate which means it
should not be much larger or much lower.

DEPT OF EI,SJCET,PALAI 77
[Link] AND SPURIOUS OPTIMA
• The optimization function of a neural network is highly nonlinear and has many local optima.
Therefore, it is essential to choose good initialization points, especially when dealing with large
parameter spaces with many local optima. One approach to improve neural network initialization is
called pretraining, where shallow sub-networks of the original network are trained either
supervised or unsupervised to create the initial weights.

• Pretraining is performed in a greedy and layerwise manner, where a single layer of the network is
trained at a time to learn the initialization points of that layer. This method provides initialization
points that ignore irrelevant parts of the parameter space to begin with. Unsupervised pretraining,
in particular, can avoid overfitting by moving the initialization point closer to the basin of good
optima in the test data. This is because some of the minima in the loss function may be spurious
optima that are only present in the training data and not in the test data. By using unsupervised
pretraining, the initialization point is closer to the basin of good optima in the test data, which
improves model generalization.

DEPT OF EI,SJCET,PALAI 78
5. COMPUTATIONAL CHALLENGES

• Assignment
• Assignment1: Implementation of ANN using Python / Neural Network Tool box- MATLAB

DEPT OF EI,SJCET,PALAI 79
DEPT OF EI,SJCET,PALAI 80