Module 1

Review of Neural Networks (Part 2)

Syllabus
Review of Neural Networks: Model of a biological neuron, McCulloch
Pitts Neuron, Activation Functions, Perceptron, Perceptron Learning
Algorithm and Convergence, Multilayer Perceptron, Back propagation,
Learning XOR, Sigmoid Neurons, Gradient Descent, Feed forward
Neural Networks.

Module 1 : Review of Neural Networks 2

Perceptron Learning Algorithm
• The Perceptron Learning Algorithm is a supervised learning algorithm
used for binary classification. (for e.g. whether a mail is spam/not
spam)
• Perceptron Learning model is a combination of 2 concepts, McCulloch
Pitts model of artificial neuron and Hebbian rule of adjusting weights.
• Perceptron network can be trained for single output unit as well as
multiple output units.
• Goal: To find a rule (a straight line) that separates two types of things
(like yes/no, spam/not spam)

Module 1 : Review of Neural Networks 3

There are four significant steps in a perceptron learning algorithm:
Step 1: Multiply all input values with corresponding weight values and then add them to determine
the weighted sum. Mathematically, we can calculate the weighted sum as follows:
∑wi∗xi=x1∗w1+x2∗w2+…+wn∗xn
Step 2: Add another essential term called bias 'b' to the weighted sum to improve the model
performance.
∑wi∗xi+b
Step 3: An activation function is applied to this weighed sum, producing a binary or a continuous-
valued output.
Y=f(∑wi∗xi+b)
Step 4: The difference between this output and the actual target value is computed to get the error
term, E, generally in terms of mean squared error. The steps up to this form the forward
propagation part of the algorithm.
MSE=E=(Ypredicted−Yactual)2
We optimize this error (loss function) using an optimization algorithm. Generally, some form of
gradient descent algorithm is used to find the optimal values of the hyperparameters like learning
rate, weight, Bias, etc. This step forms the backward propagation part of the algorithm.
Module 1 : Review of Neural Networks 4
Module 1 : Review of Neural Networks 5
Module 1 : Review of Neural Networks 6
Perceptron Learning
• The perceptron model is a more general computational model than McCulloch-Pitts neuron. It
takes an input, aggregates it (weighted sum) and returns 1 only if the aggregated sum is more
than some threshold else returns 0.

Module 1 : Review of Neural Networks 7

Perceptron Learning
• Rewriting the threshold and making it a constant input with a variable weight.

• A single perceptron can only be used to implement linearly separable functions. It

takes both real and boolean inputs and associates a set of weights to them, along with a
bias (the threshold thing I mentioned above). We learn the weights, we get the function.
Module 1 : Review of Neural Networks 8
Vector representation
• A 2-dimensional vector can be represented on a 2D plane.
• Two vectors oh size n+1, w and x, the dot product of these vectors (w.x) could be
computed as follows:

• Transpose is just to write it in matrix multiplication form.

Module 1 : Review of Neural Networks 9

Vector representation
• w and x are just two lonely arrows in an n+1 dimensional space (their dot
product quantifies how much one vector is going in the direction of the other).
• The perceptron was only computing a lame dot product (before checking if it's
greater or lesser than 0).
• The decision boundary line which a perceptron gives out that separates positive
examples from the negative ones is really just w . x = 0.

Module 1 : Review of Neural Networks 10

Angle Between Two Vectors
• Computing the dot products differently if only the angle between the vectors and
their individual magnitudes are known.

• Get the angle between two vectors, if only you know the vectors, to calculate
vector magnitudes and their vanilla dot product.
• The cosine of the angle between w and x is 0, when w being perpendicular to
arrow x in an n+1 dimensional space (here it is 2D). The dot product of two
vectors is 0, they are perpendicular to each other.

Module 1 : Review of Neural Networks 11

Angle Between Two Vectors
• To finding the line wTx = 0 which divides the input space into two halves.
• Every point (x) on this line satisfies the equation wTx = 0.

• The angle is 90° (∵ cosα =wT x/ ||w||||x|| =0)

• Since the vector w is perpendicular to every point on the line it is actually perpendicular to the line itself.
• We can thus rewrite the perceptron rule as
y=1 if wTx ≥ 0
y=0 if wTx < 0

Module 1 : Review of Neural Networks 12

• Consider some points (vectors) which lie in the positive half space of this line (i.e.,
wTx ≥ 0) The angle between any such vector and w is less than 90◦.

• What about points (vectors) which lie in the negative half space of this line (i.e.,
wTx < 0) The angle between any such vector and w is greater than 90◦

• This follows from the formula :(cosα =wTx/ ||w||||x||)

Module 1 : Review of Neural Networks 13

 • We initialize w with some
random vector.
• We then iterate over all the
examples in the data, (P U N)
both positive and negative
examples.
• Now if an input x belongs to P,
the dot product MUST be
greater than and equal to 0.
• And if x belongs to N, the dot
product MUST be less than 0.

Module 1 : Review of Neural Networks 14

 Case 1: When x belongs to P and
its dot product w.x < 0
Case 2: When x belongs to N and
its dot product w.x ≥ 0

• Only for these cases, we are

updating our randomly
initialized w.
• Otherwise, we don’t touch w at
all because Case 1 and Case 2
are violating the very rule of a
perceptron.
• i.e. we are adding x to w in Case
1 and subtracting x from w in
Case 2.
Module 1 : Review of Neural Networks 15
Convergence
• If the training data is linearly separable, the Perceptron Learning Algorithm is
guaranteed to converge after a finite number of steps (updates).
• When the data can be separated by a straight line (or plane in higher
dimensions), the perceptron moves closer to that line with every mistake.
• Each update moves the weights in the correct direction.
• Eventually, the algorithm finds a set of weights that correctly classify all training
samples.

Module 1 : Review of Neural Networks 16

Condition for Convergence
• There must exist a weight vector w∗ ; such that: 𝑦𝑖(𝑤∗⋅𝑥𝑖+𝑏)>0
• for all training samples yi (w∗⋅xi​ +b)> 0
• This means a perfect decision boundary exists that separates the classes.
• If the data is not linearly separable (e.g., XOR problem), the algorithm will never
find a perfect weight vector.
• It will keep updating forever (or until a max number of iterations is reached).

Module 1 : Review of Neural Networks 17

Major Limitation of perceptron learning
• Linearly Separable Data Only: The perceptron can only solve problems where the
data classes can be separated by a straight line (or hyperplane). It fails on
problems like XOR, which require nonlinear decision boundaries.
• Noisy or Overlapping Data: It’s sensitive to noise and outliers. Even if the data is
mostly separable, a few misclassified points can prevent convergence.
• Hard Decision Boundaries: The output is binary (0 or 1) with no probability or
confidence score. This makes it unsuitable for tasks requiring nuanced
predictions.
• Single-Layer Architecture: A basic perceptron has no hidden layers, so it lacks the
capacity to learn complex patterns or hierarchical features.
• Inability to Solve XOR Problem: This classic example highlights the perceptron’s
failure to model non-linear relationships. No single straight line can separate XOR
outputs correctly.

Module 1 : Review of Neural Networks 18

Solutions
• Use Multi-Layer Perceptrons (MLPs) with hidden layers and nonlinear
activation functions.
• Apply gradient descent and backpropagation for training.
• Incorporate probabilistic models like logistic regression or softmax
classifiers for confidence-based outputs

Module 1 : Review of Neural Networks 19

Types of Perceptron Models
Based on the number of layers, perceptrons are broadly classified into two major
categories:
Single Layer Perceptron Model:
• It is the simplest Artificial Neural Network (ANN) model.
• A single-layer perceptron model consists of a feed-forward network and includes
a threshold transfer function for thresholding on the Output.
• The main objective of the single-layer perceptron model is to classify linearly
separable data with binary labels.
• The architecture consist of a layer of input neurons fully connected to a single
layer of output neurons.

Module 1 : Review of Neural Networks 20

Types of Perceptron Models
Multi-Layer Perceptron(MLP) Model:
• The multi-layer perceptron learning algorithm has the same structure as a single-layer perceptron
but consists of an additional one or more hidden layers, unlike a single-layer perceptron, which
consists of a single hidden layer.
• Extension of multi-layer perceptron including more than one layer of trainable weights. i.e. 3
layers: input, hidden, output layer. Each connection between 2 neurons is given by a certain
weight.
• MLP can solve complex problems — like recognizing handwriting, faces, or spam emails.
Input Layer → Hidden Layer(s) → Output Layer

• Input Layer: Takes the raw data (like numbers, pixels, etc.)
• Hidden Layers: Do the actual processing by detecting patterns
• Output Layer: Gives the final answer (like "Yes/No", "Cat/Dog", etc.)

Module 1 : Review of Neural Networks 21

Module 1 : Review of Neural Networks 22
Working of MLP
• Take Inputs: Data is given to the network.
• Pass Through Layers: Data moves from input → hidden layers →
output.
• Each Neuron:
• Multiplies input with weights
• Adds bias
• Passes it through an activation function (like ReLU or Sigmoid)
• Get Output: The final result (prediction) comes from the output layer.

Module 1 : Review of Neural Networks 23

Cost Function (Error/Loss Function)
• A cost function is a measure of "how good" a neural network did with respect to
it's given training sample and the expected output.

• It also may depend on variables such as weights.

• A cost function is a single value, not a vector, because it rates how good the
neural network did as a whole.

• The key idea is that the smaller the value of Cost function, better the neural
network is performing.

• Smaller value of cost function implies that the output of the model and the
desired output are very similar.

Module 1 : Review of Neural Networks 24

Back propagation
• Backward-error propagation, used to implement the adaptive feedback required to
adjust the weights during training.
• Backpropagation is a learning algorithm used to train Multilayer Neural Networks (like
MLPs).
• It is the backbone of training deep neural networks.
• In a network with many layers, we need a way to find and correct the errors — not just in
the output layer, but also in the hidden layers.
• It is a method for updating the weights in a neural network by propagating the error
backward from the output layer to the input layer.
• Finding answer to these questions:
1. “How wrong was your prediction?”
2. “Which part of the network caused the mistake?”
3. “How should you update the weights to do better next time?”

Module 1 : Review of Neural Networks 25

Module 1 : Review of Neural Networks 26
Back propagation
• The back-propagation algorithm looks for the minimum of the cost function in
weight space using the method of gradient descent.

• The combination of weights which minimizes the cost function is considered to

be a solution of the learning problem.

• Basically, we need to figure out whether we need to increase or decrease the

weight value.

• Once we know that, we keep on updating the weight value in that direction until
error becomes minimum.

• We might reach a point, where if you further update the weight, the error will
increase. At that time you need to stop, and that is your final weight value.

Module 1 : Review of Neural Networks 27

Back propagation
What it does?
1. Calculates how much each weight contributed to the error.
2. Uses that information to adjust weights to minimize future errors.
3. Efficiently applies the chain rule from calculus across layers.

Module 1 : Review of Neural Networks 28

Step 1: Forward Pass(Forward propagation)
• Give inputs to the network.
• The network calculates an output (prediction).
Step 2: Compute Error
• Compare the prediction with the actual answer using a loss function.
• Example: Error=1/2(𝑦true−𝑦predicted)2
Step 3: Backward Pass(Backward propagation)
• Calculate how much each weight contributed to the error.
• Use chain rule of calculus to pass the error from output to hidden layers.
Step 4: Update Weights
• Adjust each weight slightly to reduce the error, using:
𝑤=𝑤−𝜂⋅∂Error/∂𝑤
• η is the learning rate (controls how fast we update weights).

Module 1 : Review of Neural Networks 29

Module 1 : Review of Neural Networks 30
Workflow of Back propagation
• We start from the input we have, we pass them through the network layer and
calculate the actual output of the model straight forwardly.
• This step is called forward-propagation, because the calculation flow is going in
the natural forward direction from the input -> through the neural network -> to
the output.
• At this stage, in one hand, we have the actual output of the randomly initialized
neural network along with the desired output from training data set.
• We are finding out how well the model performs w.r.t to training data set and
desired output.
• This can be specified as sum of squares of the difference between the actual and
the desired output known as the cost function.

Module 1 : Review of Neural Networks 31

Workflow of Back propagation
• Our goal is to generate a model with minimum value for cost function.
• Find out an optimization process that aims to minimize cost function. Or in other
words we need to find the weights that minimize the cost function.
• To find the optimal value for weights we are using the method called gradient
Descent , it deals with the derivative of the cost function.
• In mathematics, the derivative of a function at a certain point, gives the rate or
the speed with which this function is changing its values at this point.
• In gradient descent, we are calculating the derivative of cost function(error) w.r.t
to each weight wi represented as : ∂Error or ∆ wi
• Once error is known, it will be used for backward propagation and weights
adjustment.

Module 1 : Review of Neural Networks 32

Workflow of Back propagation
• Error is propagated from output layer to the hidden layers.
• With respect to each node we are calculating the rate of change of error w.r.t to
the weight represented as ∆ wi
• New weights are calculated for each node as wi = wi + ∆ wi
• Important thing is not to update any weights until all errors have been calculated
• Now repeat the process and find out the error using new weights.
• This process is repeated iteratively until we minimize the error.

Module 1 : Review of Neural Networks 33