Module 1

Review of Neural Networks (Part 3)

Syllabus
Review of Neural Networks: Model of a biological neuron, McCulloch
Pitts Neuron, Activation Functions, Perceptron, Perceptron Learning
Algorithm and Convergence, Multilayer Perceptron, Back propagation,
Learning XOR, Sigmoid Neurons, Gradient Descent, Feed forward
Neural Networks.

Review of Neural Networks 2

Exclusive OR (XOR)

Input A Input B XOR Output

0 0 0

0 1 1

1 0 1

1 1 0

Review of Neural Networks 3

Review of Neural Networks 4
Learning XOR (exclusive OR)
• XOR Learning problem demonstrates why single-layer perceptrons fall short—and
how multi-layer networks overcome those limitations.
• XOR outputs 1 only when inputs differ.
• Not linearly separable - you can’t split 1s and 0s with a straight line.
• A single-layer perceptron can only learn linear decision boundaries.
• XOR requires a non-linear boundary, which a perceptron can’t model.

Review of Neural Networks 5

How to learn XOR?
To learn XOR, we use:
• A multi-layer perceptron (MLP) with at least one hidden layer, that transforms
the data into a form that can be separated.
• Non-linear activation functions like sigmoid or ReLU.
• Backpropagation to adjust weights based on error. (Training Algorithm)

Layers Neurons Activation Function

Input 2 ; x1,x2

Hidden 2 ; h1,h2 ReLU

Output 1;y Sigmoid

Review of Neural Networks 6

How to solve XOR problem?
Assume we use sigmoid activation and the following weights:
• Hidden neuron 1 acts like an OR gate
• Hidden neuron 2 acts like an AND gate
• Output neuron subtracts them (like OR − AND = XOR)

Inputs OR output (h1) AND output (h2) Final output = OR − AND

(0,0) 0 0 0
(0,1) 1 0 1
(1,0) 1 0 1
(1,1) 1 1 0

Review of Neural Networks 7

Step 1: Input
• If input features are (x1,x2), then x1 =1 and x2=0.
Step 2: Hidden Layer Computation
• Each hidden neuron receives x1 and x2 ​, multiplies them with weights, adds bias, and passes
through an activation function:
h1​ =σ(w11⋅ x1 + w12⋅ x2 +b1​ )
h2​ =σ(w21⋅ x1 + w22⋅ x2 +b2​ )
where σ is the sigmoid or ReLU function.
Step 3: Output Layer Computation
• The output neuron receives h1​ ​and h2​ : y=σ(w1o⋅ h1 + w2o⋅ h2 +bo)
Step 4: Output Decision
• If the final output y is close to 1 → Predict 1
• If y is close to 0 → Predict 0

Review of Neural Networks 8

Symbol Meaning
w₁₁ Weight from input x₁ to hidden neuron h₁
w₁₂ Weight from input x₂ to hidden neuron h₁
w₂₁ Weight from input x₁ to hidden neuron h₂
w₂₂ Weight from input x₂ to hidden neuron h₂
w₁ₒ Weight from hidden neuron h₁ to output neuron y
w₂ₒ Weight from hidden neuron h₂ to output neuron y

Review of Neural Networks 9

Review of Neural Networks 10
Sigmoid Neuron
• A sigmoid neuron is a type of artificial neuron used in neural networks that applies the
sigmoid activation function (Logistic) to its input.
• Commonly used to squash the output between 0 and 1.
• Output function is much smoother than the step function.
• A small change in the input weights only causes a small change in the output as opposed
to the stepped output, which is essential for learning.
• The inputs to the sigmoid neuron can be real numbers unlike the boolean inputs in MP
Neuron and the output will also be a real number between 0–1, interpreted as a
probability between x and y.
Use case:
1. Output layers of binary classification models
2. Logistic Regression
3. Smooth gradient updates during training

Review of Neural Networks 11

Sigmoid Neuron
• The sigmoid function is defined as:

Where, x is the weighted sum of inputs

e is Euler’s number (~2.718)

• No sharp transition at the threshold b.

• This function creates an S-shaped curve (called a “sigmoid”).

Review of Neural Networks 12

Sigmoid Neuron
• Takes inputs: x1​, x2​,…, xn
• Multiplies them with their weights: w1​ , w2 ,…, wn
• Sums them all and adds bias: z=w1​.x1​ +w2​.x 2​ +⋯+wn.x n​ +b
• Applies sigmoid function to compute the output:

• Large negative input →output is near 0

• 0 input →output is 0.5
• Large positive input →output is near 1

Review of Neural Networks 13

Gradient
• Gradient function f:R →R is a vector valued function that gives the direction and
rate of the fastest increase or decrease of the function at any point of time.
• The gradient is simply a derivative vector for a multivariate function.
• “How steep something is and Which direction you should go to increase or
decrease a value?”
• A gradient shows:
1. How much the loss/error changes when you change the weights
2. Which direction to move the weights to reduce the loss.
• The gradient of the loss function tells us how to improve the model.

Review of Neural Networks 14

Gradient
• The gradient of a function f is denoted by ∇f, vector of partial derivatives of a function f
with respect to all its input variables.
• The gradient of function f(x1,x2,x3,…,xn) is

• how much the function changes when you change each input
• ∇f(xk) : points to the direction of steepest ascent.
• -∇f(xk) : points to the direction of steepest descent.
• : Partial derivative of f with respect to x1

• ∇f=0 at any point. i.e. critical point.

Review of Neural Networks 15

Example of Gradient
f(x,y)=w12+w22

=2w1

=2w2

∇f=[2w1, 2w2]
At a point (1,2), ∇f=[2.(1), 2.(2)]=[2,4]
Gradient vector of (1,2) →(2,4)
This vector shows the direction of steepest increase in the function.

Review of Neural Networks 16

Gradient Descent
• Gradient descent (GD) is an iterative first-order optimisation algorithm used to
find a local minimum/maximum of a given function.
• To minimize the function by iterative moving in the direction of negative gradient
of function.
• Goal: To minimise a cost/loss function.
• In gradient descent, we update parameters in the opposite direction of the
gradient: wk+1= wk−η⋅∇f(w k) {Weight update rule in gradient descent}
Where: η = learning rate, positive scalar that control step size
∇f(w) = gradient of the loss function w.r.t. weight vector w k
• Gradient descent algorithm does not work for all functions. A function has to be:
1. Differentiable
2. Convex

Review of Neural Networks 17

• A function is differentiable it has a derivative for each point in its domain.

Review of Neural Networks 18

• For a univariate function, convex means that the line segment connecting two
function’s points lays on or above its curve.

• Another way, calculate the second order derivative and check it is greater than 0.

• As the second order derivative is greater than 0 (2>0), function is strictly convex.
Review of Neural Networks 19
Term Description

Local Minimum Lowest point in a small region

Global Minimum The absolute lowest point in the entire graph

Local Maximum Highest point in a small region

Global Maximum The absolute highest point overall

Review of Neural Networks 20

Gradient Descent Algorithm
Step 1: Start with initial point w0.
Step 2: Update the point using gradient.
wk+1= wk−η⋅∇f(w k)
Step 3: Repeat until convergence, i.e. gradient become too small, maximum
number of iteration reached.

η too small → slow convergence

η too large → algorithm overshoots or diverge

Review of Neural Networks 21

Feed forward Neural Networks (FNN)
• FNN is one of the simplest and most foundational types of artificial neural networks.
• Data flows in one direction— Without looping back — no cycles, no feedback.

• Training a Feed forward Neural Networks:

1. Forward Pass: Input layer → hidden layer(s) → output layer. Then passed through an activation
function like ReLU, sigmoid, or tanh.
z= w1x1+ w2x2 + w3x3 +…+ wnxn
Output=Activation(z)
1. Loss Calculation: Compare output with actual label using loss function(MSE or cross-entropy)
2. Backpropagation: Send error backward to compute gradients
3. Gradient Descent: Update weights to reduce error

• Use case: Pattern recognition, classification, regression.

Review of Neural Networks 22

Review of Neural Networks 23