CSE 3175: ARTIFICIAL NEURAL NETWORKS

[Module 2]

25 August 2025 1
Syllabus

25 August 2025 2
 Lecture - 6
Rosenblatt’s Perceptron Model

25 August 2025 3
• A perceptron is a fundamental building block of neural
networks.
• It’s a simple computational unit inspired by the way
biological neurons work in the human brain.
Introducing Artificial Neural Networks Using the 7 Ws and 1 H

• It is a linear classifier used to solve binary

classification problems, where the goal is to separate
two classes of data points by finding an optimal
decision boundary.

25 August 2025 4
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

25 August 2025 5
 Perceptron

• The perceptron was first proposed by Rosenblatt

(1958).
• It is a simple neuron, used to classify its input into one of
Introducing Artificial Neural Networks Using the 7 Ws and 1 H
two categories.
• It is a single processing unit of a neural network.
• It uses an activation function (Signum) that returns +1
if the weighted sum of its input is >= 0 and -1 otherwise.

25 August 2025 6
 Structure of Perceptron

Introducing Artificial Neural Networks Using the 7 Ws and 1 H

25 August 2025 7
 Components of the Neuron Model
1. Linear Combiner:
▪ The linear combiner sums up the inputs received from various synapses,
which are weighted connections between neurons.
▪ Additionally, it incorporates an externally applied bias to adjust the sensitivity
of the model.
[Link] Local Field:
▪ This resulting sum, also referred to as the induced local field, represents the
total input to the neuron after weighting and bias adjustments.
[Link] Limiter:
▪ The induced local field is then passed through a hard limiter function.
▪ If the induced local field is positive, the output of the hard limiter is +1.
▪ If the induced local field is negative, the output is -1.
25 August 2025 8
 Linear Threshold (LTU)

• Perceptron can be defined as a single artificial neuron.

• It computes its weighted input with the help of the threshold
activation function or step function.
• It is also called as a TLU (Threshold Logical Unit)(McCulloch &
Pitts 1943).

25 August 2025 9
 Components
Weighted Sum of Inputs: Each input feature is multiplied by its
corresponding weight. The products of these multiplications are
then summed up to calculate the weighted sum of inputs.

Bias: The bias is a constant term added to the weighted sum.

It helps the perceptron account for cases where all input values
are zero.

Activation Function: The activation function takes the weighted

sum plus bias as input and determines the final output of the
perceptron.
The choice of activation function can impact the perceptron's
ability to learn complex patterns.

25 August 2025 10
 Perceptron
▪ The most common AF used in perceptron's is the step function or
the sign function.
▪ The step function outputs 1 if the weighted sum plus bias is
greater than or equal to zero, and 0 otherwise.
▪ This essentially makes the perceptron act as a simple threshold-
based binary classifier.
▪ Single perceptron can only learn linear decision boundaries,
which limits its ability to handle complex data.
▪ To overcome this limitation, multi-layer neural networks with
nonlinear activation functions (e.g., sigmoid, tanh, ReLU) are
used, enabling them to learn more intricate relationships in the
data.

25 August 2025 11
 What can a Perceptron do ?
▪ The perceptron is an algorithm for supervised classification of an
input into one of several possible non-binary outputs.
▪ Perceptron can be defined as a single artificial neuron that computes
its weighted input with the help of the threshold activation function
or step function.
▪ It is used for binary Classification and only models linearly
separable classes.
▪ First, train a perceptron for a classification task.
▪ Find suitable weights in such a way that the training examples are
correctly classified.
25 August 2025 12
 Perceptron

25 August 2025 13
 Linear Separability

Linear Separable:

Linear inseparable:

Solution?

25 August 2025 14
 Linear /Non-Linear Patterns

25 August 2025 15
 Linear Separability

• Every logic function can be implemented by neural networks. So, the step function is
commonly used in primitive neural networks without a hidden layer or a widely
known name, such as a single-layer perceptron.

• This type of network can classify linearly separable problems, such as an AND
gate or an OR gate. In other words, all classes (0 and 1) can be separated by a single
linear line as illustrated below.

25 August 2025 16
25 August 2025 17
 Limitations of Perceptron

▪ The perceptron can only model linearly separable

functions, those functions which can be drawn in a 2-
dimensional graph, and a single straight line separates
values into two parts.
▪ Boolean functions given below are linearly separable:
▪ AND, OR, COMPLEMENT
▪ It cannot model the XOR function as it is nonlinearly
separable. When the two classes are not linearly
separable, it may be desirable to obtain a linear separator
that minimizes the mean squared error.

25 August 2025 18
 Perceptron Convergence Theorem

▪ The perceptron can learn to classify items accurately.

▪ It’s like a basic building block for understanding how computers make
decisions, much like our brains handle simple choices.

▪ Classification Goal: The primary objective of this neuron model is to

classify inputs x1​, x2​,…, xm into one of two classes, C1 or C2:

▪ Class C1: Inputs are classified as C1 if the output of the neuron model
is +1 (indicating a positive induced local field).

▪ Class C2: Inputs are classified as C2 if the output of the neuron model
is -1 (indicating a negative induced local field).

25 August 2025 19
25 August 2025 20
Bias b(n) is treated as a synaptic weight driven by a
fixed input equal to 1.
(m+1)-by-1 input vector, weight vector
n denotes the time-step
x(n) = [+1, x1(n), x2(n), ..., xm(n)]^T
w(n) = [b, w1(n), w2(n), ..., wm(n)]^T
the linear combiner output
25 August 2025 21
For fixed n, the equation w^T.x = 0, plotted in an m-dimensional
space (and for some prescribed bias) with coordinates x1, x2, ...,
xm, defines a hyperplane as the decision surface between two
different classes of inputs.

The training process of a perceptron involves the adjustment of the

weight vector w in such a way that the two classes c1 and c2 are
linearly separable.
That is, there exists a weight vector w such that we may state.

25 August 2025 22
Given the subsets of training vectors h1 and h2, the training
problem for the perceptron is then to find a weight vector w
satisfying the decision boundary equation.

25 August 2025 23
 Lecture-7
Activation functions and Multilayer Perceptron

25 August 2025 24
 Activation functions
▪ Activation functions determine the output of a neuron given an input. Common types
include
▪ The step function outputs a binary value (0/1) based on a threshold.
▪ The sign function outputs -1 or 1 depending on the input's sign.
▪ The sigmoid function outputs a value between 0 and 1, introducing smooth non-linearity
▪ The linear function outputs the input directly, often used in the final layer for regression
tasks.
▪ The combination of these functions helps the network learn various data patterns.
▪ The choice of activation function affects the network's capacity to capture complex
relationships.

25 August 2025 25
 Sigmoid Unit
kth sigmoid unit

x1 wk1
x.2 wk2
.
. . ok
. .
xm wkm

bk
1
ok = f ( yk ) =
1 +e- y
k

25 August 2025 26
25 August 2025 27
Example

25 August 2025 28
25 August 2025 29
 Multilayer Perceptron

25 August 2025 30
 Multilayer Perceptron (MLP)

▪ It is fully connected dense layers,

which transform any input
dimension to the desired
dimension.

▪ A neural network that has

multiple layers.

▪ A popular method for the Output Layer

training of MLP is the back-
propagation algorithm (1985). Adjustable Weights

Input Layer

25 August 2025 31
 Features of MLP

▪ The model of each neuron in the network includes a

nonlinear activation function that is differentiable.

▪ The network contains one or more layers that are

hidden from both the input and output nodes.

▪ The network exhibits a high degree of connectivity,

the extent of which is determined by the synaptic
weights of the network.

25 August 2025 32
 Types of Layers

• The input layer.

– Introduces input values into the network.
– No activation function or other processing.
• The hidden layer(s).
– Perform classification of features.
– Two hidden layers are sufficient to solve any problem.
– Features imply that more layers may be better.
• The output layer.
– Functionally just like the hidden layers.
– Outputs are passed on to the world outside the neural
network.
25 August 2025 33
 Multilayer perceptron with two hidden layers

25 August 2025 34
 Multilayer perceptron with two hidden layers

▪ The multiplayer perceptron with two hidden layers and

an output layer.
▪ To set the stage for a description of the multilayer
perceptron in its general form, the network shown is
fully connected.
▪ This means that a neuron in any layer of the network is
connected to all the neurons (nodes) in the previous
layer.
▪ Signal flow through the network progresses in a
forward direction, from left to right and on a layer-by-
layer basis.
25 August 2025 35
 Training an MLP
➢ Training stage proceeds in 2 phases – Forward and Backward phases
➢ Forward phase
➢ The synaptic weights of the network are fixed, and the input signal is
propagated through the network, layer by layer, until it reaches the
output.
➢ Changes are confined to the activation potentials and outputs of the
neurons in the network.
➢ Backward phase
➢ An error signal is produced by comparing the output of the network
with a desired response.
➢ The resulting error signal is propagated through the network, again layer
by layer, but this time the propagation is performed in the backward
direction.
➢ Successive adjustments are made to the synaptic weights of the network.
➢ Calculation of the adjustments for the output layer is straightforward,
but it is much more challenging for the hidden layers.
25 August 2025 36
 Types of signals in MLP
➢ Function signals / Input signals
➢ It’s an input signal (stimulus) that comes in at the input end of the network, propagates forward
(neuron by neuron) through the network, and emerges at the output end of the network as an
output signal.
➢ Presumed to perform a useful function at the output of the network.
➢ At each neuron of the network through which a function signal passes, the signal is calculated
as a function of the inputs and associated weights applied to that neuron.
➢ Error signals
➢ Originates at an output neuron of the network and propagates backward (layer by layer) through
the network
➢ Its computation by every neuron of the network involves an error-dependent function in one
form or another

25 August 2025 37
 Function of the Hidden Neurons

▪ The hidden neurons act as feature detectors; play a critical role in the operation
of a multilayer perceptron.
▪ As the learning process progresses across the MLP, the hidden neurons begin to
gradually “discover” the salient features that characterize the training data.
▪ They do so by performing a nonlinear transformation on the input data into a
new space called the feature space.
▪ In this new space, the classes of interest in a pattern-classification task, for
example, may be more easily separated from each other than could be the case in
the original input data space.
▪ Formation of this feature space through supervised learning distinguishes the
multilayer perceptron from Rosenblatt’s perceptron.

25 August 2025 38
 Task of hidden neurons

Perform two computations:

1. Function signal appearing at the output of each neuron

- Expressed as a continuous nonlinear function of the input

signal and synaptic weights associated with that neuron;

2. Estimate of the gradient vector (i.e., the gradients of the

error surface with respect to the weights connected to the
inputs of a neuron),

- Needed for the backward pass through the network.

25 August 2025 39
 Gradient Descent

▪ An optimization algorithm that is commonly used to

train machine learning models and neural networks.
▪ It trains machine learning models by minimizing errors
between predicted and actual results.
▪ Training data helps models learn over time, and the cost
function within gradient descent specifically acts as a
barometer, gauging its accuracy with each iteration of
parameter updates.
▪ Until the function is close to or equal to zero, the model
will continue to adjust its parameters to yield the smallest
possible error.
25 August 2025 40
 Gradient Descent

One of the most commonly used iterative optimization algorithms in

machine learning and deep learning is used to train machine learning
and deep learning models. It helps in finding the local minimum of a
function.

[Link]

25 August 2025 41
 Lecture 8 - 9

Gradient Descent and Error Function

25 August 2025 42
 Gradient Descent

▪ An optimization algorithm that is commonly used to

train machine learning models and neural networks.

▪ It trains machine learning models by minimizing errors between

predicted and actual results.

▪ Training data helps models learn over time, and the cost function
within gradient descent specifically acts as a barometer,
calculating its accuracy with each iteration of parameter updates.

▪ Until the function is close to or equal to zero, the model will

continue to adjust its parameters to yield the smallest possible
error.

25 August 2025 43
 Delta Rule

➢ The adjustment made to the weight factor of an input neuron

connection is proportional to the product of the error signal and the
input value of the connection.
➢ Error is the difference between the desired and actual output.
➢ The key idea behind the delta rule is to use gradient descent to
search the hypothesis space of possible weight vectors to find the
weights that best fit the training data.
➢ The delta rule is important because it provides the basis for the
backpropagation algorithm, which can learn networks with many
interconnected units.
➢ Widrow & Hoff Learning rule or the Least Mean Square (LMS)
rule.
25 August 2025 44
 Error – Gradient Descent

➢ It minimizes the error between predicted and actual outputs

by iteratively adjusting the weights and biases of the
network.
➢ To achieve this, it performs two steps iteratively
I. Compute the gradient (slope), the first-order derivative
of the function at that point.
II. Make a step (move) in the direction opposite to the
gradient. The opposite direction of the slope increases
from the current point by alpha times the gradient at that
point.

25 August 2025 45
 Error Function / Cost Function
▪ Delta Rule employs the error function for what is known as Gradient Descent
learning, which involves the ‘modification of weights along the most direct path
in weight-space to minimize error’, so the change applied to a given weight is
proportional to the negative of the derivative of the error with respect to that
weight.
▪ The Error/Cost function is commonly given as the sum of the squares of the
differences between all target and actual node activation for the output layer.
▪ For a particular training pattern (i.e., training case), error Ep
▪ ‘Ep’ - total error over the training pattern,
▪ ½ - value applied to simplify the function’s derivative
▪ ’n’ - all output nodes for a given training pattern
▪ ‘tj’ sub n - Target value for node n in output layer j,
▪ ‘aj’ sub n - actual activation for the same node.
▪ This particular error measure is attractive because its derivative, whose value is
needed in the employment of the Delta Rule, and is easily calculated.

25 August 2025 46
25 August 2025 47
25 August 2025 48
25 August 2025 49
25 August 2025 50
25 August 2025 51
 E =  1

wi wi 2 dD
(td − od )2
Sigmoid function
= 1   (td − od )2
2 dD wi

= 1 2(td − od ) (td − od )
2 dD wi

= −(t d − od ) od
dD wi
yd
= −(t d − od ) od chain rule
dD
yd wi
n

1 ( xiwi)

= −  (t d − od ) ( ) i=0
Sigmoid
dD yd 1+ e − yd
wi function
= −  (t d − od )od (1− od )xi Continue….
dD

25 August 2025 52
 wi = − E =   (t d − od )od (1− od )xi
wi dD

wi  wi +   (t d − od )od (1− od )xi Delta rule learning (training)

dD

wji = (td − od )od (1− od )xi

wji  wi +(td − od )od (1− od )xi

[Link]

25 August 2025 53
 How Gradient Descent Optimization Works for Minimizing Error

➢ The goal is to find the value that minimizes (the pink point in the
figure below).
➢ To achieve that point, use the gradient information, where the
gradient represents the slope of the function.
➢ Positive gradient, the curve will increase when the error increases.
➢ [Link]
Negative gradient, the curve will decrease when error increases.
➢ Thus, how to update weights so that error decreases constantly?
The answer is by subtracting the existing weight from the gradient.
➢ What step should be made in every iteration update?
➢ The learning rate is a tuning parameter that determines the step
size at each iteration of gradient descent. It determines the speed at
which we move down the slope.
25 August 2025 54
➢ Assume Y = f(x) = (1/2)x^2.
➢ How to reach the pink point
minimizing Y value, i.e., the pink
point will be x = 0.
➢ Assume the current x = 10.

25 August 2025 55
[Link]

25 August 2025 56
 Batch Learning and Online Learning

▪ Batch learning and online learning are two different

approaches in machine learning for training models
using data.

▪ They differ in how they process and update the model's

parameters as new data becomes available.

25 August 2025 57
 Batch Learning
• Batch learning, also known as offline learning or batch training,
involves training a machine learning model using the entire dataset
at once.
• In this approach, the model is trained on a fixed dataset, often
referred to as a "batch," and its parameters are updated after
processing the entire batch.
• The model learns from the patterns present in the entire batch and
adjusts its parameters to minimize a specified loss function.

25 August 2025 58
 Advantages and disadvantages of batch learning

Advantages of Batch Learning:

• Generally, leads to stable convergence since the model
learns from the complete dataset.
• Suitable for scenarios where the entire dataset can fit
into memory.
Disadvantages of Batch Learning:
• Can be computationally expensive and memory-
intensive, especially for large datasets.
• Not well-suited for scenarios where data is
continuously streaming or evolving.
25 August 2025 59
 Online Learning

• Online learning, also known as incremental learning or online

training, involves updating the machine learning model's parameters
in real-time as new data arrives.
• In this approach, the model is updated iteratively, with each new
data point influencing the model's parameters.
• Online learning is well-suited for situations where the data is
continuously generated or arrives in streams.

25 August 2025 60
 Advantages and disadvantages of online learning

Advantages of online learning:

• Efficient for large datasets or data streams that cannot be
stored entirely in memory.
• Can adapt to changing patterns in the data over time.
• Supports dynamic model updates as new data becomes
available.
Disadvantages of online learning:
• Can lead to more frequent updates, potentially causing
instability in the model.

25 August 2025 61
• In practice, the choice between batch learning and
online learning depends on the nature of the problem,
the available computational resources, the
characteristics of the data, and the desired update
frequency of the model.
• Some hybrid approaches also exist, where models are
updated in small batches at regular intervals to balance
between efficiency and model stability.

25 August 2025 62
 Lecture 10-11
The Back-Propagation Algorithm

25 August 2025 63
 Back-Propagation

▪ Backpropagation is a supervised learning algorithm used to train

a feedforward neural network. It is the most popular and widely
used algorithm for training ANN.

▪ It adjusts the weights and biases of the neural network based on

the error between the predicted output and the actual output.

▪ It works by propagating the error backwards through the

network, from the output layer to the input layer, and adjusting the
weights and biases of each neuron along the way.

▪ It consists of two phases: forward propagation and backward

propagation. During forward propagation, the input is fed into the
neural network, and the network calculates the output.

25 August 2025 64
 Back-Propagation

▪ During backward propagation, the error between the predicted

output and the actual output is calculated, and the weights and
biases of each neuron are adjusted to reduce the error.
▪ It uses the gradient descent optimization method to update the
weights and biases of the network.
▪ It is a powerful and effective method for training feedforward neural
networks and has been used successfully in many real-world
applications.
▪ It is used for a variety of applications, including image
classification, speech recognition, natural language processing, and
time series prediction.

25 August 2025 65
 The Back-Propagation Algorithm
Forward

Backward

➢ Method for learning weights in feed-forward (FF) nets uses gradient descent to
minimize the error, propagating deltas to adjust for errors backward from outputs
to hidden layers to inputs.
➢ The idea of the algorithm can be summarized as follows :
1. Computes the error term for the output units using the observed error.
2. From the output layer, repeat
• propagating the error term back to the previous layer
• updating the weights between the two layers until the earliest hidden layer is
reached.

25 August 2025 66
 Steps in the Back Propagation Algorithm

STEP ONE: initialize the STEP THREE:

weights and biases.
• The weights in the network
Propagate the inputs
are initialized to random
STEP TWO: feed the forward; we compute
numbers from the interval [-
1,1]. training sample. the net input and
• The biases are similarly output of each unit in
initialized to random the hidden and output
numbers from the interval [- layers.
1,1].

STEP FIVE: update

STEP SIX:
STEP FOUR: back weights and biases to
terminating
propagate the error. reflect the
conditions.
propagated errors.

25 August 2025 67
 Propagation through Hidden Layer (One Node )

-
x0 w0j
x1 w1j
 f
output y
xn wnj

Input weight weighted Activation

vector x vector w sum function

The inputs to unit j are outputs from the previous layer.

These are multiplied by their corresponding weights in order to form
a weighted sum, which is added to the bias associated with unit j.
A nonlinear activation function f is applied to the net input.

25 August 2025 68
 Propagate the inputs forward
For unit j in the input layer, its output is equal to its input, that is,
for input unit j.
Oj = I j

The net input to each unit in the hidden and output layers is
computed as follows.
• Given a unit j in a hidden or output layer, the net input is
I j =  wij Oi +  j
i

where wij is the weight of the connection from unit i in the previous
layer to unit j; Oi is the output of unit I from the previous layer;
j
is the bias of the unit

25 August 2025 69
 Propagate the inputs forward

Each unit in the hidden and output layers takes its net input and then
applies an activation function.

The function symbolizes the activation of the neuron represented by

the unit. It is also called a logistic, sigmoid, or squashing function.

Given a net input Ij to unit j, then

1
Oj = f(Ij), Oj = −I j
the output of unit j, is computed as
1+ e

25 August 2025 70
 Back propagate the error
When reaching the Output layer, the error is computed and
propagated backwards.
For a unit k in the output layer, the error is computed by a formula:

Errk = Ok (1 − Ok )(Tk − Ok )
Where Ok – actual output of unit k ( computed by activation function.
1
Ok = −Ik
1+ e

Tk – True output based of known class label; classification of training

sample
Ok(1-Ok) is a Derivative ( rate of change ) of the activation function.

25 August 2025 71
 Update weights and biases

Weights are updated by the following equations, where l is a constant

between 0.0 and 1.0 reflecting the learning rate; this learning rate is
fixed for implementation.
wij = (l ) ErrjOi

wij = wij + wij

• Biases are updated by the following equations

 j = (l) Errj

 j =  j +  j
25 August 2025 72
 Terminating Conditions
Updating weights and biases after the presentation of each sample
• All wij in the previous epoch are below some threshold.
case.

Epoch --- One iteration through the training set is called an epoch.
• The percentage of samples misclassified in the previous
epoch
Epoch is below
updating some threshold.
------------

Alternatively, the weight and bias increments could be

• A pre specified number of epochs has expired.
accumulated in variables, and the weights and biases updated
after all of the samples of the training set have been presented.
• In practice, several hundred thousand epochs may be
Caserequired
updating before the accurate
is more weights will converge.

25 August 2025 73
 Backpropagation Formulas

Output vector
Errk = Ok (1 − Ok )(Tk − Ok )
Output nodes
1 Err j = O j (1 − O j ) Errk w jk
Oj = −I j k
1+ e
Hidden nodes

I j =  wij Oi +  j wij  j =  j + (l) Err j

i
wij = wij + (l ) Err j Oi
Input nodes

Input vector: xi

25 August 2025 74
 Example of Back Propagation

Input = 3, Hidden Neuron

= 2 Output =1

Initialize weights :

Random Numbers from -1.0 to 1.0

Initial Input and weight

x1 x2 x3 w14 w15 w24 w25 w34 w35 w46 w56

1 0 1 0.2 -0.3 0.4 0.1 -0.5 0.2 -0.3 -0.2

25 August 2025 75
 Example ( cont.. )

Bias added to Hidden

+ Output nodes

Initialize Bias

Random Values from

-1.0 to 1.0

Bias ( Random )

θ4 θ5 θ6
-0.4 0.2 0.1

25 August 2025 76
 Net Input and Output Calculation

Unitj Net Input Ij Output Oj

4 0.2 + 0 - 0.5 -0.4 = -0.7

1
Oj = = 0.332
1 + e0.7
5 -0.3 + 0 + 0.2 + 0.2 =0.1 1
Oj = = 0.525
1 + e −0.1

(-0.3)0.332-(0.2)(0.525)+0.1= -0.105 1
6 Oj = = 0.475
1 + e0.105

25 August 2025 77
 Calculation of Error at Each Node

Unit j Error j

6 0.475 x (1-0.475)(1-0.475)=0.1311
We assume T 6 = 1

5 0.525 x (1- 0.525)x 0.1311x

(-0.2) = 0.0065

4 0.332 x (1-0.332) x 0.1311 x

(-0.3) = -0.0087

25 August 2025 78
 Calculation of Weights and Bias Updating
Learning Rate l =0.9

Weight New Values

w46 -0.3 + 0.9(0.1311)(0.332) = -0.261

w56 -0.2 + (0.9)(0.1311)(0.525) = -

0.138
w14 0.2 + 0.9(-0.0087)(1) = 0.192

w15 -0.3 + (0.9)(-0.0065)(1) = -0.306

……..similarly ………similarly
θ6 0.1 +(0.9)(0.1311)=0.218

……..similarly ………similarly

25 August 2025 79
25 August 2025 80
25 August 2025 81
25 August 2025 82
25 August 2025 83
25 August 2025 84
25 August 2025 85
25 August 2025 86
25 August 2025 87
25 August 2025 88
25 August 2025 89
 Lecture 12-15
Multilayer Perceptron Applications and
more problems on backpropagation

25 August 2025 90
 Tangent hyperbolic activation function

Also referred to as the tanh function

Shifted and stretched version of the
sigmoid function.
Input range shown [-10,10]
The output range of the tanh
function is [-1,1].
Gradient of tanh

25 August 2025 91
Perform backpropagation algorithm on the given network
using the tangent function as activation on both hidden
and output layers. I/p [1,1] Target O/p [0]

25 August 2025 92
 Multilayer Perceptron Applications

1. Image Classification
2. Speech Recognition
3. Financial Prediction and Risk Management
4. Medical Diagnosis
5. Anomaly Detection
MLPs are typically used for supervised learning problems
where the aim is to learn a function that maps input data to
appropriate output labels.
25 August 2025 93
 Image Classification with MLP

➢ Image classification is a task in computer vision that

involves classifying an image into one of several
predefined categories.
➢ Although Convolutional Neural Networks (CNNs) are
more commonly used for this task due to their efficiency
in handling image data.
➢ MLPs can also be applied, especially when working with
small or less complex datasets.

25 August 2025 94
 How MLP Works for Image Classification
Input Layer:
➢ The input layer receives the raw pixel values of the image. For example,
for a grayscale image of 28x28 pixels, the input layer would have 784
neurons (28 * 28).
➢ Each neuron in the input layer represents a pixel value, typically
normalized to a range between 0 and 1.
Hidden Layers:
➢ The hidden layers in the MLP are responsible for learning the features
from the input data.
➢ These layers apply a series of linear transformations followed by a non-
linear activation function.
➢ The depth number of hidden layers and width number of neurons per
layer are hyperparameters that can be tuned for optimal performance.

25 August 2025 95
 How MLP Works for Image Classification

Output Layer:
o The output layer is usually a SoftMax layer in image classification
tasks. If there are n classes, the output layer will have n neurons.
o The SoftMax function converts the raw output scores into
probabilities, with each neuron’s output representing the
probability that the input image belongs to a particular class.
Training:
o The MLP is trained using a labeled dataset. The training process
involves adjusting the weights and biases of the neurons using
backpropagation and gradient descent to minimize the loss
function.
o Common loss functions for classification tasks include Cross-
Entropy Loss.
25 August 2025 96
 Example of MLP in Image Classification
Let's consider the task of classifying handwritten digits from the
MNIST dataset using an MLP.
Dataset:
o The MNIST dataset consists of 60,000 training images and 10,000
testing images of handwritten digits (0-9). Each image is 28x28
pixels in size.
MLP Architecture:
o Input Layer: 784 neurons (for 28x28 pixel images)
o Hidden Layer 1: 128 neurons with ReLU activation
o Hidden Layer 2: 64 neurons with ReLU activation
o Output Layer: 10 neurons with SoftMax activation (one for each
digit 0-9)

25 August 2025 97
25 August 2025 98
25 August 2025 99
 Applications of MLP in Image Classification
Although MLPs are not the first choice for image classification, they
can still be used in scenarios such as:
• Simple Image Classification Tasks: Where the image data is small
and less complex, MLPs can perform sufficiently well.
• Feature Extraction: MLPs can be used as a feature extractor where
the final classification is done using another classifier.
• Dimensionality Reduction: MLPs can also help in reducing the
dimensionality of the data, capturing the most important features
before passing it to a more complex model like a CNN.

25 August 2025 100

 Lecture 13-14
XOR Problem

25 August 2025 101

 What is the XOR Problem?
The XOR problem is a classic problem in artificial intelligence and
machine learning.

XOR, which stands for exclusive OR, is a logical operation that takes
two binary inputs and returns true if exactly one of the inputs is true.

The XOR gate follows a specific truth table, where the output is true
only when the inputs differ.

This problem is particularly interesting because a single-layer

perceptron, the simplest form of a neural network, cannot solve it.

25 August 2025 102

Three-layer network for solving the Exclusive-OR
operation
−1

3
w13 −1
x1 1 3 w35
w23 5

5 y5
w24
x2 2 4 w45
w24
Input 4 Output
layer layer
−1
Hidden layer
 Final results of three-layer network learning

Inputs Desired Actual Error

output output
x1 x2 yd y5 e
1 1 0 Y
0.0155 −0.0155
0 1 1 0.9849 0.0151
1 0 1 0.9849 0.0151
0 0 0 0.0175 −0.0175

25 August 2025 109

 Network for solving the Exclusive-OR operation
1

-1.5
1
+1.0
x1 1 3 -2 -0.5
+1.0

5 y5

+1.0
x2 2 +1.0
4
+1.0
-0.5

1
[Link]

25 August 2025 110

25 August 2025 111
Applications
domains and tasks where neural networks are successfully used
◦ recognition
◦ control problems
◦ series prediction
◦ weather, financial forecasting
◦ categorization
◦ sorting of items (fruit, characters, …)

25 August 2025 112

 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

25 August 2025 113

 Thank You !!

25 August 2025 114