Module 5

Classification Model
Introduction
Classification in machine learning is a predictive modeling process by which
machine learning models use classification algorithms to predict the correct label
for input data.

Classification is a supervised machine learning method where the model tries to

predict the correct label of a given input data. In classification, the model is fully
trained using the training data, and then it is evaluated on test data before being
used to perform prediction on new unseen data.
Lazy Learners Vs. Eager Learners
There are two types of learners in machine learning classification: lazy and eager
learners.
Eager learners are machine learning algorithms that first build a model from the
training dataset before making any prediction on future datasets.

They spend more time during the training process because of their eagerness to
have a better generalization during the training from learning the weights, but they
require less time to make predictions.

Most machine learning algorithms are eager learners, and below are some
examples:

1. Logistic Regression.
2. Support Vector Machine.
3. Decision Trees.
4. Artificial Neural Networks.
Lazy learners or instance-based learners, on the other hand, do not create any
model immediately from the training data, and this is where the lazy aspect comes
from.

They just memorize the training data, and each time there is a need to make a
prediction, they search for the nearest neighbor from the whole training data,
which makes them very slow during prediction.

Some examples of this kind are:

1. K-Nearest Neighbor.
2. Case-based reasoning.
Types of classification
Classification-based predictive modeling tasks are distinguished from each other
based on the number of categories and the degree to which the categories are
exclusive:

1. Binary classification sorts data into two exclusive categories.

2. Multiclass classification sorts data into more than two exclusive categories.
3. Multilabel classification sorts data into nonexclusive categories.
4. Imbalanced classification has an unequal distribution of data points across
categories.
Types of classification algorithms
There are many different types of classification algorithms.

Some of the most popular classification algorithms include:

1. Logistic regression
2. Decision tree
3. Random forest
4. Support vector machine (SVM)
5. K-nearest neighbors
6. Naive Bayes
Classification learning evaluation methods
Some of the most popular evaluation metrics are:

1. Accuracy
2. Precision
3. Recall
4. F1 score
5. Confusion matrix
6. ROC curve
Artificial Neural Network
An artificial neuron is a
mathematical function conceived
as a model of a biological neuron
in a neural network.
Artificial Neural Networks
Artificial Neural Networks contain artificial neurons which are called units .

These units are arranged in a series of layers that together constitute the whole
Artificial Neural Network in a system.

In the majority of neural networks, units are interconnected from one layer to
another.

Each of these connections has weights that determine the influence of one unit on
another unit.

As the data transfers from one unit to another, the neural network learns more and
more about the data which eventually results in an output from the output layer.
 Feature Biological Neural Network (BNN) Artificial Neural Network (ANN)
Real neurons with dendrites, cell bodies, Digital units (nodes) that handle numeric weights
Building Block and axons. and activations.
Electrical impulses (action potentials) +
Signal Type chemical transmitters. Numeric values passed between layers.
Local changes in synapse strength (e.g., Data-driven adjustments of weights guided by
Learning Process neurons that fire together wire together). error calculations.
Continuous remodeling of connections Structured training phases that depend on large
Adaptation through growth or pruning (plasticity). datasets and backpropagation.
About 20 watts to run a vast parallel Often requires powerful processors, which can
Energy Use system. draw far more energy.
Can reroute signals if neurons or areas Sensitive to broken nodes unless special
Fault Tolerance are damaged. redundancies are designed.
Distributed across connected neurons; Encoded in numeric weights; may forget old
Memory Storage shaped by daily experiences. tasks when retrained.
Slower signal transfer but extensive Can compute very quickly on specialized
Speed parallelism. hardware, but often in a more linear sequence.
Hard to measure individual neuron Sometimes seen as a “black box” of weights and
Interpretability contributions in complex thoughts. biases.
Activation Functions
Activation functions are an integral building block of neural networks that enable
them to learn complex patterns in data.

They transform the input signal of a node in a neural network into an output signal
that is then passed on to the next layer.

Without activation functions, neural networks would be restricted to modeling only

linear relationships between inputs and outputs.

Activation functions introduce non-linearities, allowing neural networks to learn

highly complex mappings between inputs and outputs.
Types of activation functions
Linear Activation Function (Identity)

data scientists use linear activation

functions, also known as identity
functions, when they want the output
to be the same as the input signal.
Piecewise Linear (PL)

Piecewise linear is an iteration on the

above, except involving an affine
function, so it is also known as
piecewise affine.

It’s defined using a bound or unbound

sequence of numbers, either compact,
finite, or locally finite, and is not
differentiable due to threshold points,
so it only propagates signals in the
slope region.
Binary Step Function

The binary step function is a door that

only opens when a specific threshold
value has been met.

When an input is above that threshold,

the neuron is activated, and when not,
it’s deactivated.
Sigmoid, Logistic Activation
Functions

The Sigmoid activation function, also

known as the logistic activation
function, takes inputs and turns them
into outputs ranging between 0 and 1.

Larger, more positive inputs should

produce output values close to 1.0,
with smaller, more negative inputs
producing outputs closer to 0.0.
Tanh Function (Hyperbolic Tangent)

Tanh (or TanH), also known as the

hyperbolic tangent activation function,
is similar to sigmoid/logistic, even
down to the S shape curve, and it is
differentiable.

Except, in this case, the output range

is -1 to 1 (instead of 0 to 1). It is a
steeper gradient and also encounters
the same vanishing gradient challenge
as sigmoid/logistic.
Rectified Linear Unit (ReLU)
ReLU is more computationally efficient
For many years, researchers and data
scientists mainly used Sigmoid or
Tanh, and then when ReLU came
along, training performance increased
significantly
ReLU doesn’t activate every neuron in
sequence at the same time, making it
more efficient than the tanh or
sigmoid/logistic activation functions.
Unfortunately, the downside of this is
that some weights and biases for
neurons in the network might not get
updated or activated.
Leaky ReLU Function

One solution to the “dying ReLU”

problem is a variation on this known
as the Leaky ReLU activation function.

With the Leaky ReLU, instead of being

0 when 𝑧<0, a leaky ReLU allows a
small, non-zero, constant gradient 𝛼
(Normally, 𝛼=0.01).
Parametric ReLU Function

Parametric ReLU is another iteration

of ReLU (an advance on the above,
Leaky ReLU) except with a
parameterized slope α, and is also not
differentiable.

This activation function generally

outperforms ReLU especially when
used for image classification tasks in
deep learning.
Exponential Linear Units (ELUs)
Function

The exponential linear units (ELUs)

function is another iteration on the
original ReLU, another way to
overcome the “dying ReLU” problem,
and it’s also not differentiable.

ELUs use a log curve for negative

values instead of a straight line, with it
becoming smooth slowly until it
reaches -α.
Scaled Exponential Linear Units
(SELUs)
Similar to ELUs, the scaled version of
this is also attempting to overcome the
same challenges of ReLUs.
SELUs control the gradient more
effectively and scale the normalization
concept, and that is scales with a
lambda parameter.
SELUs remove the problem of vanishing
gradients, can’t die (unlike ReLUs), and
learn faster and better than other more
limited activation functions.
Gaussian Error Linear Units
(GELUs)

Gaussian error linear units (GELUs)

are part of the Gaussian function
mathematical family.

GELUs combines properties and

inspiration from ReLUs, dropout, and
zoneout and is considered a smoother
version of ReLU.
Soft Sign

Soft sign is equally useful in statistics

and other related fields.

It’s a continuous and differentiable

activation function with a range from
-1 to 1, so it can be used to model
bipolar data while being
computationally efficient.
Soft Plus

Soft Plus takes Soft Sign a little

further, making it an equally, if not
even more, useful activation function
for neural networks.

f(x)=ln(1+e^x)
Probit

a quantile function that’s associated

with the standard normal distribution
and works as an activation function in
neural networks and machine learning
models.
Softmax

The softmax function, also known as

the softargmax function and the
multi-class logistic regression, is one
of the most popular and well-used
differentiable layer activation
functions.

Softmax turns input values that are

positive, negative, zero, or greater
than one into values between 0 and 1.
Advantages of Neural Network
1. Non-Linearity
2. Input-output mapping
3. Adaptivity
4. Evidential response
5. Contextual Information
6. Fault Tolerance
7. Uniformity of Analysis and Design
Biological Neuron
 Neuron Structure
Dendrites

These are branch-like structures that receive messages from other neurons and allow the
transmission of messages to the cell body.

Cell Body

Each neuron has a cell body with a nucleus, Golgi body, endoplasmic reticulum, mitochondria and
other components.

Axon

Axon is a tube-like structure that carries electrical impulse from the cell body to the axon terminals
that pass the impulse to another neuron.

Synapse

It is the chemical junction between the terminal of one neuron and the dendrites of another neuron.
Neuron Types
Sensory Neurons
The sensory neurons convert signals from the external environment into
corresponding internal stimuli. The sensory inputs activate the sensory neurons and
carry sensory information to the brain and spinal cord. They are pseudounipolar in
structure.
Motor Neurons
These are multipolar and are located in the central nervous system extending their
axons outside the central nervous system. This is the most common type of neuron
and transmits information from the brain to the muscles of the body.
Interneurons
They are multipolar in structure. Their axons connect only to the nearby sensory and
motor neurons. They help in passing signals between two neurons.
The dendrites receive the impulse from the terminal button or synapse of an adjoining
neuron.

Dendrites carry the impulse to the nucleus of the nerve cell which is also called as
soma. Here , the electrical impulse is processed and then passed on to the axon.

The axon is longer branch among the dendrites which carries the impulse from the
soma to the synapse.

The synapse then , passes the impulse to dendrites of the second neuron. Thus, a
complex network of neurons is created in the human brain.

Neurons communicate with each other by sending signals, called neurotransmitters,

across a narrow space, called a synapse, between the axons of the sender neuron
and dendrites of the receiver neuron.
McCulloch-Pitts Neuron
The logic is as follows:

● If the sum of the weighted inputs exceeds the threshold value, then the neuron is said to be activated
and the output signal is 1.
● Otherwise the neuron is not activated and the output signal is 0.
The MCP Neuron, named after Warren McCulloch and Walter Pitts, is a
fundamental concept in the history of artificial neurons and neural networks.

An MCP neuron is a simplified version of a biological neuron, receiving binary

inputs (0 or 1) and generating a binary output.

These inputs are weighted equally (all weights set to 1), and the neuron’s output is
determined by applying a weighted sum to its inputs and comparing it to a
threshold.

If the weighted sum is greater than or equal to the threshold, the neuron outputs 1;
otherwise, it outputs 0.

One of the key features of the MCP neuron is its activation threshold. If the
summation of weighted inputs exceeds this threshold, the neuron “fires,”
producing an output signal.
The McCulloch-Pitts Neuron is the first computational model of a neuron. It can be
divided into two parts:

1. Aggregation: The neuron aggregates multiple boolean inputs (0 or 1).

2. Threshold Decision: Based on the aggregated value, the neuron makes a
decision using a threshold function.

The main elements of the McCulloch-Pitts model can be summarized as follow:

1. Neuron activation is binary. A neuron either fire or not-fire

2. For a neuron to fire, the weighted sum of inputs has to be equal or larger than
a predefined threshold
3. If one or more inputs are inhibitory the neuron will not fire
4. It takes a fixed one time step for the signal to pass through a link
5. Neither the structure nor the weights change over time
Limitations of McCulloch-Pitts Neuron

Despite its pioneering role, the McCulloch-Pitts Neuron has limitations:

1. Inability to handle non-boolean inputs.

2. The requirement to manually set thresholds.
3. All inputs are treated equally; no weighting mechanism.
4. Cannot handle functions that are not linearly separable like XOR.

Linear separability (for boolean functions): There exists a line (plane) such that all
inputs which produce a 1 lie on one side of the line (plane) and all inputs which
produce a 0 lie on other side of the line (plane).
Linear Separability
Linear separability, a concept tied to the perceptron model, explores the capability
of a model to separate data points of different classes with a linear boundary.

While McCulloch-Pitts neurons are limited to linearly separable problems, this

notion paved the way for more sophisticated models.

There exists a line (plane) such that all inputs which produce a 1 lie on one side of
the line (plane) and all inputs which produce a 0 lie on other side of the line
(plane).
Numericals
Characteristics of McCulloch Pitts ANN
Here's a breakdown of its key characteristics:

Binary Inputs and Outputs:

The MP neuron operates with binary inputs (0 or 1) and outputs (0 or 1),

representing "off" or "on" states.

Threshold Logic:

It sums the weighted inputs and compares the sum to a threshold value. If the sum
exceeds the threshold, the neuron "fires" (output 1); otherwise, it remains inactive
(output 0).
Mimicking Biological Neurons:
The MP neuron aims to capture the essence of how biological neurons function,
by combining multiple inputs and producing a single output based on a threshold.
Foundation of ANNs:
The MP neuron is considered the first computational model of a neuron, laying the
groundwork for more complex artificial neural networks.
Logical Operations:
By adjusting the threshold and weights, the MP neuron can be configured to
perform basic logical operations like AND, OR, and NOT.
Linear Separability:
The MP neuron, in its basic form, is limited to solving linearly separable problems,
meaning it can only separate data points with a straight line or hyperplane.
 Classes of Neural Network
There are several different architectures for ANNs, each with their own strengths
and weaknesses. Some of the most common architectures include:

Feedforward Neural Networks:

This is the simplest type of ANN

architecture, where the information flows
in one direction from input to output.

The layers are fully connected, meaning

each neuron in a layer is connected to all
the neurons in the next layer.
Recurrent Neural Networks
(RNNs):

These networks have a

“memory” component, where
information can flow in cycles
through the network.

This allows the network to

process sequences of data,
such as time series or
speech.
Convolutional Neural Networks (CNNs):

These networks are designed to process data with a grid-like topology, such as images.

The layers consist of convolutional layers, which learn to detect specific features in the
data, and pooling layers, which reduce the spatial dimensions of the data.
Autoencoders:

These are neural networks that are

used for unsupervised learning.

They consist of an encoder that

maps the input data to a
lower-dimensional representation
and a decoder that maps the
representation back to the original
data.
Generative Adversarial Networks (GANs):

These are neural networks that are used for generative modeling.

They consist of two parts: a generator that learns to generate new data samples,
and a discriminator that learns to distinguish between real and generated data.
There exist five basic types of neuron connection architecture :

1. Single-layer feed-forward network

2. Multilayer feed-forward network
3. Single node with its own feedback
4. Single-layer recurrent network
5. Multilayer recurrent network
Single-layer feed-forward network
In this type of network, we have only
two layers input layer and the output
layer but the input layer does not
count because no computation is
performed in this layer.
The output layer is formed when
different weights are applied to input
nodes and the cumulative effect per
node is taken.
After this, the neurons collectively
give the output layer to compute the
output signals.
 Multilayer feed-forward network
This layer also has a hidden layer that is internal to the network and has no direct
contact with the external layer.

The existence of one or more hidden layers enables the network to be

computationally stronger, a feed-forward network because of information flow
through the input function, and the intermediate computations used to determine
the output Z.

There are no feedback connections in which outputs of the model are fed back
into itself.
Single node with its own feedback
When outputs can be directed back as
inputs to the same layer or preceding
layer nodes, then it results in feedback
networks.

Recurrent networks are feedback

networks with closed loops.

The above figure shows a single

recurrent network having a single neuron
with feedback to itself.
Single-layer recurrent network
The network is a single-layer network with a feedback connection in which the
processing element’s output can be directed back to itself or to another
processing element or both.

A recurrent neural network is a class of artificial neural networks where

connections between nodes form a directed graph along a sequence.

This allows it to exhibit dynamic temporal behavior for a time sequence. Unlike
feedforward neural networks, RNNs can use their internal state (memory) to
process sequences of inputs.
Multilayer recurrent network
In this type of network, processing element output can be directed to the
processing element in the same layer and in the preceding layer forming a
multilayer recurrent network.

They perform the same task for every element of a sequence, with the output
being dependent on the previous computations.

Inputs are not needed at each time step.

The main feature of a Recurrent Neural Network is its hidden state, which
captures some information about a sequence.
Application of Neural Network
1. Pattern Recognition
2. Image Processing
3. Text recognition
4. Optimization and Constraint satisfaction: Supply Chain Optimization
5. Disease Diagnosis
6. Stock Market Prediction
7. Predictive analytics.
Performance measures of Neural Network
To determine if neural network training is successful, monitor metrics like loss
(training and validation), accuracy, precision, recall, F1-score, and AUC (for binary
classification), and analyze the model's performance on unseen data.
1. Loss (Training and Validation):

Training Loss:

1. This measures how well the model is learning from the training data.
2. It should generally decrease over time as the model learns.

Validation/Test Loss:

1. This measures how well the model generalizes to unseen data.

2. If the validation loss starts increasing while the training loss continues to
decrease, it indicates overfitting, meaning the model is memorizing the
training data rather than learning generalizable patterns.
2. Accuracy, Precision, Recall, and F1-Score (for classification tasks):
Accuracy:
The proportion of correctly classified instances out of the total instances.
Precision:
The ability to avoid false positives (out of all instances predicted as positive, what
proportion are actually positive).
Recall:
The ability to find all relevant instances (out of all actual positive instances, what
proportion are correctly identified).
F1-Score:
The harmonic mean of precision and recall, providing a balanced measure of
performance.
3. AUC (Area Under the Receiver Operating Characteristic Curve) (for binary
classification):

AUC measures the model's ability to discriminate between positive and negative
classes, regardless of the classification threshold.
4. Other Important Metrics:

Confusion Matrix:

A table that shows the breakdown of true positives, true negatives, false positives,
and false negatives, providing a detailed view of the model's performance.

Mean Average Precision (mAP):

For object detection tasks, mAP measures the average precision across different
categories.

Throughput and Latency:

For real-time applications, it's important to measure the model's speed and
efficiency (how many predictions can be made per second and how long each
prediction takes).
5. Data Splitting and Evaluation:

Training Set: Used to train the model.

Validation Set: Used to monitor the model's performance during training and
prevent overfitting.

Test Set: Used to evaluate the final model's performance on completely unseen
data.

Cross-validation: A technique to evaluate the model's performance by splitting

the data into multiple folds and training the model on different combinations of
folds.
6. Techniques to Improve Performance:

1. Data Augmentation: Increasing the amount and diversity of training data.

2. Regularization: Techniques like L1 and L2 regularization to prevent overfitting.
3. Dropout: Randomly dropping neurons during training to prevent overfitting.
4. Early Stopping: Stopping training when the validation loss starts increasing.
5. Hyperparameter Tuning: Optimizing the model's hyperparameters (e.g.,
learning rate, number of layers).
6. Normalization/Standardization: Scaling the input data to a common range.
7. Batch Normalization: A technique to normalize the activations of each layer.
8. Transfer Learning: Using pre-trained models on similar tasks to reduce
training time and improve performance.
9. Adversarial Training: Training the model to be robust to adversarial attacks.
Important Terminology of ANN
Artificial Neural Networks (ANNs) are complex systems used in machine learning
and artificial intelligence. Here are some important terminology related to ANNS
1. Activation Function: A mathematical function applied to the output of a
neuron, determining its firing or activation level. It introduces non-linearity to the
network and enables complex mappings between inputs and outputs. Common
activation functions include the sigmoid, ReLU (Rectified Linear Unit), and softmax
functions.
2. Weights: Each connection between neurons in an ANN is associated with a
weight. The weight represents the strength or importance of that connection,
indicating how much influence the output of one neuron has on the input of
another.
3. Bias: A bias term is an additional parameter associated with each neuron in an
ANN. It allows for the shifting of the activation function's output, providing flexibility
in the decision boundary of the network.
4. Feedforward: The process of propagating input data through the network, layer
by layer, from the input layer to the output layer. In feedforward neural networks,
information flows in one direction without any loops or feedback connections.
5. Backpropagation: A learning algorithm for adjusting the weights of an ANN
based on the difference between the network's predicted output and the desired
output. It calculates the gradient of the loss function with respect to the weights
and uses this information to update the weights in the network.
6. Loss Function: A function that measures the dissimilarity between the
predicted output of an ANN and the true output. It quantifies the error made by the
network and is used to guide the training process, minimizing the error through
weight adjustments.
7. Gradient Descent:: An optimization algorithm used in the training of neural
networks. It iteratively adjusts the weights in the direction of steepest descent of
the loss function, aiming to find the global or local minimum of the error surface.

8. Epoch: A single pass of the entire training dataset through an ANN. During one
epoch, forward propagation, backpropagation, and weight updates take place.

9. Batch Size: The number of training examples from the dataset used in a single
iteration of the gradient descent algorithm. In each batch, the weights are updated
based on the average gradient calculated from the examples in the batch.
10. Learning Rate: A hyperparameter that determines the step size or rate at
which the weights are updated during training. It controls the magnitude of weight
adjustments based on the calculated gradient, balancing the speed and accuracy
of learning.

11. Overfitting: A phenomenon in machine learning where an ANN learns to

perform well on the training data but fails to generalize to unseen or test data. It
occurs when the network becomes too complex or when the training dataset is
insufficient.

12. Regularization: Regularization techniques are used to prevent overfitting in

ANNs. They introduce additional terms to the loss function or modify the weight
update process to discourage excessive complexity in the network.
LIMITATIONS OF NEURAL NETWORK

Neural networks, while powerful, have limitations including the need for large
datasets, computational intensity, the "black box" nature of their decision-making,
and potential for overfitting.
1. Data Requirements:

Large Datasets:

Neural networks, especially deep learning models, often require massive amounts
of labeled data to train effectively. This can be a significant hurdle, especially for
tasks with limited or expensive data collection.

Data Quality:

The performance of a neural network is highly dependent on the quality and

relevance of the training data. Poor data can lead to biased or inaccurate models.
2. Computational Resources:

Computational Intensity:

Training neural networks can be computationally expensive and time-consuming,

especially for large and complex models. This requires specialized hardware like
GPUs or TPUs.

Memory Footprint:

Large neural networks can require significant memory resources, both during
training and deployment.
3. Interpretability (The "Black Box" Problem):

Lack of Explainability: Neural networks are often described as "black boxes"

because it can be difficult to understand why they make certain predictions. This
lack of transparency can be a major issue in applications where understanding the
reasoning behind the model's decisions is critical.

4. Overfitting:

Generalization Issues: Overfitting occurs when a model becomes too specialized

in the training data, leading to poor performance on new, unseen data. This can
happen if the model is too complex or trained for too long on a limited dataset.
5. Other Limitations:
Lack of Common Sense:
Neural networks often lack the ability to make decisions based on common sense
or real-world knowledge.
Difficulty with Sequential Data:
While recurrent neural networks are designed for sequential data, traditional
neural networks can struggle with tasks that require processing information in a
specific order.
Fixed-Size Inputs:
Some neural network architectures have fixed input sizes, which can be a
limitation for tasks with variable-length inputs.
Perceptron
The Perceptron is one of the simplest artificial neural network architectures,
introduced by Frank Rosenblatt in 1957. It is primarily used for binary
classification.

Perceptron is a type of neural network that performs binary classification that

maps input features to an output decision, usually classifying data into one of two
categories, such as 0 or 1.

This model can work on non-boolean values where each input connection gets
associated with a weight.
Basic Components of Perceptron
Input Features: The perceptron takes multiple input features, each representing a
characteristic of the input data.

Weights: Each input feature is assigned a weight that determines its influence on
the output. These weights are adjusted during training to find the optimal values.

Summation Function: The perceptron calculates the weighted sum of its inputs,
combining them with their respective weights.
Activation Function: The weighted sum is passed through the Heaviside step
function, comparing it to a threshold to produce a binary output (0 or 1).

Output: The final output is determined by the activation function, often used for
binary classification tasks.

Bias: The bias term helps the perceptron make adjustments independent of the
input, improving its flexibility in learning.

Learning Algorithm: The perceptron adjusts its weights and bias using a learning
algorithm, such as the Perceptron Learning Rule, to minimize prediction errors.
Perceptron Rule of Convergence

Perceptron Learning Rule states that the algorithm would automatically learn the
optimal weight coefficients.

The input features are then multiplied with these weights to determine if a neuron
fires or not.

The Perceptron receives multiple input signals, and if the sum of the input signals
exceeds a certain threshold, it either outputs a signal or does not return an output.

In the context of supervised learning and classification, this can then be used to
predict the class of a sample.
Perceptron Rule of Convergence

The Perceptron rule states that if the data is linearly separable, the Perceptron
algorithm will converge to a solution in a finite number of steps.

1 Perceptron Learning Rule

The Perceptron updates its weights based on the following rule:

Structure of a Perceptron
Working of a Perceptron
The working of a Perceptron involves the following steps:

1. Initialize Weights and Bias:

a. Weights are initialized to small random values.
b. Bias is also initialized to a small random value.
2. Feed Input to the Network:
a. The input feature vector is multiplied with the corresponding weights.
3. Compute Weighted Sum:
a. The weighted sum is calculated using:

4. Apply Activation Function:

a. The step function is applied to decide the output:
5. Update Weights:
a. During training, the Perceptron’s weights are adjusted to minimize the difference
between the predicted output and the actual output.
b. This is achieved using supervised learning algorithms like the delta rule or the
Perceptron learning rule.

6. Repeat Until Convergence:

a. The process is repeated until the output matches the target value or the maximum
number of iterations is reached.
Perceptron Learning Algorithm
The Perceptron Learning Algorithm is an iterative method used to train a
perceptron model for binary classification.

It adjusts the model's weights to minimize classification errors.

Step 1: Initialize Weights and Bias

● Initialize the weights 𝑤1,𝑤2,…,𝑤𝑛 to small random values.

● Set the bias 𝑏 to zero or a small value.
● Choose a suitable learning rate 𝛼 (commonly between 0.01 and 0.1).

Step 2

● We perform the sweets 3 to 7 until the final stopping condition is false.

Step 3

● We apply identity activation functions containing the input layer containing the
input, unit.
●
Step 4

● We Obtain the net input:

● Where ‘n’ is the number of input neurons in the input layer.

● Now to obtain the output, we apply the activation formula over the net input.
Step 5 now we compare the actual output (calculated output) and desired (target)
output:
● If y≠t, then we use the formulae:

● Else, we have

Step 6:
● We train the network until there is no weight change.
● Thai is called stopping condition for the network, If the conditions is not
satisfied, we start again from Step 4
Perceptron Training Algorithm for Multiple Output Classes
For multiple output classes, the perceptron training algorithm is as follows:

Step 1:Initialize the weights, biases and learning rate suitably.

Step 2: Check for stopping condition; if it is false, perform Steps 3-7.

Step 3: Perform Steps 4-6 for each bipolar or binary training vector pair s:t.

Step 4: Set activation (identity) of each input unit i= 1 to n:

Step 5:Calculate output response of each output unit j=1 to m; First the net input is
calculated as:
Then activations are applied over the net input to calculate the output response:

Step 6: if Make adjustment in weights and bias for j =1 to m and i= 1 to n

I.e.

else we have
Single-Layer Perceptron:
Structure:
Consists of an input layer and an output layer, with no hidden layers in between.
Function:
Can only learn linear decision boundaries, meaning it can separate data points that
are linearly separable.
Limitations:
Struggles with complex, non-linear data patterns.
Example:
A simple binary classifier that can separate two classes with a straight line.
Multi-Layer Perceptron (MLP):

Structure: Includes an input layer, one or more hidden layers, and an output layer.

Function: Can learn complex, non-linear relationships in data through the use of
hidden layers.

Advantages: More powerful than single-layer perceptrons, capable of modeling

intricate patterns and making accurate predictions on complex datasets.

Example: Image recognition, natural language processing, and speech

recognition.

Training: MLPs are trained using algorithms like backpropagation, which adjusts
the weights and biases of the network to minimize errors.
Advantages of Single Layer Perceptrons:

Simplicity and Efficiency:

Perceptrons are easy to understand, implement, and train, making them

computationally efficient.

Binary Classification:

They are well-suited for binary classification problems where the data is linearly
separable.

Clear Geometric Interpretation:

They have a clear geometric interpretation as a hyperplane that separates two

classes of data.
Disadvantages of Single Layer Perceptrons:
Limited Expressive Power:
Single-layer perceptrons struggle with non-linearly separable patterns, such as
XOR, circles, or spirals.
Sensitivity to Initial Conditions:
The performance of a single-layer perceptron can be sensitive to the initial
conditions of the weights, potentially leading to suboptimal solutions.
Limited to Binary Classification:
Single-layer perceptrons are only suitable for binary classification problems.
Convergence Issues:
If the data is not linearly separable, the perceptron might keep adjusting endlessly
without finding the right answer.
Limitations of Perceptrons:

Non-Linear Data: Single-layer perceptrons cannot learn non-linear relationships in

data.

Feature Scaling: Perceptrons can be sensitive to feature scaling, which can impact
their performance.

Lack of Probabilistic Output: Perceptrons don't provide probabilistic outputs, which

can be useful in some applications.
Advantages of Multilayer Perceptron

● Powerful Non-Linear Modeling:

○ MLP can model complex and non-linear relationships due to its multiple layers and activation
functions.
● Universal Approximation:
○ An MLP with at least one hidden layer and sufficient neurons can approximate any continuous
function.
● Flexibility:
○ MLP can be adapted for various tasks such as classification, regression, and pattern
recognition.
● Generalization:
○ With proper training, MLP can generalize well on unseen data, improving prediction accuracy.
● Versatility:
○ MLP is effective for a wide range of problems, including image recognition, speech processing,
and financial forecasting.
Disadvantages of Multilayer Perceptron

● Computational Complexity:
○ Training an MLP requires significant computational resources, especially for large networks.
● Prone to Overfitting:
○ Without proper regularization or sufficient data, MLP may memorize the training data instead
of generalizing.
● Black-Box Nature:
○ Understanding how an MLP makes decisions can be challenging due to its complex structure.
● Slow Convergence:
○ Training may require extensive iterations to converge, particularly for deep architectures.
● Need for Hyperparameter Tuning:
○ Performance heavily depends on selecting appropriate learning rates, number of layers, and
neurons.
Limitations of Multilayer Perceptron

● Linearly Separable Limitation (in a Single Layer):

○ A single-layer perceptron cannot solve non-linearly separable problems like XOR.
● Data Dependency:
○ MLP performance depends heavily on data quality, size, and feature engineering.
● Vanishing Gradient Problem:
○ In deep networks, gradients may become too small, slowing learning progress in earlier layers.
● Training Instability:
○ MLP is sensitive to initial weight values and learning rate settings, which may cause
convergence issues.
Numerical

Q. Implement AND function using perceptron network for bipolar Inputs and
targets.
Implement OR function with binary inputs and bipolar targets using Perceptron
training algorithm upto 3 epochs.

Find the weights using perceptron network for ANDNOT function when all the
inputs are presented only one time use bipolar input and target.
Adaline Network
● A network with a single linear unit is called Adaline (Adaptive Linear Neural).
● A unit with a linear activation function is called a linear unit.
● In Adaline, there is only one output unit and output values are bipolar (+1,-1).
● Weights between the input unit and output unit are adjustable.
● It uses the delta rule i.e

● Where, , and t are the weight, predicted output, and true value
respectively.
● The learning rule is found to minimize the mean square error between
activation and target values.
● Adaline consists of trainable weights, it compares actual output with
calculated output, and based on error training algorithm is applied.
The differences between the Perceptron and Adaline
Adaptive Linear Neuron Learning algorithm

Step 0: initialize the weights and the bias are set to some random values but not to
zero, also initialize the learning rate α.

Step 1 − perform steps 2-7 when stopping condition is false.

Step 2 − perform steps 3-5 for each bipolar training pair s:t.

Step 3 − Activate each input unit as follows −

Step 4 − Obtain the net input with the following relation −

Here ‘b’ is bias and ‘n’ is the total number of input neurons.
Step 5 Until least mean square is obtained (t - yin), Adjust the weight and bias as
follows −

Now calculate the error using =>

Step 7 − Test for the stopping condition, if error generated is less then or equal to
specified tolerance then stop.
Multiple Adaptive Linear Neuron (Madaline)

Madaline which stands for Multiple Adaptive Linear Neuron, is a network which
consists of many Adalines in parallel. It will have a single output unit. Some
important points about Madaline are as follows −

● It is just like a multilayer perceptron, where Adaline will act as a hidden unit
between the input and the Madaline layer.
● The weights and the bias between the input and Adaline layers, as in we see
in the Adaline architecture, are adjustable.
● The Adaline and Madaline layers have fixed weights and bias of 1.
● Training can be done with the help of Delta rule.
It consists of “n” units of input layer and “m” units of Adaline layer and “1” unit of
the Madaline layer.

Each neuron in the Adaline and Madaline layers has a bias of excitation “1”.

The Adaline layer is present between the input layer and the Madaline layer; the
Adaline layer is considered as the hidden layer.
Multiple Adaptive Linear Neuron Training Algorithm
Step 0 − initialize the weights and the bias(for easy calculation they can be set to
zero). also initialize the learning rate α(0, α, 1) for simpicity α is set to 1.

Step 1 − perform steps 2-6 when stopping condition is false.

Step 2 − perform steps 3-5 for each bipolar training pair s:t

Step 3 − Activate each input unit as follows −

Step 4 − Obtain the net input at each hidden layer, i.e. the Adaline layer with the
following relation −

Here ‘b’ is bias and ‘n’ is the total number of input neurons.
Step 5 − Apply the following activation function to obtain the final output at the
Adaline and the Madaline layer −

Output at the hidden (Adaline) unit

Final output of the network

Step 6 − Calculate the error and adjust the weights as follows −

Step 7 − Test for the stopping condition, which will happen when there is no
change in weight or the highest weight change occurred during training is smaller
than the specified tolerance.
Back Propagation Neural Networks
Back Propagation Neural (BPN) was first introduced in 1960s, It is a multilayer
neural network consisting of the input layer, at least one hidden layer and output
layer.

It is a multi-layer feed forward neural network using Gradient descent approach

which exploits the chain rule to optimize the parameter.

The error which is calculated at the output layer, by comparing the target output
and the actual output, will be propagated back towards the input layer.

The main features of Backpropagation are the iterative, recursive and efficient
method through which it calculates the updated weight to improve the network
until it is not able to perform the task for which it is being trained.
Back Propagation Neural will use binary sigmoid activation function. The training
will have the following three phases.

Phase 1 − Feed Forward Phase

Phase 2 − Back Propagation of error

Phase 3 − Updating of weights

Back Propagation Neural Networks Training Algorithm

Step 0 − initialize the weights and the bias(For easy calculation and simplicity,
take some small random values but not zero). also initialize the learning rate α(0,
α, 1).

Step 1 − Continue step 2-10 when the stopping condition is not true.

Step 2 − Continue step 3-9 for every training pair.

Phase 1

Step 3 − Each input unit receives input signal xi and sends it to the hidden unit for
all i = 1 to n
Step 4 Calculate the net input at the hidden unit using the following relation −

Now calculate the net output by applying the following sigmoidal activation
function

Step 5 Calculate the net input at the output layer unit using the following relation

Calculate the net output by applying the following sigmoidal activation function
Phase 2

Step 6 − Compute the error correcting term, in correspondence with the target
pattern received at each output unit, as follows

Then, send δk back to the hidden layer

Step 7 − Now each hidden unit will be the sum of its delta inputs from the output
units.

Phase 3

Step 8 −
Step 9 −

Step 10 − Check for the stopping condition, which may be either the number of
epochs reached or the target output matches the actual output.
And Function using perceptron