DEEP LEARNING
CSA3007
Module 2
▪ Fundamental of Deep Learning Methods:
Dr. Praveen Kumar Tyagi
Unit II -
Artificial Neural Networks:
• Artificial Neural Networks are computational models inspired by the structure
and functioning of the human brain.
• Artificial Neural Networks contain artificial neurons, which are called units.
• These units are arranged in a series of layers that together constitute the
whole Artificial Neural Network in a system.
• They consist of interconnected units called artificial neurons that process
information and learn patterns from data..
• ANNs are widely used in machine learning and AI tasks like pattern
recognition and prediction.
Artificial Neural Networks:
• Inter-layer connections exist between units:
• Each connection has an associated weight.
• Weights determine the strength and influence of one unit on another.
• In the majority of neural networks, units are interconnected from one layer to
another.
• Each of these connections has weights that determine the influence of one unit
on another unit.
• As the data transfers from one unit to another, the neural network learns more
and more about the data, which eventually results in an output from the output
layer
Artificial Neural Networks:
Biological Inspiration
Human brain → neurons connected by synapses
ANN → artificial neurons connected by weights
Learning → change in synaptic strength (weights)
ANNs try to mimic how the human brain learns from
experience
Artificial Neural Networks:
Artificial Neural Networks:
• An artificial neural network consists of interconnected nodes, or neurons,
organized in layers.
The three main types of layers in a typical ANN are:
• Input Layer: This layer receives the initial data or input features and transmits
them to the subsequent layers for processing.
• Hidden Layers: These intermediate layers perform complex computations on
the input data, extracting meaningful features and patterns.
• Output Layer: The final layer produces the network’s predictions or outputs
based on the processed information from the hidden layers.
Artificial Neural Networks:
• Each layer performs a specific transformation on the data, passing it to the
next layer.
• Every connection between these neurons has a weight- numerical value that
dictates the strength and importance of that connection.
• These weights in the network constantly fine-tunes during its learning, using
sophisticated algorithms like back propagations and gradient descent, to
achieve better accuracy.
Artificial Neural Networks:
• The more layers and neurons a neural network has, the more complex
functions it can learn.
Suppose, the input layer has three neurons, corresponding to three features
of the data. The hidden layer has four neurons, performing some computation
on the input data. The output layer has one neuron, producing the final
prediction or decision.
Single layer Perception:
• The Single-Layer Perceptron is the simplest form of a neural networks,
primarily used for pattern classification of linearly separable data.
• The perceptron is a single processing unit of any neural network.
• In 1953, Rosenblatt Introduced the perceptron, the first computational
model for supervised learning.
• Influenced by:
• McCulloch & Pitts (1943) – Introduced neural networks as computing
machines.
• Hebb (1949) – Proposed Hebbian learning (first self-organizing rule).
Single layer Perception:
• A single-layer perceptron consists of a single neuron with adjustable
synaptic weights and a bias.
• The model operates by computing a weighted sum of input features and
passing it through an activation function to produce a binary output.
• Operates on linearly separable data:
• Can classify two classes separated by a straight line or hyperplane.
• Uses a step or Signum activation function to produce binary output.
• The learning algorithm updates weights based on classification error.
Single layer Perception:
Figure 1: Components of an Perceptron with Step Function
Single layer Perception:
• Key Features:
• It can classify patterns that are linearly separable, i.e., patterns that can be
separated by a straight line (in 2D), plane (in 3D), or hyperplane (in higher
dimensions).
• The learning algorithm iteratively adjusts weights to minimize the classification
error, and it converges if the classes are linearly separable (known as the
Perceptron Convergence Theorem).
• When expanded to include more neurons in the output layer, the perceptron
can be extended to multiclass classification, provided the data is linearly
separable.
Single layer Perception:
• Limitations
• While the perceptron is effective for binary classification of linearly separable
data, it:
• Limited to linear decision boundaries (cannot solve non-linearly separable
problems like XOR).
• Cannot model complex decision boundaries, limiting its use in many
real-world applications.
• Needs extension to multi-layer networks (like MLPs) to solve more complex
problems.
Learning Laws in ANN:
• Learning laws define how weights are updated during training.
❑ Hebbian Learning Rule:
• Hebbian learning mimics how the brain strengthens or weakens neural
connections (synapses) based on activity
• Principle: “ Neurons that fire together, wire together ”
• Hebbian Learning is an unsupervised learning rule in which the synaptic
weight between two neurons increases if:
the presynaptic neuron (input) is active, and
the postsynaptic neuron (output) is also active at the same time.
There is no target output, no error correction.
Learning Laws in ANN:
Learning Laws in ANN:
• Unsupervised learning
• No error term
• Inspired by brain learning
• Used in associative memory
Hebbian Learning :
x1 x2 X3 A single neuron with three inputs is trained using the
1 -1 2 Hebbian learning rule, Update the weights and bias for one
epoch using
0 1 1
1 1 0 Initial weights: 𝑤(0)=[0.5, 1.2, 0.8] , 𝑏(0)=0.3, 𝜂=0.1,
Activation: Linear
-1 2 1
Sol:
Hebbian Learning :
Hebbian Learning :
Hebbian Learning :
Learning Laws in ANN:
❑ Perceptron Learning Rule
• The perceptron can learn from examples through a process called training.
• During training, the perceptron adjusts its weights based on observed
errors. This is typically done using a learning algorithm such as the
perceptron learning rule or a backpropagation algorithm.
• The learning process presents the perceptron with labeled examples,
where the desired output is known.
• The perceptron compares its output with the desired output and adjusts its
weights accordingly, aiming to minimize the error between the predicted
and desired outputs.
Learning Laws in ANN:
Learning Laws in ANN:
Learning Laws in ANN:
Learning Laws in ANN:
• We optimize this error (loss function) using an optimization algorithm.
Generally, some form of gradient descent algorithm is used to find the
optimal values of the hyper parameters like learning rate, weight, Bias, etc.
This step forms the backward propagation part of the algorithm.
Initial weights and bias:
Target
(x1) (x2) • w1=0
(y)
• w2=0
2 5 0
• b=0
3 6 0
Set the Learning rate (η) to small value, such as 0.1.
5 8 1
Using the step activation function, train the perceptron for
6 7 1 one epoch and determine the final values of 𝑤1, 𝑤2, and
𝑏.
Question 2: Design a single-layer perceptron for the OR Boolean function using two
epochs: 𝑤1=0.5, 𝑤2=−0.5, 𝛼=0.5, and b=[Link] activation function is the step
function.
OR gate
NOR gate
(x₁) (x₂) (d)
0 0 0
0 1 1
1 0 1
1 1 1
Limitations of Perceptron:
• The Output of a perceptron can only be a binary number (0 or 1) due to the
hard limit transfer function. Thus it is difficult to use in problems other than
binary classification, like regression or multiclass classification.
• Linear Separability:
• Basic perceptron can only solve problems where data is linearly separable
(e.g., AND, OR).
• Cannot solve XOR problem or nonlinear classification tasks.
• Single Layer Restriction:
• Cannot extract complex hierarchical features.
• Only works with a linear decision boundary.
Limitations of Perceptron:
• Convergence Issues:
• If the problem is not separable, weights may oscillate and never converge.
• No Memory:
• Perceptron processes only present input; lacks ability to remember past
information.
• Non-Differentiability:
• Hard threshold activation is not differentiable, which restricts optimization
techniques.