Module 1-notes
Deep Learning (BCA-701)
Deep learning (BCA701) Department of CSE(AI&ML)
Module-1
1. INTRODUCTION
1.1 Definition:
Neural Network or Artificial Neural Network (ANN) or Simulated Neural Network (SNN) is
derived from Biological neural networks that develop the structure of a human brain.
o Neural networks are inspired by the human brain, which is a complex, nonlinear,
parallel processing system.
o The brain can organize its structural constituents, known as neurons,
o The brain performs tasks like pattern recognition and motor control faster than
digital computers.
Examples
Human vision which is an information-processing task, it processes complex scenes in
100–200 ms.
Sonar-an active echolocation System, Bats use echolocation to detect: Distance,
Deep learning (BCA701) Department of CSE(AI&ML)
Velocity, Size, Shape, Direction (azimuth and elevation). These complex tasks are
performed by a brain as small as a plum, outperforming man-made sonar systems.
o The brain is structured at birth but develops over time through experience. Most neural
development happens in the first two years, but continues afterward. The brain exhibits
plasticity, meaning it can adapt to its environment.
Definition of Neural Network
A massively parallel distributed processor made of simple processing units (artificial
neurons).
Has a natural tendency to store and utilize experiential knowledge.
Mimics the brain in two key ways:
1. Learning from the environment.
o Stores and uses experiential knowledge.
o Learns from the environment through a learning algorithm.
2. Storing knowledge via synaptic weights (inter-neuron connections).
o Knowledge is stored in synaptic weights (interconnections).
o Learning modifies these weights to improve performance.
Like biological neurons, artificial networks can modify their structure (e.g., grow or prune
connections).
o Aim to adjust weights to meet a desired objective.
o Related to concepts in adaptive filter theory.
Example:
• A neural network can be used for predicting housing prices.
• Email spam detection
• Used for sentiment analysis in natural language processing.
• Facial recognition.
Benefits of Neural Networks
Deep learning (BCA701) Department of CSE(AI&ML)
Neural Network derives its computing power through,
1. Neural networks leverage massively parallel processing, enhancing computational
efficiency.
2. They can learn from data and generalize to unseen inputs, making them powerful in
dynamic environments.
o Their combined processing capabilities allow them to approximate solutions to large-scale,
intractable problems.
o Neural networks work best when integrated into a larger system, assigned to specific tasks
that match their strengths.
o Complex problems are broken into simpler tasks, which neural networks handle effectively
within their design limits.
Neural networks offer the following useful properties and capabilities:
1. Nonlinearity
2. Input–Output Mapping
3. Adaptivity
4. Evidential Response
5. Contextual Information.
6. Fault Tolerance.
7. VLSI Implementability.
8. Uniformity of Analysis and Design
9. Neurobiological Analogy.
1. Nonlinearity
o Neural networks use nonlinear neurons.
o Suitable for modelling complex, real-world nonlinear processes (e.g., speech).
2. Input–Output Mapping (Supervised Learning)
o Learns from labelled data by adjusting synaptic weights.
Deep learning (BCA701) Department of CSE(AI&ML)
o Builds input-output mapping without assuming a fixed statistical model.
o Related to nonparametric statistical inference.
3. Adaptivity
o Adapts to environmental changes by updating weights.
o Balances stability vs. plasticity (the stability–plasticity dilemma).
o Useful in dynamic/nonstationary environments.
4. Evidential Response
o Provides confidence levels in decisions.
o Can reject ambiguous inputs to improve reliability.
5. Contextual Information
o Neurons influenced by overall network activity.
o Naturally captures and utilizes contextual knowledge.
6. Fault Tolerance
o Tolerates failures without catastrophic performance drop.
o Information is distributed—small damage has minimal effect.
o Requires robust training algorithms.
7. VLSI Implementability
o Neural networks' parallel structure fits well in VLSI hardware.
o Enables high-speed and efficient processing.
8. Uniformity of Analysis and Design
o Consistent structure (neurons/connections) across applications.
o Supports modularity and algorithm reuse.
o Provides a unified design framework.
9. Neurobiological Analogy
o Inspired by the human brain's parallel and fault-tolerant processing.
o A research tool for both engineers and neurobiologists.
1.2 HUMAN BRAIN
• The human nervous system may be viewed as a three-stage system.
Figure 1: Block diagram representation of nervous system
• Two sets of arrows are shown in the figure.
• Those pointing from left to right indicate the forward transmission of information-bearing
signals through the system.
• The arrows pointing from right to left signify the presence of feedback in the system.
• The receptors convert stimuli from the human body or the external environment into
electrical impulses that convey information to the neural net (brain).
• Central to the system is the brain, represented by the neural (nerve) net, which continually
receives information, perceives it, and makes appropriate decisions.
• The effectors convert electrical impulses generated by the neural net into discernible
responses as system outputs.
• Typically, neurons are five to six orders of magnitude slower than silicon logic gates; events
in a silicon chip happen in the nanosecond range, whereas neural events happen in the
millisecond range. However, the brain makes up for the relatively slow rate of operation of
a neuron by having a truly staggering number of neurons (nerve cells) with massive
interconnections between them.
• It is estimated that there are approximately 10 billion neurons in the human cortex, and 60
trillion synapses or connections.
Deep learning (BCA701) Department of CSE(AI&ML)
Biological Neuron Network
Figure 2: The
pyramidal cell.
As shown in the above diagram, a typical neuron consists of the following four parts, with the
help of which we can explain its working −
Dendrites − They are tree-like branches, responsible for receiving the information from other
neurons it is connected. In another sense, we can say that they are like the ears of a neuron.
Soma (or Cell Nucleus) − It is the cell body of the neuron and is responsible for processing of
information they have received from dendrites.
Axon − It is just like a cable through which neurons send the information.
Synapses − It is the connection between the axon and other neuron dendrites. Synapses, or nerve
endings, are elementary structural and functional units that mediate the interactions between
neurons.
The most common kind of synapse is a chemical synapse, which operates as follows: A
presynaptic process liberates a transmitter substance that diffuses across the synaptic junction
between neurons and then acts on a postsynaptic process. Thus, a synapse converts a
presynaptic electrical signal into a chemical signal and then back into a postsynaptic electric al
signal.
1.3 MODELS OF A NEURON
A neuron is an information-processing unit that is fundamental to the operation of a
neural network.
The block diagram of Fig. 3 shows the model of a neuron, which forms. The basis for designing
a large family of neural networks.
6
Deep learning (BCA701) Department of CSE(AI&ML)
Figure 3: Nonlinear model of a neuron, labeled k.
The three basic elements of the neural model:
1. synapses, or connecting links
2. adder
3. activation function
1. A set of synapses, or connecting links, each of which is characterized by a weight or
strength of its own. Specifically, a signal xj at the input of synapse j connected to neuron k is
multiplied by the synaptic weight wkj.
In the weight notation wkj:
The first subscript (k) → refers to the neuron receiving the input.
The second subscript (j) → refers to the input source or synapse index.
Unlike the weight of a synapse in the brain, the synaptic weight of an artificial neuron may lie
in a range
that includes negative as well as positive values.
2. An adder for Sums all input signals after they are weighted by their respective synaptic
strengths.
Performs a linear combination of the form:
This summation is linear and forms the net input to the neuron.
3. An activation function(Squashing Function) for limiting(squashes) the amplitude of the
neuron's
output signal to a finite range.
Typically,
Deep learning (BCA701) Department of CSE(AI&ML)
Introduces non-linearity, essential for learning complex patterns.
Output:
Bias Term (bk)
Externally applied parameter to shift activation threshold.
The bias bk adjusts the net input to the activation function.
It causes an affine transformation of the linear combiner output uk, resulting in
a new value called the induced local field or activation potential vk
Adds flexibility to the neuron's response:
Alters the graph of vk vs uk, so it no longer passes through the origin.
The bias bk has the effect of increasing or lowering the net input of the activation function,
depending on whether it is positive or negative, respectively.
In mathematical terms,
where
Deep learning (BCA701) Department of CSE(AI&ML)
Figure 5 Affine transformation produced by the presence of a bias; note that vk = bk at uk = 0.
• If bk > 0, the activation function shifts right; if bk < 0, it shifts left.
• Due to this shift, the graph of vk vs. uk does not pass through the origin, as shown in Fig.
6
• This flexibility helps neurons learn complex decision boundaries during training.
Reformulated as:
Graphically and mathematically equivalent to previous model, more convenient in
implementation (like in backpropagation).
In Eq. (4), we have added a new synapse. Its input is
Deep learning (BCA701) Department of CSE(AI&ML)
Types of Activation Function
An activation function is a mathematical equation that determines the output of a
neuron in a neural network. It takes the weighted sum of the inputs (plus a bias) and
transforms it into an output signal that is then passed to the next layer of neurons.
10
Deep learning (BCA701) Department of CSE(AI&ML)
2. Sigmoid Function
The sigmoid function is the most common activation function in neural networks, known
for its S-
shaped curve.
Where, a: Slope parameter, controls the steepness of the curve.
v: Induced local field (input to activation function).
Deep learning (BCA701) Department of CSE(AI&ML)
1.4 NEURAL NETWORKS VIEWED AS DIRECTED GRAPHS
The various elements that constitute the model of an artificial neuron.
Deep learning (BCA701) Department of CSE(AI&ML)
Deep learning (BCA701) Department of CSE(AI&ML)
Neural Network Definition as a Directed Graph:
A neural network is a directed graph consisting of nodes with interconnecting synaptic and
activation links.
Or
A directed graph of nodes connected by synaptic (linear) and activation (possibly nonlinear)
links.
Four properties characterize it:
1. Each neuron is represented by a set of linear synaptic links, an externally applied
bias, and a nonlinear activation link. The bias is represented by a synaptic
link connected to an input fixed at +1.
2. Input signals are weighted by synaptic links.
3. The weighted sum of the input signals defines the induced local field of the neuron.
4. The activation link squashes the induced local field of the neuron to produce a final
output.
Partially Complete Graph (Architectural Graph)
14
Deep learning (BCA701) Department of CSE(AI&ML)
1.5 FEEDBACK
1.6 NETWORK ARCHITECTURE
There are several different architectures for ANNs, each with their own strengths and
weaknesses. Some of the most common architectures include:
Feedforward Neural Networks: This is the simplest type of ANN architecture, where the
information flows in one direction from input to output. The layers are fully connected,
meaning each neuron in a layer is connected to all the neurons in the next layer.
There are three fundamental classes of ANN architectures:
1. Single layer feed forward architecture
2. Multilayer feed forward architecture
3. Recurrent networks architecture
Single-layer Feedforward Networks:
The simplest kind of neural network is a single-layer Feed-forward network, which consists
of a single layer of output nodes; the inputs are fed directly to the outputs via a series of
weights.
Deep learning (BCA701) Department of CSE(AI&ML)
In this way, it can be considered the simplest kind of feed-forward network.
Layer is formed by taking processing elements and combining it with other
processing elements.
• Input and output are linked with each other.
• Inputs are connected to the processing nodes with various weights, resulting
in series of outputs per node.
• We do not count the input layer of source nodes because no computation is
performed there.
• The sum of the products of the weights and the inputs is calculated in each
node, and if the value is above some threshold (typically 0), the neuron
fires and takes the activated value (typically 1); otherwise, it takes the
deactivated value (typically -1).
• Neurons with this kind of activation function are also called artificial neurons or linear
threshold units.
• In the literature, the term perceptron often refers to networks consisting of just one of
these units.
• Warren McCulloch and Walter Pitts described a similar neuron in the 1940s.
Multilayer Feedforward Networks:
A Multilayer feedforward neural network, as its name suggests, is made up of multiple
layers.
These architectures typically include:
An input layer, which receives the raw data.
• One or more intermediary (hidden) layers, and
• An output layer, which produces the final prediction or result.
Architectures of this class not only process inputs and outputs but also include hidden layers
intermediary layers between input and output.
• The neurons in these layers are called hidden neurons or hidden units, so named
because they are not directly seen from either the input or the output of the network.
Deep learning (BCA701) Department of CSE(AI&ML)
• These hidden neurons perform useful intermediary computations, allowing the
network to transform and extract meaningful patterns from the
input before passing information to the output layer.
• Adding one or more hidden layers enables the network to
extract higher-order features from the input and gain a
global perspective despite the typically local connectivity of
neurons.
• It does not have any loops, no feedback, and no signal
moves in backward directions.
• The source nodes in the input layer supply the activation
pattern (input vector).
• These signals are passed forward to the first hidden layer.
• The outputs of each hidden layer serve as inputs to the next
layer, continuing this pattern until the output layer is reached.
• The final output layer provides the overall response of the
network to the given input.
Connectivity of the Network
• A network is called fully connected when every node in a layer is connected to every
node in the
adjacent forward layer.
• If some of these synaptic connections are missing, the network is considered partially
connected.
Example Architectures, A network described as 10–4–2 has:
• 10 input (source) nodes,
• 4 hidden neurons in a single hidden layer,
• 2 output neurons.
More complex networks with multiple hidden layers are m–h1–h2–q, where:
• m = number of input nodes,
• h1 = number of neurons in the first hidden layer,
• h2 = number of neurons in the second hidden layer,
• q = number of output neurons.
Recurrent Neural Network (RNNs) Architecture:
Unlike feedforward networks, Recurrent Neural Networks (RNNs) include at least
one feedback loop.
An RNN may consist of a single layer of neurons where each neuron’s output is fed
back to the inputs of all other neurons in the layer. This setup enables the network to maintain
a form of memory,
where the output from one time step influences subsequent time [Link]-feedback can also
Deep learning (BCA701) Department of CSE(AI&ML)
occur, where
a neuron’s output is fed back into its own input.
• This structure shows an RNN where there are no self-feedback loops—meaning that
each neuron’s
output is fed into the other neurons but not back into itself.
Self-feedback occurs when a neuron’s output is fed back into its own input, a characteristic
sometimes found in more complex.
RNN architectures.
• It has no hidden neurons.
• It only involves feedback between the output neurons,emphasizing the difference in
structure compared to feedforward networks, which typically have hidden layers for
intermediate computation.
In more complex RNNs, as shown there are hidden neurons.
These hidden neurons, like the output neurons, contribute to the feedback connections.
The feedback in this case originates from both the hidden neurons and the output
neurons, which further enriches the network's capability to store and propagate
information.
The presence of feedback loops profoundly impacts the learning capability and
performance of the network.
Feedback allows the network to maintain a form of memory of past inputs, which is crucial
for tasks such as time-series prediction, speech recognition, and sequence modeling.
These feedback loops incorporate elements like unit-time delay elements (denoted by
z−1).
These delay elements introduce nonlinear dynamic behaviour into the network,
Deep learning (BCA701) Department of CSE(AI&ML)
especially when the neural network contains nonlinear units.
This dynamic behaviour helps the network learn more complex patterns, unlike the static
nature of feedforward networks.
1.7 The Perceptron: A Basic Pattern Classifier
The Perceptron is the simplest form of a neural network and is used for classifying
linearly separable patterns—i.e., patterns that can be separated into two classes by a
hyperplane.
• The Perceptron consists of a single neuron with adjustable synaptic weights and a
bias.
• The basic perceptron with a single neuron can only handle binary classification—two
classes.
Multi-Class Classification-----However, by expanding the output layer of the
perceptron to include more neurons, it can handle multi-class classification. Each neuron
would correspond to a different class, but for the perceptron to work effectively, the
classes still need to be linearly separable.
1.7.1 Neural Model of the Perceptron
The perceptron is based on a McCulloch-Pitts model of a neuron, which combines:
1. A linear combiner that computes the weighted sum of inputs.
2. A hard limiter (the signum function) after calculating the weighted sum of
inputs, producing an output of +1 or -1, depending on whether the summed
value is positive or negative.
This is illustrated in Fig. 1.1, where the inputs are multiplied by their respective weights,
summed together, and then passed through the hard limiter to generate the output.
Deep learning (BCA701) Department of CSE(AI&ML)
Limitation of the Perceptron
The perceptron is limited to solving classification problems where the classes are linearly
separable.
If the data is not linearly separable (i.e., no hyperplane can separate the classes), the
perceptron will fail to converge, and no solution will be found.
Deep learning (BCA701) Department of CSE(AI&ML)
1.8. Perceptron Convergence Theorem
This theorem proves convergence of the perceptron for linearly separable data. If training
patterns are from two linearly separable classes, the algorithm converges.
o It positions a decision surface (hyperplane) between the classes.
o It adjusts the free parameters (weights and bias) based on training data.
Neuron Architecture:
1. Linear Combiner:
Computes a weighted sum of inputs.
Bias is added to this sum.
2. Induced Local Field:
The total sum (input-weight products + bias) is called the induced local field.
3. Hard Limiter (Signum Function):
The hard limiter applies a nonlinear decision rule to the linear combination.
The induced local field is the input to the hard limiter.
Acts on the induced local field.
Produces a binary output (+1 or -1, typically).
n is the time step (iteration number).
The leading +1 accounts for the bias in a unified vector form.
Weight Vector:
Deep learning (BCA701) Department of CSE(AI&ML)
The first component , represents the bias weight.
Linear Combiner Output (Induced Local Field)
This simplifies the computation of the neuron's pre-activation value.
Decision Surface:
The equation defines a hyperplane in m-dimensional space.
This hyperplane acts as the decision boundary between two classes.
Classes are well separated = a straight line (2D) can divide them.
Start with considering,
Deep learning (BCA701)
Department of CSE(AI&ML)
Deep learning (BCA701)
Department of CSE(AI&ML)
Therefore, we can write,
Deep learning (BCA701)
Department of CSE(AI&ML)
Deep learning (BCA701) Department of CSE(AI&ML)
1.9 Relation between the perceptron and Bayes classifier for a
Gaussian environment
The perceptron bears a certain relationship to a classical pattern classifier known as the Bayes
classifier. When the environment is Gaussian, the Bayes classifier reduces to a linear classifier.
This is the same form taken by the perceptron. However, the linear nature of the perceptron is not
contingent on the assumption of Gaussianity.
Bayes Classifier
In the Bayes classifier, or Bayes hypothesis testing procedure, we minimize the average risk, denoted
by
Eq(1)
For a two-class problem, represented by classes c1 and c2, the average risk is defined by Van
Trees (1968) as
Deep learning (BCA701) Department of CSE(AI&ML)
Eq(2)
The first two terms on the right-hand side of Eq.(2) represent correct decisions (i.e., correct
classifications).The last two terms represent incorrect decisions (i.e., misclassifications).
Each decision is weighted by the product of two factors:
1. The cost involved in making the decision
2. The relative frequency (i.e., prior probability) with which it occurs
Accordingly, we may rewrite Eq. (2) in the equivalent form
Eq(3)
Hence, Eq. (3) reduces to
Eq(4)
The first two terms on the right-hand side of Eq.(4) represent a fixed cost.
Deep learning (BCA701) Department of CSE(AI&ML)
Eq(5)
Likelihood ratio,
Eq(6)
Threshold of the test
Eq(7)
Deep learning (BCA701) Department of CSE(AI&ML)
Deep learning (BCA701) Department of CSE(AI&ML)
Eq(8)
Assume the following:
Specifically, by substituting Eq. 8 into 6 and taking the natural logarithm, we get
If above equation is linear classifier, as described by the relation
Figure : Signal-flow graph of Gaussian classifier.
Deep learning (BCA701) Department of CSE(AI&ML)