Introduction to Deep Learning

Difference between ML and DL

Why DL is becoming more popular
• 2005  Orkut, Facebook, Whatsapp, instagram, twitter. – web2.0
• 2008  Big data (to store data efficiently)
• 2013 Huge amount of data available
• Hardware Advancement - GPUs
Neural Network
• A neural network is a massively parallel distributed processor made up of
simple processing units that has a natural propensity for storing
experiential knowledge and making it available for use.
• It resembles the brain in two respects:
1. Knowledge is acquired by the network from its environment through a
learning process.
2. Interneuron connection strengths, known as synaptic weights, are used
to store the acquired knowledge.
The procedure used to perform the learning process is called a learning
algorithm, the function of which is to modify the synaptic weights of the
network in an orderly fashion
Benefits of Neural Networks
• neural network derives its computing power through
• massively parallel distributed structure
• its ability to learn and therefore generalize.
• Generalization refers to the neural network’s production of reasonable
outputs for inputs not encountered during training (learning).

Neural networks offer the following useful properties and capabilities:

1. Nonlinearity. 5. Contextual Information.
2. Input–Output Mapping. 6. Fault Tolerance.
3. Adaptivity. 7. VLSI Implementability.
4. Evidential Response. 8. Uniformity of Analysis and Design.
[Link] Analogy.
NonLinearity
• An artificial neuron can be linear or nonlinear.
• A neural network, made up of an interconnection of nonlinear
neurons, is itself nonlinear.
• the nonlinearity is of a special kind in the sense that it is distributed
throughout the network.
• Nonlinearity is a highly important property, particularly if the
underlying physical mechanism responsible for generation of the
input signal (e.g., speech signal) is inherently nonlinear.
Input–Output Mapping
• In supervised learning, input–output mapping refers to the process by
which a neural network learns to connect each input (features or signals)
to its correct output (target label) using a set of labeled examples.
• During training, the network repeatedly adjusts its weights to reduce the
error between its predicted output and the desired output, gradually
forming a mapping that can generalize to unseen inputs.
• This mapping is often nonparametric, meaning no fixed mathematical
form is assumed; instead, the decision boundaries are learned directly
from data.
• In pattern classification, for example, the mapping defines how feature
vectors (like image pixels or audio signals) are transformed into class
labels, enabling the network to make accurate predictions even in
complex, nonlinear scenarios.
Adaptivity
• Neural networks can adapt their synaptic weights to changes in the
environment, allowing them to be retrained for minor variations or even
adjust in real time in nonstationary environments where data patterns
change over time.
• This makes them powerful tools for adaptive pattern classification, signal
processing, and control tasks.
• However, adaptivity must be balanced with stability—too much rapid
adaptation can cause the network to react to random noise (spurious
disturbances), reducing performance.
• To avoid this, the stability–plasticity dilemma must be managed by setting
adaptation rates (time constants) long enough to ignore irrelevant changes
but short enough to respond quickly to meaningful changes.
Evidential Response
• In the context of pattern classification, a neural network can be designed to
provide information not only about which particular pattern to select, but
also about the confidence in the decision made.
• This latter information may be used to reject ambiguous patterns, should
they arise, and thereby improve the classification performance of the
network.
• Prediction: [Cat: 0.85, Dog: 0.1, Rabbit: 0.05] High confidence –cat
• Prediction: [Cat: 0.35, Dog: 0.33, Rabbit: 0.32] Low confidence/
• The evidential response helps us know how much we trust the prediction
of a neural network.
It’s not just about being right — it’s about knowing when the model might
be wrong, and handling that carefully.
Contextual Information.
• A neural network doesn’t just look at one neuron or one piece of data
in isolation—every neuron’s output can be influenced by the activity
of many others.
• This means the network can naturally take into account the context in
which information appears. For example, when recognizing a word in
a sentence, the meaning depends on the surrounding words.
• In the same way, a neural network’s structure and the current
activation of all neurons together represent knowledge, allowing it to
process information with context automatically, rather than needing
special rules for it.
Fault Tolerance
•Fault tolerance is the ability of a neural network to continue functioning with
reduced performance even when some neurons or connections are damaged.
•Distributed storage – Information is stored across many neurons and
connections, not in a single location.
•Graceful degradation – Damage causes gradual performance loss rather
than sudden catastrophic failure.
•Example – If a neuron or its links are damaged, recall quality decreases, but
the network still produces useful results.
•Extent of damage – Serious performance degradation occurs only when
damage is extensive.
•Ensuring fault tolerance – May require designing the training algorithm to be
robust.
VLSI Implementability

•Massively parallel nature of neural networks makes them well-suited for

hardware implementation.
•VLSI (Very-Large-Scale Integration) can place millions of artificial neurons
and connections on a single chip.
•Enables faster computation for certain tasks compared to software
execution.
•Hierarchical design capability of VLSI matches the layered structure of
neural networks.
•Suitable for real-time applications requiring high-speed processing.
•Provides a way to capture complex behavior efficiently in hardware.
Uniformity of Analysis and Design.
• Neural networks enjoy universality as information processors.
• the same notation is used in all domains involving the application of
neural networks.
• This feature manifests itself in different ways:
• Neurons, in one form or another, represent an ingredient common to all
neural networks.
• This commonality makes it possible to share theories and learning algorithms
in different applications of neural networks.
• Modular networks can be built through a seamless integration of modules
Neurobiological Analogy
• neural networks are inspired by the human brain. The brain shows us that
parallel, fault-tolerant, and powerful computation is possible, so engineers
design artificial neural networks with similar principles. This analogy works
both ways:
• For neuroscientists, artificial neural networks help model and understand
brain functions (e.g., modeling the vestibulo-ocular reflex to understand
how eye movement responds to head movement).
• For engineers, studying the brain gives ideas for building better and more
efficient systems (e.g., neuromorphic chips that work like the retina to
detect edges and motion).
The key point: brain-inspired design helps science and engineering improve
each other.
 THE HUMAN BRAIN
• The human nervous system may be viewed as a three-stage system,

• Central to the system is the brain, represented by the neural (nerve) net, which continually
receives information, perceives it, and makes appropriate decisions.
• Two sets of arrows are shown in the figure.
• Those pointing from left to right indicate the forward transmission of information-bearing
signals through the system.
• The arrows pointing from right to left (shown in red) signify the presence of feedback in the
system.
• The receptors convert stimuli from the human body or the external environment into
electrical impulses that convey information to the neural net (brain).
• The effectors convert electrical impulses generated by the neural net into discernible
responses as system outputs.
The Human Brain
• the brain works through neurons connected by synapses, which act like tiny
switches that pass signals between cells.
• Most synapses are chemical synapses, where an electrical signal is turned into a
chemical signal and then back into an electrical one in the next neuron.
• Neurons send their messages as action potentials (spikes), which are short
voltage pulses that travel along the axon without losing strength—this is
important because axons are long and signals would fade otherwise.
• The brain has a hierarchy of organization: starting from synapses, to
microcircuits, to neurons, to local circuits, to interregional circuits, and finally to
topographic maps that process sensory information in an organized way. Unlike
digital computers, the brain’s organization is extremely complex, and our artificial
neural networks are still far simpler, but inspired by these principles.
 The Human Brain
• The synapses represent the most fundamental level,
depending on molecules and ions for their action.
• At the next levels, neural microcircuits, dendritic trees,
and then neurons.
• A neural microcircuit refers to an assembly of synapses
organized into patterns of connectivity to produce a
functional operation of interest.
• A neural microcircuit may be likened to a silicon chip made
up of an assembly of transistors. The smallest size of
microcircuits is measured in micrometers ( m), and their
fastest speed of operation is measured in milliseconds.
• The neural microcircuits are grouped to form dendritic
subunits within the dendritic trees of individual neurons.
• The whole neuron, about 100 m in size, contains several
dendritic subunits.
 The Human Brain
• At the next level of complexity, local circuits (about 1 mm in
size) made up of neurons with similar or different properties;
these neural assemblies perform operations characteristic of
a localized region in the brain.
• They are followed by interregional circuits made up of
pathways, columns, and topographic maps, which involve
multiple regions located in different parts of the brain.
• Topographic maps are organized to respond to incoming
sensory information. These maps are often arranged in
sheets, as in the superior colliculus, where the visual,
auditory, and somatosensory maps are stacked in adjacent
layers in such a way that stimuli from corresponding points in
space lie above or below each other.
• At the final level of complexity, the topographic maps and
other interregional circuits mediate specific types of behavior
in the central nervous system.
MODELS OF A NEURON
• A neuron is an information-processing unit that is fundamental to the
operation of a neural network.
 MODELS OF A NEURON
• three basic elements of the neural model:
1. A set of synapses, or connecting links, each of which is characterized by a
weight or strength of its own. Specifically, a signal xj at the input of synapse j
connected to neuron k is multiplied by the synaptic weight wkj.
2. An adder for summing the input signals, weighted by the respective synaptic
strengths of the neuron; the operations described here constitute a linear
combiner.
3. An activation function for limiting the amplitude of the output of a neuron. The
activation function is also referred to as a squashing function, in that it
squashes (limits) the permissible amplitude range of the output signal to some
finite value.
Typically, the normalized amplitude range of the output of a neuron is written
as the closed unit interval [0,1], or, alternatively, [-1,1].
It also includes an externally applied bias, denoted by bk. The bias bk has the effect
of increasing or lowering the net input of the activation function, depending on
whether it is positive or negative, respectively.
Mathematical model
• Neuron k is depicted using 2 equations:

• where x1, x2, ..., xm are the input signals; wk1, wk2, ..., wkm are the respective
synaptic weights of neuron k; uk is the linear combiner output due to the
input signals; bk is the bias; ϕ(·) is the activation function; and yk is the
output signal of the neuron. The use of bias bk has the effect of applying an
affine transformation to the output uk of the linear combiner in the model.
Mathematical model
• The use of bias bk has the effect of applying an affine transformation
to the out put uk of the linear combiner in the model, as shown by eq
Mathematical Model

added a new synapse. Its input is

x0 =+1
and its weight is wk0 = bk
Types of Activation Function
• The activation function, denoted by ϕ(v), defines the output of a
neuron in terms of the induced local field v.
• two basic types of activation functions:
• 1. Threshold Function. For this type of activation function,

• This form of a threshold function is commonly referred to as a

Heaviside function. Correspondingly, the output of neuron k
employing such a threshold function is expressed as
Types of Activation Function
Types of Activation Function
• The sigmoid function, whose graph is “S”-shaped, is by far the most common
form of activation function used in the construction of neural networks. It is
defined as a strictly increasing function that exhibits a graceful balance between
linear and nonlinear behavior. An example of the sigmoid function is the logistic
function, defined by

• where a is the slope parameter of the sigmoid function. By varying the parameter
a, we obtain sigmoid functions of different slopes, as illustrated in Fig. 8b. In fact,
the slope at the origin equals a/4. In the limit, as the slope parameter approaches
infinity, the sigmoid function becomes simply a threshold function. Whereas a
threshold function assumes the value of 0 or 1, a sigmoid function assumes a
continuous range of values from 0 to 1.
NEURAL NETWORKS VIEWED AS DIRECTED GRAPHS
• signal-flow graphs - simplify the appearance of the model.
• Signal-flow graphs, with a well-defined set of rules, were originally
developed by Mason (1953, 1956) for linear networks.
• A signal-flow graph is a network of directed links (branches) that are
interconnected at certain points called nodes.
• A typical node j has an associated node signal xj.
• A typical directed link originates at node j and terminates on node k; it has
an associated transfer function, or transmittance, that specifies the manner
in which the signal yk at node k depends on the signal xj at node j. The flow
of signals in the various parts of the graph is dictated by three basic rules:
NEURAL NETWORKS VIEWED AS DIRECTED GRAPHS
• The flow of signals in the various parts of the graph is dictated by
three basic rules:
• Rule 1. A signal flows along a link only in the direction defined by the
arrow on the link.
• Rule 2. A node signal equals the algebraic sum of all signals entering
the pertinent node via the incoming links.
• Rule 3. The signal at a node is transmitted to each outgoing link
originating from that node, with the transmission being entirely
independent of the transfer functions of the outgoing links.
NEURAL NETWORKS VIEWED AS DIRECTED GRAPHS
• Two different types of links may be distinguished:
• Synaptic links, whose behavior is governed by a linear input–output relation.
Specifically, the node signal xj is multiplied by the synaptic weight wkj to
produce the node signal yk, as illustrated in Fig. 9a.
• Activation links, whose behavior is governed in general by a nonlinear input–
output relation. This form of relationship is illustrated in Fig. 9b, where ϕ(·) is
the non linear activation function.
NEURAL NETWORKS VIEWED AS DIRECTED GRAPHS
• This second rule is illustrated in Fig.c for the case of synaptic
convergence, or fan-in.
• This third rule is illustrated in Fig. d for the case of synaptic
divergence, or fan-out.
NEURAL NETWORKS VIEWED AS DIRECTED GRAPHS
• mathematical definition of a neural network:
• A neural network is a directed graph
consisting of nodes with interconnecting
synaptic and activation links and is
characterized by four properties:
1. Each neuron is represented by a set of linear
synaptic links, an externally applied bias, and
a possibly nonlinear activation link. The bias
is represented by a synaptic link connected to
an input fixed at 1.
2. The synaptic links of a neuron weight their
respective input signals.
3. The weighted sum of the input signals defines
the induced local field of the neuron in
question
4. The activation link squashes the induced
local field of the neuron to produce an
output.
FEEDBACK

• Feedback is said to exist in a dynamic system whenever the output of

an element in the system influences in part the input applied to that
particular element, thereby giving rise to one or more closed paths
for the transmission of signals around the system.
• the signal-flow graph of a single loop feedback system, where the
input signal xj(n), internal signal x’j (n), and output signal yk(n) are
functions of the discrete-time variable n. The system is assumed to be
linear, consisting of a forward path and a feedback path that are
characterized by the “operators” A and B, respectively
FEEDBACK
Feedback
• A is a fixed weight w and B is a unit-delay operator z-1, whose output
is delayed with respect to the input by one time unit.
Feedback
Feedback
NETWORK ARCHITECTURES
• Single-Layer Feedforward Networks
• neurons are organized in the form of layers.
• In the simplest form of a layered network, have an input
layer of source nodes that projects directly onto an output
layer of neurons (computation nodes), but not vice versa.
• This network is strictly of a feedforward type.
• for the case of four nodes in both the input and output
layers. Such a network is called a single-layer network,
with the designation “single-layer” referring to the output
layer of computation nodes (neurons).
• We do not count the input layer of source nodes because
no computation is performed there.
•.
• Multilayer Feedforward Networks
• neural network where information flows in one direction only:
Input → Hidden layers → Output.
• it has hidden layers that transform the input before it reaches the
output.
•“Hidden” neurons are not directly visible from input or output.
•They are the internal workers of the network, transforming raw
input into something more meaningful.
•With just inputs and outputs, the network can only learn simple
relationships.
•Hidden layers allow the network to learn complex, higher-order
patterns:
•Detect edges in images, Recognize letters from shapes, Understand
meaning in speech.
•Input layer supplies raw features (numbers, pixels, measurements).
•Hidden layer neurons take combinations of those inputs and apply
activation functions (e.g., sigmoid, ReLU).
•Their outputs feed into the next layer.
•Finally, the output layer produces the network’s prediction/decision.
•.
• Recurrent Networks
• A recurrent neural network distinguishes itself from a feedforward neural network in that
it has at least one feedback loop.
• a recurrent network may consist of a single layer of neurons with each neuron feeding its
output signal back to the inputs of all the other neurons.
• there are no self-feedback loops in the network; self-feedback refers to a situation where
the output of a neuron is fed back into its own input.
• The presence of feedback loops, has a profound impact on the learning capability of the
network and on its performance. The feedback loops involve the use of particular branches
composed of unit-time delay elements (denoted by z-1),which result in a nonlinear dynamic
behavior, assuming that the neural network contains nonlinear units.
Rosenblatt’s Perceptron
Introduction
• In the formative years of neural networks (1943–1958), several
researchers stand out for their pioneering contributions:
• McCulloch and Pitts (1943) for introducing the idea of neural
networks as computing machines. Rosenblatt’s Perceptron
• Hebb (1949) for postulating the first rule for self-organized
learning.
• Rosenblatt (1958) for proposing the perceptron as the first
model for learning with a teacher (i.e., supervised learning).
Perceptron
• is the simplest form of a neural network used for the classification of
patterns said to be linearly separable.
• it consists of a single neuron with adjustable synap tic weights and bias.
• The algorithm used to adjust the free parameters of this neural network
first appeared in a learning procedure developed by Rosenblatt
(1958,1962) for his perceptron brain model.
• Rosenblatt proved that if the patterns (vec tors) used to train the
perceptron are drawn from two linearly separable classes, then the
perceptron algorithm converges and positions the decision surface in
the form of a hyperplane between the two classes. The proof of
convergence of the algorithm is known as the perceptron convergence
theorem.
PERCEPTRON
• a neural modeling consists of a linear combiner followed by a hard limiter.
• The summing node of the neural model computes a linear combination of the
inputs applied to its synapses, as well as incorporates an externally applied
bias.
• The resulting sum, that is, the induced local field, is applied to a hard limiter.
• Accordingly, the neuron produces an output equal to 1 if the hard limiter
input is positive, and -1 if it is negative.
PERCEPTRON
• In the signal-flow graph model of Fig. 1.1,
• the synaptic weights of the perceptron are denoted by w1,w2,...,wm.
• the inputs applied to the perceptron are denoted by x1, x2, ..., xm.
• The externally applied bias is denoted by b.
• From the model, the hard limiter input, or induced local field, of the neuron is
PERCEPTRON
• The goal of the perceptron is to correctly classify the set of externally applied
stimuli x1, x2,...,xm into one of two classes,c1 or c2.
• The decision rule for the classification is to as sign the point represented by
the inputs x1, x2,...,xm to class c1 if the perceptron output y is +1 and to class c2
if it is -1.
• To develop insight into the behavior of a pattern classifier, it is customary to
plot a map of the decision regions in the m-dimensional signal space spanned
by the m input variables x1, x2, ..., xm. In the simplest form of the perceptron,
there are two decision regions separated by a hyperplane, which is defined by
 Perceptron
• For the case of two input variables x1 and x2, for which
the decision boundary takes the form of a straight line.
• A point (x1, x2) that lies above the boundary line is
assigned to class c1,and a point (x1,x2) that lies below
the boundary line is assigned to class c2.
• the effect of the bias b is merely to shift the decision
boundary away from the origin.
• The synaptic weights w1,w2,...,wm of the perceptron can
be adapted on an iteration by-iteration basis.
• For the adaptation, we may use an error-correction rule
known as the perceptron convergence algorithm
RELATION BETWEEN THE PERCEPTRON AND BAYES
CLASSIFIER FOR A GAUSSIAN ENVIRONMENT