Neural Network: Introduction
• It was more than 2000 years ago, our
ancestors had started to discover the
architecture and behavior of human
brain.
• Ramon Y. Cajal and Hebb continued the
work of Aristotle and tried to build the
artificial "thinking machine".
• Based on the information about the
functions of the brain and the quest for
obtaining a mathematical model for our
learning habits, a new technology
Artificial Neural Networks was started.
Neural Network: Introduction
• Our brain can process information quickly
and accurately. You can recognize your
friend's voice in a noisy railway station. How
is the brain able to process the voice signal
added with the noise and retrieve the
original signal?
• Can we duplicate this amazing process
through a machine? Can we make a machine
to duplicate some learning habits of a
human? Can a machine be made to learn
from experience?
• We will get answer during the study of Neural
Network.
Neural Network: Definition
• An artificial neural network is an
information processing system that has
been developed as a generalization of the
mathematical model of human cognition
(sense of knowing).
• A neural network is a network of
interconnected neurons, inspired from the
studies of the biological nervous system.
In other words, neural network functions
in a way similar to the human brain.
• The function of a neural network is to
produce an output pattern when
presented with an input pattern.
Neural Network: Definition
• Neural network is the study of
networks consisting of nodes
connected by adaptable weights,
which store experimental knowledge
from task examples through a
process of learning.
• The nodes of the brain are
adaptable; they acquire knowledge
through changes in the node weights
by being exposed to samples.
Neural Network: Biological Neural
Net.
• Neural network architectures are
motivated by models of the human brain
and nerve cells. Our current knowledge of
human brain is limited to its anatomical
and physiological information.
• Neuron (from Greek, meaning nerve cell)
is the fundamental unit of the brain. The
neuron is a complex biochemical and
electrical signal processing unit that
receives and combines signals from many
other neurons through filamentary input
paths, the dendrites (Greek: tree links).
Neural Network: Biological Neural
Net.
• A biological neuron has three types
of components namely dendrites,
soma and axon. Dendrites are
bunched into highly complex
"dendritic trees", which have an
enormous total surface area. The
dendrites receive signals from other
neurons.
• Dendritic trees are connected with
the main body of the neuron called
the soma (Greek: body).
Neural Network: Biological Neural
Net.
• The soma has a pyramidal or
cylindrical shape. The soma sums
the incoming signals. When
sufficient input is received, the cell
fires.
• The output area of the neuron is a
long fiber called axon. The impulse
signal triggered by the cell is
transmitted over the axon to other
cells.
Neural Network: Biological Neural
Net.
• The connecting point between a neuron's
axon and another neuron’s dendrite is
called a synapse (Greek: contact). The
impulse signals are then transmitted
across a synaptic gap by means of a
chemical process.
• A single neuron may have 1000 to 10000
synapses and may be connected with
around 1000 neurons. There are 100
billion neurons in our brain, and each
neuron has 1000 dendrites.
Neural Network: Biological Neural
Net.
Neural Network: Artificial Neuron
• The artificial neuron (also called
processing element or node) mimes the
characteristics of the biological neuron. A
processing element possesses a local
memory and carries out localized
information processing operations.
• The artificial neuron has a set of ‘n’ inputs
xi, each representing the output of
another neuron.
• The subscript i in xi takes values between
i and n and indicates the source of the
vector input signal.
Neural Network: Artificial Neuron
• The inputs are collectively referred to as
X.
• Each input is weighed before it reaches
the main body of the processing element
by the connection strength or the weight
factor (or simply weight) analogous to the
synaptic strength.
• The amount of information about the
input that is required to solve a problem is
stored in the form of weights. Each signal
is multiplied with an associated weight w1,
w2, w3,...,wn before it is applied to the
Neural Network: Artificial Neuron
• In addition, the artificial neuron has
a bias term w0, a threshold value ‘θ ‘
that has to be reached or extended
for the neuron to produce a signal, a
nonlinear function 'F' that acts on
the produced signal 'net' and an
output 'y' after the nonlinearity
function.
Neural Network: Artificial Neuron
Neural Network: Artificial Neuron
• The following relation describes the transfer
function of the basic neuron model.
• y = F (net)
• Where
• net = w0 + x1w1 + x2w2 + x3w3 +...... + xnwn
• or n
net w0 xi wi
i 0
x w i i
• and the neuron firing condition is:
i 0
F (net ) [for linear activation function],
x0=1
• or
Neural Network: Classification
• Artificial neural networks can be
classified on the basis of
• 1. Pattern of connection between
neurons, (architecture of the
network)
• 2. Activation function applied to the
neurons
• 3. Method of determining weights on
the connection (training method)
Neural Network: ARCHITECTURE
• The neurons are assumed to be arranged
in layers, and the neurons in the same
layer behave in the same manner.
• All the neurons in a layer usually have the
same activation function. Within each
layer, the neurons are either fully
interconnected or not connected at all.
• The neurons in one layer can be
connected to neurons in another layer.
• The arrangement of neurons into layers
and the connection pattern within and
between layers is known as network
architecture.
Neural Network: ARCHITECTURE
• lnput layer:
• The neurons in this layer receive the
external input signals and perform no
computation, but simply transfer the input
signals to the neurons in another layer.
• Output layer:
• The neurons in this layer receive signals
from neurons either input layer or in the
hidden layer.
• Hidden layer:
• The layer of neurons that are connected
in between the input layer and the output
layer is known as hidden layer.
Neural Network: ARCHITECTURE
• Neural nets are often classified as single
layer networks or multilayer networks.
• The number of layers in a net can be
defined as the number of layers of
weighted interconnection links between
various layers.
• While determining the number of layers,
the input layer is not counted as a layer,
because it does not perform any
computation.
• The architecture of a single layer and a
multilayer neural network is shown in the
Neural Network: ARCHITECTURE
• Single Layer Network
• A single layer network consists of one layer of
connection weights. The net consists of a layer of
units called input layer, which receive signals
from the outside world and a layer of units called
output layer from which the response of the net
can be obtained.
• This type of network can be used for pattern
classification problems
Neural Network: ARCHITECTURE
• Multilayer Network:
• A multilayer network consists of one or
more layers of units (called hidden layers)
between the input and output layers.
Multilayer networks may be formed by
simply cascading a group of layers; the
output of one layer provides the input to
the subsequent layer.
• A multilayer net with nonlinear activation
function can solve any type of problem.
• However training a multilayer neural
network is very difficult.
Neural Network: ARCHITECTURE
• Multilayer Network:
Neural Network: ACTIVATION
FUNCTIONS
• The purpose of nonlinear activation
function is to ensure that the neuron's
response is bounded - that is, the actual
response of the neuron is conditioned or
damped, as a result of large or small
activating stimuli and thus controllable.
• Further, in order to achieve the
advantages of multilayer nets compared
with the limited capabilities of single layer
networks, nonlinear functions are
required.
• Different nonlinear functions are used,
depending upon the paradigm and the
algorithm used for training the network.
Neural Network: ACTIVATION
FUNCTIONS
• The various activation functions are:
• Identity function (Linear function):
• Identity function can be expressed:
• f(x) = x for all x.
• Binary step function: Binary step function is
defined as:
Neural Network: ACTIVATION
FUNCTIONS
Neural Network: ACTIVATION
FUNCTIONS
Neural Network: ACTIVATION
FUNCTIONS