Artificial
Neural Networks
Machine Learning
BS(CS/SE/IT)
Lecture 5
Syed Muhammad Ali Imran
University of Agriculture Faisalabad Sub Campus TTS
CONTENT
Introduction
What is a Neural Network?
Basic Component of ANN (Neuron, Layers, Connections,
Activation Funstions)
Mathematical Model of a Neuron
Network Architectures
Training
Commonly Used Activation Functions
Typical Problem Areas
Welcome to the World of
Artificial Neural network
Brain
A marvelous piece of
architecture and design.
In association with a nervous
system, it controls the life
patterns, communications,
interactions, growth and
development of hundreds of
million of life forms.
There are about 1010 to 1014 nerve cells (called neurons) in
an adult human brain.
Neurons are highly connected with each other. Each nerve
cell is connected to hundreds of thousands of other nerve
cells.
Passage of information between neurons is slow (in
comparison to transistors in an IC). It takes place in the
form of electrochemical signals between two neurons in
milliseconds.
Energy consumption per neuron is low (approximately 10-6
Watts).
Look more like some blobs
of ink… aren’t they!
Taking a more closer look reveals
that there is a large collection of
Different molecules, working
Together coherently, in an organized
manner.
Put together, they form the
best information processing
system in the known universe.
Nucleus
Axon
Synapse
Axons from
another
neurons Cell Body
Synapse Dendrites
Mind you, a neuron is a three dimensional entity!
Flow of Information
A few Neurons and their synaptic junctions
An artificial neural network is an information processing system that
has certain performance characteristics in common with biological
neural networks.
An ANN can be characterized by:
1. Architecture: The pattern of connections between different neurons.
2. Training or Learning Algorithms: The method of determining weights
on the connections.
3. Activation Function: The nature of function used by a neuron to
become activated.
There are two basic categories:
1. Feed-forward Neural Networks
◦ These are the nets in which the signals flow from the input units to the
output units, in a forward direction.
◦ They are further classified as:
1. Single Layer Nets
2. Multi-layer Nets
2. Recurrent Neural Networks
◦ These are the nets in which the signals can flow in both directions from the
input to the output or vice versa.
w11
w21
X1 Y1
w31
w12
X2 w22 Y2
l w32 l
l l
l l
w13
Xn w2m Ym
wnm
Input Output
Units Units
w11 v11
X1 Y1
wi1 vj1
Z1
l wn1 vp1 l
l l l
l l l
l
w1j v1k
Xi wij Zj vjk Yk
wnj l vpk
Biological l
l
l l
Neurons l
l
w1p v1m l
l
In Action wip
Zp vjm
Xn wnp vpm Ym
Input Hidden Output
Units Units Units
1 w11 1
X1 Y1
w1n
v1n v1m
w1m
Xn Ym
wnm
1 1
Supervised Training
Training is accomplished by presenting a sequence of training
vectors or patterns, each with an associated target output
vector.
The weights are then adjusted according to a learning algorithm.
During training, the network develops an associative memory. It
can then recall a stored pattern when it is given an input vector
that is sufficiently similar to a vector it has learned.
Unsupervised Training
A sequence of input vectors is provided, but no traget vectors
are specified in this case.
The net modifies its weights and biases, so that the most similar
input vectors are assigned to the same output unit.
1. Binary Step Function
0 if x 0
(a “threshold” or “Heaviside” function) f ( x)
1 if x 0
2. Bipolar Step Function
1 if x 0
f ( x)
3. Binary Sigmoid 1 if x 0
Function (Logistic
; f ' ( x) f ( x)1 f ( x)
1
f ( x)
Sigmoid) 1 exp( x)
1 exp( x)
3. Bipolar Sigmoid g ( x) ; g ' ( x) 1 g ( x)1 g ( x)
1 exp( x) 2
4. Hyperbolic - Tangent h( x) 1 exp( 2 x) ; h' ( x) 1 h( x)1 h( x)
1 exp( 2 x)
The number of application areas in which artificial neural networks are
used is growing daily.
Some of the representative problem areas, where neural networks have
been used are:
1. Pattern Completion:
ANNs can be trained on a set of visual patterns represented by pixel values.
◦ If subsequently, a part of an individual pattern (or a noisy pattern) is
presented to the network, we can allow the network’s activation to
propagate through the net till it converges to the original (memorized)
visual pattern.
◦ The network is acting like a “content- addressable” memory.
2. Classification: An early example of this type of network
was trained to differentiate between male and female faces.
◦ It is actually very difficult to create an algorithm to do so yet,
an ANN has shown to have near-human capacity to do so.
3. Optimization: It is notoriously difficult to find algorithms
for solving optimization problems (e.g. TSP).
◦ There are several types of neural networks which have been
shown to converge to ‘good-enough’ solutions i.e. solutions
which may not be globally optimal but can be shown to be
close to the global optimum for any given set of parameters.
4. Feature Detection: An early example of this is the phoneme
producing feature map of Kohonen:
The network is provided with a set of inputs and must learn to
pronounce the words; in doing so, it must identify a set of features
which are important in phoneme production.
5. Data Compression: There are many ANNs which have been
shown to be capable of representing input data in a compressed
format loosing as little of the information as possible.
6. Approximation: Given examples of an input to output mapping,
a neural network can be trained to approximate the mapping so that
a future input will give approximately the correct answer i.e. the
answer which the mapping should give.
7. Association: We may associate a particular input with a
particular output so that given the same (or similar) input again,
the net will give the same (or a similar) output again.
8. Prediction: This task may be stated as: given a set of previous
examples from a time series, such as a set of closing prices of
Karachi Stock Exchange, to predict the next (future) sample.
9. Control: For example to control the movement of a robot arm
(or truck, or any non-linear process) to learn what inputs (actions)
will have the correct outputs (results)