INTRODUCTION TO ANN AND MACHINE

LEARNING

Prof. ACS Rao

Department of Computer Science & Engineering, Indian
Institute of Technology (ISM),
Dhanbad 826004
email: acsrao@[Link]
Artificial Neural Networks (ANN): Basics of artificial neural networks, Characteristics and
Comparison with biological neural networks. Advantages of ANNs, Synaptic dynamics,
Applications of ANNs, (03)
Basic Models of ANNs: Mc-Culloch Pitt’s model, Single Layer Perceptron model of neural
networks, Hebb’s model, Learning Laws, (04)
Learning: Supervised, unsupervised, and Reinforcement Law of learning. Comparison and
differences among learning laws. LMS and Delta Learning, Gradient descent method,
Multilayer Perceptron Model (MLP), Back propagation algorithm for weight updates,
classification problem using MLP. (04)
Multilayer Perceptron Model (MLP), Back propagation algorithm for weight updates,
classification problem using MLP. (04)
Architecture for complex pattern recognition tasks, Competitive Learning: Hopfield model,
Self Organizing Feature Map, ART; (04)
How do our brains work?
▪ A processing element

Dendrites: Input
Cell body: Processor
Synaptic: Link
Axon: Output
How do our brains work?
▪ A processing element

A neuron is connected to other neurons through about 10,000

synapses
 How do our brains work?
▪ A processing element

A neuron receives input from other neurons. Inputs are combined.

 How do our brains work?
▪ A processing element

Once input exceeds a critical level, the neuron discharges a spike ‐

an electrical pulse that travels from the body, down the axon, to
the next neuron(s)
How do our brains work?
▪ A processing element

The axon endings almost touch the dendrites or cell body of the
next neuron.
 How do our brains work?
▪ A processing element

Transmission of an electrical signal from one neuron to the next is

effected by neurotransmitters.
How do our brains work?
▪ A processing element

Neurotransmitters are chemicals which are released from

the first neuron and which bind to the
Second.
 How do our brains work?
▪ A processing element

This link is called a synapse. The strength of the signal that

reaches the next neuron depends on factors such as the amount of
neurotransmitter available.
How do ANNs work?

An artificial neuron is an imitation of a human neuron

THE FOLLOWING TABLE SHOWS THE COMPARISON BETWEEN ANN AND BNN
BASED ON SOME CRITERIA MENTIONED
 NEURAL NETWORKS
• A neuron can have any number of inputs from one to
n, where n is the total number of inputs.
• The inputs may be represented therefore as x1, x2,
x3… xn.
• And the corresponding weights for the inputs as w1,
w2, w3… wn.
• Output a = x1w1+x2w2+x3w3... +xnwn

13
Why Artificial Neural Networks?
(An exemplar based system without any pre-programmed instructions for problem solving )
There are two basic reasons why we are interested in building
artificial neural networks (ANNs):

• Technical viewpoint: Some problems such as character

recognition or the prediction of future states of a system require
massively parallel and adaptive processing.

• Biological viewpoint: ANNs can be used to replicate and

simulate components of the human (or animal) brain, thereby
giving us insight into natural information processing.
 NEURAL NETWORKS
● ARTIFICIAL NEURAL NETWORK (ANN) IS A MACHINE LEARNING APPROACH
THAT MODELS HUMAN BRAIN AND CONSISTS OF A NUMBER OF ARTIFICIAL
NEURONS.

● NEURON IN ANNS TEND TO HAVE FEWER CONNECTIONS THAN BIOLOGICAL

NEURONS.
● EACH NEURON IN ANN RECEIVES A NUMBER OF INPUTS.

● AN ACTIVATION FUNCTION IS APPLIED TO THESE INPUTS WHICH RESULTS IN

ACTIVATION LEVEL OF NEURON (OUTPUT VALUE OF THE NEURON).
● KNOWLEDGE ABOUT THE LEARNING TASK IS GIVEN IN THE FORM OF
EXAMPLES CALLED TRAINING EXAMPLES.
 CONTD..

● AN ARTIFICIAL NEURAL NETWORK IS SPECIFIED BY:

− NEURON MODEL: THE INFORMATION PROCESSING UNIT OF THE NN,
− AN ARCHITECTURE: A SET OF NEURONS AND LINKS CONNECTING
NEURONS. EACH LINK HAS A WEIGHT,
− A LEARNING ALGORITHM: USED FOR TRAINING THE NN BY MODIFYING THE
WEIGHTS IN ORDER TO MODEL A PARTICULAR LEARNING TASK CORRECTLY
ON THE TRAINING EXAMPLES.
● THE AIM IS TO OBTAIN A NN THAT IS TRAINED AND GENERALIZES
WELL.
● IT SHOULD BEHAVES CORRECTLY ON NEW INSTANCES OF THE
LEARNING TASK.
 How do ANNs work?
• Now, let us have a look at the model of an artificial neuron.
 How do ANNs work?

..........
Input xm
..
x2 x1

Processing ∑
∑= X1+X2 + ….+Xm =y

Output y
 NEURON

● THE NEURON IS THE BASIC INFORMATION PROCESSING

UNIT OF A NN. IT CONSISTS OF:
1 A SET OF LINKS, DESCRIBING THE NEURON INPUTS, WITH
WEIGHTS W1, W2, …, WM
2 AN ADDER FUNCTION (LINEAR COMBINER) FOR COMPUTING THE
WEIGHTED SUM OF THE INPUTS: m
(REAL NUMBERS) u =  wjxj

j =1

3 ACTIVATION FUNCTION FOR LIMITING THE AMPLITUDE OF THE

y =  (u + b)
NEURON OUTPUT. HERE ‘B’ DENOTES BIAS.
 How do ANNs work?
Not all inputs are equal
xm .......... x2 x1
..
Input
wm ....
weights .
w2 w1

Processing ∑ ∑= X1w1+X2w2 + ….+Xmwm

=y

Output y
 How do ANNs work?
The signal is not passed down to the
next neuron verbatim
xm .......... x2 x1
..
Input
wm .... w2
weights .
w1

Processing ∑
Transfer Function
(Activation Function) f(vk)

Output y
The output is a function of the input, that is
affected by the weights, and the transfer
functions
Three types of layers: Input, Hidden, and
Output
 THE NEURON DIAGRAM
Bias
b
x1 w1
Activation
Induced function
Field


Output
x2 w2
v  (−) y
Input
values
  Summing
function

xm wm
weights
 NEURON MODELS
● THE CHOICE OF ACTIVATION FUNCTION  DETERMINES THE
NEURON MODEL.
EXAMPLES: a if v  c
 (v ) = 
● STEP FUNCTION: b if v  c

a if v  c

 ( v ) = b if v  d
● RAMP FUNCTION: a + (( v − c )( b − a ) /( d − c )) otherwise


1
● SIGMOID FUNCTION WITH Z,X,Y PARAMETERS  (v ) = z + 1 + exp( − xv + y )

1  1  v −  2 
 (v ) = exp  −   

● GAUSSIAN FUNCTION: 2   2   
Step Function

c
Ramp Function

c d
 Sigmoid function
The logistic curve
• The Gaussian function is the probability function of the
normal distribution. Sometimes also called the frequency
curve.
 BENEFITS AND LIMITATIONS OF NEURAL NETWORKS
BENEFITS OF ANNS
• USEFULNESS FOR PATTERN RECOGNITION, CLASSIFICATION, GENERALIZATION,
ABSTRACTION AND INTERPRETATION OF IMCOMPLETE AND NOISY INPUTS. (E.G.
HANDWRITING RECOGNITION, IMAGE RECOGNITION, VOICE AND SPEECH
RECOGNITION, WEATHER FORECASING).
• PROVIDING SOME HUMAN CHARACTERISTICS TO PROBLEM SOLVING THAT ARE
DIFFICULT TO SIMULATE USING THE LOGICAL, ANALYTICAL TECHNIQUES OF
EXPERT SYSTEMS AND STANDARD SOFTWARE TECHNOLOGIES. (E.G. FINANCIAL
APPLICATIONS).

• ABILITY TO SOLVE NEW KINDS OF PROBLEMS. ANNS ARE PARTICULARLY

EFFECTIVE AT SOLVING PROBLEMS WHOSE SOLUTIONS ARE DIFFICULT, IF NOT
IMPOSSIBLE, TO DEFINE. THIS OPENED UP A NEW RANGE OF DECISION
SUPPORT APPLICATIONS FORMERLY EITHER DIFFICULT OR IMPOSSIBLE TO
COMPUTERIZE.
30
• ROBUSTNESS. ANNS TEND TO BE MORE ROBUST THAN THEIR CONVENTIONAL
COUNTERPARTS. THEY HAVE THE ABILITY TO COPE WITH IMCOMPLETE OR
FUZZY DATA. ANNS CAN BE VERY TOLERANT OF FAULTS IF PROPERLY
IMPLEMENTED.
• FAST PROCESSING SPEED. BECAUSE THEY CONSIST OF A LARGE NUMBER OF
MASSIVELY INTERCONNECTED PROCESSING UNITS, ALL OPERATING IN
PARALLEL ON THE SAME PROBLEM, ANNS CAN POTENTIALLY OPERATE AT
CONSIDERABLE SPEED (WHEN IMPLEMENTED ON PARALLEL PROCESSORS).
• FLEXIBILITY AND EASE OF MAINTENAINCE. ANNS ARE VERY FLEXIBLE IN
ADAPTING THEIR BEHAVIOR TO NEW AND CHANGING ENVIRONMENTS. THEY
ARE ALSO EASIER TO MAINTAIN, WITH SOME HAVING THE ABILITY TO LEARN
FROM EXPERIENCE TO IMPROVE THEIR OWN PERFORMANCE.

31
Limitations of ANNs
ANNs do not produce an explicit model even though
new cases can be fed into it and new results obtained.
ANNs lack explanation capabilities. Justifications for
results is difficults to obtain because the connection
weights usually do not have obvious interpretaions.
 7. SOME ANN APPLICATIONS

ANN APPLICATION AREAS:

• TAX FORM PROCESSING TO IDENTIFY TAX FRAUD
• ENHANCING AUDITING BY FINDING IRREGULARITES
• BANKRUPTCY PREDICTION
• CUSTOMER CREDIT SCORING
• LOAN APPROVALS
• CREDIT CARD APPROVAL AND FRAUD DETECTION
• FINANCIAL PREDICTION
• ENERGY FORECASTING
• COMPUTER ACCESS SECURITY (INTRUSION DETECTION AND CLASSIFICATION
OF ATTACKS)
• FRAUD DETECTION IN MOBILE TELECOMMUNICATION NETWORKS

33
 NEURAL NETWORK ARCHITECTURE

There are three fundamental classes of ANN architectures:

• Single layer feed forward architecture
• Multilayer feed forward architecture
• Recurrent networks architecture

Before going to discuss all these architecture first discuss the mathematical
details of a neuron at a single level , To do this , Let us first consider the
AND problem and its possible solution with neural network.

34
 THE AND PROBLEM and NEURAL NETWORK

The simple Boolean AND operation with two input variables x1 and x2 is
shown in the truth table.
Here ,we have four patterns:00,01,10,11
For the first three pattern the output is 0 and the last pattern the output is 1.

INPUT OUTPUT
0 0 0
0 1 0
1 0 0
1 1 1 35
The AND problem and Its neural network

Alternatively ,the AND problem can be thought as a perception problem

where we have to receive four different pattens as input and perceive the
result as 0 or 1.

36
The AND problem and Its neural network
A possible neuron specification to solve the And problem is given in the
following table . In this solution, when the input is 11, the weight sum exceeds
the threshold(=0.9) leading to the output 1 else it gives the output 0

INPUT OUTPUT
0 0 0 Here
y=∑wixi – θ
0 1 0 0.9 W1 = 0.5
W2=0.5
1 0 0 θ = 0.9
1 1 1
37
The AND problem is linearly separable
INPUT OUTPUT
0 0 0
0 1 0
1 0 0
1 1 1

38
The XOR problem is linearly non-separable
INPUT OUTPUT
0 0 0
0 1 1
1 0 1
1 1 0

39
BASICS OF ANN
Models of Neuron
In this section we will consider three classical models for an artificial neuron or
processing unit.

McCulloch-Pitts Model
In MP model (Figure1.2) the activation(x) is given by a weighted sum of its M input
values(a1) and bias term (theta).
The output signal is typically a nonlinear function of the activation value x. The
following equations describe the operation of an MP model

Three commonly used nonlinear function (binary ,ramp and sigmoid) are shown in the
below figure, although only the binary function used in the original MP model.
Shows some examples of logic circuits realized using the MP model. In this model a binary
output function is used with the following logic
F(x) =1, x>0
=0, x<=0
PERCEPTRON LEARNING RULE EXPLAINED WITH
SOLVED EXAMPLE
Adaline
Adaptive Linear Element is a computing model proposed by and is shown in Figure below. The main
distinction between the Rosenblatt's perceptron model and the Adaline model is that, in the Adaline the analog
activation value is compared with the target output In other words, the output is a linear of the activation
value The equations that describe the operation of an Adaline are as follows:

Activation:

Output signal: s = F(x) =x

Error: b – s =b - x
Weight change: =
Hebbian Learning Algorithm (Explained With
Solved Example)
Hebbian Learning Rule

• As we know that updation of weight matrix or updation of weights in ANN is

known as learning.

• Hebb's learning rule was introduced by Donald Hebb in 1949. It is a feed

forward and unsupervised learning rule. Hebb's learning rule works on
bipolar data. In Hebbian learning, weights are changed according to the
principle Neurons which fire together, wire together.
• The Hebb's Learning Rule is a learning rule that specifies how much the
weight of the connection between two units should be increased or
decreased in proportion to the product of their activation. Hebb states that
the connections between two neurons might be N strengthened if the
neurons fire simultaneously.

• Hebb's principle describes how to alter the weights between neurons. The
weight between two neurons increases if the two neurons activate
simultaneously, and reduces if they activate separately. Nodes that tend to
be either both positive or both negative at the same time have strong
positive weights, while those that tend to be opposite have strong negative
weights.

• It is used in pattern association ,pattern recognition or pattern classification

problem.
Delta Learning Rule
Delta Learning Rule
• As we know that updation of weight matrix or updation of weights and biases in artificial
neural network is known as learning. This is also known as training. The purpose of the
learning rule is to train the network to perform some task or to reduce the error. There are
many types of neural network learning rules. They fall into three broad categories:
supervised learning, unsupervised learning and reinforcement (or graded) learning.

• Delta learning rule is a gradient descent learning rule for updating the weights of the
inputs. Delta rule uses an error function to perform gradient descent learning. It is introduced
by Widrow and Hoff. Delta Learning Rule is a supervised learning rule. This rule is
applicable only for continuous activation function. Partial derivative of error with respect to
weights is used for training the neural network. The aim of applying delta rule is to minimize
the mean-squared error between the activation (net input) and the target value.

• Delta rule is also called continuous perceptron training rule or Least mean squares (LMS)
algorithm or the Widrow-Hoff rule.
 Topology
Artificial neural networks are useful only
when the processing units are organised in
a suitable manner to accomplish a given
pattern recognition task. This section
presents a few basic structures which will
assist in evolving new architectures. The
arrangement of the processing units,
connections , and pattern is referred to as
topology.
Summary of Basic learning Laws
 LEARNING OF NEURAL NETWORKS: TOPICS
✓ Concept of learning
• Learning in
✓ Single layer feed forward neural network
• Multilayer feed forward neural network
• Recurrent neural network
• Types of learning in neural networks
 THE CONCEPT OF LEARNING

• The learning is an important feature of human computational

ability.
• Learning may be viewed as the change in behaviour acquired
due to practice or experience, and it lasts for relatively long
time.
• As it occurs, the effective coupling between the neuron is
modified. 9
2
 THE CONCEPT OF LEARNING

• In case of artificial neural networks, it is a process of modifying neural

network by updating its weights, biases and other parameters, if any.
• During the learning, the parameters of the networks are optimized and as a result
process of curve fitting.
• It is then said that the network has passed through a learning Phase.

9
3
TYPES OF LEARNING
• There are several learning techniques.
• A taxonomy of well known learning techniques are shown in the following.

9
4

• In the following, we discuss in brief about these learning techniques.

 SUPERVISED LEARNING
• In this learning, every input pattern that is used to train the network is
associated with an output pattern.
• This is called training set of data. Thus, in this form of learning, the
input-output relationship of the training scenarios are available.
• Here, the output of a network is compared with the corresponding
target value and the error is determined.
• It is then feed back to the network for updating the same. This results in
an improvement.
• This type of training is called learning with the help of teacher.
 UNSUPERVISED LEARNING

• If the target output is not available, then the error in prediction can not
be determined and in such a situation, the system learns of its own by
discovering and adapting to structural features in the input patterns.

• This type of training is called learning without a teacher.

9
6
 REINFORCED LEARNING

• In this techniques, although a teacher is available, it does not tell the

expected answer, but only tells if the computed output is correct or
incorrect. A reward is given for a correct answer computed and a penalty for
a wrong answer. This information helps the network in its learning process.

• Note: Supervised and unsupervised learnings are the most popular forms of
learning. Unsupervised learning is very common in biological systems.

• It is also important for artificial neural networks: training

9
7
data are not always
available for the intended application of the neural network.
 GRADIENT DESCENT LEARNING

• The learning technique is based on the minimization of error E defined in terms of

the weights and the activation function of the network .
• Also, it is required that the activation function employed by the network is
differentiable, As the weight update is dependent on the gradient of the error E.

9
8
GRADIENT DESCENT LEARNING

9
9
 STOCHASTIC LEARNING

• Stochastic learning
• In this method, weights are adjusted in a probabilistic fashion.

• Simulated annealing is an example of such learning (proposed by boltzmann and

cauch)

1
0
0
 COMPETITIVE LEARNING

• Competitive learning:
• In this learning method, those neurons which responds strongly to input
stimuli have their weights updated.
• When an input pattern is presented, all neurons in the
layer compete and the winning neuron undergoes weight
adjustment.
• This is why it is called a winner-takes-all strategy.
1
0
• In this course, we discuss a generalized approach of supervised learning to train
1

different type of neural network architectures.

LEARNING ANNS

In this course, we discuss a generalized approach of supervised learning

to train different type of neural network architectures.
 SINGLE LAYER FEED FORWARD NN (SLFFNN) TRAINING

• We know that, several neurons are arranged in one layer with

inputs and weights connect to every neuron.
• Learning in such a network occurs by adjusting the weights
associated with the inputs so that the network can classify the
input patterns.

• A single neuron in such a neural network is called perceptron.

1
0
3
SINGLE LAYER FEED FORWARD NN TRAINING

1
0
4
SINGLE LAYER FEED FORWARD NN TRAINING

Debasis Samanta
CSE
IIT Kharagpur 19
SINGLE LAYER FEED FORWARD NN TRAINING

Debasis Samanta
CSE
IIT Kharagpur 20
SINGLE LAYER FEED FORWARD NN TRAINING

21
 LEARNING OF NEURAL NETWORKS: TOPICS

✓ CONCEPT OF LEARNING

✓ LEARNING IN

✓ SINGLE LAYER FEED FORWARD NEURAL NETWORK

• MULTILAYER FEED FORWARD NEURAL NETWORK

• RECURRENT NEURAL NETWORK
TRAINING MLFFNNS
TRAINING MULTILAYER FEED FORWARD NEURAL NETWORK
SPECIFYING A MLFFNN
SPECIFYING A MLFFNN
SPECIFYING A MLFFNN

7
SPECIFYING A MLFFNN
LEARNING A MLFFNN
INPUT LAYER COMPUTATION
INPUT LAYER COMPUTATION
HIDDEN LAYER COMPUTATION
HIDDEN LAYER COMPUTATION

13
OUTPUT LAYER COMPUTATION
OUTPUT LAYER COMPUTATION
OUTPUT LAYER COMPUTATION

16
BACK PROPAGATION ALGORITHM
 METHOD OF STEEPEST DESCENT

• Supervised learning is, in fact, error-based learning.

• In other words, with reference to an external (teacher) signal(i.e. Target output)
it calculates error by comparing the target output and computed output.
• Based on the error signal, the neural network should modify its configuration,
which includes synaptic connections, that is , the weight matrices.
 METHOD OF STEEPEST DESCENT

• It should try to reach to a state, which yields minimum error.

• In other words, its searches for a suitable values of parameters
minimizing error, given a training set.
• Note that, this problem turns out to be an optimization
problem.
METHOD OF STEEPEST DESCENT

5
METHOD OF STEEPEST DESCENT
METHOD OF STEEPEST DESCENT
METHOD OF STEEPEST DESCENT
METHOD OF STEEPEST DESCENT
CALCULATION OF ERROR IN A NEURAL NETWORK
CALCULATION OF ERROR IN A NEURAL NETWORK
SUPERVISED LEARNING : BACK-PROPAGATION
ALGORITHM
SUPERVISED LEARNING : BACK-PROPAGATION
ALGORITHM
SUPERVISED LEARNING : BACK-PROPAGATION
ALGORITHM
SUPERVISED LEARNING : BACK-PROPAGATION
ALGORITHM
SUPERVISED LEARNING : BACK-PROPAGATION ALGORITHM
9
‘BACK-PROP’ ALGORITHM SUMMARY
(WITH MATHS!) (NOT EXAMINABLE)

153
‘BACK-PROP’ ALGORITHM SUMMARY (WITH NO MATHS!)

154
BACK PROPAGATION ALGORITHM EXPLAINED WITH
SOLVED EXAMPLE
The BAM and the Hopfield Memory
Introduction
• The concept of an associative memory is a fairly intuitive one: Associative
memory appears to be one of the primary functions of the brain. We easily
associate the face of a friend with that friend's name, or a name with a
telephone number.

• Many devices exhibit associative-memory characteristics. For example, the

memory bank in a computer is a type of associative memory: it associates
addresses with data. An object-oriented program (OOP) with inheritance can
exhibit another type of associative memory. Given a datum, the OOP asso-
ciates other data with it, through the OOP's inheritance network.
ASSOCIATIVE-MEMORY DEFINITIONS

In this section, we review some basic definitions and concepts related to as-
sociative memories. We shall begin with a discussion of Hamming distance, not
because the concept is likely to be new to you, but because we want to relate it to
the more familiar Euclidean distance, in order to make the notion of Hamming
distance more plausible. Then we shall discuss a simple associative memory called
the linear associator
Hamming Distance

• Hamming space can be defined by the expression

Hn = {x =( x1,x2, … , xn)t ∈ Rn : xi6 (± 1)

• Let x =(x1,x2,..,Xn) and y = (Y1,Y2,..,Yn) be two vectors in n- dimensional

Euclidean space, subject to the restriction that so that x and y are also
vectors in n-dimensional Hamming space. The Euclidean distance between the
two vector endpoints is
 Since xi,yi belongs to (± 1) {0,4} :
2
(Xi – Yi) = 0 Xi =Yi
=4 Xi !=Yi
Thus, the Euclidean distance can be written as

d= {4(#mismatched components of x and y)}0.5

We define the Hamming distance as

h = # mismatched components of x and y (4.2) or the
number of bits that are different between x and
 The Hamming distance is related to the Euclidean distance by the equation
d =2 * h0.5
h = (d*d) / 4

The Linear Associator

THE BAM :The BAM consists of two layers of processing elements that
are fully interconnected between the layers.
BAM Architecture
We can borrow the procedure from the linear-associator model to
construct the weight matrix. Given L vector pairs that constitute the set
of exemplars that we would like to store, we can construct the matrix:

w = y1x1t + y2x2t + ….. + yLxLt

We can make BAM into
autoassociative memory
By constructing the weight matrix as

w = x1x1t + x2x2t + ….. + xLxLt

BAM Mathematics
BAM Energy Function

or
SELF ORGANIZING FEATURE MAPS
 OUTLINE
1) Self-Organization
2) Unsupervised Learning
3) Feature Maps
4) Self Organizing Feature Maps (SOFMs)
1) Network Architecture
2) Network in Operation
5) Network Initialization and Training Techniques
6) Mathematical Foundations
7) Pros & Cons of SOFMs
8) Example Implementations
 1) SELF-ORGANIZATION
system - a group of interacting parts functioning as a whole and
distinguishable from its surroundings (environment) by recognizable
boundaries.

system property - the resultant system no longer solely exhibits

the collective properties of the parts themselves (“the whole is more
than the sum of its parts”)

organization - the arrangement of selected parts so as to promote a

specific function
external organization - system organization imposed by external
factors
self organization - evolution of a system into an organized
form in the absence of external constraints.
 WHAT IS AN ATTRACTOR?
A preferred position for the system, such that if the system is
started from another state it will evolve until it arrives at the
attractor, and will then stay there in the absence of other
factors.
Examples:
- point (e.g. swinging pendulum)
- path (e.g. a planetary orbit)
- series of states (e.g. the metabolism of a cell)

Studying self-organization is equivalent to investigating the

attractors of the system; a complex system can have many
attractors and these can alter with changes to the system
interconnections;
 2) UNSUPERVISED LEARNING
information processing system - a system organized in such a way
to have a property of processing the input from its surroundings and
producing the output
Examples: computer, neural network

computer - external constraints imposed upon its organization; lack

of system properties such as intelligence, learning etc.

(unsupervised) neural network - no external constraints; mimics

biological neural systems; consists of a collection of neurodes and
their interconnections;
Expectations: given sufficient complexity, the same
properties that occur in the brain will also occur in the network:
ability to learn (possible today), self-awareness, imagination (distant
future, if ever)
3) FEATURE MAPS
What is the result of brain’s self-organization?

- formation of feature maps in the brain that have a linear or

planar topology (that is, they extend in one or two dimensions)

Examples:
- tonotopic map - sound frequencies are spatially mapped into
regions of the cortex in an orderly progression from low to high
frequencies.
- retinotopic map - visual field is mapped in the visual cortex
(occipital lobe) with higher resolution for the centre of the visual
field
- somatosensory map - mapping of touch
Why are feature maps important?

Sensory experience is multidimensional

E.g. sound is characterised by pitch, intensity, timbre, noise
etc.,

The brain maps the external multidimensional representation of

the world (including its spatial relations) into a similar 1 or 2 -
dimensional internal representation.

That is, the brain processes the external signals in a topology-

preserving way

So, if we are to have a hope of mimicking the way the brain

learns, our system should be able to do the same thing.
4.) Self-Organizing Feature Maps (SOFMs)

a.k.a as Kohonen networks, competitive filter associative memories

- represents the embodiment of the ideas we have discussed so far

- named after Dr. Eng. Teuvo Kohonen
4.1) NETWORK ARCHITECTURE
Input Layer

- accepts multidimensional input pattern from the environment

- an input pattern is represented by a vector.

e.g. a sound may consist of pitch, timbre, background

noise, intensity, etc.
- each neurode in the input layer represents one dimension of the
input pattern
- an input neurode distributes its assigned element of the input
vector to the competitive layer.
 COMPETITIVE LAYER
- each neurode in the competitive layer receives a sum of weighted
inputs from the input layer
- every neurode in the competitive layer is associated with a
collection of other neurodes which make up its
'neighbourhood’
- competitive Layer can be organized in 1 dimension, 2 dimensions, or
... n dimensions
- typical implementations are 1 or 2 dimensions.
- upon receipt of a given input, some of the neurodes will be
sufficiently excited to fire.
- this event can have either an inhibitory, or an excitatory effect on
its neighborhood
- the model has been copied from biological systems, and is known
as 'on-center, off-surround' architecture, also known as lateral
feedback / inhibition.
Output layer

- organization of the output layer is application-dependent

- strictly speaking, not necessary for proper functioning of a

Kohonen network

- the "output" of the network is the way we choose to view the

interconnections between nodes in the competitive layer

- if
nodes are arranged along a single dimension, output can be
seen as a continuum:
4.2) Network in Operation

Competition in a SOFM (The Emergence of Order from Chaos)

- each neurode in the competitive layer receives a (complex)

mixture of excitatory and inhibitory signals from the neurodes
in its neighbourhood, and from the input layer.
- lateral inhibition is used to stabilize the network and prevent
"meltdown" due to over - excitation in the competitive layer,
or starvation due to poor selection of the threshold value.
- for a given input, the neurode which responds most strongly will be
permitted to adjust the weights of the neurodes which make up its
neighbourhood, including itself.
- this is a "winner takes all" strategy to the learning process.

- neurodes in this layer are competing to 'learn' the input

vectors.
5) Network Initialization and Training Techniques

Initialization

- originally was thought acceptable to randomize input weights

and neighbourhood associations
- this may lead to the rise of dead vectors: neurodes which will
never fire…
- if we know something about the distribution of inputs, it may
be useful to initialize the neurodes to mirror this
distribution
- ideally, we want each neurode to be able to 'win', or be in the
neighbourhood of a winning neurode, for at least one
input from the training set
 SELECTION OF NETWORK SIZE
Q: "How many neurodes do I need in the competitive Layer?”

A: "How much error can you live with?”

- generally speaking, more neurodes equals less error.

- higher numbers of neurodes will support finer-grained

classification of the inputs.

- the size of the competitive layer is governed by the number of

parameters represented in the input vector

- caveat: the more neurodes present, the longer training will

take...
 THE TRAINING SET
- MOST IMPORTANT ELEMENT OF THE NETWORK PLANNING PROCESS

- WE WANT THE TRAINING SET TO MIRROR THE PROBABILITY DISTRIBUTION OF INPUTS IN THE
ENVIRONMENT

- WITHIN THE TRAINING SET, WE SHOULD RANDOMIZE THE PRESENTATION OF

• INPUT CLASSES

- WE MAY NEED TO ADJUST THE ELASTICITY OF THE LEARNING RATE TO

• ACCOMMODATE LARGE SETS

• NOTE: NO GUARANTEE OF CONVERGENCE FOR NEURAL NETWORKS WITH MORE

THAN ONE DIMENSION IN THE COMPETITIVE LAYER
7) Pros & Cons of SOFMs

Pros:

- excellent for classification problems

- can greatly reduce computational complexity

- high sensitivity to frequent inputs

- new ways of associating related data

- no need of supervised learning rules

Cons:
- system is a black box

- error rate may be unacceptable

- no guarantee of network convergence for higher dimension

networks
- many problems can't be effectively represented by a SOFM

- a large training set may be required

- for large classification problems, training can be lengthy

- can be difficult to come up with the input vector.

- associations developed by SOFM not always easily understood

by people.
20