Neural networks

Neural networks

Plan

Here are the topics that we will cover:

Definition and basic concepts.

The perceptron.

Multilayer networks with backpropagation of the error.

Practical examples of neural network coding in Matlab:

XOR gate training using the backpropagation technique
propagation.
FunctionY trainingj=i)2using predefined functions
of matlab (newff).
Basic concepts

Modeling of the neuron

The real neuron
In a brain, there are 10^12
neurons with 10^3 to 10^4
connections by neuron.
Dendrite: receptor of
messages
Body: generates potential
action (the response)
Axon: transmits the signal to
following cells
Synapse: axon junction -
dendrite (more or less
passerby
Neuron: autonomous element
devoid of intelligence
 Modeling of the neuron
The modeling of the biological nervous system is based on the following correspondence

The weights
x1:Entry 1 The transfer function
x1 w1
Or activation
y : Output
f(x)
x2Entry 2
x2 w2

Nervous system Neural computing system

Neuron Processor
Dendrite Elements of entries
Body of the neuron Transfer function
Axon Output element
Synapse Weight
 DEFINITION
Definition
Artificial neural networks are highly interconnected networks of
elementary processors operating in parallel. Each elementary processor
calculates a unique output based on the information it receives. Any structure
Hierarchical network is obviously a network.

Couch entrance Hidden layer Output layer

x1Entry 1 w11
x1 f(x) w11
w12 w12 y1Exit 1
w1n f(x)
x2Entry 2 f(x)
x2 y2Exit 2
wij. . . wjk
. . f(x)
. . . wn1
. .
wn2 . .
xnEntry n wn2
xn f(x)
wnn
 Basic notions

Modeling of the neuron

The conventional graphical representation of a neuron

formula modeled by Mc Culloch and Pitts.

X1 w1
X2 w2 g
w3
X3
w4 Σf Y

X4 wn

Xn
 Basic concepts

Modeling of the neuron

The constituent elements of the artificial neuron

The entries "X" of the neuron

come from either others X1 w
elements 'processors' 1
or from the environment.
The weights 'W' determineX2 w2
the influence of each g
w3
entry.
The combination function
X3
w4 ΣF f Y
"g" combines the entries and
the weights. X 4 wn
The transfer function
calculate the output 'Y' of the Xn
neuron according to the
combination input.
FUNCTION FUNCTION
of COMBINATION of TRANSFER
 Basic concepts

Modeling of the neuron

The Combination Function calculates the influence of each

entry taking into account its weight. It sums up the
weighted entries: X1w1

g = ∑ WiXI X2 w2 g
Wi: w3
Weight of the
X3
w4
Σf Y

connection to X4 wn
the entry i.
•Xi: Xn
Signal of the input i. FUNCTION
of
COMBINATION
 Basic concepts
Modeling of the neuron

The Transfer Function determines the state of the neuron (in

exit
X1 w1
Output calculation:
Y = f(g) X2 w2 g
or also: w3
Y = f(∑ WiXi) X3
w4 Σf Y

The function of X4 wn
transfer 'f' can
to have several Xn
forms.
FUNCTION
of TRANSFER
 Basic Concepts

Modeling of the neuron

The function 'f' can be of the form:

Step function.

Piecewise linear function.

Differentiable function (sigmoid).

 Basic concepts

Modeling the neuron

Step transfer function:

Y = f(g)

+1

g
THRESHOLD

-1
 Basic notions

Modeling of the neuron

Piecewise linear transfer function:
Y = f(g)

+1

g
THRESHOLD

-1
 Basic Concepts

Modeling of the neuron

Differentiable transfer function (sigmoid):
Y = f(g)

+1

g
THRESHOLD

-1
 Basic concepts

Types of learning
The purpose of neural networks is to learn to respond correctly to
different entries.
Method: modification of weights through supervised learning, or not
supervised.
Supervised learning:an 'instructor' system
correct the incorrect answers.

supervisor desired output

error

network output obtained

 Basic concepts

Types of learning

Unsupervised learning: the neural system learns by itself

by forming entry classes with common responses.

network output obtained

 Basic concepts

Types of learning
Supervised learning (e.g., OCR)
Imposed association between
an input vector (shape
multidimensional) and a
output vector (the
desired response.

The error is calculated at

each attempt in order to
correct the weights.

The weights are modified

up to the minimum error,
or no error.
a
 Basic concepts
Types of learning
Supervised learning (e.g., OCR)

The expected answer is the 'a'. Only one and

unique output vector must be activated.
a

NETWORK b

c
Σf
d

z
 Basic concepts

Types of learning
Unsupervised learning

No indication of errors.
The network detects the characteristics
common from different entrances.
He thus tries to form 'classes' of
in an autonomous way.
He learns to give answers to
classes
Neural networks

The perceptron
 The perceptron

Outline of the part

Here are the chapters we will cover:

The learning phases

utilization
The learning algorithm
Learning functions
logics
The limits of the perceptron
 The perceptron

The learning-using phases

•Phase 1: LEARNING, the concept of the Perceptron
is based on a learning algorithm that
the goal is to correct the weighting of the
inputs in order to provide an activity (output) in
alignment with the elements to be learned.

Phase 2: UTILIZATION, once the examples are learned,

each neuron activates or not its output (in
correspondence with the acquired domain), depending on
elements applied at input.
 The perceptron

The learning-usage phases

Any use of the network must be preceded by a phase

of learning during which typical examples are presented to him.

USE
LEARNING of the NETWORK

PHASE 1 PHASE 2
 The perceptron

The learning algorithm

Classes and linear separability

The perceptron is a
X1 w +1
linear classifier 1
He is making a score of his. X g
2 w2 ↑
input space (X1,…,Xn) in
two or more classes C1, w3
... linear separable Cm.
X3
w4 Σf Y

We consider two classes: X 4 wn ↓

–C1 (Y = +1) Threshold
–C2 (Y = -1) Xn -1

-1
 The perceptron

The learning algorithm

Perceptron algorithm (principle algorithm)

1:INITIALISATION:W1 et W2:[-1;+1], SEUIL et PAS:[0;+1]

(X1;X2) -> "Correct output"
ET logique:(+1;+1)->+1;(-1;+1)->-1;(-1;-1)->-1;(+1;-1)->-1
2:REPEATER
For each example from the database: (X1;X2) -> "Y_correct
4: Calculation of the output 'Y_calculated' for each example:
4a: Combination function: (its output "g": potential)
g = (W1 x X1) + (W2 x X2) - THRESHOLD
4b: Activation function:
IF("g") >= 0 THEN "Y_calculated" = +1
IF "Y_calculated" = -1
If the output 'Y_calculated' is different from the output
Y_correcte"(ERROR = "Y_correcte" - "Y_calculated")
THEN Modification of weights:
5a: W1(t+1)= W1(t) + ( X1 x PAS x ERROR )
5b: W2(t+1)= W2(t) + ( X2 x PAS x ERROR )
6: Go back to 4 to recalculate the output
As long as an error persists, return to 4
 The perceptron

Learning logical functions

Simulation of the logical AND function through learning:

In the case of the AND function X1

the separation of the two
classes is done by a line – +
right
-W1xX1 + W2xX2 - THRESHOLD = 0 X2

Two classes (two answers): – –

C1 (Y = +1):
For the entries: (+1; +1)
-C2 (Y = -1): "AND" logic
For the entries:
{(-1;-1); (+1;-1); (-1;+1)}
 The perceptron

Learning logical functions

Simulation of the logical OR function through learning:

X1
In the case of the OR function,
a line always allows the + +
separation of the two classes.
X2
For two classes:
C1 (Y = +1): – +
•{(+1;+1); (+1;-1);(-1;+1)}

-C2 (Y = -1):
•(-1;-1) logical OR
 The perceptron

The limits of the perceptron

The limitations of the perceptron: the exclusive OR logical function.

In the case of the

X1
exclusiveOR-function,
the separation of the two + –
classes cannot take place
by a line but by X2
a curve. – +
–C1 (Y = +1):
•{(-1;+1); (+1;-1)}
C2 (Y = -1): OR-Exc logic
[{"coordinates":"(+1;+1)"},{"coordinates":"(-1;-1)"}]
The perceptron

The limits of the perceptron

The perceptron is a linear classifier.

It cannot handle nonlinear problems

of the OU EXCLUSIVE type, COMPARATIVE (and well
others...)

The structuring of a neural network

(made up of several layers), and
the joint use of an algorithm
appropriate learning will allow
address the limitations identified here.
Neural networks

Multilayer networks with feedback

error propagation
 Multilayer networks with backpropagation of error

Part plan

Here are the chapters we will cover:

Neural networks
formulas
Learning of a network
multilayer
The learning algorithm
 Multilayer networks with error backpropagation

Formal neural networks

Single layer neural network

F(x)
UNIQUE COUCH

F(x)

F(x)

The inputs of the networks are either outputs of other neurons or

direct inputs into the network (for example, pixels).
 Multilayer networks with backpropagation of error

Formal neural networks

Two-layer neural network

F(x)
F(x)
F(x)
F(x)
LAYING OUT

MATTRESS AT THE ENTRANCE*

The inputs of the networks are either outputs from other neurons or ...
direct entries into the network (for example, pixels).
 Multilayer networks with backpropagation of error

Formal neural networks

Network with a hidden layer (3 layers of neurons).

F(x)
F(x) ?
F(x)
F(x)
BED LAYING OUT
CACHEE
BED IN ENTRY
The inputs to the networks are either outputs from other neurons or direct inputs.
in the network (for example, pixels).
 Multilayer networks with backpropagation of error

Learning of a multilayer network

General operating principle.

ACTIVATION

ACTIVATION
F(x) E
F(x) F(x) R
R
F(x) E
F(x) F(x) U
R
F(x) LAYER Bedding Out
CACHEE
Couch in entry CORRECTION
CORRECTION of the ERROR
of the ERROR
 Multilayer networks with error backpropagation

Learning of a multilayer network

Notations :
x1, x2, x3, … , xk: the forms presented as input.
•xk: vector with I elements. x11 x12 x1I
X: matrixI Where to learn forms. x21 x22☹ x 2I
X
Each layer provides a 'response vector':
•hk: vector to elements, response of the hidden x K1x K2Rxlayer to akthform.
AI
•ok: vector to elements, output layer response to akthform.
•tk: vector with J elements, desired (theoretical) response of the output layer to
wetthform.
•T: matrixJ Where desired answers (theoretical).
matrixL Weights of connections between the inputs and the layer
hidden (W l,i: input connection to hidden neuron).
Z: matrixJ The weights of connections between the hidden layer and the
output layer (Z j,l: hidden neuron connection - output neuron).
 Multilayer networks with backpropagation of error

Learning of a multilayer network

The modifiable synapses (and their matrix W and Z).

W Z
x1 F(x) E
F(x) R
R
x2 F(x) E
F(x) U
R
xn
h
F(x)
LAY
x The neurons o
ENTREES OUTFIT LAYING
I J neurons
 Multi-layer networks with error backpropagation

Learning a multilayer network

Non-linear activation principle.

•Let a neuron (from a hidden layer or

of exit) and its potential named;
o f(an)
the output will be nin the form:

with its transfer function (non-linear,

derivable).
Common examples of transfer1 function:
f(x)
1  e x
The logistic function (or sigmoid):
 Multilayer networks with backpropagation of the error

Learning of a multilayer network

Graphical representation of transfer functions.

The logistic function (or sigmoid):

1
f(x)
1  e x

•f(x)and itsederivative
x
1: e x
f(x)1 f(x)
 1 e 
x 2
1 e 1 e
x x
 Multilayer networks with error backpropagation

The learning algorithm

Backpropagation algorithm of the error,

Step 1: Transmission of the signal between the input and the output via the layer
hidden

Let the vector xk be

the entrance (form k). hk f(Wxk)
The response of the
hidden cell is the ok f(Zhk)
vector :
The response of
cells of the layer
the output is the
 Multilayer networks with backpropagation of error

The learning algorithm

Backpropagation algorithm of the error,

Step 2: Calculation of the error at the output

We compare the given answer.

(vector ok) to the theoretical response
ek (tk ok)
(vector tk).
Error for the k-th form:
The error signal results from weighting
f(Zhk)  ek ok (1 ok)  (tk ok)
the error is by the activation state of
lefte,k

 product each(from
cellHadamar)
(a strong of activation
two matricesis
f(Zh):more harmful
k intensity than a weak
of output activation).
cell activation
 Multilayer networks with backpropagation of error

The learning algorithm

Backpropagation algorithm of the error,

Step 3: Adjusting the weights of the 'hidden/output' connections

The connection matrix Z is

corrected by successive iterations.
Z t1 Z t . exite,[Link] Z t tZ
:positive real number (the training step)
 Multilayer networks with error backpropagation

The learning algorithm

Error backpropagation algorithm,

Step 4: Calculation of the error at the output of the hidden layers

The problem is to estimate the error of

the (unknown) activity of the cells
hidden (for an expected response and
known only in output!)
There is no ideal answerexit, k
available.
We estimate it based on:
–du signal d’erreur :
 Multilayer networks with backpropagation of error

The learning algorithm

Backpropagation algorithm of the error,

Step 4 (continued): Calculation of the error at the output of the hidden layers

The error oute,k spreads in the direction

inverse (back-propagation) f(Wx through k ) the
connections Z.
It is weighted by activation
hidden,k
hidden
f(Wx k )  
cells. Z T
t sorte,k  hk  1 hk    Z tT exitede,k

The error signal:
 Multilayer networks with error backpropagation

The learning algorithm

Backpropagation algorithm of the error,

Step 5: Correction of the weights of the 'input/hidden' connections

Calculation of the weights of input neurons:

Wt1 W t . hidden,k .x T
k W W
tt
 Neural networks
Summary of the module

Retro-
NEURON propagation of
Perceptron : the error:
principle of =
Combination estimation of
separability the mistake in
linear +
Transfer hidden couches

Function Transfer Function:

Combination: in stages, or
piecewise linear, or
∑ WiXi derivable