Helmi ayari

23/08/2019

39
Deep Learning– Principles et techniques

NEURAL NETWORKS

Neural networks are a technique of machine learning, which is predicated

on this idea of learning from example.

We give the computer a model with which it can evaluate examples, and a
small set of instructions to modify the model when it makes a mistake.

We expect that, over time, a well-suited model would be able to solve the
problem extremely accurately.

40
Deep Learning– Principles et techniques

20
 23/08/2019

NEURAL NETWORKS

Let’s define our model to be a function h (x, θ) . The input x is an example

expressed in vector form.

For example, if x were a grayscale image, the vector’s components would

be pixel intensities at each position, as shown in the following figure:
0
0
 
0
 
0.5
0
0 0 1 0  
1
0.5
0 1 0.5 0  
1
1
0 0.5 1 0.5  
0.5
 
1
0.5 1 0.5 1 0.5
 
0
0
 
0.5
1
41  
Deep Learning– Principles et techniques

NEURAL NETWORKS

The input θ is a vector of the parameters that our model uses. Our machine
learning program tries to perfect the values of these parameters as it is
exposed to more and more examples.

0
0
 
0
 
0.5
0
0 0 1 0  
1
0.5
0 1 0.5 0  
1
1
0 0.5 1 0.5  
0.5
 
1
0.5 1 0.5 1 0.5
 
0
0
 
0.5
1
42  
Deep Learning– Principles et techniques

21
 23/08/2019

NEURAL NETWORKS

As we move on to much more complex problems, such as object

recognition and text analysis, our data becomes extremely high
dimensional, and the relationships we want to capture become highly
nonlinear.
.

To accommodate this complexity, recent research in machine learning has
attempted to build models that resemble the structures utilized by our
brains.

43
Deep Learning– Principles et techniques

NEURAL NETWORKS

Synaptic
terminal

Dendrites

Axon
Cell body

Inputs Strengths Sum Transform Outputs

A functional description of a biological neuron’s structure

44
Deep Learning– Principles et techniques

22
 23/08/2019

NEURAL NETWORKS

We can translate this functional understanding of the neurons in our brain into
an artificial model that we can represent on our computer.

x1 w1
x2

xn
w2
..
wn
. f y

Schematic for a neuron in an artificial neural net

45
Deep Learning– Principles et techniques

NEURAL NETWORKS

The artificial neuron takes in some number of inputs, x1, x2, ... , xn, each
of which is multiplied by a specific weight, w1, w2, ... ,wn. These weighted
inputs are, as before, summed together to produce the logit of the neuron:
n

∑w x
i =0
i i

In many cases, the logit also includes a bias, which is a constant (not
shown in the figure). The logit is then passed through a function f to
produce the output y = f (z) . This output can be transmitted to other
neurons.

46
Deep Learning– Principles et techniques

23
 23/08/2019

NEURAL NETWORKS

x = [x1 x2 … xn]
w = [w1 w2 … wn]
y = f (x.w + b)
b is the bias term

A single neuron is not nearly expressive enough to solve complicated

learning problems.

For example, it is impossible for a single neuron to differentiate handwritten

digits.

47
Deep Learning– Principles et techniques

NEURAL NETWORKS

The brain contains neurons that are a bit like organic switches. These can
change their output state depending on the strength of their electrical or
chemical input.

Neurons in the human brain are organized in layers.

The human cerebral cortex (the structure responsible for most of human
intelligence) is made up of six layers.

Information flows from one layer to another until sensory input is converted
into conceptual understanding.

For example, the bottommost layer of the visual cortex receives raw visual
data from the eyes. This information is processed by each layer and
passed on to the next until, in the sixth layer, we conclude whether we are
looking at an animal, a vehicle, etc.

48
Deep Learning– Principles et techniques

24
 23/08/2019

NEURAL NETWORKS

The neural network in a person's brain is an extremely interconnected

neural network, where the output of a given neuron can be the input of
thousands of other neurons.

Learning occurs by repeatedly activating some neural connections over
others, which reinforces these connections. This makes them more likely to
produce a desired result based on a specified intake.

This learning involves feedback: when the desired result is achieved, the
neural connections that result in this result are reinforced.

49
Deep Learning– Principles et techniques

NEURAL NETWORKS

A neural network comes about when we start hooking up neurons to each

other, the input data, and to the output nodes, which correspond to the
network’s answer to a learning problem.

The demonstrates a simple example of an artificial neural network, similar

to the architecture described in McCulloch and Pitt’s work in 1943

50
Deep Learning– Principles et techniques

25
 23/08/2019

NEURAL NETWORKS
o1 o2 o3

(2 ) (2 )
w3,1 w1,2 w(2
3,2
) (2 )
w1,3
(2 ) (2 )
w1,1 w2,1 w(2
2,2
)
w(2
2,3
)
w(2
3,3
)

A simple example of a feed-

forward neural network with
three layers (input, one hidden, (1) (1)
w3,1 w1,2 w(1 ) (1)
w1,3
and output) and three neurons 3,2
(1) (1)
per layer w1,1 w2,1 w(1
2,2
)
w(1
2,3
)
w(1
3,3
)

i1 i2 i3
51
Deep Learning– Principles et techniques

NEURAL NETWORKS

The bottom layer of the network pulls in the input data.

The top layer of neurons (output nodes) computes our final answer.

The middle layer(s) of neurons are called the hidden layers.

(k)

wi, j is the weight of the connection between the ith neuron in the kth layer
with the jth neuron in the k+1st layer.

These weights constitute our parameter vector, θ, and just as before, our
ability to solve problems with neural networks depends on finding the
optimal values to plug into θ.

52
Deep Learning– Principles et techniques

26
 23/08/2019

NEURAL NETWORKS

In this example, connections only traverse from a lower layer to a higher

layer.

There are no connections between neurons in the same layer, and there
are no connections that transmit data from a higher layer to a lower layer.

These neural networks are called feed-forward networks, they are the
simplest to analyze.

53
Deep Learning– Principles et techniques

NEURAL NETWORKS

Although in this example every layer has the same number of neurons, this
is neither necessary nor recommended. More often than not, hidden layers
have fewer neurons than the input layer to force the network to learn
compressed representations of the original input.

For example, while our eyes obtain raw pixel values from our surroundings,
our brain thinks in terms of edges and contours.

This is because the hidden layers of biological neurons in our brain force
us to come up with better representations for everything we perceive

54
Deep Learning– Principles et techniques

27
 23/08/2019

NEURAL NETWORKS

It is not required that every neuron has its output connected to the inputs of
all neurons in the next layer (experience)

The inputs and outputs are vectorized representations. For example, you
might imagine a neural network where the inputs are the individual pixel
RGB values in an image represented as a vector. The last layer might have
two neurons that correspond to the answer to our problem:

[1, 0] if the image contains a male face,

[0, 1] if the image contains a female face,

[1, 1] if it contains both, and

[0, 0] if it contains neither.

55
Deep Learning– Principles et techniques

NEURAL NETWORKS

Let’s consider the input to the ith layer of the network to be a vector x =
[x1 x2 ... xn].

Problem: find the vector y = [y1 y2 ... ym] produced by propagating the
input through the neurons.

« We can express this as a simple matrix multiply if we construct a weight

matrix W of size n × m and a bias vector of size m.

In this matrix, each column corresponds to a neuron, where the jth element
of the column corresponds to the weight of the connection pulling in the jth
element of the input.

(
y = f WTx + b )
56
Deep Learning– Principles et techniques

28
 23/08/2019

NEURAL NETWORKS

The biological neuron is simulated in an ANN by an activation function.

In classification tasks (for example, spam identification), this activation

function must have an activation characteristic.

In other words, once the input is greater than a certain value, the state of
the output must change state, for example from 0 to 1, from -1 to 1 or from
0 to >0. This simulates the activation of a biological neuron.

57
Deep Learning– Principles et techniques

NEURAL NETWORKS

The function is "activated", that is, it goes from 0 to 1 when the input x is
greater than a certain value

58
Deep Learning– Principles et techniques

29
 23/08/2019

NEURAL NETWORKS

x1

f
x2
y = hw,b(x)
x3

+1

The circle in the image above represents the node. The node is the “seat” of
the activation function, and takes the weighted inputs, sums them, then inputs
them to the activation function. The output of the activation function is shown
as h in the above diagram.

59
Deep Learning– Principles et techniques

NEURAL NETWORKS

x1

f
x2
y = hw,b(x)
x3

+1

What about this “weight” idea that has been mentioned? The weights are real
valued numbers, which are multiplied by the inputs and then summed up in
the node. So, in other words, the weighted input to the node above would be:

x1w1 + x2w2 + x3w3 + b

60
Deep Learning– Principles et techniques

30
 23/08/2019

NEURAL NETWORKS

x1

f
x2
y = hw,b(x)
x3

+1

The weights are the variables that are changed during the learning process,
and, along with the input, determine the output of the node. The b is the
weight of the +1 bias element – the inclusion of this bias enhances the
flexibility of the node.

61
Deep Learning– Principles et techniques

NEURAL NETWORKS

Let’s take an extremely simple node, with only one input and one output :

x1
f y = hw(x)

The input to the activation function of the node in this case is simply x1w1.
What does changing w1 do in this simple network?
62
Deep Learning– Principles et techniques

31
 23/08/2019

NEURAL NETWORKS

Here we can see that changing the weight changes the slope of the output of
the sigmoid activation function, which is obviously useful if we want to model
different strengths of relationships between the input and output variables.

63
Deep Learning– Principles et techniques

NEURAL NETWORKS

However, what if we only want the output to change when x is greater

than 1? This is where the bias comes in – let’s consider the same network
with a bias input:

x1
w1

+1
b
f y = hw,b(x)

64
Deep Learning– Principles et techniques

32
 23/08/2019

NEURAL NETWORKS

In this case, the w1 has been increased to simulate a more defined “turn on”
function. As you can see, by varying the bias “weight” b, you can change
when the node activates. Therefore, by adding a bias term, you can make the
node simulate a generic if function, i.e. if (x > z) alors 1 sinon 0.

65
Deep Learning– Principles et techniques

NEURAL NETWORKS

The network structure consists of an input layer, a hidden layer, and an output
layer. An example of such a structure can be seen below:

the (+1) bias is connected to each of the nodes in the subsequent layer
66
Deep Learning– Principles et techniques

33
 23/08/2019

NEURAL NETWORKS

i refers to the node number of the connection in layer l + 1 and j

wij(l ) : refers to the node number of the connection in layer l.
(1 )
w21 : the connection between node 1 in layer 1 and node 2 in layer 2.

bi(l ) : i is the node number in the layer l + 1

The weight on the connection between the bias in layer 1 and the
b2(1) : second node in layer 2

h (lj ) : j denotes the node number in layer l of the network

h2( 2 ) : the output of node 2 in layer 2

67
Deep Learning– Principles et techniques

NEURAL NETWORKS

h1( 2 ) = f ( w11(1) x1 + w12(1) x2 + w13(1) x3 + b1(1) )

h2( 2 ) = f ( w21
(1)
x1 + w22
(1)
x2 + w23
(1)
x3 + b2(1) )

h3( 2 ) = f ( w31
(1)
x1 + w32
(1)
x2 + w33
(1)
x3 + b3(1) )

hWb (x ) = h1( 3) = f ( w11(1) h1( 2) + w12( 2 ) h2( 2 ) + w12( 2 ) h3( 2 ) + b1( 2 ) )

68
Deep Learning– Principles et techniques

34
 23/08/2019

NEURAL NETWORKS

GRADIENT DESCENT AND OPTIMISATION

Setting of the values of the weights which link the layers in the network is
what constitutes the training of the system.

In supervised learning, the idea is to reduce the error between the input
and the desired output.

If we have a neural network with one output layer, and given some input x
we want the neural network to output a 2, yet the network actually
produces a 5, a simple expression of the error is |(2 − 5)| = 3

The idea of supervised learning is to provide many input-output pairs of
known data and vary the weights based on these samples so that the error
expression is minimized.

69
Deep Learning– Principles et techniques

NEURAL NETWORKS

GRADIENT DESCENT AND OPTIMISATION

We can specify these input-output pairs as {(x1 , y1 ),..., (xm , y m )}

m is the number of training samples that we have on hand to train the
weights of the network.

In training the network with these (x, y) pairs, the goal is to get the neural
network better and better at predicting the correct y given x.

This is performed by varying the weights so as to minimize the error.

70
Deep Learning– Principles et techniques

35
 23/08/2019

NEURAL NETWORKS

GRADIENT DESCENT AND OPTIMISATION

The way we figure out the gradient of a neural network is via the famous
backpropagation method

71
Deep Learning– Principles et techniques

NEURAL NETWORKS

THE BACKPROPAGATION

This method allows us to “share” the error to all the weights in the network –
or in other words, it allows us to determine how much of the error is caused
by any given weight.

72
Deep Learning– Principles et techniques

36