Unit II

Multi-Layer Perceptron
 Going Forwards
• Just as it did for the Perceptron, training the
MLP consists of two parts: The outputs are for
the given inputs
• The current weights, and then updating the
weights according to the error.
• They are two types of networks:
• These are generally known as going forwards
and backwards through the network.
 Going Forwards cont…
• Biases
• We need to include a bias input to each
neuron.
• An extra input that is permanently set to -1,
and adjusting the weights to each neuron as
part of the training.
• Thus, each neuron in the network (whether it
is a hidden layer or the output) has 1 extra
input, with fixed value.
 Going Backwards: Back Propagation
Error
• It is in the backwards part of the algorithm
that things get tricky.
• Computing the errors at the output is no more
difficult than it was for the Perceptron, but
working out what to do with those errors is
more difficult.
• The method that we are going to look at is
called back-propagation of error.
 Going Backwards: Back Propagation
Error cont…
• an error function for each neuron k:
Ek = yk − tk
The error function that we used for the
Perceptron was
 Going Backwards: Back Propagation
Error cont…
• We can do this in a few different ways, but the
one that will turn out to be best is the sum-of-
squares error function, which calculates the
difference between y and t for each node,
squares them, and adds them all together
 Going Backwards: Back Propagation
Error cont…
• Imagine a ball rolling around on a surface that
looks
Going Backwards: Back Propagation
Error cont…
Going Backwards: Back Propagation
Error cont…
Going Backwards: Back Propagation
Error cont…
 Multi-layer Perceptron in Practice
• The choices that can be made about the
network in order to use it for solving real
problems.
• We will then apply these ideas to using the
MLP to find solutions to four different types of
problem:
• regression,
• classification,
• time-series prediction,
• and data compression.
 Multi-layer Perceptron in Practice
cont…
Amount of Training Data
• For the MLP with one hidden layer there are (L
+ 1) × M + (M + 1) × N weights, where L, M, N
are the number of nodes in the input, hidden,
and output layers, respectively.
• The extra +1s come from the bias nodes,
which also have adjustable weights.
 Multi-layer Perceptron in Practice
cont…
• A rule of thumb that has been around for
almost as long as the MLP itself is that you
should use a number of training examples that
is at least 10 times the number of weights.
 Multi-layer Perceptron in Practice
cont…
Number of Hidden Layers
• There is a sketchy demonstration that two hidden
layers are sufficient using pictures in Figure 4.9.
• The basic idea is that by combining sigmoid
functions we can generate ridge-like functions,
and by combining ridge-like functions we can
generate functions with a unique maximum.
• The way that the MLP does this is shown in Figure
4.10.
Multi-layer Perceptron in Practice
cont…
Multi-layer Perceptron in Practice
cont…
 Multi-layer Perceptron in Practice
cont…
When to Stop Learning
• At some stage the error on the validation set
will start increasing again, because the
network has stopped learning about the
function that generated the data, and started
to learn about the noise that is in the data
itself (shown in Figure 4.11).
• At this stage we stop the training. This
technique is called early stopping.
Multi-layer Perceptron in Practice
cont…
 EXAMPLES OF USING THE MLP
we shall look at the four types of problems that
are generally solved using an MLP:
– regression,
– classification,
– time-series prediction,
– and data compression/data denoising.
EXAMPLES OF USING THE MLP Cont…
A Regression Problem
• The regression problem we will look at is a
very simple one.
• We will take a set of samples generated by a
simple mathematical function, and try to learn
the generating function (that describes how
the data was made) so that we can find the
values of any inputs, not just the ones we
have training data for.
EXAMPLES OF USING THE MLP Cont…
• The reason why we have to use the reshape()
method is that NumPy defaults to lists for
arrays that are N ×1
• compare the results of the [Link]() calls
below, and the effect of the transpose
operator .T on the array:
 EXAMPLES OF USING THE MLP Cont…
>>> x = [Link](0,1,40)
>>> [Link](x) (40,)
>>> [Link](x.T) (40,)
>>> >>> x =[Link](0,1,40).reshape((1,40))
>>> [Link](x) (1, 40) >>> [Link](x.T) (40, 1)
 EXAMPLES OF USING THE MLP Cont…
• You can plot this data to see what it looks like
(the results of which are shown in Figure 4.12)
using:
• >>> import pylab as pl
• >>> [Link](x,t,’.’)
EXAMPLES OF USING THE MLP Cont…
 EXAMPLES OF USING THE MLP Cont…
• train = x[0::2,:]
• test = x[1::4,:]
• valid = x[3::4,:]
• traintarget = t[0::2,:]
• testtarget = t[1::4,:]
• validtarget = t[3::4,:]
EXAMPLES OF USING THE MLP Cont…
 EXAMPLES OF USING THE MLP Cont…
Classification with the MLP
• There are a couple of choices for the outputs.
The first is to use a single linear node for the
output, y, and put some thresholds on the
activation value of that node.
• For example, for a four-class problem, we
could use:
EXAMPLES OF USING THE MLP Cont…
 EXAMPLES OF USING THE MLP Cont…
Time-Series Prediction
• There is a common data analysis task known as time-series
prediction, where we have a set of data that show how
something varies over time, and we want to predict how
the data will vary in the future.
• It is quite a difficult task, but a fairly important one.
• It is useful in any field where there is data that appears over
time, which is to say almost any field.
• Most notable (if often unsuccessful) uses have been in
trying to predict stock markets and disease patterns.
• The problem is that even if there is some regularity in the
time-series, it can appear over many different scales.
EXAMPLES OF USING THE MLP Cont…
• Figure 4.14 shows an example of a time-series
with τ = 3 and k = 4, with a set of datapoints
that make up an input vector marked as white
circles, and the target coloured black.
EXAMPLES OF USING THE MLP Cont…
EXAMPLES OF USING THE MLP Cont…
Data Compression
• We are now going to consider an interesting
variation of the MLP.
• Suppose that we train the network to
reproduce the inputs at the output layer
• The network is trained so that whatever
 EXAMPLES OF USING THE MLP Cont…
• you show it at the input is reproduced at the
output, which doesn’t seem very useful at
first, but suppose that we use a hidden layer
that has fewer neurons than the input layer
(see Figure 4.17)
• This bottleneck hidden layer has to represent
all of the information in the input, so that it
can be reproduced at the output
EXAMPLES OF USING THE MLP Cont…
EXAMPLES OF USING THE MLP Cont…
EXAMPLES OF USING THE MLP Cont…
 DERIVING BACK-PROPAGATION
• In fact, there are only three things that you
really need to know. One is the derivative
(with respect to x) of 1 /2 x 2
• Another is the chain rule, which says that
dy /dx = dy/ dt dt/ dx .
• The third thing is very simple: dy /dx = 0 if y is
not a function of x.
DERIVING BACK-PROPAGATION Cont…
• The Network Output and the Error
– The output of the neural network (the end of the
forward phase of the algorithm) is a function of
three things:
• The current input (x)
• The activation function g(·) of the nodes of the network
• The weights of the network (v for the first layer and w
for the second)
DERIVING BACK-PROPAGATION Cont…
• The Error of the Network
DERIVING BACK-PROPAGATION Cont…
DERIVING BACK-PROPAGATION Cont…
• Back-Propagation of Error
That’s fine, because we can use the chain
rule again
DERIVING BACK-PROPAGATION Cont…
• where g(·) is the activation function
• An activation function (initially a threshold
function) that decides whether the neuron
fires (‘spikes’) for the current inputs.
DERIVING BACK-PROPAGATION Cont…
DERIVING BACK-PROPAGATION Cont…
• Note that we can do exactly the same
computations if the network has extra hidden
layers between the inputs and the outputs.
• It gets harder to keep track of which functions
we should be differentiating, but there are no
new tricks needed.
DERIVING BACK-PROPAGATION Cont…
DERIVING BACK-PROPAGATION Cont…
• The Output Activation Functions
the three functions are:
 Radial Basis Functions
• In the Multi-layer Perceptron, the activations
of the hidden nodes were decided by whether
the inputs times the weights were above a
threshold that made the neuron fire.
• The product of the inputs and the weights was
summed, and if it was well above the
threshold then the neuron fired, if it was well
below the threshold it did not, and between
those values it acted linearly.
 Radial Basis Functions Cont…
• This has the result that the activity in the
hidden layer is distributed over the neurons
there, and it is this pattern of activation that
was used as the inputs to the next layer.
• This has the result that the activity in the
hidden layer is distributed over the neurons
there, and it is this pattern of activation that
was used as the inputs to the next layer.
 Radial Basis Functions Cont…
• In order to understand this better we are going to
need two concepts, one from machine learning,
weight space, and one from neuroscience,
receptive fields.
• weight space is abstractly plot them in the same
set of dimensions as the inputs, and to have
neurons that were ‘closer’ to the input being
more highly activated.
• Receptive fields is the area of the input that
directly influences the neuron's output.
Radial Basis Functions Cont…
 Radial Basis Functions Cont…
• The effect of radial basis functions in weight
space.
• The points show the position of the RBF in
weight space, while the circle around each
point shows the receptive field of the node.
 Radial Basis Functions Cont…
• We do not typically use a real Gaussian
function for the activation function, ignoring
the normalisation to get an approximation to
it written as:
 Radial Basis Functions Cont…
• The choice of σ in this equation is quite
important, since it controls the width of the
Gaussian.
 spline
• A spline curve is a mathematical
representation for which it is easy to build an
interface that will allow a user to design and
control the shape of complex curves and
surfaces.
• The general approach is that the user enters a
sequence of points, and a curve is constructed
whose shape closely follows this sequence.
The points are called control points.
THE RADIAL BASIS FUNCTION (RBF)
NETWORK
 THE RADIAL BASIS FUNCTION (RBF)
NETWORK Cont…
• The Radial Basis Function network consists of
input nodes connected by weights to a set of RBF
neurons, which fire proportionally to the distance
between the input and the neuron in weight
space.
• The activations of these nodes are used as inputs
to the second layer, which consists of linear
nodes.
• The schematic looks very similar to the MLP
except for the lack of a bias in the hidden layer.
THE RADIAL BASIS FUNCTION (RBF)
NETWORK Cont…
 THE RADIAL BASIS FUNCTION (RBF)
NETWORK Cont…
• We can space out RBF nodes to cover the
whole of space by continuing this pattern
everywhere, so that the network acts as a
universal approximator, since there is an
output for every possible input.
 THE RADIAL BASIS FUNCTION (RBF)
NETWORK Cont…
• Training the RBF Network
– So we can split the training into two parts:
position the RBF nodes, and then use the
activations of those nodes to train the linear
outputs. This makes things much simpler
– training an RBF network can be reduced to using
two other algorithms that are commonly used in
machine learning, one after the other.
– This is known as a hybrid algorithm, since it
combines supervised and unsupervised learning.
 The Radial Basic Function Algorithm
• Position the RBF centres by either:
– using the k-means algorithm to initialise the positions
of the RBF centres OR
– setting the RBF centres to be randomly chosen
datapoints
• Calculate the actions of the RBF nodes
• Train the output weights by either:
– using the Perceptron OR
– computing the pseudo-inverse of the activations of
the RBF centres (this will be described shortly)
 THE RADIAL BASIS FUNCTION (RBF)
NETWORK Cont…
• the width of the Gaussians should be set
according to the maximum distance between
the locations of the hidden nodes (d) and the
number of hidden nodes.
• The most common choice is to pick the width
of the Gaussian as σ = d/√ 2M, where M is the
number of RBFs.
 THE RADIAL BASIS FUNCTION (RBF)
NETWORK Cont…
• It is a modification of Gaussian function and it
looks like the soft-max function:
 INTERPOLATION AND BASIS
FUNCTIONS
• One of the problems that we looked at is function
approximation:
– Given some data, find a function that goes through
the data without overfitting to the noise, so that
values between the known datapoints can be inferred
or interpolated.
– The RBF network solves this problem by each of the
basis functions making a contribution to the output
whenever the input is within its receptive field.
– So several RBF nodes will probably respond for each
input.
 INTERPOLATION AND BASIS
FUNCTIONS Cont…
• If each function just returns the average value
within its patch, then for one-dimensional
data we get a histogram output.
INTERPOLATION AND BASIS
FUNCTIONS Cont…
 INTERPOLATION AND BASIS
FUNCTIONS Cont…
• Top: Curve showing a function.
• Second: A set of datapoints from the curve.
Third: Putting a straight horizontal line
through each point creates a histogram that
describes an approximation to the curve.
Bottom: That approximation
 INTERPOLATION AND BASIS
FUNCTIONS Cont…
• Representing the points by straight lines that
aren’t necessarily horizontal (so that their first
derivative matches at the point) gives a better
approximation.
 INTERPOLATION AND BASIS
FUNCTIONS Cont…
• Making the straight lines meet so that the
function is continuous gives a better
approximation.
 INTERPOLATION AND BASIS
FUNCTIONS Cont…
• Using cubic functions to connect the points
gives an even better approximation, and the
curve is also continuous at the points where
the sections join up
 Support Vector Machine
• Support Vector Machine (SVM) is a supervised
machine learning algorithm used for both
classification and regression.
• Though we say regression problems as well it’s
best suited for classification.
• The main objective of the SVM algorithm is to
find the optimal hyperplane in an N-
dimensional space that can separate the data
points in different classes in the feature space
 Support Vector Machine Cont…
Support Vector Machine Terminology
– Hyperplane: Hyperplane is the decision boundary
that is used to separate the data points of
different classes in a feature space. In the case of
linear classifications, it will be a linear equation
i.e. wx+b = 0.
– Support Vectors: Support vectors are the closest
data points to the hyperplane, which makes a
critical role in deciding the hyperplane and
margin.
Support Vector Machine Cont…
– Margin: Margin is the distance between the
support vector and hyperplane. The main
objective of the support vector machine algorithm
is to maximize the margin. The wider margin
indicates better classification performance.
– Kernel: Kernel is the mathematical function. Use
in non-linear SVM
 Support Vector Machine Cont…
Types of Support Vector Machine
Based on the nature of the decision boundary, Support
Vector Machines (SVM) can be divided into two main parts:

• Linear SVM: Linear SVMs use a linear decision boundary to

separate the data points of different classes.
• When the data can be precisely linearly separated, linear
SVMs are very suitable.
• This means that a single straight line (in 2D) or a
hyperplane (in higher dimensions) can entirely divide the
data points into their respective classes.
• A hyperplane that maximizes the margin between the
classes is the decision boundary.
Support Vector Machine Cont…
Support Vector Machine Cont…
 Support Vector Machine Cont…
• Non-Linear SVM: Non-Linear SVM can be used to
classify data when it cannot be separated into
two classes by a straight line (in the case of 2D).
• By using kernel functions, nonlinear SVMs can
handle nonlinearly separable data.
• The original input data is transformed by these
kernel functions into a higher-dimensional
feature space, where the data points can be
linearly separated.
• A linear SVM is used to locate a nonlinear
decision boundary in this modified space.
Support Vector Machine Cont…
 Support Vector Machine Cont…

• Say, our data is shown in the figure above. SVM solves

this by creating a new variable using a kernel.
• We call a point xi on the line and we create a new
variable yi as a function of distance from origin o.
• so if we plot this we get something like as shown below
Support Vector Machine Cont…
 Support Vector Machine Cont…
Algorithm
Steps
• Load the breast cancer dataset from
[Link]
• Separate input features and target variables.
• Build and train the SVM classifiers using RBF
kernel.
• Plot the scatter plot of the input features.
• Plot the decision boundary.
 Support Vector Machine Cont…
• Advantages of SVM
– Effective in high-dimensional cases.
– Its memory is efficient as it uses a subset of
training points in the decision function called
support vectors.
– Different kernel functions can be specified for the
decision functions and its possible to specify
custom kernels.