Recognizing Patterns In addition to function fitting, neural networks are also good

at recognizing patterns. For example, suppose you want to classify a tumor as

benign or malignant, based on uniformity of cell size, clump thickness, mitosis,etc.
You have 699 example cases for which you have 9 items of data and the correct
classification as benign or malignant. As with function fitting, there are two ways
to solve this problem:Use the nprtool GUI, as described in Using the Neural
Network Pattern Recognition Tool. Use a command-line solution, as described in
Using Command-Line Functions. It is generally best to start with the GUI, and
then to usethe GUI to automatically generate command-line scripts. Before using
either method, the first step is to define the problem by selecting a data set. The
next section describes the data format.

Defining a Problem

To define a pattern recognition problem, arrange a set of Q input vectors as

columns in a matrix. Then arrange another set of Q target vectors so that they
indicate the classes to which the input vectors are assigned (see "Data Structures"
for a detailed description of data formatting for static and time series data). There
are two approaches to creating the target vectors.

One approach can be used when there are only two classes; you set each scalar
target value to either 1 or 0, indicating which class the corresponding input belongs
to. For instance, you can define the exclusive-or classification problem as follows:

inputs = [0 1 0 1; 0 0 1 1];
targets = [0 1 1 0];

Alternately, target vectors can have N elements, where for each target vector, one
element is 1 and the others are 0. This defines a problem where inputs are to be
classified into N different classes. For example, the following lines show how to
define a classification problem that divides the corners of a 5-by-5-by-5 cube into
three classes:

 The origin (the first input vector) in one class

 The corner farthest from the origin (the last input vector) in a second class
 All other points in a third class
 inputs = [0 0 0 0 5 5 5 5; 0 0 5 5 0 0 5 5; 0 5 0 5 0 5 0 5];
 targets = [1 0 0 0 0 0 0 0; 0 1 1 1 1 1 1 0; 0 0 0 0 0 0 0 1];
Classification problems involving only two classes can be represented using either
format. The targets can consist of either scalar 1/0 elements or two-element
vectors, with one element being 1 and the other element being 0.

The next section shows how to train a network to recognize patterns, using the
neural network pattern recognition tool GUI, nprtool. This example uses the cancer
data set provided with the toolbox. This data set consists of 699 nine-element input
vectors and two-element target vectors. There are two elements in each target
vector, because there are two categories (benign or malignant) associated with each
input vector.

Using the Neural Network Pattern Recognition Tool

[Link] needed, open the Neural Network Start

GUI with this command: nnstart

2. Pattern Recognition Tool to open the Neural Network Pattern Recognition

Tool. (You can also use the command nprtool.)

3. Click Next to proceed. The Select Data window opens.

4. Click Load Example Data Set. The Pattern Recognition Data Set Chooser
window opens.

5. Select Example and click Import. You return to the Select Data window.

[Link] Next to continue to the Validation and Test Data window.

alidation and test data sets are each set to 15% of the original data. With these
settings, the input vectors and target vectors will be randomly divided into three
sets as follows:

 70% are used for training.

 15% are used to validate that the network is generalizing and to stop training
before overfitting.
 The last 15% are used as a completely independent test of network
generalization.

(See "Dividing the Data" for more discussion of the data division process.)

7. Under the Plots pane,click Confusion in the Neural Network

Pattern Recognition [Link] next figure shows the confusion matrices for
training, testing, and validation,and the three kinds of data combined. The network
outputs are very accurate, as you can see by the high numbers of correct responses
in the green squares and the low numbers of incorrect responses inthe red squares.
The lower right blue squares illustrate the overallaccuracies.

[Link] the Receiver Operating Characteristic (ROC) curve. Under the Plots pane,
click Receiver Operating Characteristic in the Neural Network Pattern
Recognition Tool.

Using Command-Line Functions

The easiest way to learn how to use the command-line functionality of the toolbox
is to generate scripts from the GUIs, and then modify them to customize the
network training. As an example, let's look at the simple script that was created at
step 14 of the previous section.

% Solve a Pattern Recognition Problem with a Neural Network

% Script generated by NPRTOOL
%
% This script assumes these variables are defined:
%
% cancerInputs - input data.
% cancerTargets - target data.

inputs = cancerInputs;
inputs = cancerTargets;

% Create a Pattern Recognition Network

hiddenLayerSize = 10;
net = patternnet(hiddenLayerSize);

% Set up Division of Data for Training, Validation, Testing

[Link] = 70/100;
[Link] = 15/100;
[Link] = 15/100;

% Train the Network

[net,tr] = train(net,inputs,targets);

% Test the Network

outputs = net(inputs);
errors = gsubtract(targets,outputs);
performance = perform(net,targets,outputs)

% View the Network

view(net)

% Plots
% Uncomment these lines to enable various plots.
%figure, plotperform(tr)
%figure, plottrainstate(tr)
%figure, plotconfusion(targets,outputs)
%figure, ploterrhist(errors)

You can save the script, and then run it from the command line to reproduce the
results of the previous GUI session. You can also edit the script to customize the
training process. In this case, follow each step in the script.

1. The script assumes that the input vectors and target vectors are already
loaded into the workspace. If the data are not loaded, you can load them as
follows:
2. load cancer_dataset
3. inputs = cancerInputs;
4. targets = cancerTargets;
5. Create the network. The default network for function fitting (or regression)
problems, patternnet, is a feedforward network with the default tan-sigmoid
transfer functions in both the hidden and output layers. You assigned ten
neurons (somewhat arbitrary) to the one hidden layer in the previous section.
 The network has two output neurons, because there are two target
values (categories) associated with each input vector.
 Each output neuron represents a category.
 When an input vector of the appropriate category is applied to the
network, the corresponding neuron should produce a 1, and the other
neurons should output a 0.

To create the network, enter these commands:

 hiddenLayerSize = 10;
net = patternnet(hiddenLayerSize);
Note The choice of network architecture for pattern recognition
problems follows similar guidelines to function fitting problems. More
neurons require more computation, and they have a tendency to overfit
the data when the number is set too high, but they allow the network
to solve more complicated problems. More layers require more
computation, but their use might result in the network solving
complex problems more efficiently. To use more than one hidden
layer, enter the hidden layer sizes as elements of an array in the
patternnet command.

6. Set up the division of data.

7. [Link] = 70/100;
8. [Link] = 15/100;
9. [Link] = 15/100;

With these settings, the input vectors and target vectors will be randomly
divided, with 70% used for training, 15% for validation and 15% for testing.

(See "Dividing the Data" for more discussion of the data division process.)

[Link] the network. The pattern recognition network uses the default Scaled
Conjugate Gradient (trainscg) algorithm for training. To train the network,
enter this command:
11.[net,tr] = train(net,inputs,targets);

During training, as in function fitting, the training window opens. This

window displays training progress. To interrupt training at any point, click
Stop Training.

Train the Network

Once the network weights and biases are initialized, the network is ready for
training. The multilayer feedforward network can be trained for function
approximation (nonlinear regression) or pattern recognition. The training
process requires a set of examples of proper network behavior—network
inputs p and target outputs t. The process of training a neural network
involves tuning the values of the weights and biases of the network to
optimize network
performance, as defined by the network performance function
[Link]. The default performance function for feedforward networks
is mean square error mse—the average squared error between the network
outputs a andthe target outputs t. It is defined as follows:

(Individual squared errors can also be weighted. See Error Weighting.)

There are two different ways in which training can be implemented:
incremental mode and batch mode. In incremental mode, the gradient is
computed and the weights are updated after each input is applied to the
network. In batch mode, all the inputs in the training set are applied to the
network before the weights are updated. This topic describes batch mode
training with the train command. Incremental training with the adapt
command is discussed in Incremental Training with adapt.

For most problems, when using the Neural Network Toolbox™ software,
batch training is significantly faster and produces smaller errors than
incremental [Link] training multilayer feedforward networks, any
standard numerical optimization algorithm can be used to optimize the
performance function, but there are a few key ones that have shown
excellent performance for neural network training. These optimization
methods use either the gradient of the network performance with respect to
the network weights, or the Jacobian of the network errors with respect to
the [Link] gradient and the Jacobian are calculated using a technique
called the backpropagation algorithm, which involves performing
computations backward through the network. The backpropagation
computation is derived using the chain rule of calculus and is described in
Chapters 11 (for the gradient) and 12 (for the Jacobian) of
[HDB96].Training AlgorithmsAs an illustration of how the training works,
consider the simplest optimization algorithm — gradient descent. It updates
the network weights and biases in the direction in which the performance
function decreases most rapidly, the negative of the gradient. One iteration
of this algorithm can be written as where xk is a vector of current weights
and biases, gk is the current gradient, and αk is the learning rate. This
equation is iterated until the network converges.A list of the training
algorithms that are available in the Neural Network Toolbox software and
that use gradient- or Jacobian-based methods, is shown in the following
table. For a detailed description of several of these techniques, see also
Hagan, M.T., H.B. Demuth, and M.H. Beale, Neural Network Design,
Boston, MA: PWS Publishing, 1996, Chapters [Link] 12.
FunctionAlgorithm
trainlmLevenberg-Marquardt

trainbrBayesian Regularization

trainbfgBFGS Quasi-Newton

trainrpResilient Backpropagation

trainscgScaled Conjugate Gradient

traincgbConjugate Gradient with Powell/Beale Restarts

traincgfFletcher-Powell Conjugate Gradient

traincgpPolak-Ribiére Conjugate Gradient

trainossOne Step Secant

traingdxVariable Learning Rate Gradient Descent

traingdmGradient Descent with Momentum

traingdGradient Descent

The fastest training function is generally trainlm, and it is the default

training function for feedforwardnet. The quasi-Newton method, trainbfg, is
also quite fast. Both of these methods tend to be less efficient for large
networks (with thousands of weights), since they require more memory and
more computation time for these cases. Also, trainlm performs better on
function fitting (nonlinear regression) problems than on pattern recognition
[Link] training large networks, and when training pattern
recognition networks, trainscg and trainrp are good choices. Their memory
requirements are relatively small, and yet they are much faster than standard
gradient descent [Link] Multilayer Training Speed and Memory for
a full comparison of the performances of the training algorithms shown in
the table [Link] a note on terminology, the term "backpropagation is
sometimes used to refer specifically to the gradient descent algorithm, when
applied to neural network training. That terminology is not used here, since
the process of computing the gradient and Jacobian by performing
calculations backward through the network is applied in all of the training
functions listed above. It is clearer to use the name of the specific
optimization algorithm that is being used, rather Than to use the term
backpropagation alone. Also, the multilayer network is sometimes referred
to as a backpropagation network. However, the backpropagation technique
that is used to compute gradients and Jacobians in a multilayer network can
also be applied to many different network architectures. In fact, the gradients
and Jacobians for any network that has differentiable transfer functions,
weight functions and net input functions can be computed using the Neural
Network Toolbox software through a backpropagation process. You can
even create your own custom networks and then train them using any of the
training functions in the table above. The gradients and Jacobians will be
automatically computed for you. Efficiency and Memory Reduction There
are some network parameters that are helpful when training large networks
or using large data sets. For example, the parameter [Link].
memoryReduction can be used to reduce the amount of memory that you use
whiletraining or simulating the network. If this parameter is set to 1 (the
default), the maximum memory is used, and the fastest trainin times will be
achieved. If this parameter is set to 2, then the datais divided into two parts.
All calculations (like gradients and Jacobians) are done first on part one, and
then later on part two. Any intermediate variables used in part 1 are released
before the part 2 calculations are done. This can save significant memory,
especially for the trainlm training function. If memory Reduction is set to N,
then the data is divided into N parts, which are computed separately. The
larger the value of N, the larger the reduction in memory use, although the
amount of reduction diminishes as N is [Link] is a drawback to
using memory reduction. A computational overhead is associated with
computing the Jacobian and gradient in submatrices. If you have enough
memory available, then it is better to leave memory Reduction set to 1 and
to compute the full Jacobian or gradient in one step.

If you have a large training set, and you are running out of memory,then
you should set memoryReduction to 2 and try again. If you still run out of
memory, continue to increase memoryReduction. GeneralizationProperly
trained multilayer networks tend to give reasonable answers when presented
with inputs that they have never seen. Typically, a new input leads to an
accurate ouput, if the new input is similarto inputs used in the training set.
This generalization property makes it possible to train a network on a
representative set of input/target pairs and get good results without training
the network on all possible
 input/output pairs. There are two features of the Neural Network Toolbox
software that are designed to improve network generalization: regularization
and early stopping. These features and their use are discussed in detail in
Improving Generalization. A few comments on using these techniques are
given in the [Link] default generalization feature for the multilayer
feedforward network is early stopping. Data are automatically divided into
training, validation and test sets, as described in Dividing the Data. The
erroron the validation set is monitored during training, and the training is
stopped when the validation increases over [Link].max_fail
iterations. If you wish to disable early stopping, you can assign no data to
thevalidation set. This can be done by setting [Link] to
zero. An alternative method for improving generalization is
[Link] can be done automatically by using the
Bayesian regularization training function trainbr. This can be done by setting
[Link] to 'trainbr'. This will also automatically move any data in the
validation set to the training set.

Time Series Prediction

Dynamic neural networks are good at time series prediction.

Suppose, for instance, that you have data from a pH neutralization process. You
want to design a network that can predict the pH of a solution in a tank from past
values of the pH and past values of the acid and base flow rate into the tank. You
have a total of 2001 time steps for which you have those series.

You can solve this problem in two ways:

 Use a graphical user interface, ntstool, as described in Using the Neural

Network Time Series Tool.
 Use command-line functions, as described in Using Command-Line
Functions.

It is generally best to start with the GUI, and then to use the GUI to automatically
generate command-line scripts. Before using either method, the first step is to
define the problem by selecting a data set. Each GUI has access to many sample
data sets that you can use to experiment with the toolbox. If you have a specific
problem that you want to solve, you can load your own data into the workspace.
The next section describes the data format.

Defining a Problem
To define a time series problem for the toolbox, arrange a set of TS input vectors
as columns in a cell array. Then, arrange another set of TS target vectors (the
correct output vectors for each of the input vectors) into a second cell array (see
"Data Structures" for a detailed description of data formatting for static and time
series data). However, there are cases in which you only need to have a target data
set. For example, you can define the following time series problem, in which you
want to use previous values of a series to predict the next value:

targets = {1 2 3 4 5};

The next section shows how to train a network to fit a time series data set, using
the neural network time series tool GUI, ntstool. This example uses the pH
neutralization data set provided with the toolbox.