Matrusri Engineering College

3rd year Semester 6

Teacher: [Link] Reddy

UNIT 2​
THE MULTI LAYER
PERCEPTRON
___

Notes

The Multi-Layer Perceptron (MLP)

Limitations of Linear Models

●​ Linear models always depend on data being linearly separable

●​ XOR will not work for a simple perceptron model
●​ All real-world problems cannot be linearly separable — we cannot always find a solution
with a straight line

Improving Neural Networks

Two ways to improve a neural network:

1.​ Add backward connections → Recurrent Network

2.​ Add more neurons (extra layers) → Multi-Layer Network
Multi-Layer Network Structure

●​ Add hidden neurons between the input and output layers

●​ This creates an additional layer called the hidden layer
●​ Structure: Input Layer → Hidden Layer → Output Layer
●​ Taking inputs, weights are assigned, summation of weights is computed, and the output
is given

Why Multi-Layer Works

●​ The extra layers make the network more powerful

●​ It can solve problems that linear models cannot
●​ XOR can be solved by the multi-layer perceptron model

Training the MLP — Two Main Phases

Phase 1 — Forward Pass (going forward)

●​ Input values are given and outputs are computed layer by layer
●​ Hidden layer activations are calculated first
●​ Output layer activations are calculated similarly (done for multiple layers)
●​ After the forward pass, compare network output with targeted output and calculate
error

Phase 2 — Backward Pass (updating weights)

●​ Error is used in the backward pass to update the network's weights

Biases

●​ A bias is an additional input assigned to each neuron

●​ It allows the neuron to shift its activation threshold independently of the inputs
Activation Functions

1. Sigmoid — range (0, 1)

σ(x) = 1 / (1 + e⁻ˣ)

2. Tanh — range (-1, 1)

tanh(x) = (eˣ − e⁻ˣ) / (eˣ + e⁻ˣ)

3. ReLU (Rectified Linear)

R(x) = x if x ≥ 0, else 0

Backpropagation of Error

Backpropagation is the key algorithm used to train artificial neural networks. It adjusts the
weights so that the model's predictions become more accurate over time.

Main Steps in Backpropagation:

Step 1 — Forward Propagation

y = f(Σ wᵢxᵢ + b)

Step 2 — Compute Loss

Mean Squared Error: E = ½ (target − output)²

Step 3 — Compute Gradients Using the chain rule of calculus:

∂E / ∂w

Step 4 — Back Propagation

●​ Compute gradients using the chain rule

●​ Propagate the error backward
●​ Calculate how much each weight contributed to the error

Step 5 — Update Weights Using gradient descent:

w_new = w_old − μ (∂E / ∂w)

Step 6 — Repeat Repeat forward and backward propagation until either:

1.​ Error becomes sufficiently small

2.​ Maximum iterations are reached

The MLP Algorithm — Step by Step

Step 1 — Initialise the Network

●​ Choose number of input neurons

●​ Choose number of hidden layers
●​ Choose number of outputs
●​ Initialise weights and biases randomly

Step 2 — Forward Propagation For each training sample:

●​ Multiply input by weights

●​ Add bias
●​ Apply activation function

z = Σ(wᵢxᵢ) + b y = f(z)

Output is passed layer by layer until the final output is produced.

Step 3 — Compute Error

E = ½ (target − output)²

Step 4 — Back Propagation

●​ Compute gradients using chain rule

●​ Propagate error backward
●​ Calculate how much each weight contributed to the error

Step 5 — Update Weights

w_new = w_old − μ (∂E / ∂w)

Step 6 — Repeat Until error becomes small or maximum iterations are reached.
Sequential vs Batch Training

There are two main ways to update weights when training an MLP using backpropagation:

1. Sequential Training (ST)

●​ Updates weights after processing each individual training example

Advantages:

●​ Faster learning at the beginning

●​ Requires less memory
●​ Good for large datasets

2. Batch Training (BT)

●​ Updates weights only after processing the entire dataset

Advantages:

●​ More accurate gradient estimate

●​ More stable convergence

Local Minima

●​ A local minimum is a point on the error surface where the error value is smaller than
all surrounding points but not the smallest possible value overall
●​ The algorithm stops improving because all nearby directions increase the error
●​ However, there may still exist a better minimum — called the global minimum

Mini-Batches and Stochastic Gradient Descent

When training neural networks like the MLP, weights are updated using gradient descent. Two
common methods are used:

1. Stochastic Gradient Descent (SGD)

●​ Weights are updated after each individual training example

2. Mini-Batch Gradient Descent (MBG)

●​ Training dataset is divided into small groups called mini-batches

●​ Weights are updated after processing each mini-batch

Momentum

Momentum is a technique used in training neural networks to:

●​ Speed up learning
●​ Help the algorithm escape local minima during backpropagation

How it works:

In normal gradient descent, weights are updated based only on the current gradient:

w_new = w_old − μ (∂E / ∂w)

With momentum, the update also considers the previous weight change:

Δw(t) = μ (∂E / ∂w) + α Δw(t−1) w_new = w_old + Δw(t)

This means the algorithm keeps moving in the same direction if it was already going downhill
— like a ball rolling down a slope that builds up speed. This helps the algorithm push through
small local minima and converge more stably.

The MLP In Practice

How Much Training Data Do You Need?

The total number of weights in a one-hidden-layer MLP is:

(L+1) × M + (M+1) × N

Where L, M, N are the number of input, hidden, and output nodes respectively. The +1s account
for bias nodes.

This can be a very large number of adjustable parameters. A well-known rule of thumb is:
 Number of training examples ≥ 10 × number of weights

This means neural network training requires a lot of data and is computationally expensive —
the network needs to see all those examples many times. Unfortunately there is no exact
formula for the minimum data required since it depends entirely on the problem.

How Many Hidden Layers Do You Need?

The good news is that you almost never need more than two hidden layers for normal MLP
learning. In fact it can be shown mathematically that:

One hidden layer with enough hidden nodes is sufficient to approximate any
smooth function

This is known as the Universal Approximation Theorem.

How does one hidden layer achieve this?

By combining sigmoid functions in stages:

1.​ A single sigmoid neuron produces an S-shaped curve

2.​ Combining multiple sigmoids (including reversed ones) produces a ridge/hill shape
3.​ Adding another ridge at 90° produces a bump shape
4.​ The bump can be sharpened to any extent
5.​ Any function can be approximated by combining enough of these bumps in the output
layer

This means the MLP can approximate any decision boundary — not just the straight lines that
the Perceptron was limited to.

How Many Hidden Nodes Do You Need?

Unfortunately there is no theory to guide this choice. The only approach is to:

●​ Train networks with different numbers of hidden nodes

●​ Compare their performance
●​ Choose the one that gives the best results

This trial-and-error approach is standard practice and will be explored further in Section 4.4.
Overfitting

Even with the right number of layers, training for too long or using too many hidden nodes can
cause overfitting — where the network learns the training data too well, including its noise,
and performs poorly on new unseen data.

This is shown in Figure 4.11, where the training error keeps decreasing but the validation
error (error on unseen data) starts to increase again after a certain point. The solution is early
stopping — halting training at the point where the validation error is at its lowest, before the
network starts overfitting.

Key Takeaways

Decision Guideline

Training data At least 10× the number of weights

Hidden layers 1–2 is almost always sufficient

Hidden nodes Experiment and compare

Training Use early stopping to avoid overfitting

duration

The MLP is powerful enough to approximate any continuous function, but using it well requires
careful choices about network size, training data, and training duration.

4.3.3 When to Stop Learning

The Problem

Training the MLP requires running over the entire dataset many times, with weights updating
after each iteration. Deciding when to stop is not straightforward — the two most obvious
options both have serious problems:
 ●​ Stop after N iterations → risk of overfitting if N is too large, or not learning enough if
N is too small
●​ Stop when error reaches a minimum threshold → the algorithm may never
terminate, or may overfit before reaching that threshold

Using both together helps, as does stopping when the error stops decreasing — but none of
these alone is fully reliable.

The Better Solution — Early Stopping

The most effective approach uses a validation set to monitor how well the network is
generalising to unseen data throughout training. The process works as follows:

1.​ Train the network for a predetermined number of iterations

2.​ Check the validation error — how well the network performs on data it hasn't trained
on
3.​ Continue training for a few more iterations and check again
4.​ Repeat this process

What typically happens is:

●​ Training error keeps decreasing throughout

●​ Validation error decreases at first, then starts increasing again at some point

The moment the validation error starts rising is when the network has stopped learning the
underlying pattern and has started memorising the noise in the training data — this is
overfitting. At this point, training should be stopped immediately.

This technique is called early stopping and is illustrated in Figure 4.11.

Key Takeaway

Stop training at the point where validation error is at its lowest — not when
training error is at its lowest.

These two points are different, and confusing them is one of the most common mistakes when
training neural networks.
4.4 Examples of Using the MLP

The MLP is too complex to trace weight changes by hand the way we did with the Perceptron.
Instead, we look at practical demonstrations of how the network learns from real data.

The four types of problems the MLP is commonly used for are:

●​ Regression — predicting a continuous output value

●​ Classification — assigning inputs to discrete categories
●​ Time-series prediction — predicting future values based on past sequences
●​ Data compression / denoising — compressing data into a lower-dimensional form or
removing noise from data

4.4.1 A Regression Problem

The Setup

A simple sine wave with some random noise added is used as the dataset. The goal is to train
the MLP to learn and reproduce this underlying function despite the noise.

Splitting the Data

The dataset is divided into three parts:

●​ Training set — used to train the network and update weights

●​ Validation set — used to monitor generalisation and decide when to stop
●​ Test set — used to evaluate final performance after training is complete

The data is split roughly in a 70:25 ratio between training and testing, with some held aside for
validation.

Training the Network

●​ 1 input, 1 output (linear — since this is regression, not classification)

●​ A small number of hidden nodes in one hidden layer
 ●​ The network is trained using backpropagation with early stopping

Early Stopping

Instead of training for a fixed number of iterations, the validation error is monitored
continuously. Training stops when the validation error stops improving across two consecutive
checks. Tracking two consecutive changes (rather than just one) prevents the algorithm from
stopping prematurely due to small temporary fluctuations in the error.

Choosing the Right Number of Hidden Nodes

There is no formula — experimentation is the only approach. Key findings from testing different
network sizes:

●​ Too few hidden nodes → network cannot capture the complexity of the function
●​ Too many hidden nodes (e.g. 50) → network becomes unstable with very high
variation between runs
●​ Best results come from a small number of nodes, roughly between 2 and 10 for this
problem
●​ Since weights are initialised randomly, each run produces slightly different results —
running multiple networks and comparing their validation errors is the recommended
approach

4.4.2 Classification with the MLP

Output Encoding

For classification problems, the inputs are simply the normalised feature values. The key
decision is how to encode the outputs. There are two approaches:

1. Single Linear Output Node Use one output node and apply thresholds to determine the
class. For example, for 4 classes:

●​ y ≤ -0.5 → Class 1
●​ -0.5 < y ≤ 0 → Class 2
●​ 0 < y ≤ 0.5 → Class 3
 ●​ y > 0.5 → Class 4

The problem with this approach is that it becomes impractical as the number of classes grows,
and boundary cases are ambiguous — the network gives no indication of how confident it is.

2. 1-of-N Encoding (recommended) A separate output node is used for each class. Target
vectors contain all zeros except for a single 1 in the position corresponding to the correct class.
For example, with 6 classes, the 4th class is represented as (0, 0, 0, 1, 0, 0). All outputs are
binary (0 or 1).

Making a Classification Decision

After training, classification is done by simply picking the output node with the highest
activation value — this is called the hard-max activation function. It is almost always
unambiguous since it is very unlikely that two output neurons will have exactly the same
highest value.

An alternative is the soft-max function, which scales all outputs so they sum to 1, making them
interpretable as probabilities:

●​ A clear winner will have a value close to 1

●​ If multiple outputs are similar, they will each have a value close to 1/p, where p is the
number of similar outputs — indicating the network is uncertain

Class Imbalance Problem

One important issue with classification is class imbalance — when the dataset has many more
examples of one class than another. For example, in medical data where 90% of tests are
negative, the network can learn to always predict the negative class and still be 90% accurate —
but this is completely useless as a classifier.

Two solutions:

1. Balance the dataset — ensure approximately equal numbers of each class in the training
set, even if this means discarding data from the over-represented class

2. Novelty detection — train the network only on the negative class and treat anything that
looks different as a positive example. This avoids the imbalance problem entirely.
4.4.3 Classification Example: The Iris Dataset

The Dataset

The Iris dataset is a classic machine learning dataset from the UCI repository. The task is to
classify three types of iris flowers based on four measurements — the length and width of their
sepals and petals. It was originally analysed by statistician R.A. Fisher in the 1930s.

Preprocessing

The class labels in the raw file are text (e.g. "Iris-setosa") rather than numbers, so they must be
converted to numerical values (0, 1, 2) before loading. Once loaded, the four input features are
normalised using the maximum and minimum values so all inputs fall within the same range.

Output Encoding

The class labels are converted to 1-of-N encoding — a matrix of zeros with a single 1 in the
column corresponding to the correct class. So class 0 becomes (1,0,0), class 1 becomes (0,1,0),
and class 2 becomes (0,0,1).

Splitting the Data

There are 150 examples, evenly split across three classes (50 each), so class imbalance is not a
concern here. The data is split into:

●​ Training set — half the data

●​ Validation set — one quarter
●​ Test set — one quarter

The data is randomly shuffled before splitting to ensure no single class is over-represented in
any set.

Training and Results

The network is trained with:

●​ 5 hidden nodes
●​ Softmax output activation (appropriate for multi-class classification)
●​ Early stopping using the validation set

The resulting confusion matrix shows:

Predicted C1 Predicted C2 Predicted C3

Actual C1 16 0 0

Actual C2 0 12 2

Actual C3 0 1 6

Overall accuracy: 91.9%

4.4.4 Time-Series Prediction

What is Time-Series Prediction?

A time-series is a sequence of data points recorded over time. The goal is to use past values to
predict future values. The MLP can be used for this by treating past measurements as inputs
and the future value as the target output.

Key Parameters

Two important values need to be chosen:

●​ τ (tau) — how far ahead you want to predict (the prediction horizon)
●​ k — how many past datapoints to use as inputs

The relationship is:

y = x(t + τ) = f(x(t), x(t−1), ..., x(t−kτ))

Choosing the right values of τ and k is a key part of the problem — there is no fixed rule and
experimentation is needed.

The Ozone Dataset

The example dataset contains daily measurements of the ozone layer thickness above
Palmerston North, New Zealand, between 1996 and 2004. Ozone thickness is measured in
Dobson Units at 0 degrees Celsius and 1 atmosphere of pressure.

The dataset has 4 elements per reading: year, day of year, ozone level, and sulphur dioxide level.
There are 2855 readings in total. The task is to predict future ozone levels and detect any
overall drop over time — which is relevant to global warming and increased skin cancer risk.

Data Preparation

●​ The input vector is assembled by picking k values from the array with spacing τ
●​ The data must be carefully randomised before splitting to ensure even-indexed and
odd-indexed datapoints are not systematically separated into different groups
●​ The dataset is split into training, validation, and test sets as usual

Training the Network

●​ Since this is a regression problem, linear output nodes are used (no classes, no
thresholds)
●​ The confusion matrix is not useful here — instead sum-of-squares error is used to
measure performance
●​ Early stopping is applied using the validation set
●​ You need to experiment with different values of:
○​ Number of input nodes (determined by k)
○​ Number of hidden nodes
○​ Values of τ and k

Here's a clear explanation of time-series prediction:

Time-Series Prediction

What is a Time Series?

A time series is simply a sequence of data points collected over time at regular intervals.
Examples include:

●​ Daily temperature readings

●​ Stock prices over months
●​ Heart rate measured every second
●​ Ozone levels measured daily

The key characteristic is that the order matters — each value is related to the values that came
before it.

What is Time-Series Prediction?

Time-series prediction means using past values to predict future values. For example:

●​ Given the last 7 days of temperature, predict tomorrow's temperature

●​ Given the last month of stock prices, predict next week's price

The underlying assumption is that patterns in the past will continue into the future —
which is true for many real-world phenomena.

How Does the MLP Learn Time Series?

The MLP itself doesn't inherently understand time. Instead, we convert the time-series problem
into a standard input-output problem by:

1.​ Taking a window of past values as the input vector

2.​ Using the future value we want to predict as the target output

For example, if we want to predict the value 3 steps ahead using the last 4 readings:

Input: (x(t), x(t-1), x(t-2), x(t-3)) Target: x(t+3)

The MLP then learns the relationship between the input window and the future value through
training — exactly as it would for any other regression problem.

Key Parameters

Two parameters control how the input vector is constructed:

τ (tau) — Prediction Horizon How far ahead you want to predict. A larger τ means predicting
further into the future, which is generally harder and less accurate.

k — Window Size How many past datapoints to include as inputs. A larger k gives the network
more historical context but also increases the number of input nodes and weights, requiring
more training data.

There is no formula for choosing τ and k — they must be found through experimentation.

Challenges of Time-Series Prediction

1. Choosing the right window size Too small a window and the network doesn't have enough
context to detect patterns. Too large a window and the network becomes unnecessarily
complex.

2. Non-stationarity Many real-world time series change their behaviour over time — for
example, stock markets behave differently during a recession. The MLP may struggle to
generalise if the underlying pattern shifts.

3. Noise Real data always contains random fluctuations. The network must learn the
underlying trend without overfitting to the noise — which is why early stopping and validation
sets are especially important here.

4. Data preparation The input vectors must be assembled carefully from the time series. The
data must also be randomised before splitting into training, validation, and test sets —
otherwise systematic patterns in the ordering can bias the learning.

Time-Series vs Standard Regression

 Standard Regression Time-Series Prediction

Inputs Independent features Past values of the same

sequence

Output Any continuous value Future value of the sequence

Data order Doesn't matter Critically important

Key challenge Feature selection Choosing τ and k

Network type MLP with linear output MLP with linear output

Despite these differences, from the MLP's perspective both are treated as regression problems
— the only extra work is in how the input vectors are assembled.

Real World Applications

Time-series prediction is one of the most practically important applications of neural networks:

●​ Weather forecasting — predicting temperature, rainfall, wind

●​ Financial markets — predicting stock or currency prices
●​ Energy — predicting electricity demand
●​ Healthcare — predicting patient vital signs
●​ Environmental monitoring — predicting pollution or ozone levels as in the textbook
example