Regression — Unit II Study Notes

REGRESSION
Unit II — Machine Learning Models

Comprehensive Study Notes

Linear Models • Multi-Layer Perceptron • Forward & Backward Propagation Back-Propagation Derivation •
Radial Basis Functions & Splines RBF Networks • Curse of Dimensionality • Interpolation & Basis Functions
Support Vector Machines

Page 1 of 16
 Regression — Unit II Study Notes

Table of Contents
TOC \h \o "1-3"

Page 2 of 16
 Regression — Unit II Study Notes

1. Linear Models for Regression

1.1 What is Regression?

Regression is a supervised learning task in which the goal is to learn a mapping f from an input vector x to a
continuous output (target) variable t (or y). Given a training set of N examples {(x_n, t_n)}, the model learns
parameters so that predictions f(x) are as close as possible to the true targets, and generalise well to unseen
inputs.
Key Idea: Regression predicts a continuous numeric value, unlike classification which predicts discrete class
labels.

1.2 Simple Linear Regression

The simplest model assumes a straight-line relationship between a single input variable x and the target t:

y(x) = w0 + w1 x
where w0 is the intercept (bias) and w1 is the slope (weight). The parameters w0 and w1 are learned by
minimising an error/loss function computed over the training data, most commonly the Sum-of-Squares Error:

E(w) = (1/2) Σn [ y(xn, w) − tn ]²

Minimising E(w) with respect to w0 and w1 (by setting partial derivatives to zero) gives the classical least-
squares solution.

1.3 Multiple Linear Regression

When there are D input variables x1, x2, ..., xD, the linear model generalises to:

y(x, w) = w0 + w1x1 + w2x2 + ... + wDxD = w0 + wᵀx

In matrix form, stacking all N training examples into a design matrix X (N × (D+1), with a column of 1's for the
bias) and target vector t, the least-squares solution is obtained in closed form as:

w* = (XᵀX)⁻¹ Xᵀ t
This is called the Normal Equation. It requires XᵀX to be invertible; when it is not (e.g., due to multicollinearity or
more features than samples), regularisation or pseudo-inverse methods are used.

1.4 Linear Basis Function Models

A purely linear model in x is often too restrictive. Linear Basis Function Models keep the model linear in the
parameters w but allow non-linearity in x by first transforming the input through a set of fixed, non-linear basis
functions φj(x):

y(x, w) = w0 + Σj=1..M wj φj(x) = wᵀφ(x)

Because y is still linear in w, the same least-squares machinery (normal equations) applies, even though the
mapping from x to y can be highly non-linear. Common choices of basis function include:
• Polynomial basis: φj(x) = x^j

Page 3 of 16
 Regression — Unit II Study Notes

• Gaussian basis: φj(x) = exp( − (x − μj)² / 2s² )

• Sigmoidal basis: φj(x) = σ( (x − μj) / s ), where σ is the logistic sigmoid
• Fourier basis (sine/cosine terms)

1.5 Regularisation: Ridge and Lasso

As model complexity (e.g., polynomial order M, or number of basis functions) increases, the model can overfit —
fitting noise in the training data rather than the underlying trend. Regularisation adds a penalty term to the error
function to discourage large weights:

E(w) = (1/2) Σn [ y(xn,w) − tn ]² + (λ/2) ‖w‖² (Ridge / weight decay)

This is known as ridge regression or weight decay, and has the closed-form solution w* = (XᵀX + λI)⁻¹Xᵀt. Using an
L1 penalty (Σ|wj|) instead gives Lasso regression, which additionally drives some weights exactly to zero,
performing automatic feature selection. λ is a hyperparameter controlling the strength of regularisation,
typically chosen via cross-validation.

1.6 Bias–Variance Trade-off

The generalisation error of a model can be decomposed into three components:

Expected Error = Bias² + Variance + Irreducible Noise

Regime Bias Variance Typical cause

Underfitting High Low Model too simple (e.g.,

low-order polynomial)
Good fit Low Low Model complexity
matched to data
Overfitting Low High Model too complex / too
little data
Increasing model flexibility (more basis functions, higher-degree polynomials, larger neural networks) reduces
bias but increases variance. Regularisation and cross-validation are used to find the sweet spot.

1.7 Assumptions of Linear Regression

• Linearity: the relationship between predictors and target is linear in the parameters.
• Independence: observations (and residuals) are independent of one another.
• Homoscedasticity: constant variance of residuals across all levels of x.
• Normality: residuals are approximately normally distributed (needed for inference, not for point
estimation).
• No (or low) multicollinearity among predictors.

1.8 Limitations of Linear Models

Even with fixed basis functions, linear models suffer from the curse of dimensionality if the basis functions must
be spread throughout a high-dimensional input space (see Section 3.3), and the basis functions φj must typically
be chosen in advance. This motivates models such as the Multi-Layer Perceptron, where the basis functions
themselves are adaptive and learned from data.

Page 4 of 16
 Regression — Unit II Study Notes

Page 5 of 16
 Regression — Unit II Study Notes

2. Multi-Layer Perceptron (MLP)

2.1 Overview
A Multi-Layer Perceptron is a feed-forward artificial neural network composed of an input layer, one or more
hidden layers, and an output layer. Unlike linear basis function models where φj(x) is fixed, an MLP learns its
own internal basis functions (the hidden unit activations) directly from data — this is what gives it the power to
model complex, non-linear relationships, including for regression tasks.
Universal Approximation: A feed-forward network with a single hidden layer containing a sufficient (possibly
very large) number of units, with non-linear activation functions, can approximate any continuous function on a
compact domain to arbitrary accuracy.

2.1.1 Structure of a Neuron

Each neuron (unit) computes a weighted sum of its inputs, adds a bias, and passes the result through a non-
linear activation function:

a = Σi wi xi + w0 = wᵀx + b
z = g(a)
where g is the activation function. Common choices:

Activation Formula Range Notes

Sigmoid (logistic) g(a) = 1 / (1 + e^−a) (0, 1) Classic choice; saturates

for large |a|
Tanh g(a) = (e^a − e^−a)/(e^a (−1, 1) Zero-centred, often
+ e^−a) trains faster than
sigmoid
ReLU g(a) = max(0, a) [0, ∞) Cheap, reduces
vanishing gradient,
default in deep nets
Linear (identity) g(a) = a (−∞, ∞) Used at the output layer
for regression

2.2 Going Forwards: Forward Propagation

Forward propagation is the process of computing the network's output given an input, by successively applying
the weighted sum + activation operation, layer by layer. For a network with one hidden layer:
Step 1 — Hidden layer activations (pre-activation):

aj = Σi=1..D w(1)ji xi + w(1)j0 for j = 1, ..., M

Step 2 — Hidden layer outputs, via a non-linear activation function h:

zj = h(aj)
Step 3 — Output layer pre-activations, formed as a linear combination of the hidden outputs:

Page 6 of 16
 ak = Σj=1..M w(2)kj zj + w(2)k0 for k = 1, ..., K
Regression — Unit II Study Notes

Step 4 — Output activations, via an output activation function σ:

yk = σ(ak)
For regression problems, σ is normally taken to be the identity function so that yk = ak, allowing the network to
output any real value. For binary classification a logistic sigmoid is used, and for multi-class classification a
softmax is used. Combining all the steps for a single hidden layer network gives the overall network function:

yk(x, w) = σ( Σj w(2)kj · h( Σi w(1)ji xi + w(1)j0 ) + w(2)k0 )

This expression shows explicitly that the MLP is simply a nonlinear function from a set of input variables {xi} to a
set of output variables {yk}, controlled by the vector w of all weight and bias parameters, and can be built up by
evaluating this function for successive layers of a network — each layer's outputs are used as the inputs to the
next.

2.3 Going Backwards: The Back-Propagation Algorithm

Training the MLP means choosing the weights w to minimise an error function E(w) that measures the
discrepancy between network outputs yk and targets tk over the training set, e.g. the sum-of-squares error:

E(w) = (1/2) Σn Σk [ yk(xn, w) − tnk ]²

Because E(w) is a complicated, non-linear function of the weights, it is minimised using iterative gradient-based
optimisation (typically gradient descent or a variant such as stochastic gradient descent). This requires
computing the gradient ∂E/∂w for every weight in the network. Back-propagation is an efficient algorithm for
computing exactly this gradient, using the chain rule of calculus to propagate error information backwards from
the output layer to the input layer.
Why it's needed: A naive numerical computation of the gradient (finite differences) would require O(W)
forward passes for W weights — extremely costly for large networks. Back-propagation computes all gradients
in a single forward pass plus a single backward pass, i.e. O(W) total work instead of O(W²).

2.3.1 The Two Phases of Back-Propagation

1. Forward phase: propagate the input through the network to compute all hidden and output activations (as
in Section 2.2), and evaluate the error E.
2. Backward phase: propagate 'error signals' (δ terms) backwards from the output layer through the network,
using them to evaluate the derivatives of E with respect to every weight.

2.4 Deriving Back-Propagation

Consider a single training pattern and error term En = (1/2) Σk (yk − tk)². We want ∂En/∂wji for a general weight
wji connecting unit i to unit j. Because wji only influences En through the summed input aj = Σi wji zi (where zi is
the output of the sending unit, or xi if i is an input unit), the chain rule gives:

∂En/∂wji = (∂En/∂aj) · (∂aj/∂wji)

Define the error term (delta):

δj ≡ ∂En/∂aj
Since aj = Σi wji zi, we have ∂aj/∂wji = zi, so:

Page 7 of 16
 ∂En/∂wji = δj zi
Regression — Unit II Study Notes

This says: the derivative of the error with respect to a weight is simply the product of the δ (error signal) at the
receiving unit and the activation z at the sending unit. The whole algorithm therefore reduces to finding δ for
every unit in the network.

2.4.1 Output Layer δ's

For output units, using the sum-of-squares error and identity output activation (regression case), yk = ak, so:

δk = ∂En/∂ak = yk − tk
2.4.2 Hidden Layer δ's — Back-Propagation Formula
For a hidden unit j, aj influences En only via the aks of all the units k to which unit j sends connections. Applying
the chain rule and summing over all such k:

δj = ∂En/∂aj = Σk (∂En/∂ak)(∂ak/∂aj) = h'(aj) Σk wkj δk

This is the central back-propagation formula. It states that the δ for a hidden unit is obtained by taking the δ's of
the units in the next layer forward, weighting them by the connecting weights wkj, summing them, and
multiplying by h'(aj), the derivative of the hidden unit's activation function. This is precisely why the algorithm is
called 'back-propagation': the δ values are literally propagated backwards through the network, from the output
layer toward the input layer, using the same weights that were used going forward.

2.4.3 Summary of the Algorithm

1. Apply an input vector xn to the network and forward-propagate through the network to find the
activations of all hidden and output units.
2. Evaluate the δk for all output units using δk = yk − tk (for sum-of-squares error with identity/linear output).
3. Back-propagate the δ's using δj = h'(aj) Σk wkj δk to obtain δj for every hidden unit.
4. Evaluate the required derivatives: ∂En/∂wji = δj zi.
5. Update each weight, e.g. using gradient descent: wji ← wji − η ∂En/∂wji, where η is the learning rate.
Repeat over all training patterns (an epoch) and iterate until convergence.

2.4.4 Derivative of Common Activation Functions

Activation h(a) h'(a)

Sigmoid: 1/(1+e^−a) h(a) · (1 − h(a))

Tanh 1 − h(a)²
ReLU: max(0,a) 1 if a > 0, else 0

2.5 Multi-Layer Perceptron in Practice

2.5.1 Practical Training Considerations
• Weight initialisation: weights are initialised to small random values (not zero) to break symmetry between
hidden units.
• Learning rate (η): too large causes divergence/oscillation; too small causes very slow convergence.
Adaptive methods (e.g. Adam, RMSProp) adjust η automatically per-parameter.

Page 8 of 16
 Regression — Unit II Study Notes

• Momentum: adds a fraction of the previous update to the current one, helping escape shallow local
minima and speeding convergence.
• Batch vs stochastic vs mini-batch gradient descent: batch uses the whole dataset per update (stable but
slow); stochastic uses one pattern at a time (noisy but fast, can escape local minima); mini-batch is the
common practical compromise.
• Early stopping: training is halted when validation error starts increasing, to prevent overfitting.
• Regularisation: weight decay (L2 penalty on weights) or dropout can be added to reduce overfitting,
analogous to ridge regression.
• Number of hidden units / layers: controls model capacity; chosen via validation performance.
• Local minima and saddle points: the error surface is non-convex, so gradient descent can converge to a
local minimum; multiple random restarts or momentum-based methods help mitigate this.

2.5.2 Examples of Using the MLP

Regression example — House price prediction: Inputs x = (area, number of rooms, location index, age of
property, ...). A single hidden layer MLP with, say, 10 tanh hidden units and one linear output unit is trained by
back-propagation on historical sale prices to predict the price of a new house. The linear (identity) output
activation is essential here so the network can output unbounded real values.
Regression example — Curve fitting: An MLP with a handful of hidden sigmoidal units can be trained to
approximate a smooth non-linear function such as sin(2πx) from noisy samples, illustrating the universal
approximation property directly — the network learns its own adaptive basis functions zj = h(aj) which, when
linearly combined, reconstruct the curve.
Classification example: For a two-class problem, the MLP uses a single output unit with logistic sigmoid
activation, trained with cross-entropy error instead of sum-of-squares; for K > 2 classes, K output units with a
softmax activation and cross-entropy error are used. The forward and backward propagation equations are
structurally identical — only the output activation function and error function change.
Time-series / sensor data example: MLPs are used for short-term forecasting (e.g., predicting the next value of a
temperature or stock-price series from a fixed window of past values) by treating the lagged values as the input
vector x.

Page 9 of 16
 Regression — Unit II Study Notes

3. Radial Basis Functions and Splines

3.1 Concepts
Radial Basis Function (RBF) methods are another way of building flexible, non-linear regression models on top of
the linear basis-function framework of Section 1.4. The key idea is to use basis functions φj(x) that depend on
the input only through its distance from a centre (or 'prototype') point μj:

φj(x) = φ( ‖x − μj‖ )
so that the basis function's value depends only on the radial distance from x to μj, hence 'radial' basis function.
The most common choice is the Gaussian:

φj(x) = exp( − ‖x − μj‖² / (2 sj²) )

Other common radial functions include the multiquadric φ(r) = √(r² + c²), the inverse multiquadric, and the thin-
plate spline φ(r) = r² log r.

3.1.1 Splines and Exact Interpolation

Historically, RBFs arose from the problem of exact interpolation: given N data points {xn, tn}, find a function f
such that f(xn) = tn exactly for every n. Splines address a related 1-D version of this problem by fitting smooth
piecewise polynomials between data points, subject to continuity constraints on the function value and its
derivatives at the 'knots' (join points). The RBF exact-interpolation solution generalises this idea to arbitrarily
many dimensions by placing one basis function centred exactly at each data point:

f(x) = Σn=1..N wn φ( ‖x − xn‖ )

Solving f(xn) = tn for all n gives a linear system Φw = t, where Φnj = φ(‖xn − xj‖), which can be solved exactly for
w provided Φ is invertible (guaranteed for many choices of φ, e.g. Gaussian, by Micchelli's theorem). In practice,
exact interpolation of noisy data is undesirable (it fits the noise exactly), so this is used more as a conceptual
bridge to the RBF network below, which uses far fewer basis functions than data points and includes
regularisation.

3.2 RBF Networks

An RBF Network is a feed-forward network with a single hidden layer of RBF units and a linear output layer. Its
structure and training differ from the MLP in important ways:

Aspect MLP RBF Network

Hidden unit activation Sigmoidal / global, depends on Radial / local, depends on ‖x − μj‖
wᵀx (a hyperplane) (distance from a centre)
Hidden layer training All weights trained jointly by back- Centres μj & widths sj usually set
propagation (supervised, first (unsupervised, e.g. k-means /
iterative) clustering); output weights fit
afterwards
Output layer Non-linear in general Linear combination of hidden
outputs
Nature of representation Distributed, global features Localised, local features (each unit

Page 10 of 16
 Regression — Unit II Study Notes

Aspect MLP RBF Network

responds to a region)
The output of an RBF network for regression is simply a weighted sum of the RBF activations:

y(x) = Σj=1..M wj φj(x) + w0

Because y is linear in the output weights w (even though φj(x) is highly non-linear in x), once the centres μj and
widths sj are fixed, the optimal output weights can be found by ordinary linear least squares — exactly as in
Section 1.4 — making training of the output layer fast, and in principle avoiding many of the local-minima issues
that back-propagation faces in the MLP.

3.2.1 Two-Stage Training Procedure

1. Stage 1 — Determine the basis functions (unsupervised): choose the number M of hidden units, and set
their centres μj (e.g. via k-means clustering of the input data, or by randomly selecting a subset of training
points) and widths sj (e.g. based on the average distance to nearby centres).
2. Stage 2 — Determine the output weights (supervised): with φj(x) now fixed, solve the linear least-squares
problem for w (optionally with ridge-style regularisation) exactly as for a linear basis-function model.

3.2.2 RBF vs MLP — Practical Comparison

• RBF networks typically train much faster because output-weight fitting is a single linear solve, not iterative
gradient descent.
• MLPs tend to generalise better in high dimensions because their sigmoidal units span the whole input
space (global support), whereas RBF units are local and their number can grow rapidly with dimensionality.
• RBF networks are naturally suited to problems where locality/interpolation-like behaviour is desired (e.g.,
function approximation with smooth, localised structure).
• MLPs are usually preferred for very high-dimensional, complex tasks (e.g. deep learning applications), while
RBF networks remain popular for smaller-scale interpolation and control-system applications.

3.3 The Curse of Dimensionality

The curse of dimensionality refers to the collection of problems that arise when working with data in high-
dimensional input spaces, and is central to understanding why localised basis functions (like those in RBF
networks) become inefficient as dimensionality D grows.
• Volume growth: to cover a fixed fraction of the volume of a D-dimensional space with local regions of fixed
size, the number of regions needed grows exponentially with D. E.g., dividing each of D input variables into
a fixed number of intervals partitions the space into a number of cells that grows as (that number)^D.
• Data sparsity: for a fixed number of training examples N, the data becomes increasingly sparse relative to
the volume of the space as D increases — most of the volume of a high-dimensional space is 'empty', so
local methods (e.g., nearest-neighbour, RBF with local support) see very few training points near any given
query point.
• Concentration of distance: in high dimensions, the ratio between the distance to the nearest and farthest
neighbour tends to 1, i.e. distances between points become less discriminative, undermining any method
relying on notions of 'closeness' (this directly affects RBF centres/widths and k-NN-style methods).
• Number of RBF units required: to maintain the same resolution/coverage of an RBF network as
dimensionality increases, the number of basis functions M required grows exponentially, making purely
localised approaches impractical in high-D without dimensionality reduction or very careful centre
placement.
Page 11 of 16
 Regression — Unit II Study Notes

Why it matters here: It explains why RBF networks (relying on local basis functions) can struggle in high
dimensions, motivating the use of globally-supported hidden units (as in the MLP) or dimensionality-reduction
techniques as preprocessing.
Despite this pessimistic geometric picture, real-world high-dimensional data is often well-behaved because: (i)
the data typically lies on, or near, a lower-dimensional manifold embedded in the high-dimensional space (the
manifold hypothesis); and (ii) real data typically exhibits strong correlations between input variables, so the
effective (intrinsic) dimensionality is much smaller than D. Both linear basis-function models and neural
networks can exploit this structure to give useful predictions in practice.

3.4 Interpolation and Basis Functions

This section ties together the basis-function view of regression seen throughout the unit.

3.4.1 The General Basis-Function Regression Model

Any of the models discussed (polynomial regression, RBF networks, and even, loosely, an MLP with one hidden
layer and linear output) can be written in the common form:

y(x) = Σj=0..M wj φj(x) (φ0(x) = 1, the bias term)

The difference between models lies purely in how the basis functions φj are chosen and how flexible/adaptive
they are:

Model Basis functions φj(x) Adaptive?

Polynomial regression x^j Fixed, chosen in advance

Gaussian basis regression exp(−‖x−μj‖²/2s²) Centres/widths often fixed in
advance
RBF network Radial φ(‖x − μj‖) Centres set by unsupervised stage
(semi-adaptive)
MLP (1 hidden layer) h(wjᵀx + w0j) Fully adaptive — learned jointly
with output weights

3.4.2 Interpolation vs Approximation

Exact interpolation (Section 3.1.1) fits the training data exactly, which requires as many basis functions as data
points and is highly sensitive to noise. Approximation (used by regularised linear models, RBF networks with M
<< N, and MLPs) uses far fewer basis functions than data points and relies on optimisation (least squares /
gradient descent) plus regularisation to trade off fit against smoothness, giving much better generalisation to
unseen data. This trade-off is the same bias–variance principle introduced in Section 1.6.

3.4.3 Splines as Piecewise Basis Functions

A spline of degree p is a piecewise polynomial function that is continuous and has continuous derivatives up to
order p−1 at the knots. It can itself be expressed as a linear combination of local basis functions (e.g., B-splines),
each non-zero only over a small number of adjacent intervals:

f(x) = Σj wj Bj(x)
This again reduces the fitting problem to a linear least-squares problem in the coefficients wj once the basis
functions Bj (i.e. the knot positions) are fixed — directly paralleling the RBF network's two-stage approach.

Page 12 of 16
 Regression — Unit II Study Notes

4. Support Vector Machines (SVM)

4.1 Motivation
Support Vector Machines were originally developed for classification, but the same margin-based, kernel-based
framework extends naturally to regression, known as Support Vector Regression (SVR). SVMs are included in this
unit as a contrasting approach to MLPs and RBF networks: rather than minimising a sum-of-squares or cross-
entropy error over all points, SVMs are formulated as a convex optimisation problem, guaranteeing a unique
global solution (no local minima, unlike MLP training).

4.2 Maximum Margin Classifier

For a linearly separable two-class dataset {(xn, tn)}, tn ∈ {−1, +1}, an SVM seeks the separating hyperplane wᵀx +
b = 0 that maximises the margin — the distance to the nearest training points of either class. Points lying exactly
on the margin boundaries are called support vectors, since they alone determine the position of the optimal
hyperplane.

maximise (over w, b): margin = 2 / ‖w‖ subject to tn(wᵀxn + b) ≥ 1 for all n

This is equivalent to minimising ‖w‖² subject to the same constraints — a convex quadratic programming
problem with a unique solution.

4.3 Soft Margin and the C Parameter

Real data is rarely perfectly separable. The soft-margin SVM introduces slack variables ξn ≥ 0 that allow some
points to violate the margin (or even be misclassified), penalised in the objective:

minimise (1/2)‖w‖² + C Σn ξn subject to tn(wᵀxn + b) ≥ 1 − ξn, ξn ≥ 0

The regularisation constant C controls the trade-off between maximising the margin (small C) and minimising
training misclassifications/slack (large C) — directly analogous to the role of λ in ridge regression, but acting in
the opposite direction.

4.4 The Kernel Trick

For data that is not linearly separable in the original input space, SVMs implicitly map inputs into a higher-
dimensional feature space φ(x) where a linear separator may exist. Because the optimisation problem (and the
resulting decision function) depends on the data only through inner products, one can replace the inner product
with a kernel function without ever computing φ(x) explicitly:

k(xn, xm) = φ(xn)ᵀφ(xm)

Common kernels include:

Kernel Formula Notes

Linear k(x,x') = xᵀx' Equivalent to the original linear

SVM
Polynomial k(x,x') = (xᵀx' + c)^d Captures interaction terms up to
degree d

Page 13 of 16
 Regression — Unit II Study Notes

Kernel Formula Notes

Gaussian / RBF k(x,x') = exp(−‖x−x'‖²/2σ²) Same functional form as an RBF

basis function — connects directly
back to Section 3
Sigmoid k(x,x') = tanh(κ xᵀx' + θ) Resembles an MLP-style
activation
Connection across the unit: The RBF kernel gives an SVM decision function of the form f(x) = Σn αn tn k(x, xn) +
b, which is structurally identical to the RBF network output y(x) = Σj wj φj(x) — an SVM with a Gaussian kernel
can be viewed as an RBF network where the centres are the (automatically selected) support vectors, rather
than being chosen up-front by clustering.

4.5 Support Vector Regression (SVR)

SVR adapts the SVM idea to regression by defining an ε-insensitive loss function: errors smaller than ε are
ignored (not penalised at all), and only errors larger than ε contribute to the loss, linearly:

Lε(y, t) = 0 if |y − t| ≤ ε Lε(y, t) = |y − t| − ε otherwise

This creates a 'tube' of width 2ε around the regression function within which errors are tolerated. As with
classification SVMs, only points outside (or on the boundary of) this tube — the support vectors — influence the
fitted function, giving a typically sparse solution (many αn coefficients are exactly zero). The optimisation
problem again reduces to a convex quadratic program, solvable with kernels exactly as in Section 4.4.

4.6 SVM vs MLP vs RBF Network — Summary Comparison

Property MLP RBF Network SVM / SVR

Optimisation Non-convex (gradient Convex for output layer Convex (quadratic

descent / back-prop) (linear LS); centres set programming) — global
heuristically optimum
Basis functions Adaptive, learned jointly Semi-adaptive (centres Implicit, via kernel;
via clustering) effectively centred at
support vectors
Risk of local minima Yes No (once centres fixed) No
Scalability Good for large datasets, Can suffer from curse of Training can be costly for
deep architectures dimensionality very large N (kernel
matrix is N×N)
Interpretability of Low (distributed Medium (localised Medium (sparse set of
solution weights) centres) support vectors)

Page 14 of 16
 Regression — Unit II Study Notes

5. Quick Revision Summary

5.1 Core Formulas at a Glance

Topic Key Formula

Linear regression (matrix form) w* = (XᵀX)⁻¹ Xᵀt

Ridge regression w* = (XᵀX + λI)⁻¹ Xᵀt
Linear basis function model y(x,w) = Σj wj φj(x)
MLP forward pass (1 hidden layer) yk = σ( Σj w(2)kj h(Σi w(1)ji xi + w(1)j0) + w(2)k0 )
Back-prop: output δ δk = yk − tk
Back-prop: hidden δ δj = h'(aj) Σk wkj δk
Back-prop: gradient ∂En/∂wji = δj zi
Gaussian RBF φj(x) = exp(−‖x−μj‖²/2sj²)
RBF network output y(x) = Σj wj φj(x) + w0
SVM margin objective minimise ‖w‖² s.t. tn(wᵀxn+b) ≥ 1
SVR ε-insensitive loss Lε = max(0, |y−t| − ε)

5.2 Conceptual Map

All models in this unit can be seen as instances of one underlying idea — predicting a target as a (possibly non-
linear) combination of basis functions of the input — differing chiefly in how those basis functions are chosen:
• Linear regression: basis functions = raw inputs (or fixed transformations), weights found by least squares.
• MLP: basis functions = hidden unit activations, learned adaptively via back-propagation (non-convex
optimisation).
• RBF network: basis functions = localised radial functions, centres set by clustering, output weights by least
squares (convex for stage 2).
• SVM/SVR: basis functions implicit via a kernel; a sparse subset of training points (support vectors) define
the solution via convex quadratic programming.

5.3 Suggested Practice Questions

1. Derive the normal equation for multiple linear regression from the sum-of-squares error function.
2. Starting from En = (1/2)Σk(yk − tk)², derive the back-propagation formulas for δk and δj step by step.
3. Explain, with a diagram, the difference between forward propagation and backward propagation in an
MLP.
4. Compare RBF networks and MLPs in terms of training procedure, basis function locality, and suitability for
high-dimensional data.
5. Explain the curse of dimensionality and its specific implications for RBF networks.
6. What is the kernel trick? Show how a Gaussian kernel SVM relates to an RBF network.

Page 15 of 16
 Regression — Unit II Study Notes

7. Explain the role of the regularisation parameter λ (ridge regression) and C (SVM) and how each affects the
bias–variance trade-off.

Page 16 of 16