Mathematics for Machine Learning

Comprehensive Week-by-Week Study Notes

Intuition · Derivations · Theorems · Worked Problems

How to use these notes. Each week builds on the previous one. Boxes with a
blue border are theorems or formal results. Orange boxes highlight key ideas that
are often skipped over. Red boxes flag common misconceptions. Green boxes give
precise definitions. Purple boxes contain a worked problem with a full solution.
Prerequisites that you should review before a week’s material appear in grey boxes.

Contents
1 Week 1 – Introduction to Machine Learning 4
1.1 What Is Machine Learning? . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Taxonomy of ML Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.1 Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.2 Unsupervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.3 The Training–Generalisation Distinction . . . . . . . . . . . . . . . 4
1.3 The Supervised Learning Pipeline . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 The Bias–Variance Trade-off (Conceptual) . . . . . . . . . . . . . . . . . . 5

2 Week 2 – Calculus 5
2.1 Why Calculus for ML? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Derivatives and Their Meaning . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 Partial Derivatives and the Gradient . . . . . . . . . . . . . . . . . . . . . 6
2.4 The Chain Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.5 Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.6 Taylor Series and Local Approximations . . . . . . . . . . . . . . . . . . . 7

3 Week 3 – Linear Algebra: Least Squares Regression 8

3.1 The Linear Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2 Ordinary Least Squares (OLS) . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2.1 Deriving the Normal Equations . . . . . . . . . . . . . . . . . . . . 8
3.3 Geometric Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4 Week 4 – Linear Algebra: Eigenvalues and Eigenvectors 10

4.1 Motivation: What Does a Matrix Do? . . . . . . . . . . . . . . . . . . . . 10
Mathematics for Machine Learning 2

4.2 Finding Eigenvalues: The Characteristic Equation . . . . . . . . . . . . . . 10

4.3 The Spectral Theorem (Symmetric Matrices) . . . . . . . . . . . . . . . . . 10
4.4 Positive (Semi-)Definite Matrices . . . . . . . . . . . . . . . . . . . . . . . 11

5 Week 5 – Linear Algebra: Symmetric Matrices 11

5.1 Why Symmetric Matrices Are Everywhere in ML . . . . . . . . . . . . . . 11
5.2 The Spectral Decomposition – Deeper Look . . . . . . . . . . . . . . . . . 12
5.3 Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
5.4 Matrix Functions via Spectral Decomposition . . . . . . . . . . . . . . . . 12

6 Week 6 – SVD and Principal Component Analysis 13

6.1 Singular Value Decomposition (SVD) . . . . . . . . . . . . . . . . . . . . . 13
6.2 Low-Rank Approximation (Eckart–Young Theorem) . . . . . . . . . . . . . 14
6.3 Principal Component Analysis (PCA) . . . . . . . . . . . . . . . . . . . . . 14
6.3.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
6.3.2 PCA via Eigendecomposition of the Covariance . . . . . . . . . . . 14
6.4 PCA for Image Compression / Reconstruction . . . . . . . . . . . . . . . . 14

7 Week 7 – Unconstrained Optimisation 15

7.1 Necessary and Sufficient Conditions for Optimality . . . . . . . . . . . . . 15
7.2 Gradient Descent – Formal Analysis . . . . . . . . . . . . . . . . . . . . . . 16
7.3 Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.4 Stochastic Gradient Descent (SGD) . . . . . . . . . . . . . . . . . . . . . . 16

8 Week 8 – Convex Sets, Functions, and Optimisation 17

8.1 Convex Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
8.2 Convex Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
8.2.1 Equivalent Characterisations . . . . . . . . . . . . . . . . . . . . . . 18
8.3 Examples Relevant to ML . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
8.4 Convex Optimisation Problems . . . . . . . . . . . . . . . . . . . . . . . . 18

9 Week 9 – Constrained Optimisation and Lagrange Multipliers 19

9.1 The Constrained Optimisation Problem . . . . . . . . . . . . . . . . . . . . 19
9.2 Equality Constraints: Lagrange Multipliers . . . . . . . . . . . . . . . . . . 19
9.3 Logistic Regression as an Optimisation Problem . . . . . . . . . . . . . . . 20
9.4 KKT Conditions (Inequality Constraints) . . . . . . . . . . . . . . . . . . . 20

10 Week 10 – Probabilistic Models in Machine Learning 21

10.1 Why Probability? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
10.2 Gaussian Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
10.3 Maximum Likelihood Estimation (MLE) . . . . . . . . . . . . . . . . . . . 21
10.4 MLE for the Gaussian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
10.5 Linear Regression as MLE . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

11 Week 11 – Exponential Family of Distributions 22

11.1 The Exponential Family . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
11.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
11.2.1 Bernoulli as Exponential Family . . . . . . . . . . . . . . . . . . . . 23
11.3 Key Properties of the Exponential Family . . . . . . . . . . . . . . . . . . 23
Mathematics for Machine Learning 3

12 Week 12 – Parameter Estimation and Expectation Maximisation 24

12.1 The Problem of Latent Variables . . . . . . . . . . . . . . . . . . . . . . . 24
12.2 Jensen’s Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
12.3 The Evidence Lower Bound (ELBO) . . . . . . . . . . . . . . . . . . . . . 25
12.4 EM for Gaussian Mixture Models (GMMs) . . . . . . . . . . . . . . . . . . 25

Appendix: Quick-Reference Summary 27

Mathematics for Machine Learning 4

1 Week 1 – Introduction to Machine Learning

1.1 What Is Machine Learning?
Machine learning (ML) is the study of algorithms that improve their performance on a
task through experience. Unlike classical programming where you hard-code rules, in ML
you provide data and a learning algorithm finds the rules automatically.

The Three Pillars of Machine Learning

ˆ Task (T ): What the system should do (e.g. classify emails as spam or not-spam).
ˆ Experience (E): Data the system learns from (e.g. labelled emails).
ˆ Performance measure (P ): How well the system does the task (e.g. classification
accuracy).
A program is said to learn if its P on T improves with E.

1.2 Taxonomy of ML Problems

1.2.1 Supervised Learning
You are given pairs {(xi , yi )}ni=1 where xi is an input vector (features) and yi is the
corresponding label. Goal: learn a mapping f : x 7→ y.
ˆ Regression: y ∈ R (e.g. predicting house price).
ˆ Classification: y ∈ {1, . . . , K} (e.g. digit recognition).

1.2.2 Unsupervised Learning

You only have inputs {xi }; there are no labels. Goal: discover structure (clusters, latent
factors, density, . . . ).

1.2.3 The Training–Generalisation Distinction

Training Error vs. Generalisation Error

The training error is how well the model fits the data it was trained on. The
generalisation error (test error) is how well it performs on new, unseen data.
The ultimate goal of ML is low generalisation error, not low training error. A model
that perfectly memorises training data but fails on new data has overfit.

1.3 The Supervised Learning Pipeline

features f (·;θ) L(ŷ,y) optimiser
| {zdata} −−−−→ |x ∈
Raw d
{zR} −−−→ |{z}
ŷ −−−→ |{z}
loss −−−−−→ θ∗
|{z}
collection feature vector prediction scalar learned params

The loss function L quantifies prediction error. For regression the most common choice
is squared error: L(ŷ, y) = (ŷ − y)2 . The optimiser adjusts the parameters θ to minimise
the total loss over the training set.
Mathematics for Machine Learning 5

1.4 The Bias–Variance Trade-off (Conceptual)

Intuition
Imagine fitting a curve to noisy data.
ˆ High bias (underfitting): Your model is too simple (e.g. a straight line
through data that curves). It makes the same systematic error regardless of the
training set.
ˆ High variance (overfitting): Your model is too complex and wiggles through
every training point. Small changes in the training set cause large changes in
the learned function.
The sweet spot lies in the middle. This trade-off is formalised in Week 3 once we have
the tools of linear algebra.

Common Misconception
“More data always makes the model better.” – Partially true. More data reduces
variance but does not fix high bias (a structurally wrong model). And data quality
matters as much as quantity.

Problem 1.1 – Spotting Overfitting/Underfitting

Problem. A model trained on 1000 examples achieves 99% training accuracy but
only 60% test accuracy. A second model achieves 75% training accuracy and 74% test
accuracy.
(a) Which model is overfitting? Which is underfitting (if any)?
(b) What action would you take to fix each problem?

Solution.
(a) Model 1 shows a 39% gap between training and test accuracy – a classic sign
of overfitting: the model memorised the training data but does not generalise.
Model 2 has nearly equal (and moderate) training and test accuracy; neither is
very high, suggesting the model has high bias (underfitting) – it is not complex
enough to capture the true pattern.
(b) For Model 1: regularise (add a penalty on large weights, e.g. L2/ridge), use
dropout, reduce model capacity, or collect more training data. For Model 2:
increase model complexity (more features, higher-degree polynomial, deeper
network), engineer better features, or train longer.

2 Week 2 – Calculus
Prerequisites
Familiarity with functions of one variable, limits, and basic derivative rules (power
rule, chain rule).
Mathematics for Machine Learning 6

2.1 Why Calculus for ML?

Training a model means minimising a loss function L(θ). To know which direction to
move θ to decrease L we need derivatives – which tell us the slope of the loss landscape.

2.2 Derivatives and Their Meaning

For a function f : R → R the derivative at a point x0 is
f (x0 + h) − f (x0 )
f ′ (x0 ) = lim .
h→0 h
Geometrically it is the slope of the tangent line at x0 .

Sign of the Derivative as a Navigation Signal

ˆ f ′ (x0 ) > 0: f is increasing at x0 . Move left (decrease x) to decrease f .
ˆ f ′ (x0 ) < 0: f is decreasing at x0 . Move right (increase x) to decrease f .
ˆ f ′ (x0 ) = 0: Possible minimum, maximum, or saddle point.

2.3 Partial Derivatives and the Gradient

When f : Rd → R depends on several variables θ1 , . . . , θd , the partial derivative with
respect to θj is obtained by treating all other variables as constants:
∂f f (θ + h ej ) − f (θ)
(θ) = lim ,
∂θj h→0 h
where ej is the j-th standard basis vector.

Gradient
The gradient of f at θ is the vector of all partial derivatives:
 
∂f /∂θ1
∇θ f = ∇f (θ) =  ...  ∈ Rd .
 
∂f /∂θd

The gradient points in the direction of steepest ascent of f .

2.4 The Chain Rule

If g : R → R and f : R → R are differentiable, then
d
[f (g(x))] = f ′ (g(x)) · g ′ (x).
dx
In the multivariate setting: if z = g(x) and L = f (z), then
∂L X ∂L ∂zk
= .
∂xi k
∂zk ∂xi

This is the foundation of backpropagation in neural networks.

Mathematics for Machine Learning 7

2.5 Gradient Descent

Gradient Descent Update Rule
To minimise L(θ), repeatedly apply:

θ (t+1) ← θ (t) − η ∇θ L θ (t) ,

where η > 0 is the learning rate (step size). The negative sign ensures we move in
the direction of steepest descent.

Learning Rate Misconception

“A smaller learning rate always gives a better result.” – False. A very small η makes
convergence painfully slow and can trap you in bad local minima for a very long time.
A very large η causes the updates to overshoot and diverge. Choosing η is a critical
hyperparameter decision.

2.6 Taylor Series and Local Approximations

A smooth function f near a point x0 can be approximated as:
f ′′ (x0 ) 2
f (x0 + δ) ≈ f (x0 ) + f ′ (x0 ) δ + δ + ···
2
The second-order term tells us about curvature: if f ′′ (x0 ) > 0 then x0 is a (local)
minimum.

Second Derivative Test

For f : R → R:
ˆ f ′ (x0 ) = 0 and f ′′ (x0 ) > 0: x0 is a local minimum.
ˆ f ′ (x0 ) = 0 and f ′′ (x0 ) < 0: x0 is a local maximum.
ˆ f ′ (x0 ) = 0 and f ′′ (x0 ) = 0: Test is inconclusive (could be an inflection point).

Problem 2.1 – Gradient Descent by Hand

Problem. Let f (θ) = (θ − 3)2 + 1. Starting at θ(0) = 0 with learning rate η = 0.4,
perform 3 steps of gradient descent and verify you are approaching the minimum.

Solution. First, compute the derivative: f ′ (θ) = 2(θ − 3).

Step 0 → 1:

θ(1) = θ(0) − η f ′ (θ(0) ) = 0 − 0.4 · 2(0 − 3) = 0 + 2.4 = 2.4.

Step 1 → 2:

θ(2) = 2.4 − 0.4 · 2(2.4 − 3) = 2.4 − 0.4 · (−1.2) = 2.4 + 0.48 = 2.88.

Step 2 → 3:

θ(3) = 2.88 − 0.4 · 2(2.88 − 3) = 2.88 + 0.096 = 2.976.

Mathematics for Machine Learning 8

The iterates 0 → 2.4 → 2.88 → 2.976 → · · · are converging to the true minimum
θ∗ = 3. Each step reduces the gap to θ∗ by a factor of |1 − 2η| = |1 − 0.8| = 0.2.

3 Week 3 – Linear Algebra: Least Squares Regression

Prerequisites
Vectors, dot products, matrix multiplication. If unfamiliar, think of a vector as a list
of numbers and matrix multiplication as a generalised dot product.

3.1 The Linear Model

Given n data points {(xi , yi )} with xi ∈ Rd and yi ∈ R, a linear model predicts:

ŷi = θ ⊤ xi = θ1 xi1 + · · · + θd xid .

Stacking all predictions into a matrix equation:

 
x⊤ 1
ŷ = Xθ, X =  ...  ∈ Rn×d .
 
x⊤
n

X is called the design matrix.

3.2 Ordinary Least Squares (OLS)

We want to find θ that minimises the residual sum of squares (RSS):
n
X
L(θ) = ∥y − Xθ∥2 = (yi − θ ⊤ xi )2 .
i=1

3.2.1 Deriving the Normal Equations

We minimise L by setting its gradient to zero. First, expand:

L(θ) = (y − Xθ)⊤ (y − Xθ) = y⊤ y − 2θ ⊤ X⊤ y + θ ⊤ X⊤ Xθ.

Taking the gradient with respect to θ and setting it to zero:

∇θ L = −2X⊤ y + 2X⊤ Xθ = 0.

Normal Equations
The minimiser of the OLS problem satisfies:
X⊤ X θ ∗ = X⊤ y.
If X⊤ X is invertible (i.e. the columns of X are linearly independent), the unique
Mathematics for Machine Learning 9

solution is:
θ ∗ = (X⊤ X)−1 X⊤ y.
The matrix (X⊤ X)−1 X⊤ is called the Moore–Penrose pseudoinverse of X.

3.3 Geometric Interpretation

Intuition
The predictions ŷ = Xθ must lie in the column space of X (the set of all vectors you
can write as Xθ for some θ). Since y may not be in this subspace, OLS finds the point
in the column space closest to y. This closest point is the orthogonal projection of
y onto the column space. The residual r = y − ŷ is perpendicular to every column of
X, which is exactly the condition X⊤ r = 0, i.e. the normal equations.

The Hat Matrix / Projection Matrix

The matrix H = X(X⊤ X)−1 X⊤ is called the hat matrix because ŷ = Hy (it “puts a
hat” on y). It is an orthogonal projection matrix, satisfying H2 = H (idempotent)
and H⊤ = H (symmetric). These two properties characterise any projection matrix.

Common Misconception
“To fit a linear model you just add a column of ones.” – Almost right. Adding a column
of ones to X allows the model to include a bias/intercept θ0 , so ŷ = θ0 + θ1 x1 + · · · .
Without this column the regression line is forced through the origin, which is rarely
appropriate. Always include the intercept unless you have a very specific reason not to.

Problem 3.1 – Deriving OLS for Simple Linear Regression

Problem. For n data points (x1 , y1 ), . . . , (xn , yn ) in R2 , fit the model ŷ = ax + b. Use
the normal equations to derive closed-form expressions for a∗ and b∗ .

x1 1  
 .. ..  a
Solution. Write X =  . .  and θ = .
b
xn 1
Compute: P 2 P  P 
⊤ xi xi ⊤ xi yi
X X= P , X y= P .
xi n yi
Let x̄ = n1 xi and ȳ = n1
P P
yi . Solving the 2 × 2 system gives:
Pn
(x − x̄)(yi − ȳ)
∗
Pn i
a = i=1 2
, b∗ = ȳ − a∗ x̄.
i=1 (xi − x̄)

Interpretation: a∗ is the ratio of the covariance of x and y to the variance of x. The

fitted line always passes through the centroid (x̄, ȳ).
Mathematics for Machine Learning 10

4 Week 4 – Linear Algebra: Eigenvalues and Eigen-

vectors
Prerequisites
Matrix–vector multiplication, determinants (2×2 case), concept of linear independence.

4.1 Motivation: What Does a Matrix Do?

A matrix A ∈ Rn×n acts on a vector v to produce a new vector Av. In general, this both
rotates and scales the input. Eigenvectors are the special directions that are only scaled –
not rotated.

Eigenvalue / Eigenvector
A non-zero vector v ∈ Rn is an eigenvector of matrix A with associated eigenvalue
λ ∈ R (or C) if:
Av = λ v.
The matrix A stretches v by factor λ along the same direction. If λ < 0 the direction
is reversed.

4.2 Finding Eigenvalues: The Characteristic Equation

Rearrange: (A − λI)v = 0. For a non-trivial solution v ̸= 0, the matrix (A − λI) must be
singular :
det(A − λI) = 0.
This polynomial in λ is the characteristic polynomial. Its roots are the eigenvalues.

Intuition
Why det = 0? A matrix is singular iff it maps some non-zero vector to 0. We want
(A − λI) to map v to 0 – so it must be singular, hence det = 0.

4.3 The Spectral Theorem (Symmetric Matrices)

Spectral Theorem
If A ∈ Rn×n is symmetric (A⊤ = A), then:
(i) All eigenvalues of A are real.
(ii) Eigenvectors corresponding to distinct eigenvalues are orthogonal.
(iii) A can be decomposed as A = QΛQ⊤ , where Q is orthogonal (Q⊤ Q = I) and
Λ = diag(λ1 , . . . , λn ).

This decomposition is called the eigendecomposition or spectral decomposition. It

says: every symmetric matrix is just a rotation (Q⊤ ), a rescaling (Λ), and the reverse
rotation (Q).
Mathematics for Machine Learning 11

4.4 Positive (Semi-)Definite Matrices

Positive Definite / Positive Semi-Definite

A symmetric matrix A is:
ˆ Positive definite (PD) if v⊤ Av > 0 for all v ̸= 0. Equivalently, all eigenvalues
λi > 0.
ˆ Positive semi-definite (PSD) if v⊤ Av ≥ 0 for all v. Equivalently, all
eigenvalues λi ≥ 0.

Why PD Matters for ML

The matrix X⊤ X from OLS is always PSD. It is PD (and hence invertible) iff the
columns of X are linearly independent (no multicollinearity). Covariance matrices
are always PSD. The Hessian of a convex function is always PSD – a fact we exploit
heavily in optimisation.

Problem 4.1 – Eigendecomposition of a 2 × 2 Matrix

 
4 2
Problem. Let A = . Find all eigenvalues and eigenvectors. Is A positive
2 1
definite?

Solution.
Step 1 – Eigenvalues.

det(A − λI) = (4 − λ)(1 − λ) − 4 = λ2 − 5λ + 4 − 4 = λ2 − 5λ = λ(λ − 5) = 0.

So λ1 = 0 and λ2 = 5.
Step
 2 – Eigenvectors.
 For λ1 = 0: (A −0 · I)v
 = Av = 0. Row reduce:
4 2 2 1 1
→ , giving 2v1 + v2 = 0, so v1 = .
2 1 0 0   −2  
−1 2 2
For λ2 = 5: (A − 5I)v = v = 0. Row reduce: v1 = 2v2 , so v2 = .
2 −4 1
Check orthogonality: v1⊤ v2 = 1 · 2 + (−2) · 1 = 0. ✓
Step 3 – Positive Definiteness. Since λ1 = 0 (not strictly positive), A is positive
semi-definite, but not positive definite. Indeed, A is singular (det A = 0), confirming
it is not invertible.

5 Week 5 – Linear Algebra: Symmetric Matrices

Prerequisites
Eigenvalues and eigenvectors (Week 4), matrix transpose.

5.1 Why Symmetric Matrices Are Everywhere in ML

Several key matrices in ML are symmetric by construction:
Mathematics for Machine Learning 12

ˆ Covariance matrix: Σ = E[(x − µ)(x − µ)⊤ ]. Note (x − µ)(x − µ)⊤ is an outer

product, always symmetric PSD.
ˆ Gram matrix / kernel matrix: Kij = k(xi , xj ).
ˆ Hessian of a twice-differentiable function: Hij = ∂ 2 f /∂θi ∂θj = Hji (Clairaut’s
theorem).
ˆ X⊤ X from OLS.

5.2 The Spectral Decomposition – Deeper Look

Pn
From the spectral theorem: A = QΛQ⊤ = k=1 λk qk q⊤
k.

Each term λk qk q⊤ k is a rank-1 matrix (an outer product) that projects onto the k-th
eigenvector direction and scales by λk . The symmetric matrix is thus a weighted sum of
rank-1 projections.

Low-Rank Approximation
If only the top r < n eigenvalues are large (the rest near zero), we can approximate:
r
X
A≈ λk q k q ⊤
k.
k=1

This low-rank approximation is at the heart of dimensionality reduction methods like

PCA (Week 6).

5.3 Quadratic Forms

A quadratic form is an expression Q(v) = v⊤ Av. Using the spectral decomposition
with u = Q⊤ v:
n
X
⊤ ⊤ ⊤ ⊤ ⊤ ⊤ ⊤
v Av = v (QΛQ )v = (Q v) Λ(Q v) = u Λu = λk u2k .
k=1

This is always non-negative when all λk ≥ 0 – confirming that PSD ⇔ v⊤ Av ≥ 0.

5.4 Matrix Functions via Spectral Decomposition

Because the eigendecomposition diagonalises A, we can define functions of matrices
naturally:
Ak = QΛk Q⊤ , A−1 = QΛ−1 Q⊤ (if all λk ̸= 0), A1/2 = QΛ1/2 Q⊤ (if all λk ≥ 0).

Trace and Determinant via Eigenvalues

For any square matrix A with eigenvalues λ1 , . . . , λn :
n
X n
Y
tr(A) = λk , det(A) = λk .
k=1 k=1

These are coordinate-free invariants: they do not change if you change basis.
Mathematics for Machine Learning 13

Problem 5.1 – Covariance Matrix and Its Eigenstructure

 
5 4
Problem. You have two-dimensional data with sample covariance Σ = . Find
4 5
the eigenvalues and eigenvectors. What do they tell you about the shape of the data?

Solution.
2
Characteristic polynomial: (5 − λ) − 16= 0 ⇒ (5 − λ) = ±4, giving λ 1 =9, λ2 = 1.
−4 4 1
For λ1 = 9: (Σ − 9I)v = 0 ⇒ v = 0 ⇒ v1 = v2 , so q1 = √12 .
  4 −4 1
1 1
For λ2 = 1: q2 = 2 √ .
−1
Interpretation. The eigenvectors√are the principal axes of the data’s covariance
ellipse. Along direction q1 = (1, 1)/ 2 (the 45◦ diagonal) the variance is λ1 = 9 – the
data spreads widely. Along the perpendicular direction q2 the variance is only λ2 = 1.
The
√ data
√ lives in an elongated ellipse pointing north-east, 3 times as long as it is wide
( 9/ 1 = 3).

6 Week 6 – SVD and Principal Component Analysis

Prerequisites
Eigendecomposition (Weeks 4–5), covariance matrices.

6.1 Singular Value Decomposition (SVD)

Eigendecomposition only works for square matrices. The SVD generalises it to any m × n
matrix.

SVD Theorem
Every matrix A ∈ Rm×n can be written as:

A = UΣV⊤ ,

where:
ˆ U ∈ Rm×m is orthogonal (columns uk : left singular vectors).
ˆ Σ ∈ Rm×n is diagonal with non-negative entries σ1 ≥ σ2 ≥ · · · ≥ 0 (singular
values).
ˆ V ∈ Rn×n is orthogonal (columns vk : right singular vectors).

SVD as Three Geometric Steps

Ax = UΣV⊤ x can be read as three operations applied in sequence:
1. V⊤ x: Rotate the input to align with the right singular vectors.
2. Σ(·): Scale each coordinate by the corresponding singular value.
3. U(·): Rotate into the output space.
Mathematics for Machine Learning 14

The singular values quantify how much A stretches space along each principal direction.

6.2 Low-Rank Approximation (Eckart–Young Theorem)

Eckart–Young–Mirsky Theorem
The best rank-r approximation to A in the Frobenius norm (or spectral norm) is:
r
X
Ar = σk uk vk⊤ .
k=1

Pmin(m,n)
The approximation error is ∥A − Ar ∥2F = k=r+1 σk2 .

Application: image compression. A 1000 × 1000 image (matrix) can be well-approximated

by keeping only the top r = 20 singular values, reducing storage by a factor of 1000/(2 ·
20 + 20) ≈ 17.

6.3 Principal Component Analysis (PCA)

Goal: Find a low-dimensional representation of high-dimensional data that captures
maximum variance.

6.3.1 Setup
Let X ∈ Rn×d be the centred data matrix (subtract the mean from each column). The
1
sample covariance is S = n−1 X⊤ X.

6.3.2 PCA via Eigendecomposition of the Covariance

Compute the eigendecomposition S = QΛQ⊤ with λ1 ≥ · · · ≥ λd .

Principal Components
The k-th principal component (PC) direction is the k-th eigenvector qk of S.
The PC scores (projected coordinates) are the columns of Z = XQ. The variance P
explained by the k-th PC is λk , and the fraction of variance explained is λk / j λj .

PCA via SVD (numerically preferred)

1
Since S = n−1 X⊤ X, the eigenvectors of S are exactly the right singular vectors V
of X, and λk = σk2 /(n − 1). So computing SVD of X directly (without forming X⊤ X)
is more numerically stable and is how modern software implements PCA.

6.4 PCA for Image Compression / Reconstruction

For each image x ∈ Rd , compute the r-dimensional projection:

z = Q⊤ r
r (x − x̄) ∈ R , x̂ = x̄ + Qr z ∈ Rd ,
Mathematics for Machine Learning 15

where Qr = [q1 , . . . , qr ]. The reconstruction error decreases as r increases.

Problem 6.1 – PCA from Scratch (Tricky)

 
2 1
Problem. Centred data: X = −1 2 . (a) Compute the 2 × 2 sample covariance
−1 −3
matrix S. (b) Find the first principal component direction. (c) Compute the fraction
of variance explained by PC1.

Solution.
(a) Verify centring: column means are x̄1 = (2−1−1)/3 = 0 and x̄2 = (1+2−3)/3 = 0.
Good.
 
  2 1    
1 ⊤ 1 2 −1 −1  1 6 −3 3 −3/2
S= X X= −1 2  = = .
n−1 2 1 2 −3 2 −3 14 −3/2 7
−1 −3

(b) Eigenvalues: (3 − λ)(7 − λ) − 94 = 0 ⇒ λ2 − 10λ + 21 − 49 = 0 ⇒ λ2 − 10λ + 75

4
= 0.
√
10 ± 100 − 75 10 ± 5
λ= = .
2 2
So λ1 = 7.5, λ2 = 2.5.
For λ1 =  (3 − 7.5)v1 − 32 v2 = 0 ⇒ −4.5v1 = 32 v2 ⇒ v2 = −3v1 . Unit eigenvector:
7.5: 
1
q1 = √110 .
−3
(c) Fraction of variance: λ1 /(λ1 + λ2 ) = 7.5/10 = 75%.

7 Week 7 – Unconstrained Optimisation

Prerequisites
Gradients (Week 2), positive definite matrices (Week 5), Taylor series.

7.1 Necessary and Sufficient Conditions for Optimality

First-Order Necessary Condition (FONC)

If θ ∗ is a local minimiser of differentiable f , then ∇f (θ ∗ ) = 0. Such a point is called
a stationary point (or critical point).

Hessian Matrix
The Hessian H(θ) ∈ Rd×d is the matrix of second-order partial derivatives:

∂ 2f
Hij = .
∂θi ∂θj
Mathematics for Machine Learning 16

By Clairaut’s theorem the Hessian is symmetric when f is twice continuously differen-

tiable.

Second-Order Sufficient Condition (SOSC)

If ∇f (θ ∗ ) = 0 and the Hessian H(θ ∗ ) is positive definite, then θ ∗ is a strict local
minimiser.

7.2 Gradient Descent – Formal Analysis

The update θ (t+1) = θ (t) − η∇f (θ (t) ) follows the steepest descent direction. For a strongly
convex function with Lipschitz-continuous gradient (constant L):
 µ t
f (θ (t) ) − f (θ ∗ ) ≤ 1 − [f (θ (0) ) − f (θ ∗ )],
L
where µ > 0 is the strong convexity constant and the optimal step size is η = 1/L. This is
linear convergence (the error shrinks by a constant factor each step).

7.3 Newton’s Method

Use the second-order Taylor expansion to approximate f by a quadratic and minimise
exactly:
θ (t+1) = θ (t) − H(θ (t) )−1 ∇f (θ (t) ).

Newton vs. Gradient Descent

Newton’s method achieves quadratic convergence: once close to the minimum,
the number of correct decimal places roughly doubles each iteration. The cost is
computing and inverting the Hessian: O(d3 ) per step vs. O(d) for gradient descent.
For large d (millions of parameters in deep learning), Newton is impractical.

7.4 Stochastic Gradient Descent (SGD)

For ML we minimise f (θ) = n1 ni=1 ℓi (θ). Computing the full gradient costs O(n) per
P
step. SGD uses a random mini-batch B ⊂ [n] to get a noisy but cheap gradient estimate:
η X
θ (t+1) = θ (t) − ∇ℓi (θ (t) ).
|B| i∈B

Common Misconception
“SGD converges to the same minimum as full gradient descent.” – Not exactly. In
the non-convex landscape of deep learning, the noise introduced by SGD acts as a
regulariser: it tends to escape sharp minima and finds flatter minima that generalise
better. This is one reason SGD is preferred over full-batch gradient descent in practice.
Mathematics for Machine Learning 17

Problem 7.1 – Newton’s Method vs. Gradient Descent

Problem. Minimise f (θ1 , θ2 ) = θ12 + 50θ22 starting at (1, 1). Compare one Newton
step and one gradient descent step with η = 0.01.

Solution.
∇f = (2θ1 , 100θ2 ). H = diag(2, 100).
Gradient descent step (η = 0.01):

(1, 1) − 0.01 · (2, 100) = (1 − 0.02, 1 − 1) = (0.98, 0).

Newton step: H−1 = diag(1/2, 1/100).

(1, 1) − H−1 (2, 100) = (1 − 1, 1 − 1) = (0, 0).

Newton’s method reaches the exact minimum in a single step because f is exactly
quadratic. Gradient descent took a step of size 0.01 × 2 = 0.02 in the θ1 direction but
a step of 0.01 × 100 = 1.00 in the θ2 direction – accidentally zeroing θ2 in one step
while barely moving θ1 . This illustrates the ill-conditioning problem: when H has
eigenvalues of very different magnitudes (condition number 100/2 = 50 here), gradient
descent converges slowly in the flat directions.

8 Week 8 – Convex Sets, Functions, and Optimisation

Prerequisites
Gradients, Hessians (Week 7), basic set notation.

8.1 Convex Sets

Convex Set
A set C ⊆ Rd is convex if for any two points x, y ∈ C and any t ∈ [0, 1]:

tx + (1 − t)y ∈ C.

Intuitively: the line segment connecting any two points in C stays inside C.

Examples: Euclidean balls, hyperplanes, half-spaces, the cone of PSD matrices, the
probability simplex.
Mathematics for Machine Learning 18

8.2 Convex Functions

Convex Function
f : C → R on a convex domain C is convex if for all x, y ∈ C and t ∈ [0, 1]:

f (tx + (1 − t)y) ≤ t f (x) + (1 − t) f (y).

The function lies below (or on) the chord connecting any two of its points.

8.2.1 Equivalent Characterisations

Characterisations of Convexity
For a twice-differentiable f : Rd → R, the following are equivalent:
(i) f is convex (chord condition above).
(ii) First-order condition: f (y) ≥ f (x) + ∇f (x)⊤ (y − x) for all x, y. (The
function lies above every tangent hyperplane.)
(iii) Second-order condition: The Hessian H(x) ⪰ 0 (PSD) for all x.

Why Convexity Is the Holy Grail of Optimisation

If f is convex then:
1. Every local minimum is a global minimum.
2. If f is also strictly convex, the global minimum is unique.
3. Gradient descent is guaranteed to find the global minimum (given proper step
sizes).
Non-convex problems (like training deep networks) may have many bad local minima –
yet in practice SGD still works surprisingly well, which is an active research area.

8.3 Examples Relevant to ML

ˆ f (x) = ∥x∥2 : convex (Hessian = 2I ≻ 0).
ˆ L(θ) = ∥y − Xθ∥2 : convex in θ (Hessian = 2X⊤ X ⪰ 0).
ˆ f (x) = log(1 + e−x ) (logistic loss): convex.
ˆ f (x) = x4 − x2 : non-convex (has two local minima).

8.4 Convex Optimisation Problems

Convex Programme
Minimise f (x) subject to x ∈ C, where f is convex and C is a convex set. Equivalently:
minimise a convex f subject to convex inequality and affine equality constraints.

Ridge regression is a convex programme: minimise ∥y − Xθ∥2 + λ ∥θ∥2 over θ ∈ Rd

(unconstrained, but the objective is strictly convex for λ > 0).
Mathematics for Machine Learning 19

Problem 8.1 – Proving Convexity

Problem. Show that the logistic loss ℓ(z) = log(1 + e−z ) is convex by checking the
second-derivative condition.

Solution.
−e−z
ℓ′ (z) = 1+e 1
−z = −(1 − σ(z)) where σ(z) = 1+e−z is the sigmoid.
′′ ′
ℓ (z) = σ (z) = σ(z)(1 − σ(z)).
Since 0 < σ(z) < 1 for all z ∈ R, we have σ(z)(1 − σ(z)) > 0 for all z. Therefore
ℓ′′ (z) > 0 everywhere, which means ℓ is strictly convex. It has a unique global
minimum (as z → +∞, ℓ(z) → 0, so the infimum is 0 but it is not attained – there
is no finite minimiser, meaning the unconstrained logistic regression problem drives
weights to infinity for linearly separable data: this is a well-known issue addressed by
regularisation).

9 Week 9 – Constrained Optimisation and Lagrange

Multipliers
Prerequisites
Gradients, convexity (Weeks 7–8), basic understanding of equality constraints.

9.1 The Constrained Optimisation Problem

Minimise f (x) subject to gi (x) = 0, i = 1, . . . , m and hj (x) ≤ 0, j = 1, . . . , p.

9.2 Equality Constraints: Lagrange Multipliers

Why Lagrange Multipliers Work
At a constrained optimum x∗ , you can no longer move freely in all directions – you
are confined to the surface g(x) = 0. The optimum occurs where ∇f is perpendicular
to the constraint surface. But the gradient perpendicular to {g = 0} is ∇g (since ∇g
always points normal to level sets). So at the optimum ∇f = λ∇g for some scalar λ.

Lagrange Multiplier Theorem

If x∗ is a local minimum of f subject to g(x) = 0 and ∇g(x∗ ) ̸= 0, then there exists
λ∗ ∈ R such that:
∇f (x∗ ) = λ∗ ∇g(x∗ ).
Equivalently, define the Lagrangian L(x, λ) = f (x)−λ g(x) and the optimum satisfies
∇x L = 0 and ∇λ L = 0 (i.e. g(x) = 0).
Mathematics for Machine Learning 20

9.3 Logistic Regression as an Optimisation Problem

For binary classification with labels yi ∈ {0, 1} and features xi , the model predicts
p̂i = σ(θ ⊤ xi ).
Negative log-likelihood loss:
n
X
yi log σ(θ ⊤ xi ) + (1 − yi ) log(1 − σ(θ ⊤ xi )) .
 
L(θ) = −
i=1

This is a convex function of θ (by Week 8 we know log(1 + e−z ) is convex, and composition
with a linear function preserves convexity). The gradient is:

∇θ L = X⊤ (p̂ − y),

where p̂ = σ(Xθ) elementwise. This is the residuals vector (p̂i − yi ) projected by X⊤ –

strikingly similar to OLS.

9.4 KKT Conditions (Inequality Constraints)

For problems with inequality constraints hj (x) ≤ 0, the Lagrangian is:
X
L(x, µ) = f (x) + µj hj (x).
j

KKT Conditions
At an optimum x∗ (under regularity
P ∗conditions):
(i) Stationarity: ∇f (x ) + j µj ∇hj (x∗ ) = 0.
∗

(ii) Primal feasibility: hj (x∗ ) ≤ 0.

(iii) Dual feasibility: µ∗j ≥ 0.
(iv) Complementary slackness: µ∗j hj (x∗ ) = 0 for each j.
Complementary slackness means: either the constraint is inactive (hj < 0, then µj = 0)
or the constraint is active (hj = 0, then µj ≥ 0).

Problem 9.1 – Lagrange Multipliers for Maximum Variance

Problem. Given a covariance matrix S, find the unit vector w that maximises the
projected variance w⊤ Sw, subject to ∥w∥2 = 1. This connects directly to PCA (Week
6).

Solution.
Lagrangian: L(w, λ) = w⊤ Sw − λ(w⊤ w − 1).
Stationarity: ∇w L = 2Sw − 2λw = 0 ⇒ Sw = λw.
This is exactly the eigenvalue equation: w must be an eigenvector of S.
At the solution: w⊤ Sw = w⊤ (λw) = λ ∥w∥2 = λ.
To maximise the variance, we choose the eigenvector corresponding to the largest
eigenvalue λmax . This is the first principal component – confirming that PCA is the
solution to a Lagrange-multiplier problem.
Mathematics for Machine Learning 21

10 Week 10 – Probabilistic Models in Machine Learn-

ing
Prerequisites
Basic probability: sample space, events, probability mass/density functions, conditional
probability, Bayes’ rule.

10.1 Why Probability?

Most real-world data is noisy. Probabilistic models allow us to:
ˆ Quantify uncertainty in predictions.
ˆ Derive learning algorithms from principled maximum likelihood estimation.
ˆ Build generative models for data.

10.2 Gaussian Distribution

Multivariate Gaussian
x ∼ N (µ, Σ) has density:
 
1 1 ⊤ −1
p(x) = exp − (x − µ) Σ (x − µ) ,
(2π)d/2 |Σ|1/2 2

where µ ∈ Rd is the mean and Σ ∈ Rd×d is the (PD) covariance matrix.

10.3 Maximum Likelihood Estimation (MLE)

Given i.i.d. data x1 , . . . , xn from model p(x; θ):
n
Y n
X
Likelihood: L(θ) = p(xi ; θ), Log-likelihood: ℓ(θ) = log p(xi ; θ).
i=1 i=1

Log-Likelihood Trick
We maximise the log-likelihood instead of the likelihood because:
ˆ Logarithm converts products to sums – analytically much easier.
ˆ Products of many small probabilities underflow to zero on a computer; log avoids
this.
ˆ log is monotone increasing, so arg maxθ L(θ) = arg maxθ ℓ(θ).

10.4 MLE for the Gaussian

For x1 , . . . , xn ∼ N (µ, σ 2 ):
n
2 n 2 1 X
ℓ(µ, σ ) = − log(2πσ ) − 2 (xi − µ)2 .
2 2σ i=1
Mathematics for Machine Learning 22

Setting ∂ℓ/∂µ = 0 and ∂ℓ/∂σ 2 = 0:

1X 2 1X
µ̂MLE = x̄ = xi , σ̂MLE = (xi − x̄)2 .
n i n i

MLE Variance is Biased

2
σ̂MLE 2
divides by n, not n − 1. It is a biased estimator: E[σ̂MLE ] = n−1
n
σ 2 . The
unbiased sample variance (used in statistics) divides by n − 1. The difference matters
little for large n but can be important for small samples.

10.5 Linear Regression as MLE

Assume yi = θ ⊤ xi + εi where εi ∼ N (0, σ 2 ). Then yi | xi ∼ N (θ ⊤ xi , σ 2 ), and:
n 1 X
ℓ(θ) = − log(2πσ 2 ) − 2 (yi − θ ⊤ xi )2 .
2 2σ i

Maximising over θ is equivalent to minimising i (yi − θ ⊤ xi )2 – the OLS objective. Under

P
Gaussian noise assumptions, OLS = MLE.

Problem 10.1 – MLE for an Exponential Distribution

Problem. Suppose x1 , . . . , xn ∼ Exp(λ) i.i.d., with density p(x; λ) = λe−λx for x > 0.
Find the MLE for λ.

Solution.
Log-likelihood:
n
X n
X
ℓ(λ) = log(λe−λxi ) = n log λ − λ xi .
i=1 i=1

Differentiate and set to zero:

dℓ n X n 1
= − xi = 0 =⇒ λ̂MLE = P = .
dλ λ i i xi x̄

d ℓ 2
2
Check second derivative: dλ2 = −n/λ < 0, confirming this is a maximum.

Interpretation: The MLE rate is the reciprocal of the sample mean – sensible, since
the mean of Exp(λ) is 1/λ.

11 Week 11 – Exponential Family of Distributions

Prerequisites
Probability distributions, MLE (Week 10), sufficient statistics (notion of data sum-
mary).
Mathematics for Machine Learning 23

11.1 The Exponential Family

Many common distributions – Gaussian, Bernoulli, Poisson, exponential, Beta, Gamma –
share a common mathematical form.

Exponential Family
A family of distributions with parameter η ∈ Rk is an exponential family if its
density/pmf can be written as:
 
⊤
p(x; η) = h(x) exp η T(x) − A(η) ,

where:
ˆ T(x): sufficient statistic – summarises the data for estimating η.
ˆ η: naturalRparameter.
ˆ A(η) = log h(x) exp(η ⊤ T(x)) dx: log-partition function (normalising con-
stant).
ˆ h(x): base measure.

11.2 Examples
11.2.1 Bernoulli as Exponential Family
p(x; p) = px (1 − p)1−x for x ∈ {0, 1}. Write:
 
p
p(x; p) = exp x log + log(1 − p) .
1−p
p
So η = log 1−p (log-odds), T (x) = x, A(η) = − log(1 − p) = log(1 + eη ).
Note: p = σ(η) where σ is the sigmoid – this is why logistic regression uses the sigmoid!

11.3 Key Properties of the Exponential Family

Moments from the Log-Partition Function

E[T(x)] = ∇η A(η), Cov[T(x)] = ∇2η A(η).

Since ∇2 A ⪰ 0 (covariance matrix is always PSD), A is always a convex function of
η.

Sufficiency and MLE

Because A(η) is convex, the MLE for exponential family distributions is a convex
optimisation problem with a unique solution. The MLE condition becomes:
n
1X
∇η A(η̂) = T(xi ),
n i=1

i.e. the expected sufficient statistic equals the empirical (sample-average) sufficient
statistic. This elegant moment-matching condition is the basis of Generalised Linear
Mathematics for Machine Learning 24

Models (GLMs).

Fisher–Neyman Factorisation Theorem

A statistic T(x) is sufficient for η iff the density factorises as p(x; η) = g(T(x), η)·h(x).
For exponential families, T(x) is always sufficient – you lose no information about η
by compressing data to T(x).

Problem 11.1 – Gaussian as Exponential Family

Problem. Write N (µ, σ 2 ) with known σ 2 in exponential family form, identify η, T (x),
A(η), and h(x).

Solution.

(x − µ)2 x2 µ2
   
1 1 µ
p(x; µ) = √ exp − =√ exp − 2 + 2 x − 2 .
2πσ 2 2σ 2 2πσ 2 2σ σ 2σ
Identify:
µ
η= (natural parameter),
σ2
T (x) = x (sufficient statistic),
µ2 η2σ2
A(η) = 2 = (log-partition),
2σ 2 
x2

1
h(x) = √ exp − 2 .
2πσ 2 2σ

Verification: A′ (η) = ησ 2 = µ = E[T (x)] ✓. A′′ (η) = σ 2 = Var[T (x)] ✓.

12 Week 12 – Parameter Estimation and Expectation

Maximisation
Prerequisites
MLE (Week 10), exponential family (Week 11), Jensen’s inequality (see below), basic
probability over latent variables.

12.1 The Problem of Latent Variables

Sometimes our data x is generated by a process involving hidden (latent) variables
z. The complete-data likelihood p(x, z; θ) is easy to write, but we only observe x. The
marginal likelihood (observed-data likelihood) is:
Z X
p(x; θ) = p(x, z; θ) dz or p(x, z; θ).
z
Mathematics for Machine Learning 25

Directly maximising log p(x; θ) is often intractable because of the sum/integral inside the
log.

12.2 Jensen’s Inequality

Jensen’s Inequality
For a concave function f (e.g. log) and a random variable Z:

f (E[Z]) ≥ E[f (Z)].

For a convex function the inequality reverses.

Intuition
For log: the log of an average is always ≥ the average of logs. Graphically, the log
function curves downward (concave), so the chord connecting two points lies below
the function.

12.3 The Evidence Lower Bound (ELBO)

Introduce any distribution q(z) over latent variables:
X p(x, z; θ)
log p(x; θ) = log q(z)
z
q(z)
X p(x, z; θ)
≥ q(z) log (Jensen, since log is concave)
z
q(z)
=: F(q, θ) (ELBO, or free energy).

EM as Coordinate Ascent on the ELBO

The Expectation Maximisation algorithm alternates between:
ˆ E-step: Fix θ, maximise ELBO over q: set q(z) = p(z | x; θ) (posterior). This
makes the bound tight: F(q, θ) = log p(x; θ).
ˆ M-step: Fix q, maximise ELBO over θ: θ new = arg maxθ Eq(z) [log p(x, z; θ)].
Each full E-M iteration is guaranteed to not decrease the log-likelihood log p(x; θ).

12.4 EM for Gaussian Mixture Models (GMMs)

A Gaussian Mixture Model assumes data comes from K Gaussian clusters with mixing
proportions πk :
K
X
p(x; θ) = πk N (x; µk , Σk ).
k=1
The latent variable zi ∈ {1, . . . , K} indicates which component generated xi .
E-step: Compute responsibilities (soft cluster assignments):
πk N (xi ; µk , Σk )
rik = p(zi = k | xi ; θ) = PK .
j=1 πj N (xi ; µj , Σj )
Mathematics for Machine Learning 26

M-step: Update parameters using the responsibilities as weights:

µ̂k )(xi − µ̂k )⊤

P P
1X i rik xi i rik (xi −
π̂k = rik , µ̂k = P , Σ̂k = P .
n i i rik i rik

EM vs. K-Means
K-means is a hard version of EM for GMMs: in the E-step it assigns each point to
exactly one cluster (responsibility is 0 or 1), and in the M-step it recomputes centroids.
EM with GMMs is the soft version where every point has fractional membership in
every cluster. K-means corresponds to GMMs where all Σk = σ 2 I and σ 2 → 0.

EM Convergence to Global Optimum

EM is guaranteed to increase (or maintain) the likelihood at every iteration, but it is
NOT guaranteed to find the global maximum. The marginal log-likelihood for GMMs
is non-convex. EM converges to a local maximum or saddle point that depends on
initialisation. Common practice: run EM multiple times from random initialisations,
keep the result with the highest final log-likelihood.

Problem 12.1 – EM on a Toy 1D Mixture (Tricky)

Problem. Two-component Gaussian mixture: p(x) = πN (x; µ1 , 1)+(1−π)N (x; µ2 , 1)
with π = 0.5 known. You observe n = 3 points: x1 = 0, x2 = 1, x3 = 5. Current
(0) (0)
estimate: µ1 = 0, µ2 = 4. Perform one E-M iteration.

Solution.
N (xi ;0,1) 2 /2
E-step. Compute ri1 = N (xi ;0,1)+N (xi ;4,1)
for each xi . Let ϕ(z) ∝ e−z .
For x1 = 0: numerator ϕ(0) = 1, denominator 1 + ϕ(−4) ≈ 1 + e−8 ≈ 1.00034. So
r11 ≈ 0.9997, r12 ≈ 0.0003.
For x2 = 1: ϕ(−1) = e−0.5 ≈ 0.6065; ϕ(−3) = e−4.5 ≈ 0.0111. So r21 =
0.6065/(0.6065 + 0.0111) ≈ 0.982, r22 ≈ 0.018.
For x3 = 5: ϕ(−5) = e−12.5 ≈ 3.7 × 10−6 ; ϕ(−1) = e−0.5 ≈ 0.6065. So r31 ≈ 0.000006,
r32 ≈ 0.999994.
M-step.

(1) r11 x1 + r21 x2 + r31 x3 0.9997 · 0 + 0.982 · 1 + 0 · 5 0.982

µ1 = ≈ ≈ ≈ 0.496.
r11 + r21 + r31 0.9997 + 0.982 + 0 1.982

(1) 0.0003 · 0 + 0.018 · 1 + 1 · 5 5.018

µ2 ≈ ≈ ≈ 4.928.
0.0003 + 0.018 + 1 1.0183
After one iteration: µ1 moved toward the cluster of {0, 1} and µ2 moved toward 5, as
expected. Continued iterations will refine these estimates.

Appendix: Quick-Reference Summary

Mathematics for Machine Learning 27

Week Topic Key Formula / Result

1 ML Fundamentals Task, Experience, Performance
2 Calculus θ ← θ − η∇L
3 OLS θ ∗ = (X⊤ X)−1 X⊤ y
4 Eigenvalues Av = λv, det(A −P λI) = 0
5 Symmetric Matrices A = QΛQ⊤ , tr = λk
⊤
6 SVD / PCA A = UΣV ; top eigenvectors of S
7 Unconstrained Opt. ∇f = 0; Newton: θ ← θ − H−1 ∇f
8 Convexity f (tx + (1 − t)y) ≤ tf (x) + (1 − t)f (y)
9 Lagrange / KKT ∇f = λ∇g; KKT slackness
10 Probabilistic Models θ̂MLE = arg maxθ ℓ(θ)
11 Exponential Family p(x; η) = h(x) exp(η ⊤ T − A(η))
12 EM Algorithm E-step: q = p(z|x); M-step: arg max Eq [log p(x, z)]