Chapter 2

Mathematical Foundation of Machine Learning

2.1. System of Linear equations

2.2. Norms or distance between vectors or length of vector

Norms measure the "size" or "length" of a vector. The L1 norm and L2 norm are two
common types used in various applications, including machine learning,
optimization, and linear algebra.

2.2.1. Manhattan or L1 -norm of a vector:

The L1 norm of a vector 𝑥 = (𝑥1, 𝑥2, . . . . . . . . 𝑥𝑛) is the sum of the absolute
values of its components.

||X|| 1 = |x1 | + | x2 | . . . . . . . . |xn |

E.g. 𝒙 = [𝟑, −𝟒, 𝟓]

The L1 norm is: ||𝐗|| 𝟏 = |𝐱 𝟏 | + | 𝐱 𝟐 | . . . . . . . . |𝐱 𝐧 |

||𝐗|| 𝟏 = |𝟑| + | − 𝟒| + |𝟓| = 𝟏𝟐

E.g. 𝒙 = [𝟑, −𝟒, 𝟓] and y= [-1, 5, -3]

||𝐱 − 𝐲|| 𝟏 = |𝟑 − (−𝟏)| + | − 𝟒 − 𝟓| + |𝟓 − (−𝟑)| = 𝟏𝟖

Uses of L1 Norm

Feature selection in machine learning (Lasso regression): Encourages sparsity

(some coefficients become exactly zero).

Robust error measurement: Less sensitive to outliers than L2 norm.

2.2.2. Euclidean or L2 -norm of a vector

The L2 norm is the square root of the sum of the squares of a vector’s
components:

||𝐗|| 𝟐 = √𝒙𝟐𝟏 + 𝒙𝟐𝟐 +. . . . . . . . 𝒙𝟐𝒏

E.g. 𝒙 = [𝟑, −𝟒, 𝟓]

The L1 norm is: ||𝐗|| 𝟐 = √𝒙𝟐𝟏 + 𝒙𝟐𝟐 +. . . . . . . . 𝒙𝟐𝒏

||𝐗|| 𝟐 = √𝟑𝟐 + (−𝟒)𝟐 + 𝟓𝟐 = √𝟓𝟎

E.g. 𝒙 = [𝟑, −𝟒, 𝟓] and y= [-1, 5, -3]

 ||𝐱 − 𝐲|| 𝟐 = √[𝟑 − (−𝟏)]𝟐 + (−𝟒 − 𝟓)𝟐 + [𝟓 − (−𝟑)]𝟐 = √𝟏𝟔𝟏

Uses of L2 Norm
Machine learning (Ridge regression, regularization): Penalizes large weights but
does not encourage sparsity.
Optimization: Used in gradient descent because it is differentiable everywhere.

Physics and engineering: Measures actual physical distance.

2.3. Inner products

2.4. Orthogonal vectors

If dot product of two vectors results into 0, then vectors are considered as orthogonal
to each other.

2.5. Symmetric Positive Definite Matrices

2.6. Determinant
In case of matrix of size 2 X 2 :
𝟐 −𝟏
𝑨=[ ]
𝟒 𝟑
𝒅𝒆𝒕(𝑨) = [(𝟐)(𝟑) − (𝟒)(−𝟏)]

𝒅𝒆𝒕(𝑨) = [𝟔 + 𝟒] = 𝟏𝟎
𝟏 𝟐 𝟑
𝑩=[ 𝟒 −𝟓 𝟔]
−𝟐 𝟏 𝟕
−𝟓 𝟔 𝟒 𝟔 𝟒 −𝟓
𝒅𝒆𝒕(𝑩) = (𝟏) 𝒅𝒆𝒕 | | − (𝟐) | | + (𝟑) | |
𝟏 𝟕 −𝟐 𝟕 −𝟐 𝟏
𝒅𝒆𝒕(𝑩) = (𝟏)[(−𝟓)(𝟕) − (𝟏)(𝟔)] − (𝟐)[(𝟒)(𝟕) − (−𝟐)(𝟔)]
+ (𝟑)[(𝟒)(𝟏) − (−𝟐)(−𝟓)]

𝒅𝒆𝒕(𝑩) = −𝟒𝟏 − 𝟖𝟎 − 𝟏𝟖 = −𝟏𝟑𝟗

2.7. Trace
Trace of a matrix is sum of diagonal elements of matrix.
𝒏

𝒕𝒓(𝑨) = ∑ 𝑨𝒊𝒊
𝒊=𝟏

𝟐 −𝟏
𝑨=[ ]
𝟒 𝟑
𝒕𝒓(𝑨) = 𝟐 + 𝟑 = 𝟓
 𝟏 𝟐 𝟑
𝑩=[ 𝟒 −𝟓 𝟔]
−𝟐 𝟏 𝟕
𝒕𝒓(𝑩) = 𝟏 + (−𝟓) + 𝟕 = 𝟑

2.8. Eigenvalues and vectors

The eigenvalues and eigenvectors determine the geometry of a transformation
represented by A.

Eigenvalues (𝜆): Describe how the eigenvectors are scaled during the transformation.

If ∣λ∣>1 , the transformation stretches along the eigenvector direction.

If ∣λ∣<1| , the transformation compresses along the eigenvector direction.

If λ<0, the transformation includes a reflection.

Eigenvectors (v): Define the directions in which the transformation is purely scaling
(no rotation or shearing).

[note: computation of eigenvalues and corresponding eigenvectors is explained with

2.10 and 2.11 sections. please check shared PPT (module 2), for the details of matrix
and corresponding transformation based on eigenvalues and eigenvectors]

2.9. Orthogonal Projections

Suppose we want to project the point P(4,3) onto the line y=2x

Step 1: Understand the Setup

• The point P(4,3) is not on the line y = 2x

• We need to find its orthogonal projection, meaning the point P′ on the line
where the segment PP′ is perpendicular to the line.

Step 2: Find the Perpendicular Line

The given line equation is y=2x

The slope of this line is 2. A line perpendicular to this will have a negative
reciprocal slope of -½ .

So, the equation of the perpendicular line passing through P(4,3) is

𝟏
𝒚 − 𝟑 = − (𝒙 − 𝟒)
𝟐
𝟏
𝒚 = − (𝒙) + 𝟓
𝟐
Step 3: Find the Intersection

To find the projection P′, solve the system of equations:

 𝟏
𝒚 = 𝟐𝒙 (original line) and 𝒚 = − 𝟐 (𝒙) + 𝟓 (perpendicular line)

𝟏
𝟐𝒙 = − (𝒙) + 𝟓
𝟐
𝟓𝒙 = 𝟏𝟎

𝒙=𝟐

As 𝒚 = 𝟐𝒙 ,

𝒚=𝟒

So, the orthogonal projection of P(4,3) onto the line y=2x is P′(2,4).
2.10. Diagonalization
Diagonalization is also referred as EigenValue Decomposition. It restricts size of
matrix to be decomposed. Matrix needs to be of square size only i.e. n x n size. The
number of rows must be equal to the number of columns.

A full set of independent eigenvectors is necessary for diagonalizability:

The eigenvectors span the space if the matrix is diagonalizable (e.g., 𝑹𝒏 ).
The matrix does not have enough independent eigenvectors if it is not diagonalizable.
This is typical of defective matrices when the geometric multiplicity is smaller than
the algebraic multiplicity, such as a shear matrix.

Decompose A into 𝑷𝑫𝑷−𝟏

Step 1: Find eigenvalues of given matrix A.

𝐝𝐞𝐭 (𝐀 − 𝝀 𝐈 ) = 𝟎

where 𝜆 represents the eigenvalues of the matrix, and I is the identity matrix.

This equation will give eigenvalues 𝝀𝟏 , 𝝀𝟐 , 𝝀𝟑 , . . . . . . . . . . . . . . . . 𝝀𝐧 .

Step 2: Find Eigenvectors

For each eigenvalue 𝝀𝒊 , solve the equation: (𝑨 − 𝝀 𝑰 ) = 𝟎 where v is the

eigenvector corresponding to eigenvalue The eigenvectors form the columns of the
matrix P.

Step 3: Form the Matrix P:

Once the eigenvectors are determined, arrange them as columns in a matrix P.

Step 4: Diagonal Matrix D:

Construct the diagonal matrix D using the eigenvalues of A, placing each

Eigenvalue 𝝀𝒊 on the diagonal.

𝑫 = 𝒅𝒊𝒂𝒈(𝝀𝟏 , 𝝀𝟐 , 𝝀𝟑 , . . . . . . . . . . . . . . . . 𝝀𝐧 )

Step 5: Verify the Diagonalization:

Ensure that the matrix A is diagonalizable by checking:

𝑨 = 𝑷𝑫𝑷−𝟏 where 𝑷−𝟏 is the inverse of the matrix P.

Note : If the matrix has enough linearly independent eigenvectors (i.e., the number of
linearly independent eigenvectors equals the size of the matrix), the matrix is
diagonalizable.

e.g.
𝟒 𝟏
𝑨=[ ]
𝟐 𝟑
Step 1: Find Eigenvalues

𝐝𝐞𝐭(𝐀 − 𝝀𝐈) = 𝟎

Where I is an identity matrix and 𝜆 is an eigenvalue.

𝟒 𝟏 𝟏 𝟎
𝐝𝐞𝐭(𝐀 − 𝝀𝐈) = [ ] − 𝝀[ ]
𝟐 𝟑 𝟎 𝟏

𝟒−𝝀 𝟏
𝒅𝒆𝒕(𝑨 − 𝝀𝑰) = 𝒅𝒆𝒕 [ ]=𝟎
𝟐 𝟑−𝝀
𝒅𝒆𝒕(𝑨 − 𝝀𝑰) = [(𝟒 − 𝝀)(𝟑 − 𝝀) − 𝟐]

𝒅𝒆𝒕(𝑨 − 𝝀𝑰) = 𝟎 = 𝟏𝟐 − 𝟕 𝝀 + 𝝀𝟐 − 𝟐 𝝀𝟐 − 𝟕𝝀 + 𝟏𝟎 = 𝟎

(𝝀 − 𝟓 ) (𝝀 − 𝟐) = 𝟎

𝝀𝟏 = 𝟓 𝒂𝒏𝒅 𝝀𝟐 = 𝟐

Step 2: Find Eigenvectors

For 𝝀𝟏 = 𝟓 𝑭𝒐𝒓 𝝀𝟐 = 𝟐

4−𝜆 1 4−𝜆 1
[ ]𝑣 = 0 [ ]𝑣 = 0
2 3−𝜆 1 2 3−𝜆 2

−1 1 𝑥
[ ][ ] = 0
2 −2 𝑦 2 1 𝑥
[ ][ ] = 0
2 1 𝑦
-x + y =0
i.e. x = y 2x + y =0
x = -½ y

1 1
𝑣1 = [ ] 𝑣2 = [ ]
1 −2

𝟏 𝟏
𝑷=𝒗= [ ]
𝟏 −𝟐
Step 3: Construct the Diagonal Matrix D

The diagonal matrix D contains the eigenvalues on the diagonal

𝟓 𝟎
𝑫= [ ]
𝟎 𝟐
Step 4: Compute 𝑷−𝟏

Matrix P is invertible only if its determinant is non zero.

 𝑑𝑒𝑡(𝑃) = (−2)(1) − (1)(1)

𝑑𝑒𝑡(𝑃) = −3

1
𝑃−1 = 𝐴𝑑𝑗𝑢𝑔𝑎𝑡𝑒 𝑜𝑓 𝑃
𝑑𝑒𝑡(𝑃)
1 −2 −1
𝑃−1 = [ ]
𝑑𝑒𝑡(𝑃) −1 1

1 −2 −1
𝑃−1 = [ ]
−3 −1 1
2 1
𝑃 −1 = [3 3 ]
1 1
−
3 3
Step 5: Verify Diagonalization

Perform 𝑃𝐷𝑃−1 and confirm it is same as A.

2.11. Single Value Decomposition
𝟏 𝟐
𝑨 = [𝟑 𝟏]
𝟐 −𝟏
SVD decomposes A into three matrices,

𝐀 = U∑VT

𝐰𝐡𝐞𝐫𝐞, 𝐔 𝐢𝐬 𝐚𝐧 𝐨𝐫𝐭𝐡𝐨𝐠𝐨𝐧𝐚𝐥 𝐦𝐚𝐭𝐫𝐢𝐱 𝐜𝐨𝐧𝐭𝐚𝐢𝐧𝐢𝐧𝐠 𝐭𝐡𝐞 𝐥𝐞𝐟𝐭 𝐬𝐢𝐧𝐠𝐮𝐥𝐚𝐫 𝐯𝐞𝐜𝐭𝐨𝐫𝐬.

∑ 𝐢𝐬 𝐚 𝐝𝐢𝐚𝐠𝐨𝐧𝐚𝐥 𝐦𝐭𝐫𝐢𝐱 𝐜𝐨𝐧𝐭𝐚𝐢𝐧𝐢𝐧𝐠 𝐬𝐢𝐧𝐠𝐮𝐥𝐚𝐫 𝐯𝐚𝐥𝐮𝐞𝐬.

VT 𝒊𝒔 𝒂𝒏 𝒐𝒓𝒕𝒉𝒐𝒈𝒐𝒏𝒂𝒍 𝒎𝒂𝒕𝒓𝒊𝒙 𝒄𝒐𝒏𝒕𝒂𝒊𝒏𝒊𝒏𝒈 𝒕𝒉𝒆 𝒓𝒊𝒈𝒉𝒕 𝒔𝒊𝒏𝒈𝒖𝒍𝒂𝒓 𝒗𝒆𝒄𝒕𝒐𝒓𝒔.

Step 1: Compute Right Singular Matrix V from Eigenvalues of 𝑨𝑻 𝑨

𝟏 𝟐
𝟏 𝟑 𝟐
𝑨𝑻 𝑨 = [ ] [𝟑 𝟏 ]
𝟐 𝟏 −𝟏
𝟐 −𝟏
𝟏(𝟐) + 𝟑(𝟑) + 𝟐(𝟐) 𝟏(𝟐) + 𝟑(𝟏) + 𝟐(−𝟏)
=[ ]
𝟐(𝟏) + 𝟏(𝟑) + (−𝟏)(𝟐) 𝟐(𝟐) + 𝟏(𝟏) + (−𝟏)(−𝟏)

𝟏𝟒 𝟑
=[ ]
𝟑 𝟔

det(𝑨𝑻 𝑨 − 𝝀𝑰) = 𝟎

𝟏𝟒 𝟑 𝟏 𝟎
det([ ] − 𝝀[ ]) = 𝟎
𝟑 𝟔 𝟎 𝟏

𝟏𝟒 − 𝝀 𝟑
det([ ]) = 𝟎
𝟑 𝟔−𝝀

𝟏𝟒 − 𝝀 𝟑
| |=𝟎
𝟑 𝟔−𝝀

(𝟏𝟒 − 𝝀)(𝟔 − 𝝀) − (𝟑)(𝟑) = 𝟎

𝟖𝟒 − 𝟏𝟒𝝀 − 𝟔𝝀 + 𝝀𝟐 -9=0

𝝀𝟐 − 𝟐𝟎 𝝀 + 𝟕𝟓 = 𝟎
 (𝝀 − 𝟏𝟓)(𝝀 − 𝟓) = 𝟎

𝝀𝟏 = 𝟏𝟓 𝒂𝒏𝒅 𝝀𝟐 =5

For 𝝀𝟏 = 𝟏𝟓 For 𝝀𝟐 = 𝟓
𝟏𝟒 − 𝟏𝟓 𝟑 𝟏𝟒 − 𝟓 𝟑
| |𝒗 = 𝟎 | |𝒗 = 𝟎
𝟑 𝟔 − 𝟏𝟓 𝟏 𝟑 𝟔−𝟓 𝟐
−𝟏 𝟑 𝟗 𝟑
| |𝒗 = 𝟎 | |𝒗 = 𝟎
𝟑 −𝟗 𝟏 𝟑 𝟏 𝟐
𝟏 𝟑 𝟏 −𝟏
𝒗𝟏 = [ ] 𝒗𝟐 = [ ]
√𝟏𝟎 𝟏 √𝟏𝟎 𝟑

𝟑 −𝟏

𝑽 = √𝟏𝟎 √𝟏𝟎
𝟏 𝟑
[√𝟏𝟎 √𝟏𝟎]

Step 2 : Compute Singular values and corresponding diagonal matrix

𝝈𝒊 = √𝝀𝒊

𝝈𝟏 = √𝝀𝟏 = √𝟏𝟓

𝝈𝟐 = √𝝀𝟐 = √𝟓

And
𝝈𝟑 = √𝝀𝟑 = 𝟎

As there are three rows, sigma matrix is of size 3 x 3 i.e. of size U matrix.

√𝟏𝟓 𝟎 𝟎
∑=[ 𝟎 √𝟓 𝟎]
𝟎 𝟎 𝟎

Step 3: Computer Left Singular Vector U from eigenvectors of 𝑨𝑨𝑻

𝟏 𝟐
𝟏 𝟑 𝟐
𝑨𝑨𝑻 = [𝟑 𝟏 ][ ]
𝟐 𝟏 −𝟏
𝟐 −𝟏
 𝟏(𝟏) + 𝟐(𝟐) 𝟏(𝟑) + 𝟐(𝟏) 𝟏(𝟐) + 𝟐(−𝟏)
= [ 𝟑(𝟏) + 𝟏(𝟐) 𝟑(𝟑) + 𝟏(𝟏) 𝟑(𝟐) + 𝟏(−𝟏) ]
𝟐(𝟏) + (−𝟏)(𝟐) 𝟐(𝟑) + (−𝟏)(𝟏) 𝟐(𝟐) + (−𝟏)(−𝟏)

𝟓 𝟓 𝟎
= [𝟓 𝟏𝟎 𝟓]
𝟎 𝟓 𝟓
Compute Eigenvalues of 𝑨𝑨𝑻

det(𝑨𝑨𝑻 − 𝝀𝑰) = 𝟎

𝟓 𝟓 𝟎 𝟏 𝟎 𝟎
det([𝟓 𝟏𝟎 𝟓] − 𝝀 [𝟎 𝟏 𝟎]) = 𝟎
𝟎 𝟓 𝟓 𝟎 𝟎 𝟏

𝟓−𝝀 𝟓 𝟎
𝒅𝒆𝒕 ([ 𝟓 𝟏𝟎 − 𝝀 𝟓 ])=0
𝟎 𝟓 𝟓−𝝀

𝟓−𝝀 𝟓 𝟎
⌈ 𝟓 𝟏𝟎 − 𝝀 𝟓 ⌉=𝟎
𝟎 𝟓 𝟓−𝝀

𝟏𝟎 − 𝝀 𝟓 𝟓 𝟓
(𝟓 − 𝝀) | | −𝟓 | |+𝟎
𝟓 𝟓−𝝀 𝟎 𝟓−𝝀

(𝟓 − 𝝀)[(𝟏𝟎 − 𝝀)(𝟓 − 𝝀) − (𝟓)(𝟓)] − 𝟓[(𝟓)(𝟓 − 𝝀) − (𝟎)(𝟓)] = 𝟎

(𝟓 − 𝝀)[𝝀𝟐 − 𝟏𝟓𝝀 + 𝟐𝟓] − 𝟓 [𝟐𝟓 − 𝟓𝝀] = 𝟎

−𝝀𝟑 + 𝟐𝟎 𝝀𝟐 − 𝟏𝟎𝟎𝝀 + 𝟏𝟐𝟓 − 𝟏𝟐𝟓 + 𝟐𝟓 𝝀 = 𝟎

−𝝀𝟑 + 𝟐𝟎 𝝀𝟐 − 𝟕𝟓 𝝀 = 𝟎

𝝀𝟑 − 𝟐𝟎 𝝀𝟐 + 𝟕𝟓 𝝀 =

𝝀 (𝝀𝟐 − 𝟐𝟎 𝝀 + 𝟕𝟓 ) = 𝟎

𝝀 (𝝀 − 𝟏𝟓 ) (𝝀 − 𝟓) = 𝟎

𝝀𝟏 = 𝟎, 𝝀𝟐 = 𝟏𝟓 𝒂𝒏𝒅 𝝀𝟑 = 𝟓
 𝝀𝟐 = 𝟏𝟓 𝝀𝟑 = 𝟓
𝝀𝟏 = 𝟎
𝟓 − 𝟏𝟓 𝟓 𝟎 𝟓−𝟓 𝟓 𝟎
𝟓−𝝀 𝟓 𝟎 [ 𝟓 𝟏𝟎 − 𝟏𝟓 𝟓 ] 𝐮𝟐 = 𝟎 [ 𝟓 𝟏𝟎 − 𝟓 𝟓 ] 𝐮𝟐 = 𝟎
⌈ 𝟓 𝟏𝟎 − 𝝀 𝟓⌉ 𝒖𝟏 = 𝟎 𝟎 𝟓 𝟓 − 𝟏𝟓 𝟎 𝟓 𝟓−𝟓
𝟎 𝟓 𝟓 𝐱
−𝟏𝟎 𝟓 𝟎 𝟎 𝟓 𝟎 𝐱
𝟓 𝟓 𝟎 [ 𝟓 −𝟓 𝟓 ] [ 𝐲] = 𝟎 [𝟓 −𝟓 𝟓 ] [ 𝐲] = 𝟎
[𝟓 𝟏𝟎 𝟓] 𝒖𝟏 = 𝟎 𝟎 𝟓 −𝟏𝟎 𝐳 𝟎 𝟓 𝟎 𝐳
𝟎 𝟓 𝟓
-10x +5y =0 5y = 0
𝟓𝒙 + 𝟓𝒚 = 𝟎 y= 2x 5x -5y + 5z =0
𝒙 = −𝒚 5y -10z =0 5x -0 +5z = 0
𝟓𝒙 + 𝟏𝟎𝒚 + 𝟓𝒛 = 𝟎 y=2z x =- z
By substituting x=-y, i.e. x = z 5x -5y + 5z =0
𝟓𝒙 − 𝟏𝟎𝒙 + 𝟓𝒛 = 𝟎 y=x+z
−𝟓𝒙 + 𝟓𝒛 = 𝟎
𝒙=𝒛

𝟏 𝟏 𝟏 𝟏 𝟏 𝟏
𝒖𝟏 = [−𝟏] 𝐮𝟐 = [𝟐] 𝐮𝟐 = [𝟎]
√𝟑 𝟏 √𝟔 𝟏 √𝟐 −𝟏

𝟏 𝟏 𝟏
√𝟑 √𝟔 √𝟐
𝟏 𝟐
𝑈= − 𝟎
√𝟑 √𝟔
𝟏 𝟏 𝟏
−
[ √𝟑 √𝟔 √𝟐]
SVD applications
Chapter 3

Mathematical Foundation of Machine Learning

The least-squares method

Multivariate Linear Regression

Regularized

Regression

Using Least-Squares Regression for classification

Support Vector Machines