Mathematical Foundations of AI & Data Science

Introduction:
The mathematical foundations needed for Artificial Intelligence and Machine Learning
(AI/ML) primarily include linear algebra, calculus, probability and statistics, and optimization.
Linear algebra provides the tools to represent and manipulate data using vectors and matrices,
which form the backbone of data processing in machine learning algorithms. Calculus,
particularly differential calculus, is essential for understanding how models learn, especially
during gradient-based optimization used in training neural networks. Probability and statistics
help in modeling uncertainty, making predictions, and analyzing patterns within data.
Optimization techniques are critical for improving model performance by minimizing errors or
loss functions. A strong grasp of these mathematical concepts is important because they enable
practitioners to understand the inner workings of algorithms, design better models, interpret
results correctly, and troubleshoot problems effectively, rather than treating AI systems as black
boxes.

]_H Importance of Linear Algebra in Computing

Linear algebra is fundamental to computing as it provides the mathematical framework for
representing and manipulating data efficiently. In areas like computer graphics, machine
learning, artificial intelligence, and data science, linear algebra helps process high-dimensional
data through vectors, matrices, and tensor operations. It underlies algorithms for facial
recognition, natural language processing, and image compression, and is key to optimizing
complex computations. Matrix operations form the backbone of many programming libraries
and software tools used for simulations, data transformations, and numerical computations,
making linear algebra indispensable for modern computational technologies. In AI/ML studies,
linear algebra is particularly useful in the following domains

◆ 1. Data Representation
Computers handle all kinds of information—such as images, audio, video, and text—by
converting them into numbers.
Linear algebra gives a clear structure to organize this numerical data using:
Vectors: a list of numbers in a line (1D)
Matrices: rows and columns of numbers (2D)
Tensors: data in more than two dimensions (like colored images or video frames)
• Example:
¸
‘
)
An image is stored as a matrix of pixel brightness values.
A sentence is stored as a vector showing the importance of each word.

◆ 2. Efficient Computation
Modern computers, especially GPUs (Graphics Processing Units), are built to do matrix
and vector calculations very quickly.
Linear algebra makes it possible to handle large amounts of data smoothly and perform
operations like addition and multiplication on that data.
•
)
‘ Example:
¸
Training an AI model (like face detection) involves multiplying large matrices many times.
GPUs do this in seconds.

◆ 3. Transformation of Data
Linear algebra helps change the form or orientation of data using operations like rotation,
scaling, and projection.
These transformations are useful in computer graphics, animations, and when trying to
understand high-dimensional data.
• Example:
¸
‘
)
To rotate a 3D object in a game or a simulation, a matrix is used to apply the change.

◆ 4. Solving Systems of Equations

Many problems in engineering and science are based on systems of equations.
Linear algebra offers reliable methods (like matrix equations) to solve these using
computers.
•
) Example:
‘
¸
To predict weather, thousands of equations are solved every second using linear algebra in
weather models.

◆ 5. Foundation of Machine Learning and AI

 Every AI model—such as those used in self-driving cars or recommendation systems—
depends on linear algebra.
Each layer in a neural network uses matrix operations to process input data and improve its
predictions during training.
• Example:
¸
‘
)
A face recognition system checks if two images are similar by comparing their vectors
using linear algebra operations.

Vectors in AI/ML

Vectors are essential building blocks in Artificial Intelligence (AI) and Machine Learning
(ML). They are not just lines with direction and magnitude — they are the language through
which computers understand and manipulate information.

A vector is simply an ordered list of numbers. It can represent various forms of data:`

 A grayscale image as a vector of pixel brightness values.

 A document as a vector of word frequencies.
 A user’s movie preferences as a vector of ratings.

🖼️ 1. Grayscale Image as a Vector

A grayscale image is made of pixels, each with a brightness value between 0 (black) and
255 (white).

。
ˆ· Original Image (3×3 pixels):
.ç'
•

[ 0 50 100 ]
[150 200 250 ]
[ 30 60 90 ]
We can flatten this into a vector (1D array):
[0, 50, 100, 150, 200, 250, 30, 60, 90]
˛C* Now this vector can be fed into an ML model to recognize the image (e.g., a number
or a letter).
📄 2. Document as a Vector of Word Frequencies (Bag of Words)
Suppose we have 3 simple documents:
 Doc 1: "AI is powerful"
Doc 2: "AI is learning"
Doc 3: "Machine learning is useful"

Let’s build a vocabulary:

["AI", "is", "powerful", "learning", "Machine", "useful"]

Now we convert each document into a vector based on how many times each word appears
(word frequency):
Word → AI is powerful learning Machine useful
Doc 1 1 1 1 0 0 0
Doc 2 1 1 0 1 0 0
Doc 3 0 1 0 1 1 1
So,
Doc 1 = [1, 1, 1, 0, 0, 0]
Doc 2 = [1, 1, 0, 1, 0, 0]
Doc 3 = [0, 1, 0, 1, 1, 1]

These vectors help models understand the content similarity between documents.
🎥 3. User’s Movie Preferences as a Vector of Ratings
Imagine 3 movies:
["Inception", "Avatar", "Titanic"]
User A’s ratings:
Inception: 5
Avatar: 3
Titanic: 0 (not watched)
So, User A’s preference vector is:
[5, 3, 0]
User B’s ratings:
Inception: 4
Avatar: 3
Titanic: 5
So, User B’s vector is:
[4, 3, 5]
Now, using vector similarity, a recommendation system can say:
"A and B have similar tastes — recommend Titanic to User A."
Vector operations

⬛ 1. Vector Addition
Let:
𝐀 = [3, 5, 7], 𝐁 = [1, 4, 2]
𝐀 + 𝐁 = [3 + 1, 5 + 4, 7 + 2] = [4, 9, 9]
Used in: Gradient updates, combining feature vectors.

⬛ 2. Vector Subtraction
Let:
𝐀 = [5, 8, 6], 𝐁 = [2, 3, 1]
𝐀 − 𝐁 = [5 − 2, 8 − 3, 6 − 1] = [3, 5, 5]
Used in: Error calculation, tracking changes during optimization.

⬛ 3. Scalar Multiplication
Let:
𝐀 = [2, 4, 6], k = 0.5
k · 𝐀 = [0.5 × 2, 0.5 × 4, 0.5 × 6] = [1, 2, 3]
Used in: Scaling weights, adjusting learning rate effects.

⬛ 4. Dot Product
Let:
𝐀 = [1, 2, 3], 𝐁 = [4, 5, 6]
𝐀 · 𝐁 = (1 × 4) + (2 × 5) + (3 × 6) = 4 + 10 + 18 = 32
Used in: Measuring similarity, computing neuron activations.

⬛ 5. Cross Product (Only for 3D vectors)

Let:
𝐀 = [1, 2, 3], 𝐁 = [4, 5, 6]
𝐀 × 𝐁 =[(2 × 6 − 3 × 5), (3 × 4 − 1 × 6), (1 × 5 − 2 × 4)]
= [12 − 15, 12 − 6, 5 − 8] = [−3, 6, −3]
Used in: 3D spatial models, robotics, physics-based simulations.
 ⬛ 6. Norm (Magnitude)
Let:
𝐀 = [3, 4]
‖𝐀‖ = √(3² + 4²) = √(9 + 16) = √25 = 5
Used in: Measuring length of vectors, computing Euclidean distance.

⬛ 7. Normalization
Let:
𝐀 = [3, 4]
‖𝐀‖ = √(3² + 4²) = 5
𝐀 (normalized) = [3/5, 4/5] = [0.6, 0.8]
Used in: Scaling vectors for stable learning, computing cosine similarity.

Applications of vector operations in AI/ML

📸 1. Grayscale Image as a Vector
Ç Concept:
A grayscale image is simply a grid (or matrix) of numbers — each number represents
brightness of a pixel:
0 = black
255 = white
Values in between are shades of gray.

ç̇
+ Example:
⏷
<
‘
1
_
Let’s say we have a small 2×2 image:
Image A:
[ 100 150 ]
[ 200 250 ]
Each number is a pixel's brightness.
We flatten this 2D matrix row by row into a 1D vector:
𝐀→ = [100, 150, 200, 250]
This makes image data easier to use in ML models (which often take input as 1D vectors).
Now consider another image:
𝐁→ = [80, 160, 190, 240]

+ Vector Addition — Image Blending

We want to blend (combine) two images.
Mathematically:
𝐀→ + 𝐁→ = [100+80, 150+160, 200+190, 250+240]
= [180, 310, 390, 490]
Q˙ But 490 is greater than 255 (max brightness). So we typically normalize or clip values
•
to [0, 255] after addition.
Clipping vs Normalization
Problem: Pixel values above 255 are invalid for 8-bit grayscale images.

Option 1: Clipping
Values above 255 are clipped to 255:
Result = [180, 255, 255, 255]
Advantage: Simple
Disadvantage: Loses contrast or detail in bright areas

Option 2: Min–Max Normalization

Scale values into the [0, 255] range.
Step 1: Find min and max
Min = 180
Max = 490
Step 2: Apply the formula
𝑵𝒐𝒓𝒎𝒂𝒍𝒊𝒛𝒆𝒅 𝒗𝒂𝒍𝒖𝒆 = (𝒙 − 𝒎𝒊𝒏) ⁄ (𝒎𝒂𝒙 − 𝒎𝒊𝒏) × 𝟐𝟓𝟓
For each value:
180 → (180 − 180)/(490 − 180) × 255 = 0
310 → (310 − 180)/(490 − 180) × 255 ≈ 107
390 → (390 − 180)/(490 − 180) × 255 ≈ 173
490 → (490 − 180)/(490 − 180) × 255 = 255
Normalized result = [0, 107, 173, 255]
 Advantage: Preserves relative contrast
Use: Common in machine learning pipelines
* In ML:
˛
C
Used in image blending, adding filters, augmenting data, etc.

o Dot Product — Measuring Similarity

We can compute the dot product to measure how similar two images are.
Consider,
𝐀→ = [100, 150, 200, 250]
𝐁→ = [80, 160, 190, 240]
𝐀→ ⋅ 𝐁→ = (100×80) + (150×160) + (200×190) + (250×240)
= 8000 + 24000 + 38000 + 60000
= 130000
* Interpretation:
˛
C
The higher the dot product, the more similar the two images are (in terms of pixel
brightness patterns).

Vector Norms

A vector norm is a function that assigns a length or size to a vector.

In simple terms, it tells you how long or how big a vector is.

Let 𝐯 = [v₁, v₂, ..., vₙ] be a vector in ℝⁿ.

A norm ‖𝐯‖ is a non-negative number that satisfies certain properties

● Common Types of Norms

⬛ 1. L1 Norm (Manhattan Norm / Taxicab Norm)
Formula:
‖𝐯‖₁ = |v₁| + |v₂| + ... + |vₙ|
Interpretation:
Measures total distance moved along axes (like
walking in a grid).
Sensitive to sparse vectors [Sparsity refers to how many elements in a dataset, vector, or
matrix are zero or nearly zero].
Example:
𝐯 = [3, −4, 1]
‖𝐯‖₁ = |3| + |−4| + |1| = 3 + 4 + 1 = 8

⬛ 2. L2 Norm (Euclidean Norm)

Formula:
‖𝐯‖₂ = √(v₁² + v₂² + ... + vₙ²)
Interpretation:
The straight-line (as-the-crow-flies) distance from the origin.
Most commonly used in ML models.
Example:
𝐯 = [3, −4, 1]
‖𝐯‖₂ = √(3² + (−4)² + 1²) = √(9 + 16 + 1) = √26 ≈ 5.10

⬛ 3. L∞ Norm (Maximum Norm or Chebyshev Norm)

Formula:
‖𝐯‖∞ = max(|v₁|, |v₂|, ..., |vₙ|)
Interpretation:
Measures the largest absolute component of the vector.
Used in max-margin optimization.
Example:
𝐯 = [3, −4, 1]
‖𝐯‖∞ = max(3, 4, 1) = 4

● Use of Vector Norms in AI/ML

Regularization:
Helps avoid overfitting.
L1 regularization → promotes sparsity (feature selection).
L2 regularization → penalizes large weights smoothly.
Gradient Descent:
Step size and direction use L2 norms to compute magnitude of gradients.
Distance Metrics:
Vector norms are used to compute similarity/distance between feature vectors.

Use of Vector Norms in the Image Vector Examples

We had two grayscale images represented as flattened vectors:
A→ = [100, 150, 200, 250]
B→ = [80, 160, 190, 240]

⬛ 1. L2 Norm (Euclidean Norm)

This measures the magnitude (brightness intensity) of the entire image vector.
Formula for A→:
‖A→‖₂ = √(100² + 150² + 200² + 250²)
= √(10,000 + 22,500 + 40,000 + 62,500)
= √135,000 ≈ 367.42
Formula for B→:
‖B→‖₂ = √(80² + 160² + 190² + 240²)
= √(6,400 + 25,600 + 36,100 + 57,600)
= √125,700 ≈ 354.58
◆ Use: Helps understand how "bright" the images are on average.
◆ AI Use: L2 is used to calculate distances between vectors (e.g., image similarity, KNN,
clustering).

⬛ 2. L1 Norm (Manhattan Norm)

Measures the total pixel intensity regardless of direction (just a sum of absolute values).
‖A→‖₁ = |100| + |150| + |200| + |250| = 700
‖B→‖₁ = |80| + |160| + |190| + |240| = 670
◆ Use: Measures overall intensity. Can be used to compare overall pixel contribution.
◆ AI Use: L1 is common in regularization (e.g., Lasso), where simplicity is preferred by
keeping only the most important features and setting others to zero.

⬛ 3. L∞ Norm (Max Norm)

Measures the maximum pixel value in the image.
‖A→‖∞ = max(|100|, |150|, |200|, |250|) = 250
‖B→‖∞ = max(|80|, |160|, |190|, |240|) = 240
◆ Use: Helps find the brightest pixel.
◆ AI Use: Often used in bounding or thresholding operations (e.g., in adversarial image
processing or feature clipping).

⬛’ Application in Normalization
After blending the two images:
A→ + B→ = [180, 310, 390, 490]
These values exceed the valid grayscale range [0, 255], so we normalize.

◆ Min-Max Normalization Example

We scale each value between 0 and 255 using the formula:
x_scaled = ((x - min) / (max - min)) × 255
Where:
min = 180
max = 490
Now apply the formula to each pixel:
For 180:
x_scaled = ((180 − 180) / (490 − 180)) × 255 = (0 / 310) × 255 = 0
For 310:
x_scaled = ((310 − 180) / 310) × 255 = (130 / 310) × 255 ≈ 106.9
For 390:
x_scaled = ((390 − 180) / 310) × 255 = (210 / 310) × 255 ≈ 172.7
For 490:
x_scaled = ((490 − 180) / 310) × 255 = (310 / 310) × 255 = 255
Normalized vector = [0, 106.9, 172.7, 255]
◆ AI Use: Input normalization ensures all features (e.g., pixel values) contribute equally
to model training.

* Summary Table
˛
C
 Norm Type Meaning Role in AI/ML
L2 Euclidean magnitude Similarity, distance metrics, KNN, clustering
L1 Sum of absolute values Sparsity, feature selection, Lasso regularization
L∞ Maximum absolute value Thresholding, adversarial robustness

Additional read for conceptual clarity: Lasso Regularization

Lasso (Least Absolute Shrinkage and Selection Operator) is a type of regularization
technique used in machine learning and statistics to simplify models and prevent overfitting.
It does this by penalizing large coefficients and even forcing some of them to become zero,
leading to sparse models.

◆ Mathematical Form
In ordinary linear regression, we minimize the mean squared error (MSE):
𝑳𝒐𝒔𝒔 = 𝜮 (𝒚ᵢ − ŷᵢ)²
In Lasso Regression, we add a penalty term based on the absolute values of the
coefficients:
𝑳𝒐𝒔𝒔 = 𝜮 (𝒚ᵢ − ŷᵢ)² + 𝝀 𝜮 |𝒘ⱼ|
Where:
yᵢ = actual output
ŷᵢ = predicted output
wⱼ = model coefficients
λ = regularization parameter (controls the amount of shrinkage)

◆ What Lasso Does

The term Σ |wⱼ| is the L₁ norm of the coefficient vector.
As λ increases, Lasso shrinks some coefficients toward zero.
For large enough λ, some weights become exactly zero — meaning those input features
are excluded from the model.

◆ Simple Example
Imagine you're predicting house prices using:
Area (sq. ft)
 Number of rooms
Distance to school
Wallpaper color
Lasso might assign a weight of zero to "wallpaper color" if it's irrelevant. This removes
the noise and keeps only important predictors.

◆ Key Benefits of Lasso

⬛ Performs automatic feature selection
⬛ Produces sparse, interpretable models

⬛ Helps prevent overfitting, especially in models with many features

◆ Summary
Lasso regularization improves model performance by penalizing unnecessary complexity
and encouraging simpler, more generalizable solutions.

MATRICES IN AI/ ML
Matrices are fundamental tools in Artificial Intelligence (AI) and Machine Learning (ML)
because they provide a structured way to represent and manipulate large amounts of data and
mathematical operations efficiently. In these fields, almost everything—from inputs to
outputs, from features to model parameters—is stored and processed using matrices.

1. Matrix Addition
Matrix addition is done element-wise, meaning the corresponding elements of two
matrices are added together.
⬛ Condition: Both matrices must have the same dimensions.
Example:
Let:
A = [1 2] B = [4 5]
[3 4] [6 7]
Then:
A + B = [1+4 2+5] = [5 7]
 [3+6 4+7] [9 11]

o 2. Matrix Subtraction
Like addition, subtraction is also element-wise.
Example:
Let:
A = [5 7] B = [1 2]
[9 11] [3 4]
Then:
A- B = [5−1 7−2] = [4 5]
[9−3 11−4] [6 7]

◆ 3. Matrix Multiplication (Dot Product)

Matrix multiplication combines rows of the first matrix with columns of the second. This is
not element-wise.
⬛ Condition: The number of columns in the first matrix must equal the number of
rows in the second.
Example:
Let:
A = [1 2] (2×2)
[3 4]

B = [5 6] (2×2)
[7 8]
Then:
A × B = [ (1×5 + 2×7) (1×6 + 2×8) → [19 22]
(3×5 + 4×7) (3×6 + 4×8) ] [43 50]
So:
A × B = [19 22]
[43 50]

◆ 4. Scalar Multiplication
Each element of a matrix is multiplied by a single scalar value.
Example:
k=2

A = [1 2]
[3 4]

k × A = [2×1 2×2] = [2 4]
[2×3 2×4] [6 8]

◆ 5. Transpose of a Matrix
The transpose of a matrix flips rows into columns.
Example:
A = [1 2 3]
[4 5 6]

Aᵀ = [1 4]
[2 5]
[3 6]

⏷ Types of Matrices and Their Uses in Machine Learning

⬛ 1. Row Matrix
What it is: A matrix with only one row.
Use: Think of it like one student's marks across subjects.
Example: [65, 70, 85, 90] → one data point with 4 features (like marks in 4 subjects).

⬛ 2. Column Matrix
What it is: A matrix with only one column.
Use: Often used to hold weights or results.
Example:
[2]
[3]
 [4]
It's like stacking scores vertically.

⬛ 3. Square Matrix
What it is: Same number of rows and columns.
Use: Used for transformations, like rotating or scaling images.

⬛ 4. Diagonal Matrix
What it is: All values outside the diagonal are zero.
Use: Used to scale things easily. Each number on the diagonal affects a specific value — like
changing brightness per color channel in an image.

⬛ 5. Identity Matrix
What it is: A special diagonal matrix where all diagonal values are 1.
Use: Acts like the number 1 in multiplication. It keeps values the same — helpful in
simplifying equations.

⬛ 6. Zero Matrix
What it is: All values are zero.
Use: Used to reset values or as placeholders.

⬛ 7. Symmetric Matrix
What it is: The matrix is the same when flipped across the diagonal.
Use: Common in statistics to show how features relate — for example, how height and
weight vary together.

⬛ 8. Orthogonal Matrix
What it is: A matrix where rows and columns are at right angles (mathematically speaking).
Use: Useful for rotations and simplifying data in PCA (Principal Component Analysis), a
method to reduce data size.

⬛ 9. Sparse Matrix
What it is: A matrix with mostly zeros.
Use: Happens when data is huge but only a few values matter — like storing how many times
each word appears in millions of documents (most words don’t appear in most documents).

⬛ 10. Triangular Matrix

What it is: A matrix where all values above or below the diagonal are zero.
Use: Speeds up solving equations, especially when training ML models.

⬛ 11. Stochastic Matrix

What it is: A matrix where each row adds up to 1 — representing probabilities.
Use: Used in systems that predict outcomes based on previous steps — like weather
prediction or Google’s PageRank.

⬛ 12. Hessian and Jacobian Matrices

What they are: Special matrices used in math behind learning.
Use: They help machines understand how to improve and adjust during learning — like
telling how much to correct its mistake in the next step.

⌘Matrix Decomposition (Matrix Factorization)

⬛ Matrix Decomposition
Matrix decomposition is the process of breaking down a complex matrix into simpler, more
manageable components — typically a product of two or more matrices. This helps reduce
computational complexity and improves numerical stability. Just as prime factorization
simplifies a number, matrix decomposition simplifies matrices to their core building blocks.
In data science, signal processing, and AI/ML, matrix decomposition plays a vital role in
solving systems of equations, dimensionality reduction, data compression, image
recognition, and recommendation systems.

´ Importance of matrix decomposition

"◎●’
Matrix decomposition:
 Makes solving equations faster and more stable.
 Helps in numerical algorithms and software packages.
 Powers machine learning techniques such as PCA and recommender systems.
 Reduces the dimensionality of large datasets (for example, image data or text
vectors).

' 1. LU Decomposition (Lower-Upper Decomposition)

⬛c‘
–•.
Definition:
LU decomposition factors a square matrix A into the product of:
L (lower triangular matrix)
U (upper triangular matrix)
A = LU

Example:
Let
A= ⎡2 3⎤
⎣4 7⎦
We want:
L=⎡1 0⎤ U=⎡2 3⎤
⎣ l₂₁ 1 ⎦ ⎣0 1⎦
Steps:
u₁₁ = 2, u₁₂ = 3
l₂₁ = 4 ÷ 2 = 2
u₂₂ = 7 − (2 × 3) = 1
Final L and U:

L=⎡1 0⎤ U=⎡2 3⎤
⎣2 1⎦ ⎣0 1⎦
Check: LU = A ✔

k 2. QR Decomposition
×
˙
´
.
Definition:
QR decomposition breaks a matrix A into:
Q (orthogonal matrix: QᵀQ = I)
R (upper triangular matrix)
A = QR

Use Case:
Used in solving linear regression, particularly when the system has more rows than
columns (overdetermined).

Example:
Let
A= ⎡ 1 1⎤
⎣ 1 −1 ⎦

Q = (1/√2) × ⎡ 1 1⎤
⎣ 1 −1 ⎦
R = √2 × ⎡ 1 0⎤
⎣0 1⎦
Thus, A = QR ✔

« 3. Cholesky Decomposition
#
o
Definition:
If A is symmetric and positive definite, it can be decomposed into:
A = LLᵀ
Where L is a lower triangular matrix and Lᵀ is its transpose.

Example:
Let
A= ⎡ 4 2⎤
⎣2 3⎦
Assume
L= ⎡ a 0⎤
⎣b c⎦
Solving:
a² = 4 → a = 2
ab = 2 → b = 1
b² + c² = 3 → 1 + c² = 3 → c = √2
Then,

L= ⎡2 0⎤
⎣ 1 √2 ⎦
So, A = LLᵀ ✔

, 4. Eigen Decomposition
Definition:
Eigen decomposition expresses a square matrix A as:
A = P D P⁻¹
Where:
P is the matrix of eigenvectors
D is a diagonal matrix of eigenvalues
P⁻¹ is the inverse of the eigenvector matrix

Use Case:
Foundational in Principal Component Analysis (PCA), signal analysis, and system
dynamics.

Example:
Let
A= ⎡ 4 1⎤
⎣2 3⎦
Eigenvalues: λ₁ = 5, λ₂ = 2
Eigenvectors:
v₁ = ⎡ 1 ⎤ v₂ = ⎡ −1 ⎤
⎣1⎦ ⎣2⎦

P = ⎡ 1 −1 ⎤
⎣1 2⎦ D= ⎡5 0⎤
⎣0 2⎦
So, A = P D P⁻¹ ✔

# 5. Singular Value Decomposition (SVD)

/¡ç
Definition:
For any m × n matrix A:
A = U Σ Vᵀ
Where:
U: orthogonal matrix of left singular vectors (m × m)
Σ: diagonal matrix of singular values (m × n)
Vᵀ: transpose of orthogonal matrix of right singular vectors (n × n)

Use Case:
Used in image compression, noise filtering, recommendation engines, and PCA.

Example:
Let
A= ⎡ 3 1⎤
⎣1 3⎦
SVD gives:
U = ⎡ 1/√2 −1/√2 ⎤
⎣ 1/√2 1/√2 ⎦
Σ=⎡4 0⎤
⎣0 2⎦
Vᵀ = Uᵀ
Then, A = U Σ Vᵀ ✔
H
µ
_ Summary Table
–'
‘
l_
Decomposition Form Matrix Type Applications
LU A = LU Square matrix Solving equations, matrix
inversion
QR A = QR Any matrix Least squares, orthogonal
projections
Cholesky A = LLᵀ Symmetric, positive- Engineering, simulations
definite
Eigen A= Square matrix PCA, dynamics, control
PDP⁻¹ systems
SVD A= Any matrix PCA, compression, noise
UΣVᵀ filtering

Singular Value Decomposition (SVD) – Matrix Decomposition

⬛ Singular Value Decomposition (SVD) is a powerful matrix factorization technique in

linear algebra.
It expresses any real matrix A (of size m × n) as a product of three special matrices:
A = U × Σ × Vᵗ
Where:
A = Original matrix (m × n)
U = Left singular vectors (orthogonal matrix of size m × m)
Σ = Diagonal matrix (m × n) containing non-negative real numbers called singular values
Vᵗ = Transpose of an orthogonal matrix V (of size n × n)

• Importance of SVD
Q̇
Used in dimensionality reduction (PCA, LSA)
Compresses data while preserving essential features
Solves ill-conditioned or overdetermined systems
Useful in image compression, noise reduction, natural language processing
Foundation of recommender systems
⏷ U, Σ, and Vᵗ
U (Left Singular Vectors):
Columns are eigenvectors of A × Aᵗ
Represents directions in input space
Uᵗ × U = I (orthogonal)
Σ (Singular Values):
Diagonal matrix with values σ₁ ≥ σ₂ ≥ ... ≥ σᵣ > 0
Shows importance of each dimension
Vᵗ (Transpose of Right Singular Vectors):
V contains eigenvectors of Aᵗ × A
Vᵗ × V = I (orthogonal)

🔁 Geometric Interpretation
SVD shows how any linear transformation works as:
Rotation (V) → Scaling (Σ) → Rotation (U)
This means any matrix operation is equivalent to:
Rotating data
Stretching it along principal directions
Rotating it again

⌘
Step-by-Step SVD Example

Let’s take the matrix:

A=
[3 1]
[1 3]
Compute the Singular Value Decomposition (SVD) of this 2×2 matrix:
Goal:
Find matrices U, Σ, and Vᵗ such that:
A = U × Σ × Vᵗ
⬛ Step 1: Compute Aᵗ × A
Aᵗ = transpose of A

Aᵗ = [ 3 1 ]
[1 3]

Now compute Aᵗ × A:

Aᵗ × A =
[3 1] [3 1] = [ (3×3 + 1×1) (3×1 + 1×3) ]
[1 3] × [1 3] [ (1×3 + 3×1) (1×1 + 3×3) ]

= [ 9+1 3+3 ]
[ 3+3 1+9 ]

= [ 10 6]
[6 10 ]

⬛ Step 2: Find Eigenvalues of Aᵗ × A

Let the matrix be:
M = [ 10 6 ]
[ 6 10 ]
solve the characteristic equation:
det(M – λI) = 0
So,
|10 – λ 6 |
|6 10 – λ| = 0

(10 – λ)(10 – λ) – (6×6) = 0

(10 – λ)² – 36 = 0
(10 – λ)² = 36
10 – λ = ±6
Solving:
λ₁ = 10 – 6 = 4
λ₂ = 10 + 6 = 16

⬛ Step 3: Compute Singular Values

The singular values are the square roots of the eigenvalues of Aᵗ × A:
σ₁ = √16 = 4
σ₂ = √4 = 2
So the Σ matrix (2×2 diagonal) is:
Σ=[4 0]
[0 2]

⬛ Step 4: Compute V (Right Singular Vectors)

We find eigenvectors of Aᵗ × A corresponding to eigenvalues λ = 16 and λ = 4.

➤ For λ = 16:
M = [ 10 6 ]
[ 6 10 ]
Solve (M – 16I) × v = 0
M – 16I =
[10 – 16 6]
[6 10 –16] =
[ -6 6]
[ 6 –6 ]
From row 1:
–6x + 6y = 0 → x = y
So, an eigenvector is:
v₁ = [1, 1]ᵗ
Normalize:
‖v₁‖ = √(1² + 1²) = √2
v₁ (unit) = [1/√2, 1/√2]ᵗ

➤ For λ = 4:
M – 4I =
 [6 6]
[6 6]
From row 1:
6x + 6y = 0 → x = –y
So, an eigenvector is:
v₂ = [1, –1]ᵗ
Normalize:
‖v₂‖ = √(1² + (–1)²) = √2
v₂ (unit) = [1/√2, –1/√2]ᵗ

So, matrix V is:

V = [ 1/√2 1/√2]
[ 1/√2 –1/√2]
Then Vᵗ (transpose) is:
Vᵗ = [ 1/√2 1/√2 ]
[ 1/√2 –1/√2 ]

⬛ Step 5: Compute U = (1/σᵢ) × A × vᵢ

➤ Compute u₁ = (1/4) × A × v₁
Where v₁ = [1/√2, 1/√2]ᵗ
A × v₁ =
[3 1] [1/√2] = [ (3×1/√2 + 1×1/√2) ] = [4/√2]
[1 3] × [1/√2] [ (1×1/√2 + 3×1/√2) ] [4/√2]

u₁ = (1/4) × [4/√2, 4/√2]ᵗ = [1/√2, 1/√2]ᵗ

➤ Compute u₂ = (1/2) × A × v₂
Where v₂ = [1/√2, –1/√2]ᵗ
A × v₂ =
[3 1] [1/√2] = [ (3×1/√2 + 1×(–1/√2)) ] = [2/√2]
[1 3] × [–1/√2] [ (1×1/√2 + 3×(–1/√2)) ] [–2/√2]
 u₂ = (1/2) × [2/√2, –2/√2]ᵗ = [1/√2, –1/√2]ᵗ

So, matrix U is:

U = [ 1/√2 1/√2 ]
[ 1/√2 –1/√2 ]

⬛ Step 6: Final SVD

We now have:
U = [ 1/√2 1/√2 ]
[ 1/√2 –1/√2 ]
Σ= [4 0]
[0 2]
Vᵗ = [ 1/√2 1/√2 ]
[ 1/√2 –1/√2 ]

● Final Answer:
◎
’
"
´
A = U × Σ × Vᵗ
[3 1]
[1 3]=
[ 1/√2 1/√2 ] [4 0] [ 1/√2 1/√2 ]
[ 1/√2 –1/√2 ] × [ 0 2 ] × [ 1/√2 –1/√2 ]

⌘Spectral Decomposition (Eigen Decomposition)

H Introduction
_
]
Spectral decomposition, also known as eigenvalue decomposition, is a form of matrix
factorization that involves breaking a square matrix into its eigenvalues and eigenvectors.
This method is especially useful in mathematics, physics, and machine learning for
understanding the fundamental properties of a matrix.

Ç Spectral Decomposition
If a square matrix A is diagonalizable, it can be written as:
 𝐴 = 𝑃𝐷𝑃−1𝐴 = 𝑃𝐷𝑃 − 1
Where:
A is a square matrix.
P is a matrix whose columns are the eigenvectors of A.
D is a diagonal matrix containing the eigenvalues of A.
P⁻¹ is the inverse of matrix P.
Alternatively, for symmetric matrices:
A=QΛQTA = QΛQ^TA=QΛQT
Where:
Q is an orthogonal matrix (i.e., QT=Q−1Q^T = Q^{-1}QT=Q−1).
Λ is a diagonal matrix of eigenvalues.

C Why is Spectral Decomposition Important?

˛*
 Helps in diagonalizing matrices to simplify computations.
 Key tool for dimensionality reduction, such as in PCA.
 Used in solving differential equations and analyzing dynamic systems.
 Useful in image compression, signal processing, and machine learning.

o Conditions for Spectral Decomposition

«
#
 The matrix must be square (same number of rows and columns).
 The matrix must be diagonalizable.
 For real symmetric matrices, eigenvalues are real and eigenvectors are orthogonal.

´k.́ Step-by-Step Example of Spectral Decomposition

˙
×
Let’s consider the matrix:
A=| 4 1|
|1 3|

⬛ Step 1: Find the Eigenvalues

We solve the characteristic equation:
det(A − λI) = 0
 That is,
| 4−λ 1 |
|1 3−λ | = 0
Now compute the determinant:
(4−λ)(3−λ) − (1)(1) = 0
⇒ λ² − 7λ + 11 = 0
Solve the quadratic equation:
λ² − 7λ + 11 = 0
Using the quadratic formula:
λ = [7 ± √(49 − 44)] / 2 = [7 ± √5] / 2
So,
λ₁ ≈ 5.618
λ₂ ≈ 1.382

⬛ Step 2: Find Eigenvectors

For λ₁ ≈ 5.618:
We solve (A − λI)v = 0
A − λ₁I =
| 4 − 5.618 1 | ≈ | −1.618 1 |
|1 3 − 5.618 | | 1 −2.618 |
Now solve:
−1.618 * x + 1 * y = 0
⇒ y = 1.618 * x
Let x = 1, then y ≈ 1.618

Eigenvector v₁ ≈ [1, 1.618]

Normalize:

‖v₁‖ = √(1² + 1.618²) = √(1 + 2.618) ≈ √3.618 ≈ 1.902

Normalized v₁ ≈ [1 / 1.902, 1.618 / 1.902] ≈ [0.525, 0.851]
So,
v₁ ≈ [0.525, 0.851]
For λ₂ ≈ 1.382:
A − λ₂I =
| 4 − 1.382 1 | ≈ | 2.618 1 |
|1 3 − 1.382 | | 1 1.618 |
Now solve:
2.618 * x + 1 * y = 0
⇒ y = −2.618 * x
Let x = 1, then y ≈ −2.618
Eigenvector v₂ ≈ [1, −2.618]
Normalize:

‖v₂‖ = √(1² + (−2.618)²) = √(1 + 6.854) ≈ √7.854 ≈ 2.802

Normalized v₂ ≈ [1 / 2.802, −2.618 / 2.802] ≈ [0.357, −0.934]
So,
v₂ ≈ [0.357, −0.934]

⬛ Step 3: Form the Decomposition

Let:
Q (Matrix of Eigenvectors) =
| 0.525 0.357 |
| 0.851 −0.934 |
Λ (Diagonal Matrix of Eigenvalues) =
| 5.618 0 |
|0 1.382 |
Since the original matrix A is symmetric, spectral decomposition gives:
A = Q × Λ × Qᵗ

⬛ Final Result:
A≈
Q × Λ × Qᵗ ≈
| 0.525 0.357 | × | 5.618 0 | × | 0.525 0.851 |
| 0.851 −0.934 | | 0 1.382 | | 0.357 −0.934 |
This reconstruction gives a close approximation of the original matrix A.
# Application: Dimensionality Reduction via PCA
ç/¡
◎
● Principal Component Analysis (PCA)
´
’"
Principal Component Analysis (PCA) is a statistical technique used to reduce the
dimensionality of a dataset while retaining as much variability (information) as possible. It
does this by transforming the original variables into a new set of uncorrelated variables
called principal components. These components are ordered in such a way that the first few
retain most of the variation present in all of the original variables.

Q Role of Spectral Decomposition in PCA

˙
•
The core of PCA lies in spectral decomposition (also known as eigendecomposition) of a
matrix—specifically, the covariance matrix of the dataset.
Here’s how it works:
Step 1: Standardize the Data
If the dataset has multiple features with different units or scales, we first standardize the
data so that each feature has a mean of 0 and a standard deviation of 1.
Step 2: Compute the Covariance Matrix
Let X be the data matrix (rows = observations, columns = variables). The covariance
matrix C is calculated as:
C = (1/n) × XᵗX
This matrix represents how variables relate to each other.
Step 3: Perform Spectral Decomposition
We then decompose the covariance matrix C using spectral decomposition:
C = Q × Λ × Qᵗ
Q is a matrix of eigenvectors (principal components).
Λ is a diagonal matrix of eigenvalues, which represent the amount of variance captured
by each principal component.
Step 4: Select Top Components
The eigenvalues are sorted in descending order. The top k eigenvectors corresponding to
the k largest eigenvalues are chosen to form a reduced basis for the data. These top
eigenvectors capture the most significant patterns or directions of variance in the data.
Step 5: Project Data onto New Space
We project the original data onto the new basis formed by these top eigenvectors. This step
reduces the data from n dimensions to k dimensions, where k < n, with minimal loss of
information.

/ The Power of Spectral Decomposition

⬛#
Eigenvectors of the covariance matrix point to the directions of maximum variance in the
data.
Eigenvalues tell us how much variance is explained by each direction.
By selecting the eigenvectors with the highest eigenvalues, PCA ensures that the reduced
data preserves the most important features.
Thus, spectral decomposition provides a mathematical foundation for identifying and
ranking directions of variance, making PCA a reliable and efficient method for
dimensionality reduction.

˛C Real-World Applications of PCA

*
Image Compression: Reducing image data without losing important features.
Face Recognition: Identifying faces using lower-dimensional “eigenfaces”.
Genomics: Analyzing large sets of gene expression data.
Finance: Identifying factors that drive market movements.
Data Visualization: Projecting high-dimensional data into 2D or 3D space for easier
interpretation.

Dimensionality Reduction via PCA Using Eigen Decomposition

Example : A 2×2 Covariance Matrix
Suppose we have a dataset with two variables, and its covariance matrix is:
C=
[ 2.0 0.8 ]
[ 0.8 0.6 ]

Step 1: Find Eigenvalues and Eigenvectors of the Covariance Matrix

performing eigen decomposition:
 C*v=λ*v
Step 1.1: Find the Characteristic Equation
|C - λI| = 0
That means:

|2-λ 0.8 |
| 0.8 0.6 - λ | = 0
Compute the determinant:
(2 - λ)(0.6 - λ) - (0.8)^2 = 0
= λ² - 2.6λ + (1.2 - 0.64)
= λ² - 2.6λ + 0.56 = 0
Now solve this quadratic equation:
λ = [2.6 ± √(2.6² - 4×1×0.56)] / 2
= [2.6 ± √(6.76 - 2.24)] / 2
= [2.6 ± √4.52] / 2
This gives:
λ₁ ≈ 2.06
λ₂ ≈ 0.54

Step 2: Find Eigenvectors

For λ₁ ≈ 2.06:
Substitute λ = 2.06 into (C - λI):
[ 2 - 2.06 0.8 ] = [ -0.06 0.8 ]
[ 0.8 0.6 - 2.06 ] [ 0.8 -1.46 ]
Now solve the system:
-0.06x + 0.8y = 0
=> y = (0.06 / 0.8)x ≈ 0.075x
So an eigenvector corresponding to λ₁ is:
v₁ = [1, 0.075]

For λ₂ ≈ 0.54:
Substitute λ = 0.54 into (C - λI):
[ 2 - 0.54 0.8 ] = [ 1.46 0.8 ]
[ 0.8 0.6 - 0.54 ] = [ 0.8 0.06 ]
Solve the system:

1.46x + 0.8y = 0
=> y = -1.825x
So an eigenvector corresponding to λ₂ is:
v₂ = [1, -1.825]

Step 3: Principal Components

Now that we have eigenvectors, we can project the data onto these vectors.
The eigenvector corresponding to the larger eigenvalue (approximately 2.06) is the
principal component direction — it captures the maximum variance in the data.
If we want to reduce the 2D data to 1D, we can project the data points onto the first
principal component (v₁).

Practice questions

1. State the formula for Singular Value Decomposition (SVD) of a matrix.

2. List the components involved in Principal Component Analysis (PCA).
3. Recall the condition under which spectral decomposition can be applied to a matrix.
4. Name two applications of SVD in real-world problems.
5. Identify the mathematical tool used in PCA for finding directions of maximum
variance.
6. Illustrate SVD using a 3×3 numerical matrix and show its decomposition.
7. Explain dimensionality reduction using PCA through a 2D to 1D projection with
Python output.
8. Describe the process of reconstructing a matrix using its spectral decomposition.
9. Demonstrate the reduction of features in a dataset using PCA in Python (e.g., Iris
dataset).
10. Classify the roles of U, Σ, and Vᵀ in SVD when applied to image matrices.
11. Perform eigen decomposition of the matrix below and determine its principal
components.

A = [2.0 0.8]
[0.8 0.6]

12. Execute PCA in Python on a dataset with 2 features and reduce it to 1 dimension.
Display both original and reconstructed data using a plot.
13. Compute the SVD of a grayscale image using Python. Display the effect of
choosing k = 10, 50, and 100 singular values.
14. Use NumPy to reconstruct a symmetric matrix from its eigenvalues and
eigenvectors. Verify if the original and reconstructed matrices match.
15. Apply PCA to reduce the Iris dataset to 2 dimensions using Python. Visualize the
result using a scatter plot and label each class.