1.

Vector Spaces
A vector space V is a set of vectors where you can add vectors and scale them
by scalars.

Examples:

2 D vectors R 2

3 D vectors R 3

2. Norms
A norm measures the length of a vector: x ↦ ∥x∥

Properties
1. Absolutely homogeneous: ∥λx∥ = |λ|∥x∥
2. Triangle inequality: ∥x + y∥ ≤ ∥x∥ + ∥y∥
3. Positive definite: ∥x∥ ≥ 0, ∥x∥ = 0 ⟺ x = 0

Common Norms
Manhattan (L 1) norm: ∥x∥ 1 = ∑ |x i |
i

Euclidean (L 2) norm: ∥x∥ 2 = √∑ x

i
2
i

ML Relevance: Weight magnitudes, regularization, measuring distances.

3. Inner Products
An inner product ⟨x, y⟩ generalizes the dot product, measuring
similarity/alignment between vectors.

Properties
Bilinear
Symmetric
Positive definite
Dot Product
In R : ⟨x, y⟩ = x
n ⊤
y = ∑ xi yi
i

ML Relevance
Neuron outputs (weighted sums)
Similarity between features/embeddings
Defining lengths, angles, and projections

4. Vector Spaces with Inner Products

A vector space equipped with an inner product allows measurement of lengths
and angles between vectors.

Example: R with dot product.

n

5. Symmetric Positive Definite (SPD) Matrices

Definition
Symmetric: A = A ⊤

Positive definite: x ⊤
Ax > 0 for all x ≠ 0

Applications
Defines inner products in coordinates: ⟨x, y⟩ = x
^
⊤
Ay
^

Appears in kernels, Hessians, and matrix decompositions

6. Lengths and Distances

Length from Inner Product
∥x∥ = √ ⟨x, x⟩

Distance Function
d(x, y) = ∥x − y∥
Properties (Metric)
1. Positive definite
2. Symmetric
3. Triangle inequality

Cauchy-Schwarz Inequality
|⟨x, y⟩| ≤ ∥x∥∥y∥

Intuition
Inner product → similarity
Distance → separation

7. Angles Between Vectors

Angle Formula
⟨x,y⟩
cos ω =
∥x∥∥y∥

Interpretation
ω = 0 → same direction
ω = π → opposite direction
ω = π/2 → perpendicular

Example: y = 4x ⟹ ω = 0

8. Orthogonality
Definition
Vectors are orthogonal: x ⊥ y ⟺ ⟨x, y⟩ = 0

Orthonormal
Orthogonal + unit length (∥x∥ = ∥y∥ = 1)

Notes:
 0-vector is orthogonal to all vectors
Orthogonality depends on the chosen inner product

9. Orthogonal Matrices
Definition
Square matrix A is orthogonal if columns are orthonormal:
⊤ −1 ⊤
AA = I ⟹ A = A

Properties
Preserves lengths and angles:
∥Ax∥ = ∥x∥

Angle between Ax and Ay = Angle between x and y

Represents rotations/reflections

10. Orthonormal Basis (ONB)

Definition
Basis {b 1, is orthonormal if:
. . . , bn }

⟨b i , b j ⟩ = 0 for i ≠ j, and ∥b ∥ = 1 i

Gram-Schmidt Process
Converts any basis to an orthonormal basis.

Example in R : 2

[1, 1], b =
1 1
b =
1 2 [1, −1]
√2 √2

11. Orthogonal Complement

Definition
For subspace U ⊂ V , the orthogonal complement U ⊥
contains all vectors
orthogonal to U .
Properties
⊥
dim(U ) = dim(V ) − dim(U )

Any x ∈ V decomposes as:

M D−M ⊥
x = ∑ λm bm + ∑ ψj b
m=1 j=1 j

Geometric Intuition
In 3 D: plane → perpendicular line (normal vector)

Applications
Hyperplanes, projections, linear dimensionality reduction

12. Orthogonal Projections

Purpose
Linear transformations that map vectors onto subspaces, minimizing distance.

Applications: Graphics, statistics, machine learning, PCA, autoencoders, linear

regression.

Definition: Projection
Linear mapping π : V → U (subspace U ⊆ V ) is a projection if:
2
π = π ∘ π = π

Projection Matrix
Corresponding projection matrix P satisfies: π

2
P = Pπ
π

12.1 Projection onto a Line (1 D Subspace)

Let line U ⊂ R
n
with basis vector b ∈ R . n

Step 1: Find coordinate λ

Orthogonality: x − π (x) ⊥ b U ⟹ ⟨x − π U (x), b⟩ = 0

Projection: π U (x) = λb
Solve: λ =
⊤
⟨x,b⟩ b x
= ⊤
⟨b,b⟩ b b

If ∥b∥ = 1: λ = ⟨x, b⟩

Step 2: Projection vector

⊤
bb
π U (x) = λb = ⊤ x
b b

Step 3: Projection matrix

(symmetric, rank 1)
⊤
bb
Pπ = ⊤
b b

12.2 Projection onto General Subspaces

Let U ⊂ R
n
of dimension m, with basis B = [b 1, … , bm ] ∈ R
n×m
.

Projection: π U (x) = Bλ ,λ∈R m

Orthogonality: B ⊤
(x − Bλ) = 0

Solve: λ = (B ⊤
B)
−1
B
⊤
x

Projection vector: π U (x) = B(B

⊤
B)
−1
B
⊤
x

Projection matrix: P π = B(B

⊤
B)
−1
B
⊤

Special case: If B is orthonormal (B ⊤

B = I ):

⊤
π U (x) = BB x

⊤
λ = B x

12.3 Gram-Schmidt Orthogonalization

Constructs orthogonal/orthonormal basis from any basis b 1, … , bn :

u1 = b1

u k = b k − π span[u
1 ,…,u k−1 ]
(b k ) , k = 2, … , n

For ONB: e k =
uk

∥u k ∥

12.4 Projection onto Affine Subspaces

Affine subspace: L = x 0 + U , where U is a vector subspace.
Orthogonal projection:
π L (x) = x 0 + π U (x − x 0 )

Distance: d(x, L) = ∥x − π L (x)∥ = ∥(x − x 0 ) − π U (x − x 0 )∥

Applications: Linear regression, PCA, separating hyperplanes.

13. Rotations
Linear transformations that preserve lengths and angles (orthogonal
transformations).

13.1 Rotations in R 2

Standard basis: e 1 = [1, 0]

⊤
,e 2 = [0, 1]
⊤

Rotation by angle θ (counterclockwise):

cos θ − sin θ
R(θ) = [ ]
sin θ cos θ

13.2 Rotations in R 3

Rotation about an axis; occurs in plane perpendicular to axis.

About e -axis:
1

1 0 0
⎡ ⎤
R 1 (θ) = 0 cos θ − sin θ
⎣ ⎦
0 sin θ cos θ

About e -axis:
2

cos θ 0 sin θ
⎡ ⎤
R 2 (θ) = 0 1 0
⎣ ⎦
− sin θ 0 cos θ

About e -axis:
3

cos θ − sin θ 0
⎡ ⎤
R 3 (θ) = sin θ cos θ 0
⎣ ⎦
0 0 1

Convention: Counterclockwise when looking "head-on" along axis toward origin.

13.3 Rotations in n Dimensions (Givens Rotations)
Givens rotation R ij (θ) ∈ R
n×n
rotates the plane spanned by coordinates i and j.

R ij (θ) = I n with:

r ii = cos θ ,r ij = − sin θ

r ji = sin θ ,r jj = cos θ

In 2 D, reduces to R(θ).

13.4 Properties of Rotations

1. Distance-preserving: ∥x − y∥ = ∥R θx − R θ y∥

2. Angle-preserving: Angle between vectors unchanged

3. Non-commutative in ≥3 D: Order matters
4. 2 D rotations are commutative: Form an Abelian group

14. Machine Learning Relevance

Applications
1. Kernel methods: Inner products in feature spaces (kernel trick, kernel-PCA,
Gaussian processes)
2. Computer graphics: Projections for rendering shadows
3. Optimization: Orthogonal projections minimize residual errors
4. Dimensionality reduction: PCA uses projections to reduce dimensionality
5. Linear models: Linear regression uses projections for least-squares solutions

Key Relationships
Vectors → live in vector space
Inner products → lengths, angles, similarity
Norms → lengths
Distances → separation
SPD matrices → inner products in coordinates
 Orthonormal bases → independent axes
Orthogonal complements → perpendicular subspaces
Rotations/orthogonal matrices → preserve geometry

Key Takeaways
1. Projections are linear (π 2
,P
= π π
2
= Pπ ) and minimize distance
2. 1 D projection: P
⊤
bb
π =
b⊤ b

3. General subspace projection: P π = B(B

⊤
B)
−1
B
⊤

4. Orthonormal basis: P π = BB
⊤

5. Gram-Schmidt constructs orthogonal/orthonormal bases

6. Rotations preserve lengths and angles
7. Orthogonal matrices: A −1
= A
⊤

8. Angles and orthogonality depend on the inner product