Linear Algebra Notes

Meaning, Method, and Mastery (with exercises & solutions)

Md. Abrarul Hoque Saad

January 2026
Contents

Preface iv

I Vectors and Systems 1

1 Vectors, Dot Product, and Geometry 2

1.1 Vectors: object vs coordinates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Dot product and norm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Projection (first “best approximation”) . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Linear Systems and Gaussian Elimination 7

2.1 From equations to a matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Row operations (why they are legal) . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Row echelon forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.4 Worked example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

II Spaces and Structure 11

3 Vector Spaces, Subspaces, Span, Basis, Dimension 12

3.1 Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2 Span and independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3 Basis and dimension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

i
Linear Algebra Notes ii

4 Linear Transformations and Matrices 15

4.1 Linear transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.2 Kernel and image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.3 Rank-nullity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

III Orthogonality and Approximation 17

5 Orthogonality, Gram–Schmidt, and QR 18

5.1 Inner product and orthonormal sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.2 Gram–Schmidt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.3 QR idea (conceptual) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

6 Least Squares (the “closest possible” solution) 20

6.1 Meaning first . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
6.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

IV Determinants and Eigenvalues 22

7 Determinants, Inverses, and LU (practical tools) 23

7.1 Determinant meaning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
7.2 LU idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
7.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

8 Eigenvalues and Diagonalization 25

8.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
8.2 A full worked example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
8.3 Diagonalization (meaning) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
8.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

9 Symmetric Matrices and the Spectral Theorem 28

9.1 Key statements (student version) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
9.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Linear Algebra Notes iii

V SVD and Data View 30

10 SVD (Singular Value Decomposition) and PCA idea 31

10.1 Statement and meaning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
10.2 PCA (very short student view) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
10.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

VI Appendix 33

A Formula Sheet (quick revision) 34

Preface

Explanation (human voice)

This note is designed for a student who wants a complete learning path in linear algebra.
Every chapter follows a simple idea: first understand the story, then learn the definitions, then
practice until it becomes automatic.
This book is written in a friendly tone on purpose. If a concept feels abstract, that usually
means the meaning was not explained properly. So these pages try to explain the meaning
before doing heavy computation.

Why this matters

How to study from this book (seriously effective):

1. Read the Why this matters box first to know the target.

2. Read the human explanation and try to re-say it in your own words.

3. Do the worked examples line-by-line.

4. Solve exercises without looking at hints.

5. If stuck, read a hint; only then read the full solution.

Key facts to remember

Prerequisites:

• Basic algebra (equations, factoring).

• Basic functions and geometry.

• Some comfort with matrices is helpful, but not required.

iv
 Part I

Vectors and Systems

1
Chapter 1

Vectors, Dot Product, and Geometry

Why this matters

Vectors are the atoms of linear algebra. A vector can represent direction, data, force, or
anything that has multiple components. Once vectors are natural, matrices will feel like
machines that act on vectors.

1.1 Vectors: object vs coordinates

Explanation (human voice)

A vector is not only “a list of numbers”. The numbers are coordinates (a description in a
chosen coordinate system). The vector is the actual object.
In R2 , you can draw it as an arrow. In R100 , you cannot draw it, but it is still the same idea:
addition and scaling follow the same rules.

Definition 1.1: Vector in Rn

A vector in Rn is an ordered list of n real numbers:
 
v1
 
 v2 
v =  . .
 
 .. 
 
vn

2
Linear Algebra Notes 3

Definition 1.2: Addition and scalar multiplication

For u, v ∈ Rn and c ∈ R:

u + v = (u1 + v1 , . . . , un + vn ), cv = (cv1 , . . . , cvn ).

Mental picture

Geometric meaning:

• u + v: do the move u, then the move v.

• cv: stretch/shrink by |c|; reverse direction if c < 0.

Definition 1.3: Linear combination

A linear combination of v1 , . . . , vk is

c1 v1 + · · · + ck vk (ci ∈ R).

Stop & check

If someone asks “What can you build from these vectors?”, the answer is always:

All linear combinations.

That set is the span, and it will come back many times.

1.2 Dot product and norm

Explanation (human voice)

The dot product turns two vectors into one number. That number measures “alignment”.
If u · v is positive, they point roughly the same way. If u · v is negative, they point roughly
opposite. If u · v = 0, they are perpendicular (orthogonal).

Definition 1.4: Dot product

For u, v ∈ Rn ,
n
X
T
u·v =u v = ui vi .
i=1
Linear Algebra Notes 4

Definition 1.5: Norm (length)

√
∥v∥ = v · v.

Common mistake

Do not mix up ∥v∥ and ∥v∥2 .

∥v∥2 = v · v

is useful for computation, but the actual length is ∥v∥.

Theorem 1.1: Cauchy–Schwarz inequality

For all u, v ∈ Rn ,
|u · v| ≤ ∥u∥ ∥v∥ .

Proof sketch

Look at the always-nonnegative quantity ∥u − tv∥2 as a quadratic in t. A quadratic that is

never negative must have discriminant ≤ 0, which forces (u · v)2 ≤ ∥u∥2 ∥v∥2 .

Theorem 1.2: Angle formula

If u, v ̸= 0, then the angle θ between them satisfies

u·v
cos θ = .
∥u∥ ∥v∥

Worked example

Let u = (1, 2, 2) and v = (2, 1, −2). Then u · v = 2 + 2 − 4 = 0, so u ⊥ v. Also ∥u∥ = 3 and

∥v∥ = 3.

1.3 Projection (first “best approximation”)

Explanation (human voice)

Projection means: “I am only allowed to live on a line (or subspace).” So I choose the closest
point on that line to my vector.
This is the start of least squares later.
Linear Algebra Notes 5

Definition 1.6: Projection onto a nonzero vector

For u ̸= 0,
uT v
proju (v) = u.
uT u

Proposition 1.1: Orthogonality of error

If u ̸= 0 then



v = proju (v) + v − proju (v) and u ⊥ v − proju (v) .

Worked example

Project v = (1, 2) onto u = (2, 1):

uT v = 2 · 1 + 1 · 2 = 4, uT u = 22 + 12 = 5,

so
4 8 4
 
proju (v) = (2, 1) = , .
5 5 5

Mini-summary

• Dot product measures alignment.

• Norm is length.

• Orthogonal means dot product = 0.

• Projection is “closest point on a line/subspace”.

1.4 Exercises
Exercise 1.1 Dot product Difficulty: Beginner

Compute (1, 2, 3) · (2, 0, −1).

Solution
2 + 0 − 3 = −1.

Exercise 1.2 Norm Difficulty: Beginner

Compute ∥(3, 4)∥.

Linear Algebra Notes 6

Solution
√
9 + 16 = 5.

Exercise 1.3 Orthogonal set Difficulty: Intermediate

Find all (x, y) ∈ R2 such that (x, y) · (2, 1) = 0.

Solution
2x + y = 0 ⇒ (x, y) = t(1, −2).

Exercise 1.4 Projection Difficulty: Intermediate

Compute proj(1,1,0) (2, 0, 1).

Hint
uT v
Use proju (v) = uT u
u.

Solution

u = (1, 1, 0), v = (2, 0, 1). uT v = 2, uT u = 2, so proju (v) = 1 · (1, 1, 0) = (1, 1, 0).

Exam-style problem

If u ⊥ v, show that ∥u + v∥2 = ∥u∥2 + ∥v∥2 .

Solution

∥u + v∥2 = (u + v) · (u + v) = u · u + 2u · v + v · v = ∥u∥2 + ∥v∥2 .

Chapter 2

Linear Systems and Gaussian

Elimination

Why this matters

Elimination is not just solving a homework system. It is the engine behind rank, invertibility,
solution spaces, and many later theorems.

2.1 From equations to a matrix

Explanation (human voice)

A system is easier to manage when it is stored as a matrix. The matrix is like a compact “data
structure” for equations.
Write the system as Ax = b:

• A = coefficients

• x = unknowns

• b = outputs / right-hand side

Definition 2.1: Augmented matrix

The system Ax = b is stored as h i

A b .

7
Linear Algebra Notes 8

2.2 Row operations (why they are legal)

Definition 2.2: Elementary row operations

• Swap two rows.

• Multiply a row by a nonzero constant.

• Add a multiple of one row to another.

Mental picture

Each row is an equation. All three row operations keep the set of solutions unchanged, so
elimination is safe.

2.3 Row echelon forms

Definition 2.3: Pivot and pivot column

A pivot is the first nonzero entry in a nonzero row of a row-echelon matrix. A pivot column is
a column containing a pivot.

Definition 2.4: REF and RREF

REF: all nonzero rows above zero rows, pivots move to the right as you go down, zeros below
pivots.
RREF: REF plus each pivot is 1 and is the only nonzero entry in its column.

Method (do this)

Gaussian elimination (REF):

1. Choose leftmost pivot; swap a row if needed.

2. Eliminate entries below pivot.

3. Move to next row/column and repeat.

Gauss–Jordan (RREF): also eliminate above pivots and scale pivots to 1.

Common mistake
A row like [0 0 0 | 5] means 0 = 5, so the system has no solution.
Linear Algebra Notes 9

2.4 Worked example

Worked example

Solve: 
 



 x + 2y + z = 7 1 2 1 7
  
2x + 3y + z = 11 ⇒
 2 3 1 11  .




x + y + 2z = 8
 1 1 2 8

Eliminate below the first pivot:

 
1 2 1 7
 
R2 ← R2 − 2R1 , R3 ← R3 − R1 ⇒ 
 0 −1 −1 −3  .

0 −1 1 1

Clean the pivot in row 2:  

1 2 1 7
 
R2 ← −R2 ⇒ 
 0 1 1 3 .

0 −1 1 1
Eliminate below pivot 2:  
1 2 1 7
 
R3 ← R3 + R2 ⇒ 
 0 1 1 3 .

0 0 2 4
Back-substitute: z = 2, then y = 1, then x = 3.

Stop & check

Elimination gives more than an answer:

• If every variable is a pivot variable ⇒ unique solution.

• If some variables are free ⇒ infinitely many solutions.

• If you get 0 = 1 type row ⇒ no solutions.

2.5 Exercises
Exercise 2.1 Two equations Difficulty: Beginner

x + y = 5
Solve .
2x − y = 4
Linear Algebra Notes 10

Solution
Add: 3x = 9 ⇒ x = 3, then y = 2.

Exercise 2.2 Inconsistent Difficulty: Beginner


x + y = 1
Does have a solution?
2x + 2y = 3

Solution
No, because doubling the first gives 2x + 2y = 2, not 3.

Exercise 2.3 Free variable Difficulty: Intermediate

Solve x + 2y + z = 3 in parametric form.

Hint
Choose two variables as parameters.

Solution
Let y = s, z = t. Then x = 3 − 2s − t. So (x, y, z) = (3, 0, 0) + s(−2, 1, 0) + t(−1, 0, 1).
 Part II

Spaces and Structure

11
Chapter 3

Vector Spaces, Subspaces, Span, Basis,

Dimension

Why this matters

This chapter is the language chapter. Span tells you what can be built. Independence tells you
if you have redundancy. Basis and dimension tell you the smallest number of building blocks.

3.1 Subspaces

Definition 3.1: Subspace

A subset W ⊆ Rn is a subspace if:

• 0 ∈ W,

• u, v ∈ W ⇒ u + v ∈ W ,

• c ∈ R, u ∈ W ⇒ cu ∈ W .

Explanation (human voice)

Fast subspace test:

• Check zero vector.

• Check closure under addition.

• Check closure under scaling.

If any one fails, it is not a subspace.

12
Linear Algebra Notes 13

Example 3.1: A plane through the origin

W = {(x, y, z) ∈ R3 : x + y + z = 0} is a subspace.

3.2 Span and independence

Definition 3.2: Span

span(v1 , . . . , vk ) = {c1 v1 + · · · + ck vk : ci ∈ R}.

Definition 3.3: Linear independence

Vectors v1 , . . . , vk are independent if

c1 v1 + · · · + ck vk = 0 ⇒ c1 = · · · = ck = 0.

Mental picture

Independence means: none of the vectors can be built from the others. Dependence means: at
least one is “extra”.

3.3 Basis and dimension

Definition 3.4: Basis
A basis of V is a set of vectors that is (i) spanning and (ii) independent.

Definition 3.5: Dimension

dim(V ) is the number of vectors in any basis of V .

Worked example

Let W = {(x, y, z) ∈ R3 : x + y + z = 0}. Write x = −y − z:

(x, y, z) = (−y − z, y, z) = y(−1, 1, 0) + z(−1, 0, 1).

So W = span{(−1, 1, 0), (−1, 0, 1)} and dim(W ) = 2.

Linear Algebra Notes 14

3.4 Exercises
Exercise 3.1 Subspace? Difficulty: Beginner

Is W = {(x, y) ∈ R2 : x + y = 1} a subspace?

Solution
No, because (0, 0) is not in W .

Exercise 3.2 Dependence Difficulty: Intermediate

Check if (1, 0, 1), (0, 1, 1), (1, 1, 2) are independent.

Solution
They are dependent because (1, 1, 2) = (1, 0, 1) + (0, 1, 1).

Exercise 3.3 Find a basis Difficulty: Advanced

Find a basis for {(x, y, z, w) ∈ R4 : x + w = 0, y − z = 0} and its dimension.

Solution
Constraints: x = −w, y = z. So (x, y, z, w) = w(−1, 0, 0, 1) + y(0, 1, 1, 0). Basis
{(−1, 0, 0, 1), (0, 1, 1, 0)}, dimension 2.
Chapter 4

Linear Transformations and Matrices

Why this matters

A matrix is best understood as a function (a machine) that transforms vectors. This viewpoint
makes kernel, image, rank-nullity, and change of basis feel natural.

4.1 Linear transformations

Definition 4.1: Linear transformation
A function T : V → W is linear if for all u, v ∈ V and c ∈ R:

T (u + v) = T (u) + T (v), T (cu) = cT (u).

Explanation (human voice)

Two rules only: respects addition and scaling. That is why it is called “linear”. Everything else
(matrices, kernels, etc.) grows from these two rules.

4.2 Kernel and image

Definition 4.2: Kernel and image

Ker(T ) = {v ∈ V : T (v) = 0}, Im(T ) = {T (v) : v ∈ V }.

15
Linear Algebra Notes 16

Example 4.1: Kernel and image example

Let T (x, y, z) = (x + z, y − z). Then Ker(T ) solves x + z = 0 and y − z = 0, so (x, y, z) =

t(−1, 1, 1).

4.3 Rank-nullity

Theorem 4.1: Rank–nullity theorem

If T : V → W is linear and V is finite-dimensional, then

dim(Ker T ) + dim(Im T ) = dim(V ).

Explanation (human voice)

This theorem is a balance law: dimensions are “conserved”. If many vectors are crushed into 0
(large kernel), then the output space (image) must be smaller.

4.4 Exercises
Exercise 4.1 Kernel practice Difficulty: Intermediate

For T (x, y) = (x + 2y, 3x + 6y) find Ker(T ) and Im(T ).

Solution
T (x, y) = (x + 2y) (1, 3) so the image is span{(1, 3)} (dimension 1). Kernel satisfies x + 2y =
0 ⇒ (x, y) = t(−2, 1) (dimension 1).
 Part III

Orthogonality and Approximation

17
Chapter 5

Orthogonality, Gram–Schmidt, and

QR

Why this matters

Orthogonality is the “clean geometry” of linear algebra. It gives best approximations, least
squares, and numerically stable computations.

5.1 Inner product and orthonormal sets

Definition 5.1: Inner product on Rn

The standard inner product is ⟨u, v⟩ = uT v.

Definition 5.2: Orthonormal set

Vectors e1 , . . . , ek are orthonormal if ⟨ei , ej ⟩ = 0 for i ̸= j and ∥ei ∥ = 1.

Explanation (human voice)

Orthonormal vectors behave like perfect coordinate axes. This is why computations become
simpler after Gram–Schmidt.

18
Linear Algebra Notes 19

5.2 Gram–Schmidt
Method (do this)

Given independent v1 , . . . , vk :

u1 = v1 , u2 = v2 − proju1 (v2 ), u3 = v3 − proju1 (v3 ) − proju2 (v3 ), . . .

Then normalize: ei = ui / ∥ui ∥.

Worked example

uT
1 v2
Let v1 = (1, 0) and v2 = (1, 1). Then u1 = v1 . Compute proju1 (v2 ) = uT
u1 = 11 (1, 0) = (1, 0).
1 u1
So u2 = v2 − proju1 (v2 ) = (0, 1). Normalize: e1 = (1, 0), e2 = (0, 1).

5.3 QR idea (conceptual)

Explanation (human voice)

QR decomposition says: a matrix can be written as

A = QR

where Q has orthonormal columns (nice geometry) and R is upper triangular (easy to solve
with back-substitution). This is one of the most useful decompositions for solving least squares.

5.4 Exercises
Exercise 5.1 Orthonormal? Difficulty: Beginner
√ √
Are u = (1, 1)/ 2 and v = (1, −1)/ 2 orthonormal?

Solution
∥u∥ = 1, ∥v∥ = 1, and u · v = (1 − 1)/2 = 0, so yes.
Chapter 6

Least Squares (the “closest possible”

solution)

Why this matters

Often Ax = b has no solution because data is inconsistent. Least squares replaces the impossible
goal “solve exactly” with the realistic goal “make the error as small as possible”.

6.1 Meaning first

Explanation (human voice)

The vector Ax always lives in Col(A) (the column space). So even if b is outside Col(A), you
can still pick the point in Col(A) that is closest to b.
That closest point is the projection of b onto Col(A). So least squares is projection, but in
higher dimension.

Definition 6.1: Least squares

A least squares solution minimizes the error:

min ∥Ax − b∥ .
x

Theorem 6.1: Normal equations

Any least squares solution satisfies

AT Ax = AT b.

20
Linear Algebra Notes 21

Worked example

Fit a constant x to data (1, 2, 4):

   
1 1
   
1 ,
A= 
2 .
b= 
1 4

Normal equations: AT A = 3, AT b = 7, so x = 73 .

6.2 Exercises
Exercise 6.1 Check normal equation Difficulty: Intermediate
! !
1 2
Let A = and b = . Find the least squares solution.
2 0

Solution

AT A = 12 + 22 = 5, AT b = 1 · 2 + 2 · 0 = 2. So x = 2/5.
 Part IV

Determinants and Eigenvalues

22
Chapter 7

Determinants, Inverses, and LU

(practical tools)

Why this matters

Determinant answers: “is this matrix invertible?” LU answers: “can we solve many systems
fast?” These are extremely practical in computation.

7.1 Determinant meaning

Explanation (human voice)

In R2 , | det(A)| is the area scaling factor of the transformation. In R3 , it is volume scaling. If

det(A) = 0, the transformation crushes space into a lower dimension, so inverse cannot exist.

Definition 7.1: Invertible matrix

A square matrix A is invertible if there exists A−1 such that AA−1 = A−1 A = I.

Proposition 7.1: Determinant test (informal)

If det(A) ̸= 0, then A is invertible; if det(A) = 0, then A is not invertible.

23
Linear Algebra Notes 24

7.2 LU idea
Explanation (human voice)

LU decomposition writes A = LU where:

• L is lower triangular (easy forward substitution),

• U is upper triangular (easy back substitution).

This is basically elimination stored as matrices.

7.3 Exercises
Exercise 7.1 Invertible? Difficulty: Beginner
!
1 2
Is A = invertible?
2 4

Solution
det(A) = 1 · 4 − 2 · 2 = 0, so not invertible.
Chapter 8

Eigenvalues and Diagonalization

Why this matters

Eigenvectors are special directions that a matrix does not turn. Along those directions the
matrix only scales. This explains repeated processes Ak , stability, vibrations, and more.

8.1 Definitions
Definition 8.1: Eigenvalue and eigenvector

For square A, a nonzero vector v is an eigenvector with eigenvalue λ if

Av = λv.

Definition 8.2: Characteristic polynomial

Eigenvalues are roots of

det(A − λI) = 0.

25
Linear Algebra Notes 26

8.2 A full worked example

Worked example
!
2 1
Let A = .
1 2
!
2−λ 1
det(A − λI) = det = (2 − λ)2 − 1 = λ2 − 4λ + 3 = (λ − 1)(λ − 3).
1 2−λ

Eigenvalues: 1 and 3.
For λ = 3, solve (A − 3I)v = 0:
!
−1 1
v = 0 ⇒ v = (1, 1).
1 −1

For λ = 1: !
1 1
v = 0 ⇒ v = (1, −1).
1 1

8.3 Diagonalization (meaning)

Explanation (human voice)

Diagonalization means: choose a basis made of eigenvectors. In that basis, the matrix becomes
diagonal, so powers Ak become easy.
Not every matrix is diagonalizable. A good quick rule: you need enough independent eigenvec-
tors.

8.4 Exercises
Exercise 8.1 Triangular shortcut Difficulty: Beginner
!
5 1
Find eigenvalues of A = .
0 2

Solution
Eigenvalues are diagonal entries: 5 and 2.
Linear Algebra Notes 27

Exercise 8.2 Not diagonalizable Difficulty: Advanced

!
1 1
Is A = diagonalizable over R?
0 1

Hint
Find eigenvectors for λ = 1 and count how many independent ones exist.

Solution
!
0 1
Only eigenvalue is λ = 1. (A − I) = so eigenvectors satisfy y = 0, giving vectors (x, 0).
0 0
Only one independent eigenvector exists, so not diagonalizable.
Chapter 9

Symmetric Matrices and the Spectral

Theorem

Why this matters

Symmetric matrices are the “nicest” real matrices: they have real eigenvalues and orthonormal
eigenvectors. This is why so many real applications prefer symmetric matrices.

9.1 Key statements (student version)

Explanation (human voice)

For symmetric A = AT :

• Eigenvalues are real.

• Eigenvectors from different eigenvalues are orthogonal.

• You can diagonalize using an orthonormal basis.

This is a huge win: geometry + algebra become perfectly aligned.

Theorem 9.1: Spectral theorem (informal)

If A is real symmetric, then

A = QΛQT

where Q is orthogonal and Λ is diagonal (real eigenvalues).

28
Linear Algebra Notes 29

9.2 Exercises
Exercise 9.1 Orthogonality of eigenvectors Difficulty: Intermediate

Suppose A = AT and Av = λv, Aw = µw with λ ̸= µ. Show v ⊥ w.

Hint

Use v T Aw in two ways and symmetry (AT = A).

Solution

v T Aw = v T (µw) = µv T w. Also v T Aw = (AT v)T w = (Av)T w = (λv)T w = λv T w. So

(λ − µ)v T w = 0, hence v T w = 0.
 Part V

SVD and Data View

30
Chapter 10

SVD (Singular Value Decomposition)

and PCA idea

Why this matters

SVD works for every matrix, not just square ones. It explains a matrix as “rotate, stretch,
rotate”. That is why it powers compression, denoising, and PCA.

10.1 Statement and meaning

Definition 10.1: SVD

For any A ∈ Rm×n , there exist orthogonal U ∈ Rm×m and V ∈ Rn×n and a diagonal Σ
(nonnegative entries) such that
A = U ΣV T .

Explanation (human voice)

SVD mental model:

• V T rotates the input space.

• Σ stretches along perpendicular axes.

• U rotates the output space.

So SVD is the cleanest “geometry of a matrix”.

31
Linear Algebra Notes 32

10.2 PCA (very short student view)

Explanation (human voice)

PCA uses SVD (or eigenvalues of a covariance matrix) to find directions of maximum variance.
In human words: it finds the most important directions where data changes the most.

10.3 Exercises
Exercise 10.1 Rank vs singular values Difficulty: Advanced

If rank(A) = r, how many singular values are nonzero?

Solution
Exactly r.
 Part VI

Appendix

33
Appendix A

Formula Sheet (quick revision)

Key facts to remember

• Dot product: u · v =
P
ui vi .
√
• Norm: ∥v∥ = v · v.
uT v
• Projection: proju (v) = uT u
u.

• Least squares normal equation: AT Ax = AT b.

• Eigen equation: Av = λv.

34