Module 2: Eigenvalues and Eigenvectors

1 The Eigenvalue Problem

This section tackles the eigenvalue problem, a special equation involving a ma-
trix A , a scalar λ, and a vector v . Solving it reveals how the matrix transforms
vectors and identifies special vectors that only change in scale, not direction.
We will crack the code of this equation and unlock the power of eigenvalues and
eigenvectors!

Definition. If A is an n × n matrix, do there exist non-zero vectors x̄ ∈ Rn

such that
A x̄ is a scalar multiple of x̄, i.e., Ax̄ = λx̄ ? This scalar, often denoted by
the Greek letter lambda, is called the eigenvalue of A, and the non-zero vector
x̄ is called the eigenvector of A corresponding to λ.
Remark. Note that we omit the case x̄ = 0 since A0 = λ0 is true for all values
of λ. An eigenvalue λ = 0, however, is possible. The equation

Ax̄ = λx̄
is equivalent to

(A − λI)x̄ = 0, x̄ ̸= 0 (*)
∗
where I is the n × n identity matrix. If equation ( ) is to have non-zero
solutions, then λ must be chosen so that the ×n matrix A − λI is singular. That
is, det(A − λI) = 0. Therefore the eigenvalue problem consists of two parts:

1. Find all scalars λ such that the matrix A−λl is singular, i.e. det(A−λI) =
0.
2. Given that A − λI is singular, find all the non-zero vectors x̄ such that
(A − λI)x̄ = 0.

Clearly if we know an eigenvalue of A, then the elementary row operation

techniques provide an efficient way to find the eigenvectors.

2 The Spectral Decomposition

The goal of this module is to prove the Spectral Theorem, stated as follows.

2.1 Spectral Theorem

Let A ∈ Rn×n be symmetric ( A⊤ = A ). Then there exists an orthonormal
basis of eigenvectors of A, {v1 , . . . , vn }, such that

1
 A = VΛ V−1 = VΛ V⊤
 
where V = v1 · · · vn and Λ is diagonal and contains the eigenvectors
of A.
Let’s begin with a reminder of the following fundamental concept of Linear
Algebra.
Let A ∈ Rn×n . A scalar-vector pair (λ, v) ∈ R × Rn with v ≠ 0 is an
eigenvaluel eigenvector pair of A if

Av = λv
Geometrically, eigenvectors are ones for which when they are multiplied by
A, the product does not change direction (except for possibly a backward direc-
tional change), but rather the eigenvector is stretched in its same direction by
some scalar factor. First, let’s see how we can compute eigenvalue/eigenvector
pairs for a given matrix. To start, we need the definition and properties of
determinants.

 
a b
Definition. Given 2 × 2 matrix A = , its determinant is
c d
det(A) := ad − bc
n×n
For a general A ∈ R , its determinant is defined iteratively as
n
X
det(A) = (−1)k a1k det (Mk )
k=1

where the minor, Mk , is the (n − 1) × (n − 1) matrix formed by deleting the

first row and k-th column of A.
Note that det(A) is always a scalar, and has the following important prop-
erties.
Theorem 7.3. Let A, B ∈ Rn×n . Then the following hold:
(1) det(I) =
1
(2) det A⊤ = det(A)
(3) det(AB)
= det(A) det(B)
(4) det A−1 = det(A)1
provided A is invertible
n
(5) det(αA) = α det(A) for any α ∈ R Qn
(6) If A has eigenvalues λ1 , . . . , λn , then det(A) = i=1 λi
(7) det(A) ̸= 0 if and only if A is invertible.
Now eigenvalue/eigenvector pairs are distinctly related to determinants in
the following way.

2
2.2 Proposition
Let A ∈ Rn×n . Then (λ, v) ∈ R × Rn with v ̸= 0 is an eigenvalueleigenvector
pair of A if and only if

det(A − λI) = 0

2.2.1 Proof
For the forward direction, suppose that Av = λv. Then (A − λI)v = 0, which
means that v ∈ N (A − λl). Since v ̸= 0, A − λl has nontrivial nullspace, and
hence is not invertible; so by Theorem 4.4(7), det(A − λI) = 0.
To see the converse, if det(A − λI) = 0, then A is not invertible, and hence has
nontrivial nullspace. Thus there exists a nonzero v such that (A − λI)v = 0,
which implies that Av = λv. Something important to note is that even if a
matrix is real-valued, its eigenvalues can be complex. Indeed, if
 
0 1
A :=
−1 0
then the solution to det(A − λI) = λ2 + 1 = 0 is λ = ±i where i is the
imaginary unit. Nonetheless, an important class of matrices do in fact have real
eigenvalues as the following proposition shows.

2.3 Proposition
Let A ∈ Rn×n be symmetric (i.e., A⊤ = A ). Then all eigenvalues of A are real.

2.3.1 Proof
Suppose (λ, v) is an eigenvalue/eigenvector pair of A . By conjugate-linearity in
the second argument for inner products, we have

λ⟨v, v⟩ = ⟨λv, v⟩ = ⟨Av, v⟩ = v, A⊤ v = ⟨v, Av⟩ = ⟨v, λv⟩ = λ̄⟨v, v⟩ (2)

Since v ̸= 0, ⟨v, v⟩ =
̸ 0, thus (2) implies that λ = λ̄, hence λ must be real.
Next we will note that distinct eigenvalues yield orthogonal eigenvectors.

2.4 Proposition
Suppose A ∈ Rn×n is symmetric, and (λ1 , ν1 ) and (λ2 , ν2 ) are eigenvalueleigen-
vector pairs of A ∈ Rn×n . If λ1 ̸= λ2 , then ν1 is orthogonal to v2 .

3
2.4.1 Proof
Note that (since λ1 , λ2 ∈ R )

λ1 ⟨v1 , v2 ⟩ = ⟨λ1 v1 , v2 ⟩ = ⟨Av1 , v2 ⟩ = ⟨v1 , Av2 ⟩ = ⟨v1 , λ2 v2 ⟩ = λ2 ⟨v1 , v2 ⟩

This implies that (λ1 − λ2 ) ⟨v1 , v2 ⟩ = 0, but λ1 − λ2 ̸= 0 by assumption, so

v1 and v2 must be orthogonal.
This leads to the following observation:

2.5 Proposition
(Easy case of the Spectral Theorem). Suppose that A ∈ Rn×n is sym-
metric and has n distinct, nonzero eigenvalues, λ1 , . . . , λn . Then A has n or-
thonormal eigenvectors, and we may write
n
X
−1 ⊤
A = SΛS = SΛS = λi S:i S:i⊤
i=1

where the columns of S are the eigenvectors of A and Λ = diag (λ1 , . . . , λn ).

Remark. Note that this result proves the Spectral Theorem in the case that A
has distinct eigenvalues, so the only case we have left to consider is that when
A has repeated eigenvalues. We will illustrate a couple of ways one can finish
the proof of the general case.
Our first argument for proving the Spectral Theorem is a more constructive
one which boils down to repeatedly doing Gram-Schmidt on blocks of the initial
matrix. Suppose that (λ1 , ν1 ) is any eigenvalue/eigenvector pair of A and note
that without loss of generality we may assume that v1 is a unit vector (if not
divide by its norm). We know that there is an orthonormal basis for Rn con-
taining v1 , say {v1 , w2 , . . . , wn }. Let Q1 be the matrix with these basis vectors
as columns. Then by orthonormality of its columns, we have
 
λ1 0
Q1 AQ⊤ 1 =
0 B1
for some symmetric matrix B1 ∈ R(n−1)×(n−1) .
Before proceeding we need the following crucial observation

2.6 Lemma
Let A ∈ Rn×n . If Q ∈ Rn×n is an orthogonal matrix, then QAQ⊤ has the same
eigenvalues as A.

4
2.6.1 Proof
Let (λ, v) be an eigenvalue/eigenvector pair of QAQ⊤ . Then QAQ⊤ v = λv, and
by multiplying by Q⊤ on the left, we see that

AQ⊤ v = λQ⊤ v
which means that λ, Q⊤ v is an eigenvalue/eigenvector pair of A. Since

both A and QA ⊤ have the same rank and this argument works for all eigenpairs,
the claim is proven.
Remark. Now we may apply the conclusion of the lemma to note that Q1 AQ⊤ 1
has the same eigenvalues as A, hence the eigenvalues of B1 are λ2 , . . . , λn . Now
we iterate the same argument as above to find an orthogonal Qe for which
 
Q e ⊤ = λ2
e B1 Q 0
,
0 B2
and we set
 
1 0
Q2 :=
0 Q
e

Now we have that

 
λ1 0 0
Q2 Q1 AQ⊤
1 Q ⊤
2 =  0 λ2 0 
0 0 B2
Iterating this procedure, we find Q1 , . . . , Qn which diagonalize A into Λ =
diag (λ1 , . . . , λn ) as desired, and we take S := Qn . . . Q1 to get the desired de-
composition.

2.6.2 Second Proof

Our second argument is nonconstructive, but follows from somewhat more basic
Linear Algebra facts. We begin with the following observation.

2.7 Lemma
Let A ∈ Rn×n be symmetric, and let S be any subspace of Rn . If Ax ∈ S for
every x ∈ S, then Ay ∈ S ⊥ for every y ∈ S ⊥ .

2.7.1 Proof
By assumption, ⟨Ax, y⟩ = 0, but by symmetry, ⟨Ax, y⟩ = ⟨x, Ay⟩, so Ay ∈ S ⊥
since x is an arbitrary element of S.
Lemma 7.12. If A ∈ Rn×n is symmetric and S is a nonzero subspace of Rn
such that Ax ∈ S for every x ∈ S, then S contains an eigenvector of A.

5
 Proof
First of all, we know that S contains an orthonormal basis by Theorem 3.4; call
k
it {u1 , . . . , uk }. Since Auj ∈ S for all j, there are scalars {rij }i,j=1 such that
k
X
Auj = rij ui
i=1

Indeed rij = ⟨Auj , ui ⟩. Let R be the k × k matrix with entries Rij =

rij . Then note that R is symmetric because rji = ⟨Aui , uj ⟩ = ⟨ui , Auj ⟩ =
⟨Auj , ui ⟩ = rij .
Now suppose that (λ, v) is an eigenvalue/eigenvector pair of R. Then define
Pk
w := j=1 vj uj where vj is the j-th component of the vector v, and uj is still
one of the basis vectors for S. Then we have
k
X
Aw = vj Auj
j=1
k
X k
X
= vj rij ui
j=1 i=1
 
k
X Xk
=  rij vj  ui
i=1 j=1
k
X
= (Rv)i ui
i=1
k
X
= λvi ui
i=1
= λw
Hence (λ, w) is an eigenvalue/eigenvector pair of A.
Now we are ready to complete the proof of the Spectral Theorem.
Proof of the Spectral Theorem. Let (λ1 , ν1 ) be an eigenvalue/eigenvector pair
of A , and set S1 := span (ν1 ). Note that A (αν1 ) = λαν1 ∈ S1 , so by Lemma
4.10, Ay ∈ S1⊥ for every y ∈ S1⊥ . If S1⊥ ̸= {0}, then Lemma 4.11 implies that
there exists an eigenvector ν2 ∈ S⊥ 1 (which is necessarily orthogonal to ν1 ).
Now if n > 2, then set S2 := span (ν1 , ν2 ), and note that Ax ∈ S2 for every
x ∈ S 2 by a similar argument to the first case. Again apply Lemmas 4.10 and
4.11 to get an orthogonal eigenvector ν3 ∈ S⊥ 2 . Continue in this manner to
choose v1 , . . . , vn and the proof is complete.
Now let us note that the Spectral Theorem gives us a representation of a matrix
in terms of the sum of rank 1 matrices.
n
Corollary Let A ∈ Rn×n be symmetric, and let {vi }i=1 be an orthonormal
n
basis of eigenvectors of A with corresponding eigenvalues {λi }i=1 . Then

6
 n
X
A= λi vi vi⊤
i=1

Proof. First note that

 
λ1 0 ··· 0
 .. .. 
  0 λ2 . . 
VΛ = v1 ··· vn  ..

 .. .. 
 . . . 0 
0 ··· 0 λn
 
= λv1 ··· λn vn .
Since V −1 = V ⊤ , we have

v1⊤
 
n
 .  X
∨Λv −1 = λi vi vi⊤

λv1 ··· λn vn  ..  =
vn⊤ i=1

as required.

3 Python Programming Tasks

3.1 Learning Tasks for Spectral Decomposition (Eigende-
composition) with Python
These tasks will guide learners through the fundamental concepts of spectral
decomposition, hands-on implementation in Python, and the exploration of ap-
plications using visualizations. Learners will gain an understanding of eigenval-
ues, eigenvectors, matrix reconstruction, and how these are applied in various
areas of linear algebra and data science.

3.2 Task 1: Review the Theory of Spectral Decomposition

Goal: Ensure a clear understanding of the theoretical basis of spectral decom-
position, including eigenvalues and eigenvectors.

3.2.1 Instructions:
1. Define spectral decomposition and explain the role of eigenvalues and
eigenvectors in this process.
2. Show that for any matrix A, spectral decomposition takes the form A =
QQQ−1 , where Q contains eigenvectors and Λ is a diagonal matrix of
eigenvalues.

3. Derive the spectral decomposition for the following matrix by hand:

7
  
3 1
A=
0 2

4. Compute the eigenvalues and eigenvectors manually and verify the recon-
struction of the matrix.

Expected Outcome: A written explanation and manual calculations show-

ing the spectral decomposition of a simple 2 × 2 matrix.

3.3 Task 2: Implement Spectral Decomposition in Python

Goal: Write Python code to perform spectral decomposition of a square matrix
and reconstruct the matrix using eigenvalues and eigenvectors.

3.3.1 Instructions:
1. Implement a Python function that:

• Takes a square matrix A as input.

• Computes the eigenvalues and eigenvectors using NumPy’s [Link]()
function.
• Reconstructs the matrix using the formula A = QΛQ−1 .

2. Test your implementation on the following matrix:

 
4 2
A=
1 3

3. Verify that the reconstructed matrix matches the original matrix.

Expected Outcome: A working Python function that outputs the eigen-

values, eigenvectors, and the reconstructed matrix, verifying the correctness of
the decomposition.

3.4 Task 3: Explore Visualization of Eigenvectors

Goal: Use Python’s Matplotlib to visualize eigenvectors as vectors in 2D space.

3.4.1 Instructions:
1. Write Python code to:

• Plot the original matrix A as a heatmap.

• Plot the eigenvectors of the matrix as arrows (vectors) on a 2D plane.

8
 • Scale the length of each eigenvector according to the corresponding eigen-
value.
 
4 2
2. For the matrix A = , visualize the eigenvectors and label them
1 3
with their corresponding eigenvalues.
Expected Outcome: A visualization with two arrows representing the
eigenvectors of matrix A, scaled by their eigenvalues, alongside a heatmap
of the matrix.

3.5 Task 4: Spectral Decomposition in 3D

Goal: Extend your Python implementation to handle 3D matrices and visualize
eigenvectors in 3D space.

3.5.1 Instructions:
1. Modify your spectral decomposition function to handle 3 × 3 matrices.
2. Use a 3 × 3 matrix such as:

 
4 1 2
A= 1 3 1 
2 1 5

3. Write Python code to visualize the eigenvectors of this matrix in 3D using

mpl toolkits.mplot3d.
4. Display the eigenvectors as arrows originating from the origin and scaled
by their eigenvalues.
Expected Outcome: A 3D plot showing the eigenvectors of the 3 × 3
matrix, scaled by their corresponding eigenvalues.

3.6 Task 5: Numerical Stability and Symmetric Matrices

Goal: Investigate the behavior of spectral decomposition for symmetric matrices
and discuss the numerical stability of the process.

3.6.1 Instructions:
1. Explain why the eigenvalues of symmetric matrices are always real and
why the eigenvectors are orthogonal.
2. Test your spectral decomposition code with the following symmetric ma-
trix:

9
  
6 2 1
A= 2 3 1 
1 1 2

3. Verify the orthogonality of the eigenvectors by checking that their dot

products are zero.

4. Discuss the numerical stability of eigendecomposition when dealing with

symmetric matrices, and compare it with the decomposition of a non-
symmetric matrix.
Expected Outcome: A report explaining the properties of symmetric
matrices, eigenvalue behavior, and numerical stability, along with verifi-
cation of orthogonality in your Python code.

3.7 Task 6: Apply Spectral Decomposition to Dimension-

ality Reduction
Goal: Understand how spectral decomposition can be applied to dimensionality
reduction techniques such as Principal Component Analysis (PCA).

3.7.1 Instructions:
1. Research how PCA uses the eigenvalues and eigenvectors of the covariance
matrix to perform dimensionality reduction.
2. Download a standard dataset, such as the Iris dataset, and:

• Compute the covariance matrix of the dataset.

• Perform spectral decomposition of the covariance matrix.
• Use the eigenvectors to project the data onto the principal components
(reduce the dimensionality to 2D).

3. Visualize the original dataset and the reduced dataset in 2D.

Expected Outcome: A 2D plot of the original and reduced datasets, along

with a clear explanation of how PCA is related to spectral decomposition.

3.8 Task 7: Solve Practical Problems Using Spectral De-

composition
Goal: Apply spectral decomposition to solve practical problems such as solving
linear systems and computing matrix powers.

10
3.8.1 Instructions:
1. Use the following matrix A :

 
2 1
A=
1 3

2. Compute its spectral decomposition.

3. Use the decomposition to:
 
1
• Solve the linear system Ax = b for b = .
2

• Compute the matrix power A5 using the spectral decomposition.

4. Compare the result with the direct computation of A5 using np. linalg.
matrix power().

Expected Outcome: A step-by-step solution of the linear system and the

matrix power computation using both spectral decomposition and direct meth-
ods, verifying the results.

4 Content Curated Video

Visual learners may benefit from the following YouTube videos that explain
the concepts of eigenvalues, eigenvectors, and the Spectral Theorem. These
videos offer step-by-step explanations and visualizations to help solidify your
understanding of spectral decomposition.

• Video Visit the URL below to view a video:

[Link]

Eigenvalues and Eigenvectors — Explained by 3Blue1Brown

4.1 Python Code for Eigenvalue Calculation

To compute eigenvalues and eigenvectors for a given matrix, we can use Python’s
NumPy library. Here’s an example code to calculate eigenvalues and eigenvec-
tors:

11
4.1.1 Eigenvalue and Eigenvector Calculation in Python
import numpy as np
\# Define a symmetric matrix
A = [Link] ([[4, 1],
[1,3]])

# Compute eigenvalues and eigenvectors}

eigenvalues, eigenvectors = [Link] (A)
print("Eigenvalues:", eigenvalues)
print("Eigenvectors:\textbackslash n", eigenvectors)

This code computes the eigenvalues and eigenvectors of a symmetric matrix

A , which is critical for understanding the spectral decomposition process.
For more examples and code resources, visit the following links:

• [Link] Eigenvalue De-

composition Code - GitHub

This code computes the eigenvalues and eigenvectors of a symmetric matrix

A, which is critical for understanding the spectral decomposition process.

5 Recommended Supplementary Reading

For a deeper understanding of the Spectral Theorem and linear algebra concepts
related to eigenvalues and eigenvectors, refer to the following textbooks:

1. Textbook: Read Chapter 5 (”Eigenvalues and Eigenvectors”) from ”In-

troduction to Linear Algebra” by Gilbert Strang.
2. Online Article: [Link]
and Eigenvectors Explained Visually

These resources provide comprehensive coverage of matrix theory, spectral

decomposition, and various applications of eigenvalues in real-world problems.
These resources provide comprehensive coverage of matrix theory, spectral de-
composition, and various applications of eigenvalues in real-world problems.

12