A Complete Tutorial on

Eigenvalues, Eigenvectors, and

Principal Component Analysis
From Fundamentals to Applications
Prepared by Shashikant Chaudhary
[Link]

enhanced by Generative AI

A Comprehensive Guide with Examples and Visualizations

December 13, 2025

Abstract
This tutorial provides a comprehensive journey from the fundamental concepts of
eigenvalues and eigenvectors to their powerful application in Principal Component
Analysis (PCA). We begin with intuitive explanations, progress through rigorous
mathematical definitions, work through detailed examples with step-by-step solu-
tions, and conclude with practical applications in data science. Throughout the
tutorial, we include visualizations to build geometric intuition and Python code
snippets for implementation.

Contents

1 Introduction: Why Study Eigenvalues? 3

1.1 Applications Across Fields . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Matrices as Linear Transformations 3

2.1 Geometric Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3 Formal Definitions 4
3.1 Etymology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

4 Finding Eigenvalues: The Characteristic Equation 4

4.1 Deriving the Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
CONTENTS Eigenvalues & PCA Tutorial

5 Step-by-Step Examples 5
5.1 Example 1: A 2×2 Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . 5
5.2 Geometric Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
5.3 Example 2: A 3×3 Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . 7
5.4 Example 3: Complex Eigenvalues . . . . . . . . . . . . . . . . . . . . . . 7

6 Important Properties of Eigenvalues 8

7 Eigendecomposition (Diagonalization) 8
7.1 Constructing the Decomposition . . . . . . . . . . . . . . . . . . . . . . . 8

8 Transition to Principal Component Analysis 10

8.1 The Problem PCA Solves . . . . . . . . . . . . . . . . . . . . . . . . . . 10

9 Mathematical Foundation of PCA 10

9.1 Covariance Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
9.2 Key Properties of the Covariance Matrix . . . . . . . . . . . . . . . . . . 11

10 The PCA Algorithm 12

11 Complete PCA Example 12

11.1 What is PC2? (The Second Principal Component) . . . . . . . . . . . . . 16

12 Visualizing PCA Dimensionality Reduction 18

13 Choosing the Number of Components 18

13.1 Scree Plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
13.2 Cumulative Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

14 Python Implementation 19
14.1 PCA from Scratch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
14.2 Using Scikit-Learn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

15 Applications of PCA 21
15.1 Image Compression (Eigenfaces) . . . . . . . . . . . . . . . . . . . . . . . 21
15.2 Noise Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
15.3 Feature Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

16 Limitations and Considerations 22

17 Summary and Key Takeaways 22

18 Practice Problems 23

A Quick Reference: Formulas 24

2
2 MATRICES AS LINEAR TRANSFORMATIONS Eigenvalues & PCA Tutorial

1 Introduction: Why Study Eigenvalues?

Before diving into definitions, let’s understand why eigenvalues and eigenvectors matter.
Consider a matrix as a transformation that acts on vectors. When we multiply a matrix
by a vector, the vector typically changes both its direction and magnitude.
However, there exist special vectors that, when transformed by a matrix, only get
scaled —they don’t rotate or change direction. These special vectors are called eigenvec-
tors, and the scaling factors are called eigenvalues.

The Big Picture

Eigenvalues and eigenvectors reveal the intrinsic properties of linear transfor-
mations. They tell us:

• Which directions are preserved under the transformation

• How much stretching or compression occurs along those directions

• Whether the transformation is stable, oscillatory, or explosive

1.1 Applications Across Fields

Eigenvalues appear everywhere in science and engineering:

• Physics: Quantum mechanics (energy levels), vibration analysis

• Engineering: Structural stability, control systems

• Data Science: PCA, spectral clustering, PageRank algorithm

• Computer Graphics: 3D transformations, facial recognition

• Economics: Input-output models, Markov chains

2 Matrices as Linear Transformations

Before studying eigenvalues, we must understand what matrices do to vectors.

2.1 Geometric Interpretation

A 2 × 2 matrix A transforms every point in the plane to a new location. Consider:
 
2 1
A=
1 2
When we apply A to a vector v, we compute Av.

3
4 FINDING EIGENVALUES: THE CHARACTERISTIC Eigenvalues
EQUATION& PCA Tutorial

(3, 3)

Av
(1, 1)

v
x

Same direction, scaled by 3

 
1
Let’s verify: if v = , then:
1
        
2 1 1 2·1+1·1 3 1
Av = = = =3
1 2 1 1·1+2·1 3 1
 
1
The vector is special—it only gets scaled by 3!
1

3 Formal Definitions
Definition 3.1 (Eigenvalue and Eigenvector). Let A be an n × n matrix. A scalar λ is
called an eigenvalue of A if there exists a nonzero vector v such that:

Av = λv

The nonzero vector v is called an eigenvector corresponding to λ.

Critical Observation
The equation Av = λv says that applying A to v has the same effect as multiplying
v by the scalar λ. The vector v does not change direction—it only gets stretched
(if |λ| > 1), compressed (if |λ| < 1), or flipped (if λ < 0).

3.1 Etymology
The word “eigen” comes from German, meaning “own” or “characteristic.” Eigenvalues
are the characteristic values that belong intrinsically to the matrix.

4 Finding Eigenvalues: The Characteristic Equation

4.1 Deriving the Method
Starting from Av = λv, we want to find λ. Rearranging:

4
5 STEP-BY-STEP EXAMPLES Eigenvalues & PCA Tutorial

Av = λv (1)
Av − λv = 0 (2)
(A − λI)v = 0 (3)

where I is the identity matrix. This is a system of linear equations. For a nonzero
solution v to exist, the matrix (A − λI) must be singular (non-invertible). This happens
exactly when:

det(A − λI) = 0

Definition 4.1 (Characteristic Polynomial). The characteristic polynomial of matrix

A is:
p(λ) = det(A − λI)
The eigenvalues of A are the roots of p(λ) = 0, called the characteristic equation.

5 Step-by-Step Examples
5.1 Example 1: A 2×2 Matrix
Complete Worked Example
Find all eigenvalues and eigenvectors of:
 
4 2
A=
1 3

Step 1: Set up the characteristic equation

     
4 2 1 0 4−λ 2
A − λI = −λ =
1 3 0 1 1 3−λ
Step 2: Compute the determinant

det(A − λI) = (4 − λ)(3 − λ) − (2)(1) (4)

= 12 − 4λ − 3λ + λ2 − 2 (5)
= λ2 − 7λ + 10 (6)

Step 3: Solve the characteristic equation

λ2 − 7λ + 10 = 0
Using the quadratic formula or factoring:

(λ − 5)(λ − 2) = 0

Therefore: λ1 = 5 and λ2 = 2

5
5 STEP-BY-STEP EXAMPLES Eigenvalues & PCA Tutorial

Step 4: Find eigenvectors for λ1 = 5

Solve (A − 5I)v = 0:
       
4−5 2 v1 −1 2 v1 0
= =
1 3−5 v2 1 −2 v2 0
From the first equation: −v1 + 2v2 = 0 ⇒ v1 = 2v2
Let v2 = 1, then v1 = 2. So:
 
2
v1 = (or any scalar multiple)
1

Step 5: Find eigenvectors for λ2 = 2

Solve (A − 2I)v = 0:
       
4−2 2 v1 2 2 v1 0
= =
1 3−2 v2 1 1 v2 0
From the first equation: 2v1 + 2v2 = 0 ⇒ v1 = −v2
Let v2 = 1, then v1 = −1. So:
 
−1
v2 = (or any scalar multiple)
1

Verification:
      
4 2 2 10 2
Av1 = = =5 = 5v1 ✓
1 3 1 5 1
      
4 2 −1 −2 −1
Av2 = = =2 = 2v2 ✓
1 3 1 2 1

5.2 Geometric Visualization

y

Av2 = 2v2 Av1 (scaled for display)

v2 = (−1, 1) v1 = (2, 1)

■ Eigenvalue λ1 = 5
■ Eigenvalue λ2 = 2

6
5 STEP-BY-STEP EXAMPLES Eigenvalues & PCA Tutorial

5.3 Example 2: A 3×3 Matrix

Higher Dimensional Example
Find the eigenvalues of:  
1 2 0
A = 0 3 0
2 1 4

Step 1: Form A − λI
 
1−λ 2 0
A − λI =  0 3−λ 0 
2 1 4−λ
Step 2: Compute the determinant
Since the matrix has zeros, we expand along the first row:

   
3−λ 0 0 0
det(A − λI) = (1 − λ) det − 2 det +0 (7)
1 4−λ 2 4−λ
= (1 − λ)(3 − λ)(4 − λ) − 2(0) (8)
= (1 − λ)(3 − λ)(4 − λ) (9)

Step 3: Solve
The eigenvalues are: λ1 = 1, λ2 = 3, λ3 = 4

Remark 5.1. Although this matrix is not triangular, the presence of zeros in the second
row significantly simplifies the calculation. Expanding the determinant along the second
row (which contains 0, 3−λ, 0) is the most efficient method here. For triangular matrices,
the eigenvalues are simply the diagonal entries.

5.4 Example 3: Complex Eigenvalues

Rotation Matrix
Find the eigenvalues of the 90° rotation matrix:
 
0 −1
R=
1 0

Characteristic equation:
 
−λ −1
det(R − λI) = det = λ2 + 1 = 0
1 −λ
Solution:
λ2 = −1 ⇒ λ = ±i

7
7 EIGENDECOMPOSITION (DIAGONALIZATION) Eigenvalues & PCA Tutorial

Geometric Interpretation

Complex eigenvalues indicate rotation. A pure rotation (no scaling) has eigenval-
ues on the unit circle in the complex plane. The eigenvalue i = eiπ/2 corresponds
to a 90° rotation.

6 Important Properties of Eigenvalues

Theorem 6.1 (Trace and Determinant). For an n×n matrix A with eigenvalues λ1 , λ2 , . . . , λn :

1. tr(A) = λ1 + λ2 + · · · + λn (sum of eigenvalues = trace)

2. det(A) = λ1 · λ2 · · · λn (product of eigenvalues = determinant)

Verification with Example 1:

• Trace: 4 + 3 = 7 = 5 + 2 ✓

• Determinant: 4 · 3 − 2 · 1 = 10 = 5 · 2 ✓

Theorem 6.2 (Eigenvalues of Special Matrices). 1. Symmetric matrices (A = AT )

have only real eigenvalues

2. Orthogonal matrices (AT A = I) have eigenvalues with |λ| = 1

3. Positive definite matrices have all positive eigenvalues

4. Triangular matrices have eigenvalues equal to the diagonal entries

7 Eigendecomposition (Diagonalization)
Definition 7.1 (Diagonalizable Matrix). A matrix A is diagonalizable if it can be
written as:
A = P DP −1
where D is a diagonal matrix of eigenvalues and P is a matrix whose columns are the
corresponding eigenvectors.

7.1 Constructing the Decomposition

   
2 −1
Using our Example 1 with λ1 = 5, v1 = and λ2 = 2, v2 = :
1 1
   
2 −1 5 0
P = , D=
1 1 0 2
   
−1 1 1 1 1 1 1
P = =
2 + 1 −1 2 3 −1 2

8
7 EIGENDECOMPOSITION (DIAGONALIZATION) Eigenvalues & PCA Tutorial

Why Diagonalization Matters

1. Matrix powers: An = P Dn P −1 (diagonal matrices are trivial to exponen-

tiate)

2. Matrix exponential: eA = P eD P −1 (crucial for differential equations)

3. Understanding transformations: Diagonalization reveals the “natural

axes” of the transformation

9
9 MATHEMATICAL FOUNDATION OF PCA Eigenvalues & PCA Tutorial

8 Transition to Principal Component Analysis

Now we connect eigenvalues to one of the most important techniques in data science:
Principal Component Analysis (PCA).

8.1 The Problem PCA Solves

Imagine you have data with many features (dimensions). PCA answers:
1. Which directions in the data have the most variation?

2. Can we reduce dimensions while preserving important information?

3. What are the underlying patterns in the data?

The Core Insight
PCA finds the eigenvectors of the covariance matrix. These eigenvectors point
in the directions of maximum variance in the data. The corresponding eigenvalues
tell us how much variance exists along each direction.

9 Mathematical Foundation of PCA

9.1 Covariance Matrix
Definition 9.1 (Covariance — Intuitive Explanation). Covariance measures how two
variables change together.

• Positive covariance: When X goes up, Y tends to go up too

• Negative covariance: When X goes up, Y tends to go down

• Zero covariance: X and Y have no linear relationship

How to Calculate Covariance (Step by Step):

Given n data points for variables X and Y :

1. Find the mean of each variable: X̄ and Ȳ

2. Center each value: subtract the mean from each data point

3. Multiply the centered values for each pair: (xi − X̄) × (yi − Ȳ )

4. Add up all the products

5. Divide by (n − 1) to get the covariance

Covariance Calculation Example

Suppose we have 4 data points:
X = {2, 4, 6, 8}, Y = {1, 3, 4, 6}

10
9 MATHEMATICAL FOUNDATION OF PCA Eigenvalues & PCA Tutorial

Step 1: Find means: X̄ = 5, Ȳ = 3.5

Step 2 & 3: Center and multiply each pair:

xi yi (xi − 5) (yi − 3.5) Product

2 1 −3 −2.5 (−3) × (−2.5) = 7.5
4 3 −1 −0.5 (−1) × (−0.5) = 0.5
6 4 +1 +0.5 (+1) × (+0.5) = 0.5
8 6 +3 +2.5 (+3) × (+2.5) = 7.5

Step 4: Sum of products = 7.5 + 0.5 + 0.5 + 7.5 = 16

16 16
Step 5: Covariance = = ≈ 5.33
4−1 3
The positive covariance confirms: as X increases, Y increases too!

Definition 9.2 (Covariance Matrix). For a dataset with p features, the covariance
matrix Σ is a p × p table where each entry shows the covariance between two features:

Var(X1 ) Cov(X1 , X2 ) · · · Cov(X1 , Xp )

 
Cov(X2 , X1 ) Var(X2 ) · · · Cov(X2 , Xp )
Σ= .. .. . ..
 
 . . . . .


Cov(Xp , X1 ) Cov(Xp , X2 ) · · · Var(Xp )
Note: Variance is just the covariance of a variable with itself: Var(X) = Cov(X, X)

9.2 Key Properties of the Covariance Matrix

1. Symmetric: Σ = ΣT (so Cov(X, Y ) = Cov(Y, X))

2. Positive semi-definite: All eigenvalues are ≥ 0

3. Diagonal entries: Σii = Var(Xi ) (variance of feature i)

11
11 COMPLETE PCA EXAMPLE Eigenvalues & PCA Tutorial

10 The PCA Algorithm

Algorithm 1 Principal Component Analysis

Require: Data matrix X with n samples and p features
Ensure: Principal components and transformed data
1: Center the data: Subtract the mean of each feature

Xcentered = X − X̄

2: Compute covariance matrix:

1
Σ= XT Xcentered
n − 1 centered
3: Find eigenvalues and eigenvectors of Σ
4: Sort eigenvectors by decreasing eigenvalue
5: Select top k eigenvectors to form projection matrix W
6: Project data: Xnew = Xcentered · W

11 Complete PCA Example

Step-by-Step PCA Calculation
Consider the following 2D dataset with 5 points:

Point x1 x2
1 2.5 2.4
2 0.5 0.7
3 2.2 2.9
4 1.9 2.2
5 3.1 3.0

Step 1: Center the Data

Calculate means:
2.5 + 0.5 + 2.2 + 1.9 + 3.1
x̄1 = = 2.04
5
2.4 + 0.7 + 2.9 + 2.2 + 3.0
x̄2 = = 2.24
5
Centered data:  
0.46 0.16
−1.54 −1.54
 
 0.16
Xcentered =  0.66 
−0.14 −0.04
1.06 0.76
Step 2: Compute Covariance Matrix
1
The formula is: Σ = XT Xcentered where n = 5 (number of points).
n − 1 centered

12
11 COMPLETE PCA EXAMPLE Eigenvalues & PCA Tutorial

First, let’s write out both matrices clearly:

 
0.46 0.16
−1.54 −1.54
 (5 rows, 2 columns)
 
 0.16
Xcentered =  0.66 
−0.14 −0.04
1.06 0.76
 
0.46 −1.54 0.16 −0.14 1.06
T
Xcentered = (2 rows, 5 columns)
0.16 −1.54 0.66 −0.04 0.76
Now multiply Xcentered
T
× Xcentered (result will be 2 × 2):

 
0.46 0.16
  −1.54 −1.54
T 0.46 −1.54 0.16 −0.14 1.06  0.16

Xcentered Xcentered = 0.66 
0.16 −1.54 0.66 −0.04 0.76  
−0.14 −0.04
1.06 0.76

Entry (1,1): First row of X T · First column of X:

(0.46)(0.46) + (−1.54)(−1.54) + (0.16)(0.16) + (−0.14)(−0.14) + (1.06)(1.06)

= 0.212 + 2.372 + 0.026 + 0.020 + 1.124 = 3.752

Entry (1,2) and (2,1): First row of X T · Second column of X:

(0.46)(0.16) + (−1.54)(−1.54) + (0.16)(0.66) + (−0.14)(−0.04) + (1.06)(0.76)

= 0.074 + 2.372 + 0.106 + 0.006 + 0.806 = 3.362

Entry (2,2): Second row of X T · Second column of X:

(0.16)(0.16) + (−1.54)(−1.54) + (0.66)(0.66) + (−0.04)(−0.04) + (0.76)(0.76)

= 0.026 + 2.372 + 0.436 + 0.002 + 0.578 = 3.412

So:  
T 3.752 3.362
Xcentered Xcentered =
3.362 3.412
Finally, divide by (n − 1) = 4:
   
1 3.752 3.362 0.938 0.8405
Σ= =
4 3.362 3.412 0.8405 0.853

What Each Entry Means

• Σ11 = 0.938 is the variance of x1 (how spread out x1 values are)

• Σ22 = 0.853 is the variance of x2 (how spread out x2 values are)

• Σ12 = Σ21 = 0.8405 is the covariance between x1 and x2 (positive means

13
11 COMPLETE PCA EXAMPLE Eigenvalues & PCA Tutorial

they increase together)

Step 3: Find Eigenvalues

Characteristic equation det(Σ − λI) = 0:
(0.938 − λ)(0.853 − λ) − 0.84052 = 0
λ2 − 1.791λ + (0.800 − 0.706) = 0 =⇒ λ2 − 1.791λ + 0.094 = 0
Using the quadratic formula:
p √
1.791 ± 1.7912 − 4(0.094) 1.791 ± 2.832 1.791 ± 1.683
λ= = =
2 2 2

λ1 ≈ 1.737 λ2 ≈ 0.054
Step 4: Find Eigenvectors
For λ1 = 1.737:
 
−0.799 0.8405
(Σ − 1.737I)v = v=0
0.8405 −0.884
From the first equation: −0.799v1 + 0.8405v2 = 0 =⇒ v1 ≈ 1.05v2
 
0.725
v1 ≈ (normalized)
0.689
For λ2 = 0.054 (orthogonal to v1 ):
 
−0.689
v2 ≈ (normalized)
0.725
Step 5: Interpret Results
x2 (centered)

PC1
PC2

x1 (centered)

PC1: 97.0% variance

PC2: 3.0% variance

Variance Explained:
λ1 1.737
PC1 explains: = ≈ 97.0%
λ1 + λ 2 1.791
λ2 0.054
PC2 explains: = ≈ 3.0%
λ1 + λ2 1.791

14
11 COMPLETE PCA EXAMPLE Eigenvalues & PCA Tutorial

Key Finding

The first principal component (PC1) captures 97.0% of the variance in the data!
This means we could reduce from 2D to 1D and lose only 3.0% of the information.
The direction of PC1 shows that x1 and x2 are highly correlated—as one increases,
so does the other.

Step 6: Project 2D Data to 1D (The Actual Transformation!)

Now we transform our 5 points from 2D to 1D. To do this, we multiply each centered
point by the first eigenvector (PC1).

How Projection Works

To project a 2D point onto a 1D line (PC1), we compute the dot product:

New 1D value = (xcentered

1 ) × (v1 ) + (xcentered
2 ) × (v2 )
 
0.725
where v1 = is our first eigenvector.
0.689

The matrix multiplication:

 
0.46 0.16
−1.54 −1.54  
  0.725
Xnew = Xcentered × v1 =  0.16
 0.66  ×

−0.14 −0.04 0.689
1.06 0.76
Calculate each point’s new 1D coordinate:

Point 2D Centered (x1 , x2 ) Calculation 1D Value

1 (0.46, 0.16) (0.46 × 0.725) + (0.16 × 0.689) 0.334 + 0.110 = 0.444

2 (−1.54, −1.54) (−1.54 × 0.725) + (−1.54 × 0.689) −1.117 + (−1.061) = −2.178
3 (0.16, 0.66) (0.16 × 0.725) + (0.66 × 0.689) 0.116 + 0.455 = 0.571
4 (−0.14, −0.04) (−0.14 × 0.725) + (−0.04 × 0.689) −0.102 + (−0.028) = −0.129
5 (1.06, 0.76) (1.06 × 0.725) + (0.76 × 0.689) 0.769 + 0.524 = 1.292

Result:  
0.444
−2.178
(5 points, now in 1D!)
 
Xnew =
 0.571 

−0.129
1.292

15
11 COMPLETE PCA EXAMPLE Eigenvalues & PCA Tutorial

BEFORE AFTER
(2.5, 2.4) 0.444
(0.5, 0.7) −2.178
(2.2, 2.9) Center data 0.571
(1.9, 2.2) × eigenvector −0.129
(3.1, 3.0) 1.292
2 numbers 1 number
per point per point
2D (2 features) 1D (1 feature)

What Just Happened?

Each 2D point got “squashed” onto the PC1 line. The dot product measures how
far along the PC1 direction each point lies:

• Point 2 has the most negative value (−2.178) — it’s furthest in the negative
PC1 direction

• Point 5 has the most positive value (1.292) — it’s furthest in the positive
PC1 direction

• Point 4 is near zero (−0.129) — it’s close to the center

We reduced from 2 numbers per point to 1 number per point, while keeping 97%
of the information!

11.1 What is PC2? (The Second Principal Component)

PC2 Explained
PC2 is the second principal component — the direction of maximum variance
that is perpendicular (orthogonal) to PC1.
Think of it this way:

• PC1 captures the “main trend” in your data (97% of variance)

• PC2 captures “what’s left over” after PC1 (3% of variance)

Why is PC2 perpendicular to PC1?

16
11 COMPLETE PCA EXAMPLE Eigenvalues & PCA Tutorial

x2

PC1
PC2

x1

PC1 ⊥ PC2 (90° angle)

By making PC2 perpendicular to PC1, we ensure that:

1. PC2 captures different information than PC1 (no redundancy)
2. The two components are uncorrelated
Calculating PC2 values (same  method
 as PC1):
−0.689
Our second eigenvector is v2 =
0.725
PC2 value = (xcentered
1 ) × (−0.689) + (xcentered
2 ) × (0.725)

Point 2D Centered PC1 value PC2 value

1 (0.46, 0.16) 0.444 (0.46)(−0.689) + (0.16)(0.725) = −0.201

2 (−1.54, −1.54) −2.178 (−1.54)(−0.689) + (−1.54)(0.725) = −0.055
3 (0.16, 0.66) 0.571 (0.16)(−0.689) + (0.66)(0.725) = 0.368
4 (−0.14, −0.04) −0.129 (−0.14)(−0.689) + (−0.04)(0.725) = 0.067
5 (1.06, 0.76) 1.292 (1.06)(−0.689) + (0.76)(0.725) = −0.180

If we keep BOTH PC1 and PC2:

 
0.444 −0.201
−2.178 −0.055
← Still 2D, but in a new coordinate system!
 
Xtransformed =
 0.571 0.368 

−0.129 0.067 
1.292 −0.180

When to Use PC2?

• Keep only PC1: When you want maximum compression (2D → 1D). You
lose 3% of information.
• Keep PC1 + PC2: When you want to rotate your data to a new coordinate
system but keep all information. The new axes (PC1, PC2) are aligned with

17
13 CHOOSING THE NUMBER OF COMPONENTS Eigenvalues & PCA Tutorial

the “natural directions” of your data.

In higher dimensions (e.g., 100 features), you might keep PC1 through PC10 to
reduce from 100D to 10D while retaining 95% of the variance.

Original Coordinates PC Coordinates

x2

PC2
PCA
Small
x1 PC1
(3%)

Large spread (97%)

12 Visualizing PCA Dimensionality Reduction

Original 2D Space Projected to 1D (PC1)

x2

Project
x1 PC1
-2 -1 0 1 2

13 Choosing the Number of Components

13.1 Scree Plot
A scree plot shows eigenvalues in decreasing order. The “elbow” suggests where to cut
off.

18
 14 PYTHON IMPLEMENTATION Eigenvalues & PCA Tutorial

Eigenvalue (Variance Explained)

5
4
3
Elbow
2
1
0
1 2 3 4 5 6 7 8
Principal Component

13.2 Cumulative Variance

Choose k components to explain a target percentage (e.g., 95%) of variance:
Cumulative Variance Explained (%)

100
95% threshold
80 Choose 4-5
components
60
40
20
0
1 2 3 4 5 6 7 8
Number of Components

14 Python Implementation
14.1 PCA from Scratch

Listing 1: PCA Implementation from Scratch

1 import numpy as np
2

3 def pca_from_scratch (X , n_components = None ) :

4 """
5 Perform PCA on dataset X .
6

7 Parameters :
8 -----------
9 X : array - like , shape ( n_samples , n_features )
10 Input data

19
 14 PYTHON IMPLEMENTATION Eigenvalues & PCA Tutorial

11 n_components : int , optional

12 Number of components to keep
13

14 Returns :
15 --------
16 X_transformed : Transformed data
17 components : Principal component directions
18 explained_variance : Variance explained by each component
19 """
20 # Step 1: Center the data
21 X_centered = X - np . mean (X , axis =0)
22

23 # Step 2: Compute covariance matrix

24 n_samples = X . shape [0]
25 cov_matrix = ( X_centered . T @ X_centered ) / ( n_samples - 1)
26

27 # Step 3: Eigendecomposition
28 eigenvalues , eigenvectors = np . linalg . eig ( cov_matrix )
29

30 # Step 4: Sort by eigenvalue ( descending )

31 sorted_idx = np . argsort ( eigenvalues ) [:: -1]
32 eigenvalues = eigenvalues [ sorted_idx ]
33 eigenvectors = eigenvectors [: , sorted_idx ]
34

35 # Step 5: Select top k components

36 if n_components is not None :
37 eigenvectors = eigenvectors [: , : n_components ]
38 eigenvalues = eigenvalues [: n_components ]
39

40 # Step 6: Project data

41 X_transformed = X_centered @ eigenvectors
42

43 # Calculate explained variance ratio

44 total_var = np . sum ( eigenvalues )
45 explai ned _var ianc e_ra tio = eigenvalues / total_var
46

47 return X_transformed , eigenvectors , expl ain ed_v aria nce_ rat io

48

49

50 # Example usage
51 np . random . seed (42)
52 X = np . array ([
53 [2.5 , 2.4] ,
54 [0.5 , 0.7] ,
55 [2.2 , 2.9] ,
56 [1.9 , 2.2] ,
57 [3.1 , 3.0]
58 ])
59

60 X_pca , components , var_ratio = pca_from_scratch (X , n_components =1)

61 print ( " Transformed data :\ n " , X_pca )

20
 15 APPLICATIONS OF PCA Eigenvalues & PCA Tutorial

62 print ( " \ nPrincipal components :\ n " , components )

63 print ( " \ nVariance explained : " , var_ratio )

14.2 Using Scikit-Learn

Listing 2: PCA with Scikit-Learn

1 from sklearn . decomposition import PCA
2 from sklearn . preprocessing import StandardScaler
3 import matplotlib . pyplot as plt
4

5 # Load and prepare data

6 from sklearn . datasets import load_iris
7 iris = load_iris ()
8 X = iris . data
9 y = iris . target
10

11 # Standardize features ( important for PCA !)

12 scaler = StandardScaler ()
13 X_scaled = scaler . fit_transform ( X )
14

15 # Apply PCA
16 pca = PCA ( n_components =2)
17 X_pca = pca . fit_transform ( X_scaled )
18

19 # Results
20 print ( " Explained variance ratio : " , pca . exp la in ed_ va ri anc e_ ra tio _ )
21 print ( " Cumulative variance : " ,
np . cumsum ( pca . e xpl ai ned _v ar ian ce _r ati o_ ) )
22

23 # Visualize
24 plt . figure ( figsize =(10 , 6) )
25 scatter = plt . scatter ( X_pca [: , 0] , X_pca [: , 1] ,
26 c =y , cmap = ’ viridis ’ , alpha =0.7)
27 plt . xlabel ( f ’ PC1 ({ pca . ex pl ain ed _va ri an ce_ ra ti o_ [0]:.1%}
variance ) ’)
28 plt . ylabel ( f ’ PC2 ({ pca . ex pl ain ed _va ri an ce_ ra ti o_ [1]:.1%}
variance ) ’)
29 plt . title ( ’ Iris Dataset - PCA Visualization ’)
30 plt . colorbar ( scatter , label = ’ Species ’)
31 plt . show ()

15 Applications of PCA
15.1 Image Compression (Eigenfaces)
PCA can compress images by keeping only the most important components:

21
17 SUMMARY AND KEY TAKEAWAYS Eigenvalues & PCA Tutorial

PCA

Original 50 PCs 20 PCs 10 PCs

(1000 pixels) (95% var) (85% var)(70% var)

15.2 Noise Reduction

By keeping only principal components with high eigenvalues, PCA filters out noise:

Xdenoised = Xprojected · W T + X̄

15.3 Feature Engineering

PCA creates uncorrelated features, which can help many machine learning algorithms:
• Removes multicollinearity for regression
• Speeds up training by reducing dimensions
• Prevents overfitting in high-dimensional data

16 Limitations and Considerations

When NOT to Use PCA
1. Non-linear relationships: PCA only captures linear correlations. Use ker-
nel PCA or t-SNE for non-linear data.

2. Interpretability needed: Principal components are linear combinations of

all features, making interpretation difficult.

3. Different scales: Always standardize features first! Otherwise, high-

variance features dominate.

4. Categorical data: PCA is designed for continuous numerical data.

17 Summary and Key Takeaways

Chapter Summary
Eigenvalues and Eigenvectors:

• Av = λv defines the eigenvalue problem

• Found by solving det(A − λI) = 0

22
18 PRACTICE PROBLEMS Eigenvalues & PCA Tutorial

• Reveal intrinsic properties of linear transformations

• Enable matrix decomposition: A = P DP −1

Principal Component Analysis:

• Finds directions of maximum variance in data

• PCs are eigenvectors of the covariance matrix

• Eigenvalues indicate variance along each PC

• Used for dimensionality reduction, visualization, denoising

Workflow:

1. Center (and optionally standardize) data

2. Compute covariance matrix

3. Find eigenvalues/eigenvectors

4. Sort by eigenvalue magnitude

5. Keep top k components

6. Project data onto new basis

18 Practice Problems
1. Find the eigenvalues and eigenvectors of:
 
3 1
A=
1 3

Verify that eigenvectors are orthogonal (since A is symmetric).

 
2 −1
2. For the matrix B = :
1 4

(a) Find eigenvalues

(b) Find eigenvectors
(c) Compute B 5 using diagonalization

3. Given the following dataset, perform PCA by hand:

 
1 2
2 4 
X=3 5 


4 4

What percentage of variance is explained by PC1?

23
A QUICK REFERENCE: FORMULAS Eigenvalues & PCA Tutorial

4. Prove that if λ is an eigenvalue of A, then λ2 is an eigenvalue of A2 .

5. Explain why the covariance matrix is always positive semi-definite.

A Quick Reference: Formulas

Concept Formula

Eigenvalue equation Av = λv

Characteristic polynomial p(λ) = det(A − λI)

Trace tr(A) = i λi
P

Determinant
Q
det(A) = i λi

Diagonalization A = P DP −1

Matrix power An = P Dn P −1

Covariance matrix Σ= 1
n−1
XT X

Variance explained Pλk

i λi

PCA projection Xnew = Xcentered · W

24