Theory and Applications of Principal

Component Analysis (PCA) in Multivariate

Analysis

[Link]. Statistics and Data Analytics

Assignment Submission

Contents

1 Introduction 3

2 Mathematical Foundations of Multivariate Analysis 3

3 Objective and Optimization Criterion 4

4 Geometrical Interpretation of PCA 4

5 Covariance Matrix and Correlation Matrix 4

6 Derivation of Subsequent Principal Components 5

7 Sample PCA from Observed Data 5

8 Properties and Theoretical Results 5

9 Applications of PCA 6

10 Limitations and Theoretical Extensions 6

11 Relationship with MANOVA and Factor Analysis 6

12 Conclusion 6

1
13 References 6

2
1. Introduction

Principal Component Analysis (PCA) is one of the most important and fundamental
techniques in multivariate statistical analysis. It forms the basis for many complex mul-
tivariate procedures such as factor analysis, canonical correlation, discriminant analysis,
and multivariate regression. PCA is primarily a data reduction technique that aims to
reduce the dimensionality of large datasets while retaining as much variation present in
the dataset as possible.
The modern data environment often involves hundreds of variables that may be cor-
related or interdependent. In such cases, traditional univariate or bivariate statistical
techniques are inadequate. PCA seeks to represent correlated variables in terms of a
smaller set of uncorrelated linear combinations known as principal components. Each
principal component captures as much of the total variability in the dataset as possible,
subject to being uncorrelated with the previous components. The technique thus provides
an orthogonal transformation of the original coordinate system into a new system that
maximizes interpretability.

2. Mathematical Foundations of Multivariate Analysis

Consider a random vector X = (X1 , X2 , . . . , Xp )′ having mean vector µ and covariance

matrix Σ. The vector X follows a multivariate normal distribution if any linear combi-
nation of its components is normally distributed. The joint probability density function
is given by

1
 
−p/2 −1/2
f (x) = (2π) |Σ| exp − (x − µ)′ Σ−1 (x − µ) (1)
2
The covariance matrix Σ encapsulates all pairwise relationships among the variables.
The diagonal elements represent variances, and the off-diagonal elements represent co-
variances. In PCA, we are primarily interested in decomposing Σ into its eigenvalues and
eigenvectors.

3
3. Objective and Optimization Criterion

The first principal component Y1 is obtained as a linear combination of X:

Y1 = a1 ′ X (2)

where a1 is a vector of constants that maximize Var(Y1 ) under the constraint a1 ′ a1 = 1.

We maximize
Var(Y1 ) = a1 ′ Σa1 (3)

subject to a1 ′ a1 = 1. Using a Lagrange multiplier λ1 , we form

L(a1 , λ1 ) = a1 ′ Σa1 − λ1 (a1 ′ a1 − 1) (4)

Differentiating with respect to a1 and equating to zero gives

Σa1 = λ1 a1 (5)

This is the characteristic equation of Σ, implying that λ1 is an eigenvalue and a1 is

the corresponding eigenvector. The solution yields p eigenvalue-eigenvector pairs (λi , ai ),
with λ1 ≥ λ2 ≥ · · · ≥ λp .

4. Geometrical Interpretation of PCA

Geometrically, PCA can be viewed as a rotation of the coordinate axes in p-dimensional

space such that the new axes (principal components) align with the directions of max-
imum variance. The eigenvectors define these directions, while eigenvalues indicate the
amount of variance captured along each axis.

5. Covariance Matrix and Correlation Matrix

PCA can be performed on either the covariance matrix Σ or the correlation matrix R.
When the variables are measured in different units, it is appropriate to standardize them

4
and perform PCA on R. The correlation matrix is obtained as

R = D−1/2 ΣD−1/2 (6)

where D is the diagonal matrix of variances.

6. Derivation of Subsequent Principal Components

The k th component satisfies

Σak = λk ak , ak ′ ai = 0 for i < k (7)

P
ensuring orthogonality. The total variance equals tr(Σ) = i λi , and the proportion of
variance explained by the first k components is
Pk
i=1 λi
Pp (8)
i=1 λi

7. Sample PCA from Observed Data

In practice, Σ is unknown and replaced by the sample covariance matrix S. The eigen-
values and eigenvectors of S provide estimates of the population parameters. Principal
components for observation j are obtained as

Yj = A′ (Xj − X̄) (9)

8. Properties and Theoretical Results

Principal components are mutually orthogonal:

Cov(Yi , Yj ) = ai ′ Σaj = 0 (i ̸= j) (10)

They capture decreasing portions of total variance and form an orthonormal basis of Rp .

5
9. Applications of PCA

PCA finds theoretical and applied importance across disciplines: in finance (factor mod-
eling), genetics (population structure), image analysis (eigenfaces), and climatology (em-
pirical orthogonal functions). It also provides the foundation for modern dimensionality
reduction algorithms.

10. Limitations and Theoretical Extensions

PCA assumes linearity and Gaussianity. Variants like Kernel PCA, Robust PCA, and
Sparse PCA generalize it to nonlinear, high-dimensional, and noisy data.

11. Relationship with MANOVA and Factor Analysis

PCA explains total variance, whereas Factor Analysis explains shared variance. MANOVA,
on the other hand, deals with hypothesis testing across groups. PCA can serve as a pre-
processing step for MANOVA by reducing dimensionality.

12. Conclusion

Principal Component Analysis is a mathematically elegant and computationally efficient

approach for representing high-dimensional data. Its derivation through eigenvalue de-
composition links linear algebra with statistical inference. PCA remains central to the
theory and practice of multivariate analysis.

13. References

1. Jolliffe, I. T. (2002). Principal Component Analysis. Springer.

2. Anderson, T. W. (2003). An Introduction to Multivariate Statistical Analysis. Wi-

ley.

3. Johnson, R. A., & Wichern, D. W. (2007). Applied Multivariate Statistical Analysis.

Pearson.

4. Rencher, A. C. (2002). Methods of Multivariate Analysis. Wiley-Interscience.