Module II: Principal Component Analysis (PCA)
Dr. Meghdoot Ghosh [Link], PhD, UGC-NET
Introduction
Principal Component Analysis (PCA) is a classical, widely used technique for exploratory data
analysis, dimensionality reduction, and data visualization. At its core, PCA finds new
orthogonal axes (principal components) that capture the largest possible variance in the data.
By projecting high-dimensional data onto the first few principal components, you often reveal
underlying structure, remove noise, and make subsequent modeling or visualization easier.
This section explains the intuition and mathematics behind PCA, discusses practical issues
(centering, scaling, how many components to keep), and gives concrete R examples (with
prcomp and plotting) using a built-in dataset.
Intuition
Imagine a cloud of points in 3D. PCA finds the line that best fits that cloud (in the least-
squares sense) — this is the first principal component (PC1). Then it finds a second line
orthogonal to the first that explains the next largest amount of variance (PC2), and so on. The
new axes are linear combinations of the original variables. If the first two or three
components capture most of the variance, you can approximate the original data using fewer
dimensions.
Key points:
PCs are uncorrelated (orthogonal).
Each PC is a linear combination of the original variables.
PCA is sensitive to the scale of variables (so scaling is often required).
Mathematical formulation (brief)
Practical considerations
Centering: Always center the variables (subtract means). Without centering, PCA will
pick up the mean.
Scaling: If variables are measured on different scales, scale to unit variance
(standardize) before PCA (i.e., use the correlation matrix). If all variables share the
same units and scale, scaling might not be necessary.
Missing data: PCA does not handle missing values directly; impute or use PCA
methods that support missingness.
Interpreting loadings: Loadings are coefficients of original variables in each PC.
Large absolute loading means that variable contributes strongly to that component.
How many components: Use scree plot, cumulative explained variance threshold (e.g.,
80–95%), parallel analysis, or cross-validation depending on the goal.
PCA in R — step-by-step with code
Below is a reproducible example using the iris dataset (numeric measurements only). Use
prcomp() (SVD-based) — preferred for numerical stability.
Notes on the code
center = TRUE, scale. = TRUE is a typical choice when variables have different
units/ranges.
pca_res$rotation gives the loadings: columns are PCs, rows are variables.
pca_res$x gives the transformed data (scores). These are what you plot or feed to
downstream models.
biplot() visualizes both observations and variable loadings on the same plot (useful
but can be cluttered).
Interpreting results (with the iris example)
The summary(pca_res) will show how much variance each PC captures. For iris,
typically PC1 and PC2 capture most of the variance (often >95% when petal
measurements dominate).
Loadings (rotation) tell you which original features make each PC. For instance, if
[Link] and [Link] have large positive loadings on PC1, PC1 is mostly a
'petal size' axis.
Scores (x) let you see how observations separate. Plotting PC1 vs PC2 colored by
species often shows clear clustering — indicating PCA found structure aligned with
species differences.
Common applications
Data visualization: Plot high-dimensional data in 2D/3D.
Noise reduction: Keep major PCs and reconstruct data for denoising.
Preprocessing: Reduce dimensionality before clustering or classification to speed up
algorithms and reduce overfitting.
Feature engineering: Use PC scores as features in predictive models.
Exploratory data analysis: Reveal correlated groups of variables.
Pitfalls and limitations
Linearity: PCA captures linear relationships only. Nonlinear structure requires
methods like kernel PCA, t-SNE, or UMAP.
Scaling sensitivity: If variables have widely different scales, you must standardize or
interpret results carefully.
Interpretability: PCs are linear combinations and can be hard to interpret if many
variables contribute.
Variance vs. importance: PCA maximizes variance, not necessarily predictive
relevance. A high-variance PC might be irrelevant to your prediction target.
Outliers: PCA is sensitive to outliers — robust PCA variants exist.
Extensions and related approaches
Kernel PCA: captures nonlinear structure by applying PCA in a high-dimensional
feature space using kernels.
Sparse PCA: produces principal components with sparse loadings for easier
interpretation.
Factor analysis: related to PCA but with an explicit statistical model for latent factors
and unique variances.
PCA for mixed data: methods like PCA on Gower distances or using PCA for
categorical data (MCA) are used when variables are not all numeric.
Summary
PCA is a fundamental tool for reducing dimensionality, simplifying datasets, and revealing
structure. In R, prcomp() provides a stable and easy-to-use implementation. Remember to
center and usually scale your variables, check explained variance to choose the number of
components, and be mindful of PCA's limitations (linearity, sensitivity to scale and outliers).
Used thoughtfully, PCA can greatly aid data exploration and preprocessing.