Principal Component Analysis

(PCA) — Key Notes

Practitioner-focused deck: concepts,
workflow, interpretation, reporting
 What is PCA?
• A linear technique that transforms correlated
variables into uncorrelated components.
• Finds directions (principal components) that
capture maximum variance.
• Used for dimensionality reduction,
visualization, noise filtering, feature
engineering.
 When to Use PCA
• Many numeric variables with multicollinearity.
• You need a smaller set of composite variables
with minimal information loss.
• Pre-processing for clustering/regression/ML;
exploratory visualization (2D/3D).
 Data & Assumptions (Practical)
• Variables numeric and on comparable scales
(standardization often required).
• Linear relationships among variables;
components are linear combinations.
• Large-enough sample size (e.g., 5–10 cases per
variable as a rule of thumb).
• No severe outliers; handle missing values
thoughtfully.
 Standardization / Scaling
• Use z-scores when variables have different
units/scales.
• PCA on covariance matrix (raw units) vs
correlation matrix (standardized).
• Standardizing avoids dominance of high-
variance variables.
The Math: Covariance/Correlation
Matrix
• Compute Σ (covariance) or R (correlation) of
variables X.
• Eigen-decompose: Σ = QΛQᵀ, where columns
of Q are eigenvectors (loadings).
• Variance explained = eigenvalues / sum of
eigenvalues.
 The Math: Singular Value
Decomposition (SVD)
• X = U S Vᵀ; columns of V are directions
(loadings).
• Component scores = U S (or X V).
• S² (squared singular values) relate to
eigenvalues and variance explained.
 Choosing Number of Components
(k)
• Scree plot ‘elbow’ (visual).
• Kaiser criterion: eigenvalues > 1 (on
correlation matrix).
• Cumulative variance threshold (e.g., 70–90%
depending on field).
• (Advanced) Parallel analysis / permutation-
based benchmarks.
 Loadings, Communalities,
Contributions
• Loading: correlation between original variable
and component.
• Squared loading ≈ variance contributed by a
variable to a component.
• Communality (PCA): sum of squared loadings
across retained components.
• Contribution plots help see which variables
shape each component.
 Component Scores & Biplots
• Scores: coordinates of observations in
component space (projection).
• Biplot overlays variables (arrows) and
observations (points).
• Interpret direction/length of arrows
(correlation and contribution).
 PCA vs Common Factor Analysis
(CFA/FA)
• PCA: decomposes total variance; components
are exact linear combos.
• FA: models common variance + unique error;
latent factors inferred.
• Rotation common in FA; PCA sometimes
rotates for interpretability but differs
conceptually.
 Categorical/Mixed Data
Alternatives
• Binary/nominal: Multiple Correspondence
Analysis (MCA).
• Mixed numeric + categorical: Factor Analysis
of Mixed Data (FAMD).
• Ordinal: consider polychoric correlations or
non-linear embeddings.
 Outliers, Missing Data & Robust
PCA
• Outliers can tilt components—inspect
leverage; consider robust methods.
• Impute missing values
(mean/median/knn/EM); document approach.
• Winsorize/extreme value treatment only with
transparent justification.
Cross‑Validation & Stability Checks
• Split‑sample or k‑fold to test stability of
loadings and explained variance.
• Bootstrap CIs on loadings or use Procrustes
rotation to compare solutions.
• Sensitivity: re‑run excluding variables/groups;
check robustness.
 Workflow (Step‑by‑Step)
• 1) Inspect data; handle missing/outliers;
standardize.
• 2) Choose correlation vs covariance; run PCA.
• 3) Decide k via scree/criteria; extract loadings
& scores.
• 4) Interpret components; label based on
high‑loading variables.
• 5) Validate stability; visualize (scree, biplot).
• 6) Use scores in downstream analysis
(clustering/regression).
 Visualizations to Include
• Scree plot (variance by component).
• Cumulative variance plot.
• Loadings heatmap/table; contribution
barplots.
• Biplot of first two components with groups
highlighted.
 Reporting PCA in Papers/Reports
• Data prep: scaling; missing/outlier handling;
matrix used (Σ or R).
• Criteria for k; eigenvalues; variance explained
(per PC and cumulative).
• Loading table (top loadings per component)
and interpretation.
• Plots (scree, cumulative, biplot); stability
checks; limitations.
 Example Commands (SPSS / R /
Python)
• SPSS: Analyze → Dimension Reduction →
Factor → Extraction: Principal components;
Rotation (optional).
• R: prcomp(scale.=TRUE) or PCA() in
FactoMineR; autoplot/ggplot2 for visuals.
• Python: scikit‑learn PCA with StandardScaler;
matplotlib for scree/biplot.
 R Code Snippet
• library(FactoMineR); library(factoextra)
• X <- scale(df)
• res <- PCA(X, graph = FALSE)
• fviz_eig(res); fviz_pca_biplot(res, repel =
TRUE)
 Python Code Snippet
• from [Link] import
StandardScaler
• from [Link] import PCA
• X = StandardScaler().fit_transform([Link])
• pca = PCA(n_components=None).fit(X)
• expl = pca.explained_variance_ratio_
• scores = [Link](X); loadings =
pca.components_.T
 Using PCA Scores Downstream
• Regression: replace collinear predictors with
leading PC scores.
• Clustering: run clustering on PC scores to
remove noise/correlation.
• Visualization: 2D/3D scatter of first PCs to
inspect structure.
Common Pitfalls & Quick Checklist
• Forgetting to scale variables with different
units.
• Over‑interpreting weak loadings or too many
components.
• Ignoring outliers/missing data; not checking
stability.
• Confusing PCA with common factor analysis.
• ✔ Scaled? ✔ k justified? ✔ Loadings clear? ✔
Stability checked? ✔ Report complete?