Research Methodology | Principal Component Analysis (PCA)

PRINCIPAL COMPONENT ANALYSIS

Theory, Mathematics, Process & Applications in Economics

Course: Applied Research Methodology & Macroeconomics

Topic: Principal Component Analysis (PCA) — Theory, Methods & Applications
Date: June 05, 2026

1. Introduction
In modern empirical research, datasets routinely contain a large number of variables that are
interrelated with one another. Analyzing such high-dimensional data directly is not only
computationally demanding but also statistically problematic — particularly when variables are
highly correlated, giving rise to multicollinearity in regression models or making it difficult to isolate
the distinct contribution of each variable to the outcome of interest.
Principal Component Analysis (PCA) is a multivariate statistical technique designed to address
precisely this challenge. It transforms a set of possibly correlated original variables into a smaller
set of new, uncorrelated variables called Principal Components (PCs). Each principal component
is a linear combination of the original variables, constructed in such a way that it captures the
maximum possible variance in the data. The first principal component (PC1) accounts for the
largest share of total variance; the second (PC2) captures the maximum remaining variance subject
to being uncorrelated with PC1; and so on.
The result is a dramatically simplified representation of the data that retains the most analytically
relevant information while discarding redundant variation. In economics and the social sciences,
PCA finds wide application in constructing composite indices — such as a Socioeconomic Status
Index or a Financial Development Index — and in resolving multicollinearity problems in regression
analysis by replacing correlated regressors with their uncorrelated principal components.

2. Mathematical Foundations
A clear understanding of PCA requires engaging directly with its mathematical structure. The
technique is grounded in the eigendecomposition of a covariance or correlation matrix, a standard
operation in linear algebra.

2.1 The Need for Standardization

PCA is sensitive to the units and scale of measurement of the original variables. A variable
measured in millions of rupees will have a much larger numerical variance than a variable
measured as a percentage, and PCA would incorrectly treat the former as more important simply
because of its scale. To eliminate this distortion, all variables must be standardized to Z-scores
before PCA is applied. The Z-score transformation converts each observation into the number of
standard deviations it lies from its variable's mean:
Z = (X - μ) / σ

Page 1 | Principal Component Analysis — Academic Assignment

 Research Methodology | Principal Component Analysis (PCA)

After standardization, every variable has a mean of zero and a standard deviation of one, ensuring
that PCA responds to the genuine correlation structure of the data rather than to arbitrary
differences in measurement scale.

2.2 The Covariance Matrix

Once the data are standardized, PCA constructs a p × p covariance matrix (denoted Σ), where p is
the number of variables. When working with standardized data, the covariance matrix is equivalent
to the correlation matrix — each diagonal entry equals 1 and each off-diagonal entry is the Pearson
correlation between two variables. This matrix fully captures the pairwise relationships among all
variables and serves as the foundation from which principal components are extracted.

2.3 Eigendecomposition — The Core of PCA

PCA finds the principal components by solving the eigenvalue equation of the covariance matrix:
Σv = λv
In this equation, v is an eigenvector — a vector that defines the direction of a principal component
in the original variable space — and λ (lambda) is the corresponding eigenvalue, which measures
the amount of variance that the component associated with that direction explains. Because Σ is a
symmetric matrix, it yields p real eigenvalues and p orthogonal (mutually uncorrelated)
eigenvectors.
The eigenvectors are ranked by their eigenvalues in descending order. The eigenvector associated
with the largest eigenvalue defines PC1 — the direction that captures the maximum variance. The
eigenvector with the second largest eigenvalue defines PC2, and so forth. Importantly, because all
eigenvectors of a symmetric matrix are orthogonal to one another, all principal components are
guaranteed to be uncorrelated — a defining property that makes them analytically valuable.

2.4 Loading Scores and Component Scores

Each eigenvector contains p elements called factor loadings. A loading indicates how strongly the
original variable contributes to that principal component. Large absolute loadings (close to +1 or -
1) indicate a strong contribution; loadings near zero indicate a weak contribution. Once the
eigenvectors are determined, the original dataset is projected onto the new coordinate system they
define, producing component scores — the values of each observation on each principal
component. These scores serve as the new variables in subsequent analysis.

3. The PCA Process — Step by Step

PCA follows a structured sequence of operations. Each step is necessary and each has direct
implications for the quality and interpretability of the results.

Step 1 Collect and organize the raw data matrix with n observations and p
variables.

Step 2 Standardize all variables to Z-scores (mean = 0, SD = 1) to eliminate

scale distortions.

Step 3 Compute the p × p covariance (correlation) matrix from the standardized

data.

Page 2 | Principal Component Analysis — Academic Assignment

 Research Methodology | Principal Component Analysis (PCA)

Step 4 Perform eigendecomposition: extract all eigenvalues (λ) and

eigenvectors (v) of the matrix.

Step 5 Rank eigenvectors by descending eigenvalue — PC1 has the largest λ,

PC2 the second largest, and so on.

Step 6 Select the number of components to retain using decision criteria (see
Section 4).

Step 7 Project the original data onto the retained eigenvectors to obtain
principal component scores.

It is important to note that this process is entirely data-driven — PCA makes no assumptions about
a dependent variable and imposes no model structure. It is an unsupervised dimensionality
reduction technique, meaning it operates purely on the internal variance-covariance structure of
the data.

4. Interpretation and Component Selection Criteria

One of the most consequential decisions in PCA is determining how many principal components
to retain. Retaining too many defeats the purpose of dimensionality reduction; retaining too few
risks discarding meaningful information. Three established criteria guide this decision.

4.1 The Kaiser Criterion (Eigenvalue > 1 Rule)

The Kaiser Criterion, also called the eigenvalue-greater-than-one rule, is the most widely applied
selection standard in practice. It states that only those principal components with eigenvalues
greater than 1.0 should be retained when PCA is conducted on a correlation matrix.
The logical basis for this rule is intuitive: since each standardized variable has a variance of exactly
1, a principal component is only worth retaining if it explains more variance than a single original
variable does on its own. A component with λ < 1 explains less variance than any individual variable
and therefore represents a net loss of information relative to simply keeping the original variable.

4.2 The Scree Plot (Elbow Method)

A scree plot is a graphical tool that displays eigenvalues on the vertical axis against the rank order
of components on the horizontal axis, producing a descending curve. The researcher looks for the
point at which the curve transitions from a steep decline to a relatively flat slope — the so-called
'elbow' of the plot. Components to the left of the elbow (i.e., those on the steep part of the curve)
are considered to explain meaningful variance and should be retained; components on the flat
section add little and should be discarded.
The scree plot is particularly useful because it is visual and intuitive, though it can be subjective —
different researchers may identify the elbow at different points when the curvature is gradual rather
than sharp.

4.3 Cumulative Variance Explained

A third approach sets a target threshold for the cumulative proportion of total variance that the
retained components must explain collectively. In applied economics and social science research,

Page 3 | Principal Component Analysis — Academic Assignment

 Research Methodology | Principal Component Analysis (PCA)

the standard threshold is 70% to 90% of total variance. The researcher retains the minimum
number of components needed to meet this threshold.
The proportion of variance explained by each component is computed as its eigenvalue divided by
the sum of all eigenvalues (which equals p, the total number of variables). The cumulative variance
is then the running sum of these individual proportions.
Proportion of Variance (PC_k) = λk / Σλ
In practice, a combination of all three criteria — the Kaiser rule, scree plot visual inspection, and
the cumulative variance threshold — yields the most defensible component selection decision.

5. Applications in Economics and Social Sciences

5.1 Construction of Composite Indices
One of the most important applications of PCA in economics is the construction of composite
indices from a set of related indicators. When a latent concept — such as poverty, financial
development, institutional quality, or human capital — cannot be measured directly but can be
proxied through several observable variables, PCA provides a principled method for combining
those proxies into a single index.
For example, a researcher wishing to construct a Financial Development Index might collect data
on the ratio of private credit to GDP, stock market capitalization, the number of bank branches per
capita, and the ratio of broad money to GDP. These variables are all correlated with one another
because they each reflect aspects of the same underlying concept. PCA extracts the first principal
component — the linear combination of these four variables that maximizes explained variance —
as the composite Financial Development Index. This component absorbs the shared variation
among the indicators, representing the latent construct of financial development more efficiently
than any single indicator alone.
This approach is far more objective than arbitrary weighting schemes, because the weights are
determined by the data's own covariance structure rather than by the researcher's judgment.

5.2 Addressing Multicollinearity in Regression

Multicollinearity — the presence of high correlations among independent variables in a regression
model — violates the assumption of regressor independence and produces unstable, inefficient
coefficient estimates with inflated standard errors. PCA offers a rigorous solution: instead of
including the original correlated regressors, the researcher replaces them with their principal
components, which are by construction orthogonal (uncorrelated) to one another.
This approach, sometimes called Principal Components Regression (PCR), resolves
multicollinearity entirely at the cost of some interpretability, since the components are linear
combinations of the originals rather than directly observable variables. The trade-off between
interpretability and estimation stability must be weighed carefully in each application.

6. Limitations and Critical Considerations

Despite its analytical power, PCA carries several important limitations that researchers must
acknowledge.
• Assumption of Linearity: PCA captures only linear relationships among variables. If the
true relationships in the data are non-linear, PCA will fail to identify the most meaningful

Page 4 | Principal Component Analysis — Academic Assignment

 Research Methodology | Principal Component Analysis (PCA)

structure, and non-linear alternatives such as kernel PCA or autoencoders may be more
appropriate.
• Interpretability: Principal components are abstract mathematical constructs — linear
combinations of all original variables. Unlike the original variables, they rarely have a
clear, intuitive meaning. Interpreting what PC1 or PC2 'represents' requires careful
examination of the loading scores and considerable domain knowledge.
• Sensitivity to Outliers: PCA is based on variance, and extreme outliers can
disproportionately influence the direction of principal components. Robust PCA variants
exist to address this but are less commonly implemented in standard software.
• Loss of Information: Dimensionality reduction inherently discards information —
specifically, the variance not captured by the retained components. If important variation
resides in the discarded components, the analysis will be incomplete.
• MNAR Concern: PCA does not handle missing data internally. Missing values must be
addressed before PCA is applied, and the chosen missing data treatment can itself
influence the resulting components.

7. Summary Reference Table

Concept Definition / Rule Purpose

Standardization (Z-score) Z = (X − μ) / σ Removes scale bias before

PCA

Covariance Matrix (Σ) p × p matrix of variable relationships Foundation for

eigendecomposition

Eigenvalue (λ) Variance explained by each Ranks and selects

component components

Eigenvector (v) Direction of each principal Defines component

component structure

Factor Loadings Contribution of each variable to a Enables component

PC interpretation

Kaiser Criterion Retain components with λ > 1 Objective selection rule

Scree Plot Elbow point in eigenvalue curve Visual selection method

Cumulative Variance Retain until 70–90% variance Practical adequacy

explained threshold

Index Construction PC1 as composite latent factor Core application in

economics

PCR Replace correlated regressors with Solves multicollinearity

PCs

8. Conclusion

Page 5 | Principal Component Analysis — Academic Assignment

 Research Methodology | Principal Component Analysis (PCA)

Principal Component Analysis is an indispensable tool in the applied researcher's statistical toolkit.
Its ability to reduce high-dimensional, correlated data into a compact set of uncorrelated
components — without losing the bulk of the information — makes it uniquely valuable across a
wide range of analytical tasks. Whether the goal is to construct a composite index that captures a
latent economic concept, to eliminate multicollinearity from a regression model, or simply to identify
the dominant patterns of variation in a complex dataset, PCA provides a mathematically rigorous
and data-driven solution.
The technique's success, however, depends critically on correct implementation: variables must be
standardized before analysis, an appropriate number of components must be selected using
established criteria, and the resulting components must be interpreted carefully through their
loading scores. Researchers must also be transparent about the limitations of PCA — particularly
its linearity assumption and the inherent loss of interpretability — and should supplement the
analysis with domain knowledge to give the extracted components meaningful economic or social
content.
Understood and applied correctly, PCA transforms the complexity of high-dimensional data into
analytical clarity, enabling researchers to draw more precise and reliable conclusions from the
empirical evidence at their disposal.

References
Jolliffe, I. T. (2002). Principal Component Analysis (2nd ed.). Springer.
Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and
Psychological Measurement, 20(1), 141–151.
Stock, J. H., & Watson, M. W. (2002). Forecasting using principal components from a large
number of predictors. Journal of the American Statistical Association, 97(460), 1167–
1179.
Shlens, J. (2014). A tutorial on principal component analysis. arXiv preprint arXiv:1404.1100.

Page 6 | Principal Component Analysis — Academic Assignment