Probabilistic PCA and Factor Analysis (FA)
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 79
Probabilistic PCA and Factor Analysis (FA)
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 80
Probabilistic PCA and Factor Analysis (FA)
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 81
Probabilistic PCA and Factor Analysis (FA)
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 82
Probabilistic PCA and Factor Analysis (FA)
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 83
Probabilistic PCA and Factor Analysis (FA)
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 84
Probabilistic PCA and Factor Analysis (FA)
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 85
Probabilistic PCA and Factor Analysis (FA)
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 86
Probabilistic PCA and Factor Analysis (FA)
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 87
Probabilistic PCA and Factor Analysis (FA)
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 88
Probabilistic PCA and Factor Analysis (FA)
• Factor Analysis (FA): FA is a technique that is used to reduce a large number of variable
into fewer number of factors.
• it is a way to condense the data in many variables into a just a few variables.
• Factor analysis is also called dimension reduction.
• it is an example of latent variable model.
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 89
Latent Variables and Assumptions
• Latent Variables are variables that are not directly observed but are inferred from other
variables.
• Mathematical models that aim to explain observed variables in terms of latent variables are
called latent variable models.
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 90
Purpose of Factor Analysis
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 91
Types of Factor Analysis
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 92
Types of Factor Analysis
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 93
Probabilistic PCA and Factor Analysis (FA)
• When dealing with dimensionality reduction tasks, methods such as Principal Component
Analysis (PCA) and Factor Analysis (FA) are commonly employed.
• Probabilistic PCA (PPCA): PPCA extends traditional PCA by incorporating a
probabilistic framework.
• It assumes that the observed data is generated by projecting the latent variables into a high-
dimensional space, followed by the addition of Gaussian noise.
• PPCA estimates the latent variables and the noise parameters using maximum likelihood
estimation, providing a probabilistic interpretation of the low-dimensional representation.
• Factor Analysis (FA): FA assumes a generative model where the observed variables are
linear combinations of the latent variables, along with Gaussian noise.
• The goal is to estimate the latent variables and the loading matrix that represents the linear
relationships between the observed and latent variables.
• FA also provides a probabilistic interpretation of the dimensionality reduction process.
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 94
Probabilistic PCA and Factor Analysis (FA)
Features Probabilistic PCA Factor Analysis
Objective Maximizes likelihood of observed data Estimates latent factors
underlying observed data
Assumptions Gaussian distribution of observed data A linear the relationship
between observed variables
and latent factors
Dimensionality Flexible approach, captures non-linear Assumes linear relationship,
relationships may not capture complex
non-linear relationships
Data Assumes Gaussian distribution Assumes a linear
Distribution combination of latent
factors and noise term
Non-linearity Can capture non-linear transformations Assumes linear relationship,
of input data limited in capturing non-
linear relationships
Performance Good for dimensionality reduction Good for interpreting
relationships and
understanding underlying
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 95
factors
Probabilistic PCA and Factor Analysis (FA)
• In general,
• "Probabilistic PCA is suitable when dealing with high-dimensional data that may exhibit
non-linear relationships and when interpretability of the latent factors is not the primary
concern."
• "Factor Analysis, on the other hand, is preferred when interpretability and understanding the
underlying factors driving the observed data are important."
• NOTE: Probabilistic PCA is suitable when dealing with high-dimensional data and non-
linear relationships, whereas Factor Analysis is preferable when interpretability and
understanding underlying factors are paramount.
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 96
Encoder Decoder
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 97
Encoder Decoder
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 98
Encoder Decoder
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 99
Encoder Decoder
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 100
Encoder Decoder
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 101
Why Attenstion
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 102
Why Attenstion
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 103
Why Attenstion
5 November 2025 A course on Machine Learning by Sunil Kumar Gouda 104