Generative
Models
Introduction
to Generative
Models
Generative models are machine learning algorithms
that generate new data samples similar to the
training data. Unlike traditional models that focus on
classification or regression, generative models aim
to learn the underlying distribution of the data to
produce new, realistic examples.
Dimensionality Explosion: Many
The Problem: 01. datasets have hundreds or
thousands of features, making
them hard to visualize and
process.
Managing and Generating
Complex Data Data Noise and Redundancy:
In today’s data-driven world, we face
challenges in working with high-dimensional
02. High-dimensional data often
contains irrelevant or noisy
features.
and complex data:
Data Scarcity: In some cases, we
03. need to generate synthetic data
for training
purposes.
or simulation
Dimensionality
Reduction
Dimensionality reduction
simplifies datasets by reducing the number of features
while retaining the most important information.
is the process of reducing the number of features that
describe some data. (wiki)
Why Dimensionality Reduction is Important
Data Visualization: Reducing data to 2D or 3D for
easier interpretation.
Data Storage: Lower-dimensional data requires less
storage space.
Computational Efficiency: Reducing dimensions
speeds up algorithms, especially with large datasets.
Dimensionality
Reduction
This can be achieved in two main ways:
Selection: Conserving only a subset of
the existing features (only some
existing features are conserved).
Extraction: Creating a smaller set of
new features based on combinations
of the original features (a reduced
number of new features are created
based on the old features).
Key Concepts
Encoder
Maps the original high-dimensional data into a compressed representation in the latent space.
Represents the compression step of the process.
Decoder
Maps the compressed latent representation back to the original space.
Represents the decompression step of the process.
Latent Space
A lower-dimensional space where the essential features of the data are captured.
Purpose
Dimensionality reduction aims to find the best encoder-decoder pair that minimizes information loss
during compression and decompression. This ensures that the encoded representation retains the most
critical features of the data.
Optimization Goal
If we denote respectively E and D the families of encoders and decoders we are considering, then the
dimensionality reduction problem can be written :
where :
defines the reconstruction error measure between the input data x and the encoded-decoded data d(e(x)).
Principal Component Analysis (PCA)
Principal Component Analysis (PCA) is a widely used dimensionality reduction technique that transforms
data into a smaller set of orthogonal components, called principal components, which are linear
combinations of the original features. These components are chosen to retain the maximum possible
variance in the data while reducing its dimensionality.
Core Idea
PCA identifies the best linear subspace of the original space, defined by an orthogonal basis of new
features.
Data is projected onto this subspace such that the approximation error (measured as Euclidean
distance) is minimized.
Principal Component Analysis (PCA)
Core Idea
PCA identifies the best linear subspace of the original space, defined by an orthogonal basis of new
features.
Data is projected onto this subspace such that the approximation error (measured as Euclidean
distance) is minimized.
Principal Component Analysis (PCA)
Steps in PCA:
1. Calculate the covariance matrix of the dataset.
2. Find eigenvectors and eigenvalues.
3. Choose the top components with the largest eigenvalues.
4. Project the data onto these components.
Principal Component Analysis (PCA)
PCA can be framed as an optimization problem within our encoder-decoder framework:
Encoder: A matrix e in E of size (n_e, n_d) that maps data to the latent space. Rows of this matrix are
orthonormal, ensuring feature independence.
Decoder: A matrix d in D of size (n_d, n_e) that reconstructs data from the latent space.
Limitations of PCA
• Assumes linearity: Cannot handle complex, nonlinear relationships.
• Sensitive to scaling: Requires data standardization.
AutoEncoders (AE)
Autoencoders are a type of artificial neural network designed to learn compact,
efficient representations of data. Their primary purpose is dimensionality reduction
and data reconstruction. Unlike traditional algorithms like PCA, autoencoders can
model complex, non-linear relationships in the data.
AutoEncoders
Linear Autoencoders: A Connection to PCA
When both the encoder and decoder are single-layer linear transformations without non-linearities,
autoencoders behave similarly to PCA:
Both seek the best linear subspace for data projection.
Encoding and decoding matrices from PCA are valid solutions for a linear autoencoder.
Linear Autoencoders: A Connection to PCA
When both the encoder and decoder are single-layer linear transformations without non-linearities,
autoencoders behave similarly to PCA:
Both seek the best linear subspace for data projection.
Encoding and decoding matrices from PCA are valid solutions for a linear autoencoder.
Key Differences from PCA:
No Orthogonality Constraint: Autoencoders do not require new features to be orthogonal, unlike PCA.
Multiple Solutions: Autoencoders can learn multiple equivalent encoder-decoder pairs for the same optimal
reconstruction error.
Deep and Non-Linear Autoencoders
When both the encoder and decoder are deep and non-linear, autoencoders can achieve significant
dimensionality reduction while maintaining low reconstruction loss.
Theoretically, an encoder with unlimited capacity could reduce any initial dimensionality to 1, with a decoder
reversing this process without loss.
Excessive dimensionality reduction may result in a latent space lacking interpretability and structure,
making it less useful for practical tasks.
The goal is not just to minimize dimensions but to retain the structure and meaningful features of the
data in the reduced representation.
Deep and Non-Linear Autoencoders
What is the link between autoencoders and content generation?
How can we generate data with autoencoders?
Autoencoders and content generation
1 - We could sample random points from the latent space.
2 - Pass these points through the decoder to produce new content.
The decoder can act like the generator, it translates points in the latent space into meaningful
data representations.
Autoencoders and content generation
1 - We could sample random points from the latent space.
2 - Pass these points through the decoder to produce new content.
The decoder can act like the generator, it translates points in the latent space into meaningful
data representations.
If the latent space is well-organized (regular and structured by the encoder during training)
Autoencoders and content generation
“The autoencoder is solely trained to encode and decode with as few loss as possible, no matter how the
latent space is organised.”
Intuitions about the regularisation
The regularity that is expected from the latent space in order to make generative process
possible can be expressed through two main properties:
continuity : two close points in the latent space should not give two completely
different contents once decoded.
completeness : for a chosen distribution, a point sampled from the latent space
should give “meaningful” content once decoded.
Issues
Encoding inputs as distributions in VAEs is not enough to guarantee a well-organized latent
space. Without proper regularization:
Tiny Variances:
The encoder may produce distributions with very small variances, resembling single
points.
This makes the VAE behave like a classic autoencoder, losing the benefits of probabilistic
encoding and risking overfitting.
Large Mean Differences:
The encoder may generate distributions with means far apart in the latent space.
This disrupts the structure and continuity of the latent space.
The Solution: Regularization
To maintain continuity and completeness in the latent space, the VAE enforces a regularization term
during training.
Covariance Regularization:
Ensures covariance matrices are close to the identity matrix
Prevents the formation of “punctual distributions” with near-zero variance.
Mean Regularization:
Encourages means to be close to 0.
Prevents distributions from being spaced too far apart in the latent space.
Practical Implementation
Regularization is achieved by encouraging the encoded distributions to be close to a standard normal
distribution. :
Centered: Means are close to 0.
Reduced: Variances are close to 1.
This is enforced through the Kullback-Leibler (KL) Divergence term in the VAE loss function:
Variational Autoencoders (VAE)
A variational autoencoder can be defined as being an autoencoder whose training is regularised to avoid
overfitting and ensure that the latent space has good properties that enable generative process.
In order to introduce some regularisation of the latent space, we proceed to a slight modification of the
encoding-decoding process: instead of encoding an input as a single point, we encode it as a
distribution over the latent space.
Variational Autoencoders
The model is then trained as follows:
First, the input is encoded as distribution over the latent space
Second, a point from the latent space is sampled from that distribution
Third, the sampled point is decoded and the reconstruction error can be computed
Finally, the reconstruction error is backpropagated through the network
Variational Autoencoders
Encoded Distributions in VAEs :
Inputs are encoded not as single points but as probabilistic distributions (typically Gaussian).
The encoder outputs:
Mean : Center of the distribution.
Covariance Matrix : Defines the spread/variance of the distribution.
Loss Function in VAEs
The loss function used in training VAEs combines two terms:
Reconstruction Term:
Minimizes the difference between the input data and its reconstruction.
Ensures the encoding-decoding process is accurate.
Regularization Term (Kullback-Leibler Divergence):
Measures the divergence between the encoded distribution and a standard Gaussian distribution
Encourages the latent space to be well-organized and regular.
Loss Function in VAEs
The Tradeoff
Regularization introduces a tradeoff:
Lower KL Divergence: Enforces a well-structured latent space.
Higher Reconstruction Error: Slightly reduces the reconstruction accuracy on the training data.
This tradeoff can be adjusted by controlling the relative weights of the reconstruction loss and the KL divergence in
the total loss function:
Mathematical Framework of Variational
Autoencoders (VAEs)
Probabilistic Framework
Generative Assumptions
Latent Variable (z):
Encoded representation sampled from a prior distribution p(z), typically a standard Gaussian
Data Variable (x):
Data is sampled from the likelihood distribution p(x|z), which depends on the latent representation z.
Probabilistic Encoder and Decoder
Encoder ( p(z|x) ): Defines the posterior distribution of the latent variable given the input.
Decoder ( p(x|z) ): Defines the likelihood distribution of the data given the latent variable.
Bayes Theorem
Relates prior, likelihood, and posterior distributions:
Assumptions in VAEs
Prior Distribution: p(z) is standard Gaussian .
Likelihood Distribution: p(x|z) is Gaussian with:
Mean f(z) , modeled as a neural network.
Covariance cI , where c is a constant.
The Challenge of Posterior Approximation
The posterior p(z|x) , which describes the probability of the latent variable z given the input x , is central to the
probabilistic framework of VAEs. However, directly computing p(z|x) is intractable because of the integral in the
denominator of Bayes’ theorem:
The Issue: The marginal likelihood p(x) involves an integral over all possible values of u :
This integral is typically computationally expensive or impossible to evaluate directly due to the high dimensionality
and complexity of the distributions.
from more details : [Link]
25a8aa9bce29
Solution: Variational Inferenc
Variational inference is used to approximate p(z|x) with a simpler, parameterized distribution. :
Key Idea:
Instead of directly computing p(z|x) , define a family of simpler distributions (e.g., Gaussians).
Optimize the parameters of to make it as close as possible to p(z|x) .
This closeness is measured using the Kullback-Leibler (KL) Divergence:
Practical Implementation in VAEs
Assumption for :
is modeled as a Gaussian distribution with:
Mean: g(x) , parameterized by a neural network.
Covariance: h(x) , also parameterized by a neural network.
Optimization Objective:
Maximize the Evidence Lower Bound (ELBO), which is equivalent to minimizing the KL divergence
between and p(z|x) :
Practical Implementation in VAEs
Practical Implementation in VAEs
In other words, for a given input x, we want to maximise the probability to have when we sample z from the
distribution and then sample from the distribution p(x|z). Thus, we are looking for the optimal such that :
then the loss will be :
Bringing Neural Networks into the Variational
Autoencoder Model
The VAE model relies on three functions:
f(z) : Decoder, mapping latent variables z back to the data space.
g(x) : Encoder function defining the mean of the latent variable distribution q_x(z) .
h(x) : Encoder function defining the covariance of q_x(z) .
Direct optimization over all possible functions is infeasible due to the infinite functional space.
Solution: Use neural networks to parameterize f, g, h . These functions now belong to families of
functions F, G, H , defined by their respective network architectures.
Bringing Neural Networks into the Variational
Autoencoder Model : some optimization
Encoder :
Shared Architecture:
g(x) (mean) and h(x) (covariance) often share parts of their network architecture and weights to simplify
computation.
Diagonal Covariance Assumption:
Instead of a full covariance matrix, we assume q_x(z) has a diagonal covariance matrix (independent variables):
This reduces the size of h(x) to a vector with the same dimensionality as g(x) .
Bringing Neural Networks into the Variational
Autoencoder Model : some optimization
Decoder :
The decoder models p(x|z) , which is assumed to be Gaussian:
f(z) : Neural network that outputs the mean of p(x|z) .
The covariance cI is fixed (a constant), simplifying the optimization process.
Bringing Neural Networks into the Variational
Autoencoder Model : Reparameterization Trick
During training, z is sampled from the encoder’s output distribution . Direct sampling is non-differentiable :
Let’s say we want to take the gradient w.r.t. θ of the following expectation,
where p is a density. Provided we can differentiate , we can easily compute the gradient:
Bringing Neural Networks into the Variational
Autoencoder Model : Reparameterization Trick
In words, the gradient of the expectation is equal to the expectation of the gradient. But what happens if
our density p is also parameterized by θ?
Bringing Neural Networks into the Variational
Autoencoder Model : Reparameterization Trick
When we apply the reparameterization trick to our simple example :
Bringing Neural Networks into the Variational
Autoencoder Model : Reparameterization Trick
• Challenge: Sampling is non-differentiable.
• Solution: Reparameterize z as:
• Enables gradient flow for backpropagation through g(x) and h(x) .
Bringing Neural Networks into the Variational
Autoencoder Model : Reparameterization Trick
VAE Loss Function