A Tutorial on Variational Autoencoders

Intuition, Variational Inference, Mathematical Derivation, and Motivation Beyond

CDF-Based Sampling

Abstract

Variational Autoencoders (VAEs) constitute one of the foundational frameworks in mod-

ern generative modeling. They combine probabilistic latent-variable modeling with neu-
ral networks in a manner that permits efficient learning and sampling in high-dimensional
spaces. This tutorial develops the subject from first principles. It begins with the idea of
generative modeling and clarifies why direct cumulative-distribution-function (CDF) based
sampling, although conceptually clean in low dimensions, becomes inadequate for realistic
data such as images, audio, or text. It then introduces latent-variable models, derives the
central problem of posterior intractability, and motivates variational inference as a tractable
approximation strategy. The evidence lower bound (ELBO) is derived carefully, followed by
the VAE objective, the reparameterization trick, and practical interpretations of each term
in the loss.

1 Introduction
A generative model aims to describe how observed data may have been produced. Given a data
point x, such as an image, a speech segment, or a document, a generative model seeks to assign
it probability under some learned distribution pθ (x) and, ideally, to generate new samples that
resemble the training data. This is conceptually different from discriminative modeling, where
the focus is on predicting labels or outputs from inputs. In generative modeling, the central
object of interest is the distribution of the data itself.
If the data distribution were known exactly in a simple form, one could sample from it di-
rectly. In elementary probability, one common method is inverse transform sampling: if F is the
cumulative distribution function (CDF) of a scalar random variable, and if u ∼ Uniform(0, 1),
then x = F −1 (u) has the desired distribution. This idea is elegant and complete in one dimen-
sion. However, realistic generative modeling rarely operates in such a convenient regime. The
data are usually high-dimensional, their distributions are highly structured, and neither the
CDF nor its inverse is available in closed form. Even defining a workable joint CDF for very
high-dimensional data is generally impractical.
This limitation motivates richer frameworks that do not require explicit inversion of a CDF
in the observed data space. One powerful idea is to assume that the observed data are generated
from a lower-dimensional latent variable z. Rather than model the data distribution directly in
the original space, we model a simpler latent distribution and a probabilistic decoder that maps
latent causes to observations. This leads naturally to latent-variable models and, ultimately, to
the VAE.

1
2 Why CDF-Based Sampling is Not Sufficient for Modern Gen-
erative Modeling
2.1 Inverse CDF Sampling in One Dimension
For a scalar random variable x with cumulative distribution function

F (x) = P(X ≤ x),

inverse transform sampling proceeds as follows:

1. Sample u ∼ Uniform(0, 1).

2. Set x = F −1 (u).

Then x has distribution F . This method is exact when F −1 is available.

This construction is useful pedagogically because it shows that sampling from a distribution
is, in principle, possible if one can characterize its cumulative mass and invert it.

2.2 Difficulties in High Dimensions

Suppose x ∈ RD with large D, as in an image represented by thousands of pixels. In that
setting, several difficulties arise.

First, the notion of a multivariate CDF is not operationally convenient. A joint

CDF
F (x1 , . . . , xD ) = P(X1 ≤ x1 , . . . , XD ≤ xD )

exists formally, but it is not generally an object one can efficiently evaluate or invert.

Second, dependencies across coordinates are extremely complex. In images, pixels

are not independent. The value of one pixel is tied strongly to its neighbors and to global
semantic structure. A direct CDF-based representation does not naturally exploit this structure.

Third, learning the full data distribution directly in observed space is too de-
manding. The data often lie near a much lower-dimensional manifold embedded in a high-
dimensional ambient space. Direct sampling in the ambient space is inefficient and statistically
wasteful.

Fourth, exact inversion is usually impossible. Even if a distribution is specified up to a

density, obtaining its inverse CDF is rarely feasible except in special cases.

2.3 The Need for Latent Variables

A central idea in modern generative modeling is that complicated observations may be generated
from simpler hidden causes. For example, an image of a face may be determined by latent factors
such as pose, illumination, identity, and expression. These factors are not directly observed, but
they give rise to the image. Therefore, rather than sample directly in image space by inverting
a joint CDF, one may instead:

2
 1. sample a latent variable z from a simple prior, often Gaussian,

2. generate x from a conditional distribution pθ (x | z).

This is the latent-variable perspective, which provides both structure and tractability.

3 Latent-Variable Generative Models

Let z ∈ Rd denote a latent variable, with d ≪ D often chosen smaller than the dimension of
the observation space. A latent-variable model defines:

pθ (x, z) = p(z) pθ (x | z),

where:

• p(z) is the prior over latent variables,

• pθ (x | z) is the likelihood or decoder,

• θ are learnable parameters.

The marginal likelihood of a data point is then

Z Z
pθ (x) = pθ (x, z) dz = p(z) pθ (x | z) dz.

This integral sums over all possible latent explanations z that could have produced x.

3.1 Learning Objective

Given a dataset {x(i) }N
i=1 , a natural learning objective is maximum likelihood:

N
X
max log pθ (x(i) ).
θ
i=1

However, each term involves

Z
log pθ (x) = log p(z) pθ (x | z) dz,

and this integral is generally intractable when pθ (x | z) is parameterized by a neural network.

This intractability is the point at which variational inference becomes necessary.

4 Posterior Inference and Its Intractability

If we observe x, the posterior distribution over latent variables is

pθ (x, z) p(z) pθ (x | z)
pθ (z | x) = =R .
pθ (x) p(z) pθ (x | z) dz

The posterior answers the inverse question:

3
 Given an observed data point x, which latent configurations z could plausibly have
generated it?

This posterior is crucial. To learn a good latent representation and to optimize the model,
we would like access to pθ (z | x). But the denominator is the marginal likelihood, which is itself
intractable. Hence the posterior is intractable as well.
This creates a circular difficulty:

• To evaluate the posterior, one needs the marginal likelihood.

• To maximize the marginal likelihood, one would ideally use the posterior.

Variational inference resolves this by replacing the true posterior with a tractable approxi-
mation.

5 Variational Inference: Core Idea

5.1 Approximating the Posterior
Let qϕ (z | x) be a family of tractable distributions parameterized by ϕ. This distribution is
intended to approximate the true posterior pθ (z | x). The goal is to make qϕ (z | x) close to
pθ (z | x) while simultaneously learning the generative model parameters θ.
Typically, qϕ (z | x) is chosen to be simple enough to evaluate and sample from efficiently. In
VAEs, it is usually taken to be Gaussian with mean and variance produced by a neural network.

5.2 KL Divergence
A natural measure of discrepancy between distributions is the Kullback–Leibler (KL) divergence:
Z
q(z)
DKL (q(z) ∥ p(z)) = q(z) log dz.
p(z)

This quantity is always nonnegative and equals zero if and only if q(z) = p(z) almost everywhere.
In variational inference, one seeks to minimize


DKL qϕ (z | x) ∥ pθ (z | x) .

However, the true posterior is intractable, so this expression cannot be optimized directly in its
raw form. The key breakthrough is that it can be rewritten to produce a tractable lower bound
on log pθ (x).

6 Derivation of the Evidence Lower Bound (ELBO)

We now derive the ELBO carefully.
Start with the log marginal likelihood:

log pθ (x).

4
Insert the variational distribution qϕ (z | x) inside the integral:
Z
pθ (x, z)
log pθ (x) = log qϕ (z | x) dz.
qϕ (z | x)

Recognizing the integral as an expectation under qϕ (z | x),

 
pθ (x, z)
log pθ (x) = log Eqϕ (z|x) .
qϕ (z | x)

Applying Jensen’s inequality, since log is concave,

log E[Y ] ≥ E[log Y ].

Hence,  
pθ (x, z)
log pθ (x) ≥ Eqϕ (z|x) log .
qϕ (z | x)
This lower bound is called the evidence lower bound, or ELBO:

L(θ, ϕ; x) = Eqϕ (z|x) [log pθ (x, z) − log qϕ (z | x)] .

Using the factorization pθ (x, z) = p(z)pθ (x | z),

L(θ, ϕ; x) = Eqϕ (z|x) [log pθ (x | z)] − DKL qϕ (z | x) ∥ p(z) .

This is the standard VAE objective.

6.1 Alternative Derivation Linking ELBO and Posterior KL

A second derivation makes the role of posterior approximation explicit. Begin with
 

qϕ (z | x)
DKL qϕ (z | x) ∥ pθ (z | x) = Eqϕ (z|x) log .
pθ (z | x)

Substitute Bayes’ rule:

pθ (x, z)
pθ (z | x) = .
pθ (x)
Then


DKL qϕ (z | x) ∥ pθ (z | x) = Eqϕ (z|x) [log qϕ (z | x) − log pθ (x, z) + log pθ (x)] .

Since log pθ (x) does not depend on z,

DKL qϕ (z | x) ∥ pθ (z | x) = log pθ (x) − L(θ, ϕ; x).

Therefore,


log pθ (x) = L(θ, ϕ; x) + DKL qϕ (z | x) ∥ pθ (z | x) .

Since KL divergence is nonnegative,

L(θ, ϕ; x) ≤ log pθ (x).

5
 This identity is extremely important. It shows that maximizing the ELBO does two things
simultaneously:

1. it increases the marginal likelihood of the data,

2. it reduces the discrepancy between the approximate posterior and the true posterior.

7 Interpretation of the ELBO Terms

The ELBO is
L(θ, ϕ; x) = Eqϕ (z|x) [log pθ (x | z)] − DKL (qϕ (z | x) ∥ p(z)).

Each term has a distinct role.

7.1 Reconstruction Term

The first term,
Eqϕ (z|x) [log pθ (x | z)],

encourages the latent variable sampled from qϕ (z | x) to retain enough information about x so
that the decoder can reconstruct it well. This is why VAEs are often said to behave partly like
autoencoders.
If the decoder distribution is Gaussian with fixed variance, maximizing this term becomes
closely related to minimizing squared reconstruction error. If the decoder is Bernoulli, it corre-
sponds to a cross-entropy-like reconstruction loss.

7.2 Regularization Term

The second term,
DKL (qϕ (z | x) ∥ p(z)),

regularizes the approximate posterior so that it remains close to the prior. This prevents the
latent space from becoming arbitrary and disorganized. It also ensures that, after training, one
can generate new data by sampling z ∼ p(z) and decoding it.
Hence the KL term is not merely a penalty; it is what makes the latent space generative
rather than just reconstructive.

8 From Variational Inference to the Variational Autoencoder

8.1 Encoder and Decoder Networks
The VAE implements the variational distribution and the likelihood using neural networks:

• The encoder network maps x to the parameters of qϕ (z | x).

• The decoder network maps z to the parameters of pθ (x | z).

A common choice is:

qϕ (z | x) = N (z; µϕ (x), diag(σϕ2 (x))),

where the encoder outputs a mean vector µϕ (x) and variance vector σϕ2 (x).

6
 The prior is usually
p(z) = N (0, I).

The decoder may define, for example,

pθ (x | z) = N (x; fθ (z), σ 2 I),

or, for binary data,

pθ (x | z) = Bernoulli(fθ (z)).

8.2 Why the Name “Autoencoder”?

The terminology arises because the architecture resembles an autoencoder:

• the encoder compresses x into a latent representation,

• the decoder reconstructs x from that representation.

However, unlike a deterministic autoencoder, the VAE is probabilistic:

• the encoder outputs a distribution over latents,

• the decoder defines a distribution over observations,

• the training objective comes from variational inference.

9 The Reparameterization Trick

9.1 The Difficulty
The ELBO contains an expectation with respect to qϕ (z | x):

Eqϕ (z|x) [log pθ (x | z)].

To optimize this using gradient-based learning, we must differentiate through samples from
qϕ (z | x). Direct sampling introduces stochasticity in a way that obstructs naive backpropaga-
tion.

9.2 The Solution

For Gaussian latent variables, write

z = µϕ (x) + σϕ (x) ⊙ ϵ, ϵ ∼ N (0, I),

where ⊙ denotes elementwise multiplication.

This expresses sampling from qϕ (z | x) as a deterministic function of:

• the encoder outputs µϕ (x) and σϕ (x),

• an auxiliary noise variable ϵ whose distribution does not depend on ϕ.

7
 Then the expectation can be rewritten as


Eqϕ (z|x) [log pθ (x | z)] = Eϵ∼N (0,I) log pθ x | µϕ (x) + σϕ (x) ⊙ ϵ .

Now gradients can flow through µϕ (x) and σϕ (x) using standard backpropagation.

9.3 Why It Matters

The reparameterization trick is one of the central innovations that made VAEs practical. With-
out it, one would need more general stochastic gradient estimators, which often suffer from high
variance and slower training.

10 Closed-Form KL for Gaussian Posterior and Standard Nor-

mal Prior
If
qϕ (z | x) = N (z; µ, diag(σ 2 )) and p(z) = N (0, I),

then the KL divergence has the closed form

d
1X 2
µj + σj2 − log σj2 − 1 .


DKL (qϕ (z | x) ∥ p(z)) =
2
j=1

Therefore, for one data point, the VAE objective becomes

d
1X 2
µj + σj2 − log σj2 − 1 .


L(θ, ϕ; x) = Eqϕ (z|x) [log pθ (x | z)] −
2
j=1

In practice, one minimizes the negative ELBO.

11 Step-by-Step Learning Procedure in a VAE

For each input x:
1. The encoder produces µϕ (x) and σϕ (x).

2. Sample ϵ ∼ N (0, I).

3. Form
z = µϕ (x) + σϕ (x) ⊙ ϵ.

4. The decoder outputs the parameters of pθ (x | z).

5. Compute the reconstruction term and the KL term.

6. Optimize the negative ELBO by gradient descent.

After training, generation is simple:
1. sample z ∼ N (0, I),

2. decode via pθ (x | z).

8
12 Why Variational Inference is Needed
It is useful to state the necessity of variational inference clearly.

12.1 The True Posterior is Intractable

In a latent-variable model with neural-network decoder,

p(z)pθ (x | z)
pθ (z | x) =
pθ (x)

cannot generally be computed exactly because the denominator requires integrating over all
latent configurations.

12.2 Exact Bayesian Inference is Too Expensive

One could imagine using exact Bayesian inference if the model were simple and conjugate. But
for expressive nonlinear decoders, exact posterior computation becomes impossible in practice.

12.3 Variational Inference Converts Inference into Optimization

Rather than compute the true posterior analytically, variational inference introduces a tractable
family qϕ (z | x) and optimizes it to approximate the posterior. This transforms a difficult
integration problem into an optimization problem.
This is a profound shift:

Instead of solving inference exactly, we solve a best-approximation problem.

13 Why VAE is Needed Over Direct CDF-Based Sampling

The comparison can now be made more precisely.

13.1 CDF-Based Sampling is Limited to Simple or Special Cases

Inverse CDF sampling works naturally when:

• the variable is one-dimensional or very low-dimensional,

• the CDF is known analytically or numerically,

• the inverse can be computed efficiently.

These conditions fail for most real-world generative tasks.

13.2 Modern Data Have High-Dimensional Structure

Images, signals, and language have rich dependencies. A VAE addresses this by:

• introducing latent variables that capture hidden factors,

• learning a structured decoder from latent space to observation space,

• avoiding the need to represent or invert a joint CDF in the original data space.

9
13.3 VAE Learns a Useful Latent Representation
Inverse CDF sampling merely provides a method of drawing samples if the distribution is already
known. It does not, by itself, discover meaningful hidden structure in data.
A VAE, by contrast, learns:

• a latent organization of the dataset,

• an encoder for approximate posterior inference,

• a decoder for probabilistic generation,

• a prior-aligned latent space from which new samples can be drawn.

13.4 VAE Supports Amortized Inference

For each new observation x, the encoder directly predicts the approximate posterior parameters:

qϕ (z | x).

This is known as amortized inference. Instead of solving a new optimization problem for each
data point, one learns a general inference mechanism shared across all data.
Direct CDF methods offer no such mechanism.

13.5 VAE Scales to Complex Data

A VAE can be trained end-to-end with neural networks using stochastic gradient descent. This
makes it suitable for complex, high-dimensional datasets where direct probabilistic constructions
in observed space are not feasible.

14 Intuitive Summary of the VAE Philosophy

The VAE can be understood as combining three ideas:

First: hidden causes generate observations. Observed data are assumed to arise from
lower-dimensional latent variables.

Second: exact inference is hard. Given an observation, finding the posterior over latent
causes is computationally intractable.

Third: learn an approximate inference mechanism. A neural encoder learns to produce

a tractable approximate posterior, and a neural decoder learns to generate observations from
latent samples.
Thus the VAE is not merely a neural architecture; it is a principled approximation to
Bayesian latent-variable inference.

10
15 A Small Conceptual Example
Suppose we model handwritten digits. Each image x is high-dimensional, but the true factors
that shape it may be much lower-dimensional: stroke thickness, slant, style, digit identity, and
local deformation. A VAE assumes that such factors are encoded in a latent variable z.
The encoder observes an image of, say, a digit “3” and outputs a Gaussian distribution over
plausible latent descriptions:

qϕ (z | x) = N (µϕ (x), diag(σϕ2 (x))).

Sampling from this distribution gives a latent code that the decoder uses to reconstruct the
image. Because the approximate posterior is regularized toward a common prior, nearby latent
points tend to decode to visually similar digits. This gives the latent space continuity and
semantic structure.
Such behavior would be difficult to obtain from a direct inverse-CDF construction in raw
pixel space.

16 Strengths and Limitations of VAEs

16.1 Strengths
• They provide a principled probabilistic framework.

• They learn continuous and often interpretable latent spaces.

• They support efficient sampling from a simple prior.

• They combine representation learning and generative modeling.

• They scale well with neural network parameterizations.

16.2 Limitations
• Generated samples may appear blurrier than those from some adversarial or diffusion-based
models.

• The approximate posterior may be too restrictive if chosen from a simple family such as
diagonal Gaussians.

• If the decoder is overly powerful, the latent code may be underutilized, a phenomenon
related to posterior collapse.

These limitations have inspired many extensions, such as β-VAEs, hierarchical VAEs, normalizing-
flow posteriors, and more expressive priors.

17 Concluding Remarks
The Variational Autoencoder emerges from a very natural sequence of ideas. One begins with
the desire to model complex data probabilistically, recognizes that direct CDF-based sampling

11
is inadequate in high-dimensional structured domains, introduces latent variables to explain
observations more economically, and then confronts the intractability of posterior inference.
Variational inference resolves this difficulty by replacing exact inference with an optimization-
based approximation, and the VAE makes this approximation learnable through neural networks
and the reparameterization trick.
The resulting framework is elegant because each component has a clear probabilistic role: the
prior defines the latent geometry, the decoder defines the generative mechanism, the encoder
approximates posterior inference, and the ELBO ties them together into a single objective.
For this reason, the VAE remains a foundational model for understanding modern generative
learning.

Appendix: Compact Formula Sheet

Latent-variable model:
pθ (x, z) = p(z) pθ (x | z).

Marginal likelihood: Z
pθ (x) = p(z) pθ (x | z) dz.

Posterior:
p(z) pθ (x | z)
pθ (z | x) = .
pθ (x)

ELBO:
L(θ, ϕ; x) = Eqϕ (z|x) [log pθ (x | z)] − DKL (qϕ (z | x) ∥ p(z)).

ELBO identity:
log pθ (x) = L(θ, ϕ; x) + DKL (qϕ (z | x) ∥ pθ (z | x)).

Reparameterization trick:

z = µϕ (x) + σϕ (x) ⊙ ϵ, ϵ ∼ N (0, I).

KL for diagonal Gaussian posterior and standard normal prior:

d
1X 2
µj + σj2 − log σj2 − 1 .


DKL (qϕ (z | x) ∥ p(z)) =
2
j=1

12