Scribd
Upload
7 views6 pages

Generative Modeling: Foundations & Techniques

Generative models are conditional generators that produce outputs based on given inputs, with examples including text, image, and speech generators. The primary objective of generative modeling is to estimate an unknown data distribution and learn to sample from it, contrasting with discriminative models that focus on conditional probabilities. The document outlines a three-step framework for solving generative problems, involving the assumption of a parametric family, the use of deep neural networks, and optimization via divergence metrics.

Uploaded by

Tahmid Alam
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
7 views6 pages

Generative Modeling: Foundations & Techniques

Generative models are conditional generators that produce outputs based on given inputs, with examples including text, image, and speech generators. The primary objective of generative modeling is to estimate an unknown data distribution and learn to sample from it, contrasting with discriminative models that focus on conditional probabilities. The document outlines a three-step framework for solving generative problems, involving the assumption of a parametric family, the use of deep neural networks, and optimization via divergence metrics.

Uploaded by

Tahmid Alam
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Introduction

Generative Modeling Foundations


Definition and Real-World Examples : Generative models act as conditional generators.
Given an input (prompt), they generate a corresponding output. Common examples include:
Text Generators: ChatGPT, Gemini, Claude (Text → Text/Code).
Image Generators: Stable Diffusion, DALL-E (Text → Image).
Speech Generators: Text-to-Speech models (Text → Audio Waveform).
Mathematical Definition of Data :

Data is defined as a set of n points {x 1


, . . . , xn } lying in a high-dimensional real space
D.

For example, a 400 × 400 pixel RGB image has a dimensionality


D = x ∗ y ∗ 3channels = 480, 000.

Key Assumption: Data points are sampled IID (independent and identically distributed)
from an unknown underlying distribution, denoted as p . This means statistical
x

independence exists across samples (e.g., Image 1 vs. Image 100), not necessarily
across dimensions within a single sample.

Two random images are statistically independent , all images have same underlying
distribution (We assume same distribution even if it may not . For easily tackling the
problem)

The Core Objective :


Data points are treated as instantiations of a vector-valued random variable.
The primary goal of generative modeling is to estimate the unknown distribution p x

and, crucially, learn to sample from it.


Contrast: Unlike discriminative models which estimate p(y|x) (labels given data),
generative models focus on the data distribution itself.

The General Solving Recipe : To solve the generative problem, the lecture proposes a three-
step framework:
1. Assume a Parametric Family: Assume a model distribution p parameterized by θ.
θ

2. Deep Neural Networks: Due to their universal approximation capabilities, deep neural
networks are used to represent these models (p ) to capture complex, high-dimensional
θ
functions.

Optimization via Divergence Metrics :


Define a divergence metric (or distance) D(p ||p θ x) that measures the "gap" between
the model distribution and the true distribution.
Solve an optimization problem over the parameters θ to minimize this divergence.
The goal is to find θ such that p
∗
θ
∗ ≈ px .
The Sampling Mechanism :

Sampling is achieved by transforming a simple, known random variable.


Let z be a random variable from a simple known distribution (e.g., Gaussian/Normal
distribution N (0, I )){0 mean ,unit variance}.
Pass z through a deterministic function (neural network) g .
θ (z)

The output x
^ = g θ
(z) will follow a complex distribution determined by the function g . θ
Implicit Estimation :
By optimizing the network parameters θ to minimize the divergence between the
output distribution and the real data distribution, the network g implicitly estimates p
θ x

.
Once trained (g ), you can generate new valid data simply by sampling noise z and
θ
∗

passing it through the network.


!

You might also like