22AIM71 GENERATIVE AI MODULE- 2 NOTES

Probability Basics for Gen-AI

1. Introduction to Probability

 Definition: Probability quantifies uncertainty and is a measure of how likely an event is

to occur.
 Range: Probability values range from 0 to 1:
o 0 indicates an impossible event.
o 1 indicates a certain event.
o The closer the value is to 1, the more likely the event is.

Formula

2. Key Probability Concepts

 Experiment: A procedure that generates outcomes, e.g., rolling a die or flipping a coin.
 Sample Space (S): The set of all possible outcomes of an experiment.
o For a coin flip: S = {Heads, Tails}.
o For a die roll: S = {1, 2, 3, 4, 5, 6}.
 Event (A): A subset of the sample space; a single outcome or a combination of outcomes.
o E.g., rolling an even number (Event A) from a die: A = {2, 4, 6}.

3. Types of Probability

 Theoretical Probability: Based on reasoning or models without actual experimentation.

1
22AIM71 GENERATIVE AI MODULE- 2 NOTES

 Experimental Probability: Based on actual experiments or historical data.

o Example: If a die is rolled 60 times and 3 appears 15 times, then:

 Subjective Probability: Based on personal judgment or

experience rather than data or formal computation.

2
22AIM71 GENERATIVE AI MODULE- 2 NOTES

4. Probability Rules

 Complement Rule: The probability of an event not occurring is:

 Addition Rule: The probability of the union of two events A and B:

P(A∪B)=P(A)+P(B)−P(A∩B)

If A and B are mutually exclusive (cannot happen simultaneously):

P(A∪B)=P(A)+P(B)

 Multiplication Rule: For two independent events A and B, the probability of both
occurring:

P(A∩B)=P(A)×P(B)

For dependent events, the rule adjusts for conditional probability.

5. Conditional Probability

 Definition: The probability of event A occurring given that event B has occurred is
written as

It helps in scenarios where one event influences another, crucial for AI models handling
sequences or predictions.

3
22AIM71 GENERATIVE AI MODULE- 2 NOTES

6. Bayes' Theorem

 Definition: A key concept in machine learning and AI for updating probabilities based on
new information.

Used in applications like spam detection, medical diagnosis, and classification algorithms in AI.

7. Random Variables

 Definition: A random variable assigns numerical values to outcomes of a random

experiment.
o Discrete Random Variables: Take on countable values (e.g., number of heads in
coin flips).
o Continuous Random Variables: Take on an infinite number of possible values
(e.g., the height of people).

8. Probability Distributions

 Discrete Probability Distribution: Lists each possible value a random variable can take,
along with its probability.

 Continuous Probability Distribution: Describes the probabilities of a continuous

random variable. Often represented by probability density functions (PDFs).
o Example: Normal distribution in AI for feature representation.

9. Expected Value (E)

 Definition: The weighted average of all possible values of a random variable.

4
22AIM71 GENERATIVE AI MODULE- 2 NOTES

It gives the long-term average outcome and is essential for decision-making algorithms in
AI.

10. Variance and Standard Deviation

 Variance (Var): Measures the spread or variability of a random variable.

 Standard Deviation (σ): The square root of the variance, used to quantify uncertainty in
predictions.

AI systems use variance to model uncertainties in predictions and make robust models.

5
22AIM71 GENERATIVE AI MODULE- 2 NOTES

11. Applications of Probability in AI

 Classification Models (e.g., Naive Bayes): Uses probability to predict classes based on
input features.
 Markov Models: Use probabilities to predict sequences or state transitions.
 Reinforcement Learning: Probability helps in decision-making under uncertainty in
dynamic environments.
 Bayesian Networks: Represent probabilistic relationships among variables for
reasoning in complex domains.

1. Introduction to GenAI and Probability

 Generative AI (GenAI): Refers to AI models capable of creating new content such as text,
images, or audio based on patterns learned from data.
 Probability's Role in GenAI: Probability is at the core of GenAI models, helping to model
uncertainty, make predictions, and generate realistic data. It’s used in decision-making
under uncertainty, generating likely sequences, and ensuring the coherence of outputs.

2. Probability in Language Models (e.g., GPT)

 Language Generation: Probabilistic models predict the next word in a sentence based
on the likelihood of previous words.
o Chain Rule of Probability: Models like GPT use the chain rule to decompose joint
probabilities over sequences:

 Next-word Prediction: Each word generated by GenAI models is chosen based on a

probability distribution over the vocabulary.
o For example, given a prompt, the model predicts the word that has the highest
conditional probability given the previous context.

6
22AIM71 GENERATIVE AI MODULE- 2 NOTES

3. Generative Models and Probabilistic Frameworks

 Generative Models: GenAI models are typically generative, meaning they learn the
probability distribution of the training data and generate new data points by sampling
from this learned distribution.
o Variational Autoencoders (VAEs): Use probabilistic encodings of data into latent
spaces and generate data from these encodings.
o Generative Adversarial Networks (GANs): Generate new data by learning a
distribution close to real-world data using two networks: a generator and a
discriminator.

Probability in GANs:

The generator tries to maximize the probability that the discriminator will mistake generated
data for real data.

4. Markov Models in GenAI

 Markov Chain: A stochastic model where the next state only depends on the current
state, not the sequence of events preceding it.
o Used in simple generative models to predict the likelihood of transitioning from
one state to another.

Markov Process for Text Generation:

o This assumption simplifies the model but retains useful probabilistic dynamics for
generating sequences (e.g., sentences).

7
22AIM71 GENERATIVE AI MODULE- 2 NOTES

5. Bayesian Networks in Generative AI

 Bayesian Networks: A directed graphical model representing probabilistic relationships

among variables. Used in GenAI to infer hidden structures or dependencies in data.
o Probability Perspective: Bayesian networks compute conditional probabilities
and use them to generate coherent content:

o Applications: They can be used in AI systems for decision-making or generating

outputs that respect underlying probabilistic relationships.

6. Sampling Techniques in GenAI

 Monte Carlo Methods: Used in probabilistic models to approximate expectations and

generate data. This involves sampling from a probability distribution to produce new
instances.

Examples:

o Markov Chain Monte Carlo (MCMC): Generates samples from complex

probability distributions by constructing a Markov chain.
o Importance Sampling: Assigns weights to samples based on their probability to
generate new data effectively.

 Probabilistic Sampling in Text Generation:

o Top-k Sampling: Limits the next-word generation to the top k most probable
words.
o Nucleus Sampling: Samples from the smallest set of words whose cumulative
probability exceeds a certain threshold ppp, ensuring variability in the generated
text.

7. Probability Distributions in GenAI

8
22AIM71 GENERATIVE AI MODULE- 2 NOTES

 Multinomial Distribution: In text generation, the likelihood of each word in the

vocabulary is modeled as a multinomial distribution.

 Gaussian Distribution: VAEs use Gaussian distributions to encode latent variables,

representing data points in latent space.
o Gaussian Mixture Models (GMM): Used in GenAI to model complex data
distributions by representing the data as a mixture of several Gaussian
distributions.

8. Bayesian Inference in GenAI

 Bayes' Theorem in Generative Models: Used to update probabilities when new data is
encountered, enabling adaptive generation of content.

This is particularly useful in models that need to adjust their outputs based on incoming user
data or contextual changes.

 Example: Bayesian networks in AI for generating decision trees or handling uncertainty

in predictions.

9. Entropy and Information Theory in GenAI

 Entropy (H): Measures uncertainty in the probability distribution of a random variable.

9
22AIM71 GENERATIVE AI MODULE- 2 NOTES

In GenAI, entropy helps in balancing randomness and coherence in generated outputs.

Lower entropy means more predictable outputs; higher entropy means more diverse and
unexpected outputs.

 KL Divergence: Measures how one probability distribution diverges from a second,

expected distribution. Used to train generative models (e.g., VAEs).

10. Reinforcement Learning and Probability

 Probability in Decision-Making: In reinforcement learning (RL), AI models use

probability to choose actions based on expected rewards.

o GenAI models employing RL (like conversational agents) make decisions under

uncertainty and adapt their strategies based on probabilistic outcomes.

11. Applications of Probability in GenAI

 Natural Language Processing (NLP): Text generation using probabilistic models to

predict word sequences.
 Image Generation: Generating images using VAEs and GANs based on probability
distributions of pixel values.
 Data Augmentation: Probabilistic transformations (e.g., rotation, scaling) of input data
for better generalization in GenAI models.
 Creative Content Creation: Probability helps GenAI to generate diverse, realistic, and
context-aware content in art, music, and writing.

10
22AIM71 GENERATIVE AI MODULE- 2 NOTES

Probabilistic Graphical Models (PGMs) for GenAI

Introduction: Probabilistic Graphical Models (PGMs) combine probability theory and graph
theory to represent complex dependencies among variables. In Generative AI (GenAI), PGMs are
crucial for modeling relationships between observed data and hidden structures, which aids in
generating realistic content, making inferences, and predicting future outcomes.

Basics of PGMs

 Graph Theory: PGMs use graphs to represent random variables (nodes) and their
probabilistic dependencies (edges). The graphs can either be:
o Directed: Where edges indicate a directional dependency (used in Bayesian
Networks).
o Undirected: Where edges indicate mutual dependencies (used in Markov Random
Fields).
 Nodes (Variables): Represent observable data points (e.g., text in a chatbot) or latent
factors (e.g., user preferences).
 Edges (Relationships): Capture probabilistic dependencies between these variables,
helping in decision-making, content generation, and pattern recognition.

Types of Probabilistic Graphical Models

a. Bayesian Networks (BNs)

 Directed Acyclic Graph (DAG): Represents conditional dependencies between variables

using directed edges.
 Conditional Independence: The key feature of BNs, where each node is independent of
its non-descendants given its parents.

Application in GenAI:

o Natural Language Understanding (NLU): BNs can model sentence structures by

defining relationships between different parts of a sentence. For instance, in a

11
22AIM71 GENERATIVE AI MODULE- 2 NOTES

customer support chatbot, a Bayesian network can represent the relationship

between user intent and specific words used in a query.

b. Markov Random Fields (MRFs)

 Undirected Graph: MRFs represent variables using undirected edges, where each
variable depends only on its neighbors.
 Markov Property: Each variable is conditionally independent of all others, given its
neighbors.

Application in GenAI:

o Image Generation: MRFs can help in modeling pixel relationships in image

generation tasks. For example, generating realistic landscape images of Indian
landmarks (e.g., the Himalayas or Kerala backwaters) using dependencies
between nearby pixel colors.

Inference in PGMs

 Inference: The process of calculating the probability distribution of certain variables

given known values of others. For GenAI, this involves predicting unknown variables or
generating new content based on learned patterns.

Exact Inference Methods:

 Variable Elimination: Simplifies complex networks by marginalizing out certain

variables to make probability calculations tractable.
 Belief Propagation: Used to propagate information through the network, useful when
dealing with large datasets in GenAI.

Approximate Inference Methods:

12
22AIM71 GENERATIVE AI MODULE- 2 NOTES

 Monte Carlo Sampling: Involves generating samples from a probability distribution to

approximate inference. In the context of AI, Monte Carlo methods can be used to generate
novel text outputs that maintain coherence with previous inputs.

Example in Indian Context: A news-generating AI might predict the next set of

sentences about a regional event (e.g., the Kumbh Mela) using inference to ensure that all
generated content is logically connected and contextually relevant.

Learning in PGMs

 Learning Parameters: PGMs must learn the conditional probability distributions of the
variables in the network. This can be done through:
o Maximum Likelihood Estimation (MLE): Estimates parameters by maximizing
the likelihood of the observed data.
o Expectation-Maximization (EM): Useful when dealing with incomplete data, a
common scenario in AI training data.

GenAI Application: For example, learning to generate context-aware responses in a

multilingual chatbot servicing various regions of India requires understanding the
likelihood of certain phrases appearing based on the user’s previous questions or
language preferences.

Hidden Markov Models (HMMs) in GenAI

 Definition: HMMs are a specific type of Bayesian Network where the system being
modeled is assumed to follow a Markov process with hidden states.
 Key Components:
o States: Hidden variables that are not directly observable.
o Observations: Observable data related to the hidden states.
o Transition Probabilities: Probabilities of moving between hidden states.

Application in GenAI:

13
22AIM71 GENERATIVE AI MODULE- 2 NOTES

o Speech Recognition: HMMs are often used to model speech patterns. For
example, in voice assistants for regional languages in India (like Hindi, Tamil, or
Kannada), HMMs can model the transition between phonemes to recognize and
respond accurately to spoken queries.

Latent Variable Models

 Latent Variables: Variables that are not directly observed but inferred from observable
data.
 Example in GenAI: Variational Autoencoders (VAEs), used for generating realistic data
(e.g., images, text), often rely on latent variable models to represent abstract features in
the data.

GenAI Application: For generating art pieces inspired by Indian cultural motifs (e.g.,
Rangoli or Mughal patterns), a latent variable model can encode abstract patterns from
the training data and then generate new, original designs.

Applications of PGMs in GenAI

 Natural Language Processing (NLP): Probabilistic models like Bayesian Networks and
HMMs are widely used in NLP tasks such as:
o Text generation: Creating contextually accurate and coherent text based on user
inputs (e.g., AI writers or summarizers for news articles in local Indian dialects).
o Language Translation: Probabilistic models help capture linguistic structures
and relationships between source and target languages.

 Image Generation: PGMs are used in models that generate new images based on existing
ones. For instance, generating AI-based artworks that reflect Indian styles like
Madhubani or Warli paintings.
 Recommendation Systems: PGMs can model user preferences, making them useful in
AI-powered recommendation systems (e.g., recommending Bollywood movies on
streaming platforms).

14
22AIM71 GENERATIVE AI MODULE- 2 NOTES

15
22AIM71 GENERATIVE AI MODULE- 2 NOTES

Challenges and Future Directions

 Scalability: Large PGM models can become computationally expensive, making them
difficult to scale for big datasets, such as national-level datasets in multilingual contexts.
 Handling Uncertainty: While PGMs effectively handle uncertainty, ensuring they
provide accurate results in noisy or incomplete data environments (common in real-
world AI applications) is challenging.

Future Direction: Developing more efficient algorithms for inference and learning in
PGMs will enhance the capabilities of GenAI, especially in handling complex and culturally
diverse data from regions like India.

Probabilistic Graphical Models are a foundational tool in Generative AI, allowing systems to
make informed, probabilistic decisions based on learned patterns. Whether in text generation,
image creation, or speech recognition, PGMs provide the framework for modeling complex
dependencies and generating realistic, context-aware content across a wide range of AI
applications, including those relevant to diverse cultural contexts like India.

Numerical Problems & Illustrations

1. Naive Bayes Classifier in PGM

Problem:
Given a dataset where:

 40% of emails are spam.

 70% of spam emails contain the word "buy."
 10% of non-spam emails contain the word "buy."

Find the probability that an email is spam, given that it contains the word "buy."

Solution:
We use Bayes’ theorem to calculate this:

16
22AIM71 GENERATIVE AI MODULE- 2 NOTES

Let:

 P(S)=0.4 P(S) = 0.4 P(S)=0.4 (prior probability of spam).

 P(¬S)=0.6 P(\¬ S) = 0.6 P(¬S)=0.6 (prior probability of not being spam).
 P(B∣S)=0.7 P(B|S) = 0.7 P(B∣S)=0.7 (probability of "buy" given spam).
 P(B∣¬S)=0.1 P(B|\¬ S) = 0.1 P(B∣¬S)=0.1 (probability of "buy" given not spam).

We need to calculate P(S∣B) P(S|B) P(S∣B), the probability of spam given "buy."

17
22AIM71 GENERATIVE AI MODULE- 2 NOTES

2. Hidden Markov Model (HMM) for Weather Prediction

Problem:

Suppose a simple weather system has two states: Sunny (S) and Rainy (R). A person can observe two
possible activities: Walk (W) or Shop (Sh). The transition and observation probabilities are as
follows:

 Transition probabilities:
o P(S→S) =0.8P(S \to S) = 0.8P(S→S)

=0.8, P(S→R)=0.2P(S \to R)

=0.2 P(S→R)=0.2

o P(R→R)=0.7P(R \to R) = 0.7P(R→R)

=0.7, P(R→S)=0.3P(R \to S)

= 0.3P(R→S)=0.3

18
22AIM71 GENERATIVE AI MODULE- 2 NOTES

 Observation probabilities:
o P(W∣S)=0.6P(W|S) = 0.6P(W∣S)

= 0.6, P(Sh∣S)=0.4P(Sh|S) = 0.4P(Sh∣S)=0.4

o P(W∣R)=0.1P(W|R) = 0.1P(W∣R)

=0.1, P(Sh∣R)=0.9P(Sh|R) = 0.9P(Sh∣R)=0.9

Given that the first day is Sunny, calculate the probability that the sequence of observations Walk,
Shop occurs over two days.

Solution:

To find the probability of the observation sequence Walk (W) followed by Shop (Sh), we need to
compute the following:

3. Markov Random Field (MRF) - Image Denoising

Problem:

19
22AIM71 GENERATIVE AI MODULE- 2 NOTES

Consider a binary image (black and white), represented by nodes in a graph. Each node XiX_iXi can
take values 0 (black) or 1 (white). The image has noise, and the task is to find the most probable
denoised image using MRF. Suppose:

 The prior probability for each pixel Xi is given by

P(Xi=1)=0.7

P(X_i = 1) = 0.7

P(Xi=1)=0.7 and P(Xi=0)=0.3

P(Xi = 0) = 0.3 P(Xi=0)=0.3.

 Neighbors Xi and Xj prefer to have the same value, i.e.,

P(Xi=Xj)=0.9

P(Xi = Xj) = 0.9

P(Xi=Xj)=0.9.

Given noisy observations Yi, how would we compute the most likely denoised image?

20
22AIM71 GENERATIVE AI MODULE- 2 NOTES

Thus, the pixel is more likely to be white in the denoised image.

21
22AIM71 GENERATIVE AI MODULE- 2 NOTES

Gaussian Mixture Model (GMM) - Clustering

Problem:
Consider a dataset with two clusters, where data points are generated by two Gaussian distributions.
The parameters for the Gaussians are:

 Cluster 1: μ1=5\mu_1 = 5μ1=5, σ1=1\sigma_1 = 1σ1=1

 Cluster 2: μ2=10\mu_2 = 10μ2=10, σ2=2\sigma_2 = 2σ2=2

A new data point x=7x = 7x=7 is observed. Find the probability that this point belongs to each cluster.

Solution:

We use the Gaussian probability density function (PDF):

22
22AIM71 GENERATIVE AI MODULE- 2 NOTES

Hidden Markov Models (HMMs)

Hidden Markov Models (HMMs) are statistical models used to represent systems that evolve over
time, where the system's states are not directly observable (hidden), but their effects or observations
are. HMMs consist of two key components: hidden states and observable events. The system
transitions between these hidden states according to certain probabilities, and each state generates
observable events based on emission probabilities.

An HMM is characterized by:

1. States: A finite set of hidden states the system can be in (e.g., weather conditions like Sunny or Rainy).
2. Observations: A set of possible observations related to the states (e.g., observed activities like walking
or shopping).
3. Transition probabilities: The probabilities of transitioning from one state to another (e.g., the
likelihood of going from Sunny to Rainy).
4. Emission probabilities: The probabilities of observing a certain event given a particular state (e.g.,
walking when it's sunny).
5. Initial state distribution: The probability distribution for the system's initial state.

HMMs are widely used for tasks like speech recognition, natural language processing,
bioinformatics, and time series prediction. Algorithms like the Forward-Backward Algorithm (for
computing probabilities), the Viterbi Algorithm (for decoding the most likely sequence of hidden
states), and the Baum-Welch Algorithm (for training HMMs) are essential tools for working with
these models. HMMs provide a powerful framework for modeling sequential data where the
underlying process is not directly observable.

Hidden Markov Models (HMM) in Speech Recognition: Example Use Case

In speech recognition, the goal is to convert spoken language into text. Human speech is a sequential
process where sounds (phonemes) are produced over time. However, the exact sequence of phonemes
(hidden states) is not directly observable in a given speech signal (observations), making HMM an
ideal model.

23
22AIM71 GENERATIVE AI MODULE- 2 NOTES

Example: Recognizing the word "cat"

Components of HMM for Speech Recognition:

1. States (Hidden States):

o In speech recognition, the hidden states represent phonemes (distinct units of sound in
speech). For the word "cat," the phonemes might be represented as:
 /k/
 /æ/
 /t/

2. Observations (Emission Probabilities):

o The observations are features extracted from the speech signal, such as MFCCs (Mel-
frequency cepstral coefficients), which are numerical representations of short segments of
speech.
o Each hidden state (phoneme) emits a sequence of observations based on certain probabilities.
For instance, the phoneme /k/ might correspond to a certain pattern in the speech signal.

3. Transition Probabilities:
o These probabilities represent the likelihood of moving from one phoneme to the next. For
example, the probability of transitioning from /k/ to /æ/ and from /æ/ to /t/ will be modeled.

4. Initial State Probabilities:

o The initial state probability represents the likelihood that the first phoneme is /k/ (for the
word "cat").

Process:

 Step 1: Observation of Speech Signal

The system records a speech signal, which is then processed to extract features (e.g., MFCCs).
This forms the observable data.
 Step 2: Modeling with HMM
The HMM consists of states representing the phonemes /k/, /æ/, and /t/. Each state has

24
22AIM71 GENERATIVE AI MODULE- 2 NOTES

associated emission probabilities that describe how likely each sound feature (observation) is,
given the current state (phoneme).
 Step 3: Transition Between States
As the speaker transitions from one phoneme to another (e.g., from /k/ to /æ/), the HMM
transitions between states based on transition probabilities.
 Step 4: Decoding the Sequence
Using the Viterbi algorithm, the HMM determines the most probable sequence of phonemes
that generated the observed speech signal. The output sequence could be /k/ → /æ/ → /t/,
representing the word "cat."
 Step 5: Converting Phonemes to Text
Once the most likely sequence of phonemes is decoded, it can be mapped to the corresponding
word in the dictionary, which in this case is "cat."

Example Breakdown:

1. Hidden States: /k/, /æ/, /t/

2. Observations: Extracted speech features (MFCCs for different sounds).
3. Transition Probabilities: Likelihood of moving from /k/ to /æ/, and from /æ/ to /t/.
4. Emission Probabilities: Likelihood of observing a particular sound feature for each phoneme.

Summary:

In this use case, the HMM helps in recognizing the word "cat" by modeling the sequence of hidden
phonemes that generate the observable speech signal. Speech recognition systems use large-scale
HMMs, typically trained on vast datasets, to handle a wide vocabulary of words and map spoken
language into accurate text representations.

Variational Interference – Introduction

Variational Inference is a technique used in machine learning and statistics to approximate complex
probability distributions, often for Bayesian models. When working with probabilistic models,

25
22AIM71 GENERATIVE AI MODULE- 2 NOTES

calculating the exact posterior distribution can be computationally intractable due to high-dimensional
integrals. Variational Inference offers an alternative by turning the problem of computing these
distributions into an optimization task, making it faster and more scalable for real-world applications,
including in deep learning and generative models.

Key Concepts in Variational Inference

1. Posterior Distribution:
In Bayesian inference, the goal is to compute the posterior distribution p(θ∣x)p(\theta | x)p(θ∣x),
where:
o θ represents the model parameters (the hidden variables).
o x represents the observed data.

2. Challenge in Computing the Posterior:

Exact inference involves computing this marginal likelihood, which usually requires solving
high-dimensional integrals. This is especially hard in large models or when the data has many
dependencies.
3. Variational Approximation:
Instead of solving for the exact posterior, variational inference approximates the true
posterior p(θ∣x) by introducing a simpler, parameterized distribution q(θ) which is easier to
work with. The goal is to make q(θ) as close as possible to p(θ∣x) .

26
22AIM71 GENERATIVE AI MODULE- 2 NOTES

4. KL Divergence:
To measure the difference between the approximated distribution q(θ) and the true posterior
p(θ∣x), we use Kullback-Leibler (KL) divergence:

5. Evidence Lower Bound (ELBO):

Directly minimizing the KL divergence is hard because p(x) (the evidence) is often intractable.
Instead, we optimize an alternative objective called the Evidence Lower Bound (ELBO),
which is equivalent to maximizing the log evidence log p(x) minus the KL divergence:

6. Optimization Process:
Variational inference turns the problem of inference into an optimization problem. The idea is
to find the parameters of q(θ) that maximize the ELBO. The optimization is typically done
using gradient-based methods, and the complexity of q(θ) can vary. It may take the form of a
simple Gaussian or more complex distributions depending on the model and data.

Detailed Steps in Variational Inference:

27
22AIM71 GENERATIVE AI MODULE- 2 NOTES

1. Choose a Family of Distributions for q(θ)q(\theta)q(θ):

A key step is choosing a family of distributions q(θ) (e.g., Gaussian distributions with certain
mean and variance). The closer this family can approximate the true posterior, the better the
inference will be.
2. Define the ELBO:
The ELBO can be written as:

The first term represents the expected log joint likelihood under q(θ), and the second term
represents the entropy of q(θ).

3. Optimize the ELBO:

The parameters of q(θ) are adjusted to maximize the ELBO, which is done through gradient-
based optimization techniques such as stochastic gradient descent.
4. Approximation of Posterior:
Once the ELBO is maximized, the variational approximation q(θ) is taken as the
approximation to the true posterior distribution p(θ∣x)

Advantages of Variational Inference:

 Scalability: Unlike traditional methods like Markov Chain Monte Carlo (MCMC), variational
inference is computationally efficient and scales to large datasets and models.
 Deterministic: Variational inference provides a deterministic approximation of the posterior,
whereas MCMC provides stochastic estimates.

Example of Variational Inference:

Let’s take an example where you’re trying to infer parameters of a Gaussian mixture model (GMM),
a common clustering model.

28
22AIM71 GENERATIVE AI MODULE- 2 NOTES

 True Posterior: The posterior distribution of the GMM parameters given the data is intractable
due to complex integrals.
 Variational Approximation: You approximate this posterior with a simpler distribution, say a
Gaussian distribution. By maximizing the ELBO, you tune this simpler distribution to be as
close as possible to the true posterior, allowing you to make inferences about cluster
assignments and parameters efficiently.

Variational Autoencoders (VAE): A Practical Example

One popular application of variational inference is in Variational Autoencoders (VAEs), used for
generating images, data compression, and other tasks. VAEs use variational inference to learn a latent
(hidden) representation of the data, enabling the generation of new data points similar to the original
data. Here, the ELBO is maximized to ensure that the latent variables capture the true underlying
structure of the data.

Variational Inference is a powerful and flexible approach to approximate posterior distributions in

Bayesian models. By transforming complex inference problems into optimization tasks, it offers a
scalable and efficient way to handle large, real-world datasets.

29
22AIM71 GENERATIVE AI MODULE- 2 NOTES

Variational Autoencoders (VAEs)

Variational Autoencoders (VAEs) are a type of generative model used in machine learning,
particularly for tasks like data generation, dimensionality reduction, and unsupervised learning. VAEs
are a specific form of autoencoder that leverage probabilistic reasoning to model complex
distributions and generate new data points similar to the original data. Unlike traditional autoencoders,
which learn a deterministic mapping from inputs to latent representations, VAEs learn a probabilistic
distribution in the latent space, which allows them to generate diverse and smooth outputs.

Core Concepts of Variational Autoencoders:

1. Autoencoder Architecture:
o Encoder: This is a neural network that takes input data (e.g., images, text) and encodes
it into a lower-dimensional latent space (or latent variables). The encoder maps the
input x to a probability distribution over the latent variables z, usually a Gaussian
distribution.
o Decoder: The decoder is another neural network that takes samples from this latent
space and reconstructs the original data. The goal is to ensure the decoded output
closely resembles the input.
2. Latent Variable Modeling:
o In a traditional autoencoder, the encoder maps an input to a single point in the latent
space. In a VAE, however, the encoder maps the input to a distribution in the latent
space, often modeled as a Gaussian distribution parameterized by a mean and
variance. This allows the model to introduce variability and generate new data.
o Instead of directly encoding the input to a fixed latent vector, the VAE encodes the
input as the parameters of a probability distribution, typically a multivariate Gaussian
distribution.
3. Reparameterization Trick:
o To allow backpropagation through the network while sampling from a distribution,
VAEs use the reparameterization trick. Rather than sampling directly from the
learned distribution, the VAE expresses the latent variables as:

30
22AIM71 GENERATIVE AI MODULE- 2 NOTES

4. Objective Function: Maximizing the Evidence Lower Bound (ELBO): VAEs are trained by
optimizing the Evidence Lower Bound (ELBO), which has two main components:
o Reconstruction Loss: This ensures that the decoder is able to reconstruct the original
input data from the latent variables. Typically, this is done using a loss function such as
mean squared error or binary cross-entropy between the input data and the
reconstructed data.
o KL Divergence: The second component minimizes the difference between the learned
latent distribution q(z∣x) and a prior distribution p(z), which is often a standard
Gaussian N(0,1). This term is the Kullback-Leibler (KL) divergence, and it
regularizes the latent space by ensuring that the latent variables follow a normal
distribution.

The ELBO can be written as:

o The first term is the expected reconstruction error, and the second term is the KL
divergence regularization.
5. Generative Process: After training, a VAE can generate new data points by sampling from the
latent space, i.e., from the prior distribution p(z), and passing those samples through the
decoder to produce new, similar-looking data points.

31
22AIM71 GENERATIVE AI MODULE- 2 NOTES

Why Use Variational Autoencoders?

1. Generative Capabilities:
VAEs can generate new, unseen examples that resemble the training data. For instance, after
training on a dataset of images, the VAE can generate new images that are not in the training
set but have similar characteristics.
2. Smooth Latent Space:
The latent space in VAEs is structured in such a way that similar inputs are mapped to nearby
points in the latent space. This smoothness allows for interpolation between different data
points, which is useful in tasks like morphing between images.
3. Regularized Latent Space:
The KL divergence term ensures that the latent variables are regularized and follow a known
distribution (usually Gaussian). This regularization prevents overfitting and encourages the
model to use the latent variables efficiently.
4. Probabilistic Interpretation:
VAEs are based on probabilistic models and therefore allow for uncertainty estimation in the
generated data. Unlike deterministic autoencoders, which map an input to a single latent
vector, VAEs map inputs to a distribution, giving a sense of how confident the model is about
the generated data.

32
22AIM71 GENERATIVE AI MODULE- 2 NOTES

Steps in VAE Workflow:

1. Input Data:
Start with input data, such as an image or text.
2. Encoding:
The encoder network takes the input data and produces parameters for the latent space (mean
and variance). These parameters define the Gaussian distribution from which latent variables
will be sampled.
3. Reparameterization and Sampling:
Instead of sampling directly from the latent distribution, the reparameterization trick is used to
allow for gradient-based optimization during training. This generates the latent variable zzz.
4. Decoding:
The latent variable zzz is passed through the decoder, which tries to reconstruct the original
data.
5. Loss Calculation:
The loss function is computed as the sum of the reconstruction loss and the KL divergence.
The model is trained by minimizing this loss.
6. Generating New Data:
Once trained, the VAE can generate new data by sampling from the latent space and decoding
these samples back into the data space.

Applications of VAEs:

1. Image Generation:
VAEs are widely used to generate new images similar to the ones in the training dataset. For
example, after training on a dataset of handwritten digits (like MNIST), a VAE can generate
new digits that resemble real handwriting.
2. Anomaly Detection:
Since VAEs learn the underlying distribution of the data, they can be used for anomaly
detection. If a new data point does not fit well in the learned latent space, it is flagged as an
anomaly.

33
22AIM71 GENERATIVE AI MODULE- 2 NOTES

3. Data Imputation:
VAEs can be used to fill in missing data by sampling from the latent space to reconstruct the
missing portions.
4. Dimensionality Reduction:
Similar to other autoencoders, VAEs can reduce high-dimensional data into lower-dimensional
latent representations, which can be useful for tasks like clustering or visualization.
5. Semi-supervised Learning:
VAEs can be used in semi-supervised learning, where labeled data is scarce. The generative
nature of VAEs can help the model learn better representations even with a limited amount of
labeled data.

Example of VAE in Image Generation:

 MNIST Dataset:
Suppose you train a VAE on the MNIST dataset, which contains images of handwritten digits.
After training, you can sample from the latent space and generate new images of digits that
look like they were part of the dataset but are actually novel and unique.

Conclusion:

Variational Autoencoders (VAEs) are a powerful tool for learning latent representations of data and
generating new data points. They extend the basic autoencoder framework by introducing a
probabilistic approach, which enables better generalization, smooth latent spaces, and the ability to
generate new data that follows the learned distribution. VAEs have found wide application in image
generation, anomaly detection, and other fields requiring generative models.

34
22AIM71 GENERATIVE AI MODULE- 2 NOTES

Module-2 Questions for Revision:

Fundamentals of Probabilistic Models for Gen-AI

1. Probability Distributions:

 Define a probability distribution and give an example of a discrete and a continuous probability
distribution.
 Explain the probability density function (PDF) and cumulative distribution function (CDF) for a
continuous random variable.
 Numerical: A coin is flipped three times. What is the probability of getting exactly 2 heads?
 Numerical: For a Gaussian distribution with mean 0 and variance 1, calculate the probability that the
random variable takes a value between -1 and 1.

2. Bayes Theorem:

 State Bayes' Theorem and explain its significance in machine learning and artificial intelligence.
 Numerical: If 1% of a population has a certain disease and a test for the disease has a false positive
rate of 5% and a false negative rate of 1%, calculate the probability that a person who tests positive
actually has the disease using Bayes' theorem.
 Provide an example of how Bayes' theorem is used in natural language processing (NLP) models.

3. Generative Models - PGMs (Probabilistic Graphical Models):

 Explain the difference between Bayesian networks and Markov networks.

 Describe the key components of a Probabilistic Graphical Model (PGM) and their applications in AI.
 Numerical: Given a small Bayesian network with three variables A, B, and C, where P(A)=0. 2,
P(B∣A)=0.7, and P(C∣B)=0.6, calculate P(C∩A).

4. Generative Models - GMMs (Gaussian Mixture Models):

 What is a Gaussian Mixture Model (GMM), and how is it different from a standard Gaussian
distribution?
 Explain how the Expectation-Maximization (EM) algorithm is used to fit a GMM.

35
22AIM71 GENERATIVE AI MODULE- 2 NOTES

 Numerical: Consider a dataset with two components in a GMM. One has a mean of 3 and variance of
1, and the other has a mean of 7 and variance of 1. If the mixing proportion of the first component is
0.6 and the second is 0.4, calculate the likelihood that a new point with value 5 belongs to each
component.

5. Generative Models - HMMs (Hidden Markov Models):

 Define the key elements of a Hidden Markov Model (HMM).

 Describe the Viterbi algorithm and how it is used in HMMs for sequence prediction.
 Numerical: Given an HMM with 3 hidden states and 3 possible observations, calculate the most likely
hidden state sequence given an observation sequence using the Viterbi algorithm (use simple
transition and emission probabilities).

Variational Inference

6. Introduction to Variational Inference (VI):

 What is the goal of Variational Inference (VI) in machine learning?

 Describe the difference between Variational Inference and Monte Carlo methods.
 Explain the Evidence Lower Bound (ELBO) and how it is optimized in VI.

7. Variational Autoencoders (VAEs):

 Explain the architecture of a Variational Autoencoder (VAE) and its significance in generative
modeling.
 What is the role of reparameterization trick in VAEs?
 Numerical: Given a VAE with a latent space of 2 dimensions and a Gaussian prior, calculate the KL
divergence between two multivariate Gaussian distributions with given means and variances.
 Describe how reconstruction loss and KL divergence are combined to train a VAE.

Miscellaneous (Numerical Problems)

36
22AIM71 GENERATIVE AI MODULE- 2 NOTES

8. Gaussian Distribution:

 Numerical: Given a univariate Gaussian distribution N(5,42), calculate the probability that the
random variable takes a value greater than 6.
 Numerical: A dataset is known to be generated from a mixture of two Gaussian distributions with
means 0 and 5 and variances 1 and 2, respectively. The mixing proportions are 0.4 and 0.6. Calculate
the overall mean and variance of the mixture.

Conceptual

11. Compare and contrast the use of GMM and HMM for time-series modeling.
When would you prefer one over the other?

12. In PGMs, explain the role of conditional independence in reducing the complexity
of joint distributions.

13. Discuss how Bayesian Networks can be used for causal reasoning and decision-
making in AI systems.

14. Provide a high-level explanation of inference in PGMs and the challenges

associated with exact inference.

37
22AIM71 GENERATIVE AI MODULE- 2 NOTES

15. What are the advantages of using Variational Inference over traditional Markov Chain Monte
Carlo (MCMC) methods in generative models?

16. Explain the concept of variational posterior distribution in the context of VAEs.

17. Describe how latent variables in a VAE represent the structure of input data in generative
models.

18. In the context of generative AI, what are some common applications of PGMs and VAEs?

19. Compare and contrast the uses of GMM versus VAE for modeling high-dimensional data.

20. Explain how hidden states in an HMM can capture dependencies in sequential data such as text
or time series.

38