Generative AI Notes

Generative AI Notes
Generative AI
Generative AI refers to a class of machine learning models designed to generate new data
samples that resemble a given training dataset. Mathematically, generative models aim to
learn the joint probability distribution P (x, z), where x represents the observed data (e.g.,
images, text, etc.) and z represents the latent variables or features. Once trained, the model
can generate new samples x ^ by sampling from the distribution P (x|z) or approximating it.

Generative AI refers to a class of artificial intelligence models designed to create new

content, ranging from text, images, and audio to video, based on patterns learned from
existing data. Unlike traditional AI systems that classify or predict based on input data,
generative AI creates something entirely new, mimicking the style or structure of the data it
has been trained on.v

Key Technologies:

1. Deep Learning Models: At the core of generative AI are deep learning techniques,
especially neural networks. Key architectures include:
Generative Adversarial Networks (GANs): GANs consist of two competing
networks, a generator and a discriminator. The generator creates content, and the
discriminator evaluates how realistic it is. This adversarial process helps the
generator improve over time.
Variational Autoencoders (VAEs): VAEs encode data into a latent space and
then decode it back, allowing for the creation of new, similar content.
Transformers: Models like GPT (Generative Pretrained Transformer) rely on
transformers to generate coherent and contextually relevant text. They are
capable of handling long-range dependencies in sequences, making them
particularly useful for natural language generation.

Applications:

1. Text Generation: Generative AI models like GPT-4 can generate human-like text,
which is useful for content creation, writing assistance, chatbots, and code generation.
2. Image Generation: Tools like DALL·E or GANs can create realistic or artistic images
based on text prompts or learned styles.
3. Audio and Music: AI models are also capable of composing music or generating voice,
making them useful in the entertainment and media industries.

1 / 11
 Generative AI Notes

4. Video and Animation: Generative AI can synthesize videos, create animations, or

even generate deepfakes, pushing the boundaries of digital media creation.
5. Drug Discovery and Science: In scientific research, generative AI is being used to
propose new molecular structures for drug discovery and optimize experimental
designs.

Advantages:

Automation and Efficiency: Generative AI can automate content creation, saving time
and resources in industries like marketing, design, and writing.
Creativity Augmentation: These models can serve as creative tools, offering unique
ideas or expanding the possibilities for artists, writers, and designers.
Scalability: Once trained, generative models can quickly produce vast amounts of
content, making them scalable solutions for many applications.

Limitations of generative AI:

1. Data Dependency:
Quality and Bias: Generative AI models are trained on larg datasets. If the training
data is biased or flawed, the generated output will reflect those biases, leading to
inaccurate or unfair results.
Data Privacy: Large datasets used for training often raise concerns about privacy,
especially if they contain sensitive personal information.

2. Lack of Understanding:
No True Understanding: Generative AI models do not understand the context or
meaning behind the data they generate. They rely on patterns in the training data rather
than comprehension, which can result in nonsensical or inappropriate content.
Inability to Reason: These models cannot perform logical reasoning or infer new
knowledge beyond their training data.

3. Hallucinations:
Inaccurate Information: Generative AI may "hallucinate" information, providing content
that sounds plausible but is factually incorrect or fabricated. This can be problematic in
domains where accuracy is crucial, like medicine or law.

2 / 11
 Generative AI Notes

4. Resource Intensive:
Computational Costs: Training large-scale generative models requires significant
computational resources, time, and energy, leading to high costs and environmental
concerns.
Inference Costs: Even after training, running these models, especially in real-time
applications, can be resource-intensive.

5. Ethical Concerns:
Misuse: Generative AI can be misused to create deepfakes, fake news, or
misinformation, leading to ethical dilemmas and societal issues.
Intellectual Property: Models trained on copyrighted materials can inadvertently
generate content that violates intellectual property rights.

6. Limited Creativity:
Repetitive Patterns: Although generative AI can produce creative outputs, it often
relies on existing patterns and may struggle to create truly novel or groundbreaking
ideas.
Overfitting to Training Data: In some cases, the model may generate outputs that are
too similar to the training data, limiting the diversity of generated content.

7. Domain-Specificity:
Limited Generalization: Generative AI models trained on specific domains may not
generalize well to other domains. For example, a model trained on legal text may not
perform well when generating creative writing or scientific papers.

8. Lack of Real-time Feedback:

No Continuous Learning: Most generative AI models are static once trained. They do
not learn from new data in real time, meaning they can become outdated if not regularly
retrained.

9. Ethical Alignment and Safety:

Unpredictable Behavior: In certain scenarios, generative models may produce
harmful or offensive content, despite attempts to align them with human values.

3 / 11
 Generative AI Notes

Control: Ensuring that AI-generated content adheres to ethical guidelines is a

challenge, especially as models become more autonomous.

Generative Adversarial Networks (GANs)

Generative Adversarial Networks (GANs) are a class of machine learning models
introduced by Ian Goodfellow in 2014. GANs are designed to generate new data samples
that resemble a given dataset. They consist of two competing neural networks: a generator
and a discriminator, which are trained simultaneously in a process often described as a
"game."

Architecture:
A GAN consists of two core components:

1. Generator (G):
The generator network takes in random noise (usually a vector sampled from a
normal or uniform distribution) and transforms it into a data sample that mimics the
distribution of the real data.
The goal of the generator is to create data that looks realistic enough to fool the
discriminator into believing that the generated data is from the real data
distribution.
2. Discriminator (D):
The discriminator is a binary classifier that takes input data (either from the real
dataset or generated by the generator) and outputs the probability that the input is
real (i.e., from the actual dataset) or fake (i.e., generated).
The discriminator’s objective is to correctly distinguish between real and
generated data.

Objective Function:
GANs are formulated as a min-max game between the generator and discriminator. The
generator tries to "fool" the discriminator by generating realistic data, while the discriminator
tries to accurately identify whether a given input is real or fake.

The objective function can be written as:

min max V (D, G) = E x∼P (x)

[log D(x)] + E z∼P (z)
[log(1 − D(G(z)))]
data z
G D

Where:

x ∼ P data (x) represents samples from the real data distribution.

4 / 11
 Generative AI Notes

z ∼ P z (z) represents the noise input to the generator (drawn from a simple distribution
like a Gaussian).
G(z) is the output of the generator, which is meant to resemble real data.
D(x) is the output of the discriminator, which returns the probability that x is real.

Steps:

1. Discriminator training: The discriminator is trained to maximize its ability to

distinguish between real data and fake data produced by the generator.
2. Generator training: The generator is trained to minimize the discriminator's ability to
distinguish its generated data from real data, i.e., to "fool" the discriminator.

The process continues iteratively, with both networks learning and improving over time.

Training Process:
Training GANs is a competitive process:

Discriminator Training: During each iteration, the discriminator receives both real
samples and fake samples generated by the generator. It learns to differentiate
between the two by minimizing a loss function, typically the cross-entropy loss.
Generator Training: The generator's goal is to fool the discriminator. It is updated by
minimizing the log-probability that the discriminator correctly classifies the generated
samples as fake.

Challenges in Training GANs:

1. Instability: GAN training can be unstable due to the adversarial nature of the process.
If one network becomes too strong (e.g., the discriminator becomes too good at
identifying fakes), the training can collapse.
2. Mode Collapse: Sometimes, the generator learns to produce only a few types of
outputs, ignoring diversity in the data. This phenomenon is called mode collapse.
3. Non-convergence: The min-max game may not always converge, leading to
oscillatory behavior where the generator and discriminator do not improve consistently.

Solutions to Challenges:
Wasserstein GAN (WGAN): Introduced a new loss function based on the Earth
Mover’s (Wasserstein) distance, providing more stable training by ensuring that the
discriminator provides useful gradients even when it becomes too good.

5 / 11
 Generative AI Notes

Progressive Growing: In applications like image generation, starting with small

images and progressively increasing the resolution can help with stability and
convergence.

Applications of GANs:
1. Image Generation: GANs are widely used in generating realistic images, such as in
Deepfake technology or tools like StyleGAN.
2. Image-to-Image Translation: GANs can convert one type of image into another, such
as turning sketches into realistic photos (e.g., Pix2Pix), or even changing weather
conditions in images.
3. Super-Resolution: GANs are used to upscale images, enhancing low-resolution
images to high-resolution ones.
4. Data Augmentation: GANs can generate synthetic training data, particularly useful in
domains where labeled data is scarce.
5. Text-to-Image: Models like DALL·E use GAN-based techniques to generate images
based on textual descriptions.

Autoencoders and Latent Representation

Autoencoders are a type of neural network used for unsupervised learning that aim to
compress and reconstruct data. The key idea behind an autoencoder is to map input data
into a compressed, low-dimensional representation (called the latent representation) and
then decode it back to reconstruct the original data. Autoencoders are widely used for
dimensionality reduction, feature learning, and data compression.

Architecture of Autoencoders:
An autoencoder consists of four main parts:

1. Encoder:
The encoder maps the input data x to a lower-dimensional latent space z. The
encoder network compresses the input data to capture its most important features.
Mathematically, this process can be represented as:

z = f θ (x)

where f is a function (usually a neural network) parameterized by θ, and z is the

θ

latent representation or code.

2. Decoder:
The decoder takes the compressed latent vector z and attempts to reconstruct the
original input data x
^. The decoder learns how to expand or decode the latent

6 / 11
 Generative AI Notes

representation back into the input data.

This is written as:

x
^ = g ϕ (z)

where g is a neural network that decodes the latent vector, and x

ϕ ^ is the

reconstructed version of x.
3. Bottleneck:
This layer consists of compressed representation of input data which is also
known as latent representation.
4. Reconstruction loss:
This is the criterion to measure how well the network is performing.

The goal is to minimize the difference between the original input x and the reconstructed
output x
^. This is done by minimizing a loss function, typically the mean squared error

(MSE):
2
L(x, x)
^ = ∥x − x∥
^

Latent Representation:
The latent representation z is the key part of an autoencoder. It is a compressed, lower-
dimensional representation of the input data that contains the essential information needed
to reconstruct the input. In other words, the latent representation captures the underlying
features and patterns of the input data in a more compact form.

Key properties of the latent space:

1. Dimensionality Reduction: The latent space is often much smaller than the input
space, making the autoencoder useful for reducing the number of dimensions
(features) in the data. This is similar to Principal Component Analysis (PCA), but
autoencoders can learn non-linear relationships, making them more powerful for
complex data.
2. Feature Learning: The latent representation z acts as a set of learned features that
summarize the key characteristics of the data. These features can be used for
downstream tasks such as classification, clustering, or visualization.
3. Continuous and Differentiable: In contrast to discrete representations, the latent
space of an autoencoder is usually continuous and differentiable, which allows the
model to learn smooth transformations between different regions in the data space.

Loss Function:
The loss function in an autoencoder typically consists of a reconstruction loss, which
ensures that the output x^ is similar to the input x. The most commonly used loss functions

7 / 11
 Generative AI Notes

are:

Mean Squared Error (MSE): Measures the average squared difference between the
input and the reconstruction.
n
1
2
L(x, x)
^ = ∑(x i − x
^i)
n
i=1

Cross-Entropy Loss: Often used for binary data or classification tasks. It measures
the difference between the true binary labels and the predicted probabilities.

Variational Autoencoder (VAE)

The Variational Autoencoder (VAE) is a generative model that extends the traditional
autoencoder by introducing a probabilistic approach to the latent space. It models the latent
variables as distributions rather than fixed vectors, allowing for the generation of new data
by sampling from these distributions. VAEs are commonly used in tasks like image
generation and data synthesis.

VAE Architecture:

1. Encoder:
Maps the input x to the parameters of a probability distribution over the latent
space z.
Instead of outputting a single latent vector, the encoder produces two vectors:
mean μ and log-variance log σ , which define a Gaussian distribution N (μ, σ ).
2 2

The encoder’s output represents the probability distribution of the latent

variable:
2
z ∼ N (μ(x), σ (x))

2. Latent Space:
A random latent vector z is sampled from the Gaussian distribution using the
reparameterization trick:

z = μ(x) + σ(x) ⋅ ϵ

where ϵ is a random variable sampled from a standard normal distribution N (0, I ),

allowing the gradient to propagate through the stochastic process.
3. Decoder:
The decoder takes the sampled latent vector z and generates the output x
^, which

is the reconstruction of the input data.

The decoder outputs a probability distribution over the input data space, allowing
the generation of new data by sampling from this distribution.

8 / 11
 Generative AI Notes

Loss Function:
The VAE loss consists of two terms:

1. Reconstruction Loss: Measures how accurately the VAE reconstructs the input. This
is often the mean squared error (MSE) or binary cross-entropy:
2
Reconstruction Loss = ∥x − x∥
^

2. KL Divergence Loss: Ensures that the learned latent distribution q(z|x) is close to a
prior distribution, typically a standard normal distribution p(z) = N (0, I ). This
regularizes the latent space:

KL Divergence Loss = D KL (q(z|x)∥p(z))

The total VAE loss is a combination of both:

L = Reconstruction Loss + KL Divergence Loss

Illustration of VAE Architecture:

How VAE Differs from Traditional Autoencoder:
1. Probabilistic Latent Space:
In a traditional autoencoder, the latent representation zzz is deterministic. The
encoder maps the input directly to a fixed latent vector, and the decoder
reconstructs the input from this vector.
In a VAE, the latent space is probabilistic. The encoder maps the input to a
distribution (with a mean and variance), and the latent vector is sampled from this
distribution, making it a stochastic process.
2. Reparameterization Trick:
Traditional autoencoders do not require sampling, as they output a fixed latent
vector.
VAEs use the reparameterization trick to allow gradient-based optimization
through stochastic sampling by expressing the latent variable zzz as a
deterministic function of the mean, variance, and a random noise term.
3. KL Divergence:
A traditional autoencoder minimizes only the reconstruction error (e.g., mean
squared error).
A VAE adds an additional term (KL divergence) to regularize the latent space,
ensuring that the learned latent distribution approximates a prior distribution,
typically a standard Gaussian.
4. Generative Capability:

9 / 11
 Generative AI Notes

A traditional autoencoder is designed mainly for dimensionality reduction and

reconstruction, and it is not inherently a generative model.
A VAE is a generative model that can sample new data points by sampling latent
vectors from the learned distribution and decoding them. This makes VAEs
suitable for generating new, unseen data that resembles the original training data.

Pre-training:
Pre-training is the initial stage where you train a large language model on a massive
dataset without any specific task or target variable in mind.
The goal of pre-training is to allow the model to learn general language representations
and patterns that can be useful for various NLP tasks.

Key characteristics of pre-training:

1. Large dataset: Pre-training requires a massive dataset, often containing billions of

tokens (words, phrases, or sentences).
2. No specific task: Unlike fine-tuning, pre-training doesn't focus on any particular NLP
task (e.g., sentiment analysis, question answering, or text
classification).
3. Language modeling objective: The primary goal is to train a model that can predict
the next word in a sequence, given the context of the previous words.
4. Self-supervised learning: Pre-training relies on self-supervised learning, where the
model learns to represent the input data without any external labels or
guidance.

Fine-tuning:
Fine-tuning is the second stage where you adapt the pre-trained model for a specific
NLP task by using a smaller dataset and adjusting the objective function.
The goal of fine-tuning is to optimize the model's performance for the target task while
leveraging the knowledge gained during pre-training.

Key characteristics of fine-tuning:

1. Smaller dataset: Fine-tuning typically requires a much smaller dataset compared to

pre-training, often containing only thousands or tens of thousands of
samples.
2. Specific task: Fine-tuning focuses on a specific NLP task (e.g., sentiment analysis,
question answering, or text classification).
3. Task-specific objective: The goal is to optimize the model's performance for the target
task using a task-specific loss function or evaluation metric.

10 / 11
 Generative AI Notes

4. Transfer learning: Fine-tuning exploits the knowledge and representations learned

during pre-training, allowing the model to adapt to the new task more
efficiently.

Key differences between pre-training and fine-tuning:

1. Dataset size: Pre-training requires a massive dataset, while fine-tuning uses a much
smaller one.
2. Objective function: Pre-training focuses on language modeling objectives, whereas
fine-tuning targets specific NLP tasks.
3. Transfer learning: Fine-tuning relies heavily on the knowledge gained during pre-
training, whereas pre-training is more about learning general language
representations.

When to use each:

1. Pre-training: Use pre-training when you want to learn a robust language

representation that can be useful for various NLP tasks.
2. Fine-tuning: Use fine-tuning when you have a specific NLP task in mind and want to
adapt the pre-trained model to that particular task.

11 / 11