Module 1:
Introduction to
Generative AI
Overview of Artificial Intelligence, Machine
Learning, and Deep Learning,Generative AI
1 2 3
What Is Artificial Intelligence What Is Machine Learning What Is Deep Learning (DL)?
(AI)?
(ML)? Deep learning plays an essential role as a
AI is broadly defined as the ability of machines separate branch within the Artificial Intelligence
The term <MLî focuses on machines learning (AI) field due to its unique capabilities and
to mimic human behavior. It encompasses a
broad range of techniques and approaches from data without the need for explicit advancements. Deep learning is defined as a
aimed at enabling machines to perceive, programming. Machine Learning algorithms machine learning technique that teaches the
reason, learn, and make decisions. AI can be leverage statistical techniques to automatically computer to learn from the data that is inspired
rule-based, statistical, or involve machine detect patterns and make predictions or by [Link] utilizes deep neural networks
learning algorithms. Machine learning, Deep decisions based on historical data that they are with multiple layers to learn hierarchical
Learning, and Generative AI were born out of trained on. While ML is a subset of AI, the term representations of data. It automatically
Artificial Intelligence. was coined to emphasize the importance of extracts relevant features and eliminates
data-driven learning and the ability of manual feature engineering
machines to improve their performance
through exposure to relevant data.
What Is Generative AI (GenAI)?
Generative AI, a branch of artificial intelligence and a subset of Deep Learning, focuses on creating models capable of
generating new content that resemble existing data. These models aim to generate content that is indistinguishable from what
might be created by humans. Generative Adversarial Networks (GANs) are popular examples of generative AI models that use
deep neural networks to generate realistic content such as images, text, or even music.
Introduction to Generative AI and its significance
Generative artificial intelligence (generative AI) is a type of AI that can create new content and ideas, including conversations,
stories, images, videos, and music. It can learn human language, programming languages, art, chemistry, biology, or any complex
subject matter. It reuses what it knows to solve new problems. For example, it can learn English vocabulary and create a poem from
the words it processes. Your organization can use generative AI for various purposes, like chatbots, media creation, product
development, and design.
Accelerates Optimizes business
research processes
Enhances customer Boosts employee
experience productivity
GenAI landscape: discriminative vs. generative
modeling
What is a discriminative model?
A discriminative model is an AI model that can determine how to classify objects. Using supervised learning, an AI model learns
that items within a labeled data set have similar characteristics and underlying patterns. The model uses those patterns to predict
whether new inputs belong to one labeled class or another. For example, you could use an AI model to identify species of plants by
uploading an image of the specimen you¾re looking at. The AI model would consider what underlying characteristics define the
image and compare that to other images it learned about in training to predict which class it belongs to.
What are discriminative models used for?
Discriminative models are helpful tools for classification and regression tasks. They are flexible, and you can apply them to various
data sets. Three applications of discriminative models include risk assessment, fraud detection, and customer segmentation.
1 2 3
Fraud detection: AI models can analyze the Customer segmentation: AI can identify Risk assessment: You can use AI to create a risk
patterns of transactions, such as credit card customer behavior patterns and predict their next assessment model to help you make informed
purchases, and detect when transactions are actions. You can segment your customer base decisions. For example, you could use AI to
likely fraudulent. into groups and provide personalized marketing evaluate whether an investment fits into your
strategies using that insight. portfolio or if someone applying for credit will
likely pay you back.
What is a generative model?
A generative model is a type of AI model that can create new items that look similar and follow the same patterns and
characteristics of material the model saw during training. A generative AI model requires a lot of training materials. Still, once
training is complete, the model can use what it learned to create realistic-looking text, images, videos, audio, or code based on
prompts you write in natural language. For example, you could ask a generative model to write a one-act play about a character
wearing a banana suit at a job interview.
The model would consider what patterns and characteristics it knows about the words in your prompt (like <one-act play,î <banana
suit,î and <job interviewî) and predict what arrangement of words is most likely to satisfy your prompt.
What are generative models used for?
You can use generative AI models to create outputs that look like humans created them, including text, images, video, audio, and
code. The uses for generative AI are vast, including both personal day-to-day use, business applications, and even medical and
biotech research. Just a few uses of generative AI models include:
Generating business documents like Optimizing code for software Translating from one language to
memos, reports, or meeting development another
summaries
Creating marketing materials, Forming synthetic data to train other
including text, images, and AI models
animations
Information theory essentials (entropy, mutual
information)
In generative AI, information theory essentials like entropy and mutual information are crucial for controlling model uncertainty
and ensuring the generated outputs capture the essential, meaningful features of the training data
1. Entropy
Entropy is a measure of the uncertainty, randomness, or unpredictability in a set of data.
Intuition:
High entropy means the outcomes are more evenly distributed and harder to predict. A fair coin has high entropy because
you are uncertain of the outcome.
Low entropy means the outcomes are more predictable or concentrated in a few values. A trick coin that always lands on
heads has zero entropy, as the outcome is certain.
Role in Generative AI:
Balancing creativity and coherence: Generative models use entropy to balance the randomness and structure of their
output. A "temperature" parameter can be used to adjust the entropy: low temperatures produce predictable, rigid text, while
high temperatures lead to more creative, but potentially incoherent, results.
Measuring uncertainty: Entropy helps evaluate the confidence of a model's predictions. In tasks like question answering,
researchers use "semantic entropy" to measure the uncertainty of the model's output meaning, rather than just the words
used, which helps detect confabulations (hallucinations).
Guiding model learning: Cross-entropy, a related concept, is a standard loss function for training neural networks. It
measures the difference between a model's predicted probability distribution and the true distribution, guiding the model to
become more accurate.
2. Mutual information
Mutual information (MI) quantifies the amount of information that one random variable contains about another. It measures the
dependency between two variables, capturing both linear and non-linear relationships.
Intuition:
If two variables are independent, their mutual information is zero.
If knowing the value of one variable significantly reduces your uncertainty about the other, their MI is high. For example, the
MI between "raining" and "carrying an umbrella" would be high.
Role in Generative AI:
Feature selection: Mutual information is used to rank features based on their relevance to a target variable, which is a key
step in training models. It helps identify the most informative parts of the input data for generating meaningful output.
Model interpretation: MI helps explain how different parts of a model's input or internal state relate to its output, especially in
complex architectures like the attention mechanisms in large language models (LLMs).
Controlling generation: By maximizing mutual information between latent (hidden) variables and the generated data,
generative models can produce samples that more accurately reflect the input's key features, making the outputs more
controllable and relevant.
Variational and Amortized Inference
Variational inference (VI) approximates complex posterior probability distributions by converting them into an optimization
problem
Amortized variational inference (A-VI) further speeds up this process in generative AI by using a learned inference function
(often a neural network) to directly map observations to their respective latent variables' approximate posterior, instead of
running a separate optimization for each new data point.
Observed data (x): Patient symptoms (fever, cough, fatigue).
Latent variable (z): The hidden disease (flu, COVID, allergies).
Goal: Infer the disease given the symptoms ³ p(z x). #
Variational Inference (VI)
#
Idea: Instead of finding the exact posterior, approximate it with a simpler distribution q×(z x).
How it works:
Define a flexible family of distributions (e.g., Gaussians).
#
Optimize parameters × to make q×(z x) close to p(z x). #
Real-world analogy:
Imagine each time a patient comes in, the doctor runs a separate optimization process (lab tests, reasoning, probabilities) to
figure out the disease distribution.
Accurate, but time-consuming and costly
Amortized Inference (AI)
Problem with VI: For every new patient, you must start the optimization from scratch.
Solution (AI): Train a neural network (inference/encoder) that directly maps symptoms (x) to the parameters of q×(z x). #
Once trained, for a new patient, the doctor can just input the symptoms and instantly get the approximate distribution over
diseases.
Real-world analogy:
Instead of rethinking from zero every time, the doctor learns a diagnostic shortcut:
See symptoms ³ directly predict the likely disease distribution.
The <effortî of inference is amortized (spread out) across all past patients.
Sr No Variational Inference Amortized Inference
1 Optimize posterior separately for each data point Train one inference model to generalize
2 Doctor re-analyzes each patient from scratch Doctor develops diagnostic intuition from past cases
3 Expensive, repeated optimization Cheap, one forward pass
4 Theoretical backbone of VAEs Theoretical backbone of VAEs
Evaluation Metrics
1 2 3
Log-Likelihood Inception Score (IS) Fréchet Inception Distance
What it measures: What it measures: (FID)
How well the model assigns Quality + diversity of generated What it measures:
probability to real data. images. How close generated data is to real
data in feature space (from
Interpretation: Higher log- Uses a pre-trained classifier
InceptionNet).
likelihood ³ model explains the (InceptionNet).
data better. A good model should produce: Steps:
Use: Common in VAEs and Confident predictions (clear, Extract features of real &
probabilistic models. realistic images). generated images.
Limitation: Hard to compute Diverse classes (not all Fit Gaussians to both.
exactly for complex models (like images look the same). Interpretation: Lower FID =
GANs), may not correlate with
Limitation: Doesn¾t compare to real generated data closer to real data.
perceptual quality.
data, only looks at generated data. Why popular: Correlates well with
human judgment.
4 5
Precision and Recall for Human Evaluation
Distributions (PRD): What it measures: Subjective
What it measures: quality (realism, creativity,
Separates two key aspects: usefulness).
Precision: Are generated samples Types:
realistic (close to real data)? Turing test style: Can humans
Recall: Do generated samples distinguish real vs generated?
cover the diversity of real data? Likert scale ratings: Human
Example: judges score realism/diversity.
A model generating only cats (all Preference tests: <Which
realistic) ³ high precision, low image/text do you prefer?î
recall. Importance: Models might score
A model generating blurry cats, well on IS/FID but still look
dogs, and cars ³ low precision, unrealistic to humans.
high recall.
Core Components
Neural Networks:
A neural network is a machine learning model, inspired by the human brain, that uses layers of interconnected artificial neurons
to process and learn from data.
Every neural network consists of layers of nodes or artificial neurons, an input layer, one or more hidden layers, and an output
layer.
Each node connects to others, and has its own associated weight and threshold.
If the output of any individual node is above the specified threshold value, that node is activated, sending data to the next layer
of the network. Otherwise, no data is passed along to the next layer of the network.
How do neural networks work?
Think of each individual node as its own linear regression model, composed of input data, weights, a bias (or threshold), and an
output. The formula would look something like this:
3wixi + bias = w1x1 + w2x2 + w3x3 + bias
output = f(x) = 1 if 3w1x1 + b>= 0; 0 if 3w1x1 + b < 0
Once an input layer is determined, weights are assigned. These weights help determine the importance of any given variable, with
larger ones contributing more significantly to the output compared to other inputs. All inputs are then multiplied by their respective
weights and then summed. Afterward, the output is passed through an activation function, which determines the output. If that
output exceeds a given threshold, it <firesî (or activates) the node, passing data to the next layer in the network. This results in the
output of one node becoming in the input of the next node. This process of passing data from one layer to the next layer defines
this neural network as a feedforward network.
Embeddings, and Representation learning
An embedding is a way of turning high-dimensional or symbolic data (like words, images, users, items) into a low-dimensional
vector (a list of numbers).
Embeddings are representations of values or objects like text, images, and audio that are designed to be consumed by machine
learning models and semantic search algorithms. They translate objects like these into a mathematical form according to the
factors or traits each one may or may not have, and the categories they belong to.
Essentially, embeddings enable machine learning models to find similar objects. Given a photo or a document, a machine
learning model that uses embeddings could find a similar photo or document. Since embeddings make it possible for computers
to understand the relationships between words and other objects, they are foundational for artificial intelligence (AI).
For example, the documents in the upper right of this two-dimensional space may be relevant to each other:
Technically, embeddings are vectors created by machine learning models for the purpose of capturing meaningful data about each
object.
In machine learning, the use of vectors makes it possible to search for similar objects. A vector-searching algorithm simply has
to find two vectors that are close together in a vector database.
How do embeddings work?
Embedding is the process of creating vectors using deep learning. An "embedding" is the output of this process 4 in other words,
the vector that is created by a deep learning model for the purpose of similarity searches by that model.
Embeddings that are close to each other 4 just as Seattle and Vancouver have latitude and longitude values close to each other
and comparable populations 4 can be considered similar. Using embeddings, an algorithm can suggest a relevant TV show, find
similar locations, or identify which words are likely to be used together or similar to each other, as in language models.
Representation Learning
A broader concept4it's about learning how to automatically find good features/representations of raw data (instead of
manually engineering them).
Representation learning is the process where a machine learning model automatically learns useful features (representations)
from raw data so that they can be used for downstream tasks like classification, prediction, or generation.
Instead of us manually engineering features (e.g., counting word frequency, calculating pixel intensity histograms),
The model discovers compact, informative representations on its own.
< Representations are usually vectors (lists of numbers) in a lower-dimensional space than the raw input.
What is Tokenization?
Tokenization, in the realm of Natural Language Processing (NLP) and machine learning, refers to the process of converting a
sequence of text into smaller parts, known as tokens. These tokens can be as small as characters or as long as words. The
primary reason this process matters is that it helps machines understand human language by breaking it down into bite-sized
pieces, which are easier to analyze.
By converting text into tokens, algorithms can more easily identify patterns. This pattern recognition is crucial because it makes
it possible for machines to understand and respond to human input. For instance, when a machine encounters the word
"running", it doesn't see it as a singular entity but rather as a combination of tokens that it can analyze and derive meaning from.
To delve deeper into the mechanics, consider the sentence, "Chatbots are helpful." When we tokenize this sentence by words, it
transforms into an array of individual words:
["Chatbots", "are", "helpful"].
This is a straightforward approach where spaces typically dictate the boundaries of tokens. However, if we were to tokenize by
characters, the sentence would fragment into:
["C", "h", "a", "t", "b", "o", "t", "s", " ", "a", "r", "e", " ", "h", "e", "l", "p", "f", "u", "l"].
This character-level breakdown is more granular and can be especially useful for certain languages or specific NLP tasks.
Conditioning Signals
Conditioning signals are inputs or instructions provided to the AI model to guide its output, such as prompts that specify a
desired style, topic, or format, or multimodal inputs like text and images that control the generation process.
These are extra inputs that guide the model¾s generation process.
(a) Class conditioning
Model generates data conditioned on a class label.
Example: In class-conditional GANs, label "dog" makes the model generate dog images.
(b) Text conditioning
Natural language prompts guide generation.
Example: Stable Diffusion uses a text encoder (like CLIP) so <a cat wearing glassesî ³ image with a cat wearing glasses.
(c) Mask conditioning
Used in inpainting or masked language models.
Example: In BERT, we mask a word:
"I love [MASK]" ³ model predicts "NLP".
In image inpainting: Masked region of an image is filled in by the model.