Module-1: Introduction to GenAI

Introduction to GenAI: Deep Learning vs GenAI, Discriminative AI vs Generative

AI, Applications of GenAI, Transformer Architecture, why are transformers models
trending?, Attention mechanism, Decoder only models, Encoder only models, Encoder
and Decoder models, Examples.

Sample Programs: -
Develop a simple Transformer for Text Classification using HuggingFace
Develop a python code to illustrate attention mechanism visualization.

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
How AI, ML, DL and GenAI are Interrelated?

Asst. Prof. Aashitha L Shamma, CSE Department

AI, ML,DL
• AI is the umbrella field focused on creating machines that can
think, learn, and solve problems.
• Machine Learning (ML): A subset of AI that learns from data.
Supervised Learning (classification, regression)
Unsupervised Learning (clustering, dimensionality reduction)
Reinforcement Learning (agent–environment interactions)
• Deep Learning (DL):Subset of ML that uses deep neural networks
(many layers).Capable of learning high-level representations from
massive datasets.

Asst. Prof. Aashitha L Shamma, CSE Department

Deep Learning (DL)
• Subset of ML that uses deep neural networks (many layers).
• Capable of learning high-level representations from massive datasets.
• Powers computer vision, speech, NLP, health diagnostics, robotics.
Artificial Neural Networks (ANNs)
Basic architecture of DL, early foundation for deep learning.
Composed of neurons arranged in layers (input → hidden → output).
Learns patterns through backpropagation.

Convolutional Neural Networks (CNNs)

Designed for image, audio, and spatial data.
Applications: image classification, medical imaging, audio recognition.

Recurrent Neural Networks (RNNs)

Designed for sequential data (time-series, speech, text).
Captures temporal patterns.
Variants: LSTM, GRU
Enabled early progress in language modelling and speech recognition.
Asst. Prof. Aashitha L Shamma, CSE Department
Transformers: The Breakthrough
Introduced in 2017 (Vaswani et al., Attention is All You Need).
Uses self-attention, replacing recurrence.
Trained efficiently on massive datasets.
Unified architecture for NLP, vision, speech, and multimodal tasks.

• BERT: Bidirectional Transformers

Built on transformer encoder architecture.
Learns contextual meaning in both directions.
Achieved state-of-the-art in:
• Question answering
• Text classification
• Named entity recognition
Opened the door for large language models.

Asst. Prof. Aashitha L Shamma, CSE Department

Generative AI
• Uses models that create new content learned from data.
• Can generate: Images, Audio, Text, Videos, Synthetic medical
data
• Powered by advanced deep learning frameworks.

Asst. Prof. Aashitha L Shamma, CSE Department

Aspect Deep Learning (DL) Generative AI (GenAI)
Learns patterns from large datasets using Creates new data (text, images, audio,
Definition multi-layer neural networks to make video) by learning the underlying data
predictions. distribution.
Classification, prediction, pattern Content generation, creativity, data
Primary Goal
recognition. synthesis.

Discriminative models learn mapping: Generative models learn data distribution

How it Works
input → output. and create new samples.

CNNs, RNNs, LSTMs, Transformers (for GANs, VAEs, Diffusion Models, LLMs (for
Core Techniques
prediction). generation).

Training samples (text, audio, images) used

Typical Inputs Images, speech, sensor data, text.
to learn the distribution.

Labels, predictions, embeddings, Synthetic images, text, audio, designs,

Outputs
classification results. molecules, videos.
Image recognition, speech processing,
Art generation, content creation, data
Applications NLP tasks, anomaly detection,
augmentation, drug discovery, simulation.
autonomous driving.

ResNet, EfficientNet, BiLSTM, Transformer GANs (StyleGAN), VAEs, Diffusion models

Examples
classifiers. (Stable Diffusion), GPT.
Asst. Prof. Aashitha L Shamma, CSE Department
How Everything Connects (Flow Summary)

• AI → ML → DL → ANN → CNN/RNN → Transformers → BERT → Generative AI

• AI is the broadest field.
• ML is a subset of AI.
• DL is a subset of ML.
• ANN is the foundation of DL.
• CNNs & RNNs evolved from ANN for vision and sequences.
• Transformers replaced RNNs by using attention.
• BERT refined transformers for language understanding.
• Generative AI builds on transformers.
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Generative AI (GenAI ) and Discriminative AI are two types of models
training in Data Science industry.
• Discriminative AI focuses only
on learning the boundary
between different categories in
the data.
• Generative AI learns the full
patterns of the data including
how different features relate to
each other which allows it to
generate entirely new content
that looks like the original.

Asst. Prof. Aashitha L Shamma, CSE Department

 Discriminative AI model
➢Discriminative model is a type of
model which is used to predict the label
or classify the labels for the data points
and it is trained on the dataset which
contain labelled data points.

➢They learn the relationship between

features of the datapoints and their
labels.

➢Once the discriminative model has been

trained, it can be used to predict the
labels for new data points.

Asst. Prof. Aashitha L Shamma, CSE Department

Generative AI Model

➢On the other hand, Generative

AI model generates the new
data instances based on the
learned probability of
distribution of existing dataset.
Generative model generates the
new content.

➢A branch of AI focused on
creating new data similar to
existing data.

Asst. Prof. Aashitha L Shamma, CSE Department

Mathematical representation of model output

Asst. Prof. Aashitha L Shamma, CSE Department

Traditional ML Model/ Discriminative Model

Asst. Prof. Aashitha L Shamma, CSE Department

Which is Generative AI and which is not?

Asst. Prof. Aashitha L Shamma, CSE Department

GenAI Model vs Discriminative Model

Asst. Prof. Aashitha L Shamma, CSE Department

Aspect Generative AI Discriminative AI
Generates new data similar to training
Core Purpose Predicts labels/classes for given input.
data.
Image generation, augmentation, text
Medical classification, fraud detection,
Use Cases generation, voice cloning, drug
sentiment analysis, object recognition.
discovery.
Primary Task Content creation, data synthesis. Classification, prediction, detection.
Requires large datasets to learn the Works well even with moderate-sized,
Data Requirements
distribution. labeled datasets.
Type of Data Used for Often trained on unstructured or semi- Requires clean, structured, labeled
Training structured data (images, audio, text). data.
New synthetic content (images, audio,
Output A predicted label or probability.
text).
Optimize generator + discriminator /
Optimize a loss (cross-entropy, hinge
Training Objective latent modelling to produce realistic
loss) to separate classes accurately.
samples.
High computational complexity; models Typically, simpler and computationally
Model Complexity
are large and harder to train. efficient.
Speed Slower training and inference. Faster training and inference.
Decision Trees, SVM, Logistic
Examples of GAN, VAE, Diffusion Models, GPT,
Regression, Random Forest, DNN, CNN,
Algorithms LLaMA, Stable Diffusion.
RNN.
Asst. Prof. Aashitha L Shamma, CSE Department
Lab program
Develop a python code to show difference between discriminative
and generative AI
Lab program
Parallel vs sequential
• Write two timing experiments:
• (a) a simple loop that processes tokens one by one (simulating an
RNN).
• (b) a vectorized operation on the whole sequence at once (simulating a
Transformer).
• Compare runtime using [Link]() for long sequences.

This tells you why Transformers train much faster on GPUs.

Transformers process all tokens simultaneously
(by matrix multiplication X @ W)
Lab program
Develop a python code to illustrate attention mechanism
visualization.
Lab program
Develop a simple Transformer for Text Classification using Hugging
Face

Hugging Face is an open-source AI company and community.

They provide pretrained models (BERT, GPT, T5, etc.), datasets,
and easy-to-use libraries (like transformers) so we don’t have to
build big models from scratch
Lab program
Write a menu-driven Python script:
• Option 1: sentiment classification (encoder-only model).
• Option 2: text completion (decoder-only).
• Option 3: translation/summarization (encoder-decoder).
The script asks the user to type text, calls the appropriate
HuggingFace pipeline, and prints the output.

Practice this program.

Generative AI Modalities

LLM LIM

Asst. Prof. Aashitha L Shamma, CSE Department

Generative AI Capabilities

Asst. Prof. Aashitha L Shamma, CSE Department

 Generative AI Output Domains
Types of Generative AI
Generative Artificial Intelligence (AI) encompasses various models and systems designed to
create new content across multiple modalities.
1. Text Generation: Models like GPT-3, GPT-4, LaMDA, and LLaMA are trained on
extensive text datasets, enabling them to perform tasks such as natural language processing,
machine translation, and content creation.
2. Code Generation: By training on programming languages, models like OpenAI Codex can
generate source code, assisting in software development and automating coding tasks.
3. Image Generation: Systems such as DALL-E, Midjourney, and Stable Diffusion generate
images from textual descriptions, facilitating applications in art, design, and media.
4. Video Generation: Models like Meta's Make-A-Video can produce coherent video clips
based on input data, impacting entertainment and content creation industries.
5. Music Generation: AI systems like MusicLM can compose music by analyzing patterns in
audio data, offering tools for musicians and composers.
6. 3D Modeling: AI-driven tools can automate 3D model creation from text, images, or videos,
streamlining processes in design and virtual reality applications.
7. Multimodal Systems: These models, such as certain versions of GPT-4, can process and
generate content across multiple data types, like text and images, enhancing their versatility.
Asst. Prof. Aashitha L Shamma, CSE Department
Generative AI Output Domains

Asst. Prof. Aashitha L Shamma, CSE Department

INPUT: Text Prompt: Astronaut floating in space

OUTPUT: Generated Image

Asst. Prof. Aashitha L Shamma, CSE Department
• Core Technologies Behind Generative AI (Timeline)

1️⃣ VAEs (2013) – Early generative models based on latent space learning
2️⃣ GANs (2014) – Generator vs discriminator competition
3️⃣ Transformers (2017) – Self-attention; foundation of GPT/BERT
4️⃣ Diffusion Models (2020+) – Noise-to-image generation; state-of-the-art today

Asst. Prof. Aashitha L Shamma, CSE Department

 Core Technologies Behind Generative AI

1. Variational Autoencoders (VAEs) — 2013

✓ VAEs were one of the earliest deep generative models.
✓ Learn a compressed representation of data
✓ Generate new samples by sampling from a learned latent space
✓ Used in image reconstruction, anomaly detection, speech
features

Asst. Prof. Aashitha L Shamma, CSE Department

 Core Technologies Behind Generative AI

2. Generative Adversarial Networks (GANs) — 2014

✓Introduced the idea of training two neural networks in
competition.
✓Generator creates new data
✓Discriminator checks if it looks real
✓Used for image synthesis, style transfer, data augmentation

Asst. Prof. Aashitha L Shamma, CSE Department

 Core Technologies Behind Generative AI

3. Transformers — 2017
✓A major breakthrough architecture for sequence modeling.
✓Based on self-attention
✓No recurrence → fast parallel training
✓Foundation for GPT, BERT, T5
✓Used for text generation, translation, multimodal tasks

Asst. Prof. Aashitha L Shamma, CSE Department

 Core Technologies Behind Generative AI

4. Diffusion Models — 2020 onwards

✓Currently leading models for high-quality image generation.
✓Learn to remove noise step-by-step from random noise
✓Extremely stable and produce photorealistic images
✓Used in DALL·E 2, Midjourney, Stable Diffusion

Asst. Prof. Aashitha L Shamma, CSE Department

How Generative AI uses Transformer architecture
to process input and generate meaningful output

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Instead of a 3×3 2D kernel, text uses a 1×3 kernel, sliding in one direction:
[embedding1] [embedding2] [embedding3]
↑ ↑ ↑
kernel covers 3 words at a time
Asst. Prof. Aashitha L Shamma, CSE Department
From CNN → RNN → LSTM → Transformers
• But cannot capture long-range relationships
(e.g., subject–verb dependency over long distance)
Example:
“The book that you gave me yesterday was amazing.”
• CNN can’t understand this well.
• So invented RNN/LSTM.
• Highlighted limitations
→ CNN: not good for sequence
→ RNN\LSTM : slow and forgetful
• So the punchline:
“This is why Transformers were invented.”

Asst. Prof. Aashitha L Shamma, CSE Department

Introduction to Transformers

Asst. Prof. Aashitha L Shamma, CSE Department

GPT: Generative Pre-trained Transformer

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Attention Mechanism

Asst. Prof. Aashitha L Shamma, CSE Department

Why Attention Matters

Asst. Prof. Aashitha L Shamma, CSE Department

Why RNNs Are Difficult to Train

Asst. Prof. Aashitha L Shamma, CSE Department

Why we need attention?

Asst. Prof. Aashitha L Shamma, CSE Department

Attention mechanism vs RNN/LSTM

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Why Positional Encoding Matters in Self-Attention

Asst. Prof. Aashitha L Shamma, CSE Department

• Lack of sequential understanding is a significant limitation of self
attention. Without taking care of this we cannot use self attention
in NLP tasks.

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
• Self attention is not bother about positions of the tokens. How to
overcome?→ Positional encoding
• Along with the word embedding encode the positional information
to self attention
•

PROBLEM
Cant use discrete number
Cant use large unbounded numbers

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
PROBLEM

The sine function provides a smooth

and continuous representation,
but its periodic nature causes
positions far apart to have similar
values

Asst. Prof. Aashitha L Shamma, CSE Department

Solution PROBLEM

positional encoding is
expressed as a vector using
both sine and cosine
to provide a more unique and
distinguishable
representation across
positions.

Their periodic nature still

causes repetition over long
distances.

Asst. Prof. Aashitha L Shamma, CSE Department

PROBLEM

Using the same frequency

for sine and cosine functions
makes different positions
look similar, especially over
long sequences. This limits
the model’s ability to
distinguish fine-grained
position differences
and capture both short-term
and long-term
dependencies.

Asst. Prof. Aashitha L Shamma, CSE Department

SOLUTION

Use multiple frequencies — each dimension pair (sin & cos) encodes
positions at a different scale.

•Low-frequency pairs capture global, broad positional trends.

•High-frequency pairs capture fine, local variations.

By using 512 dimensions (→ 256 sin + 256 cos pairs),

the model builds a rich, multi-resolution representation of position,
allowing it to understand both nearby and distant word relationships.

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Normalization steps
Asst. Prof. Aashitha L Shamma, CSE Department
Steps of linear transformation and Relu
activation function in between:

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Why Softmax is the magic step?

Asst. Prof. Aashitha L Shamma, CSE Department

How this helps decoder prediction (very important)
Without masking
• Decoder sees future words
• Learns to “copy answers”
• Training looks good
• Inference fails
With masking
• Decoder learns:
• “Predict next word using only past words”
• Training = inference behavior
• True autoregressive generation

Asst. Prof. Aashitha L Shamma, CSE Department

While predicting “एक”
• Allowed attention: मैं
• Masked: रोबोट, हूँ
• So attention weights:
[0.497 0.503 0 0]
• The decoder:
Learns language order
Learns grammar
Learns causal structure

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Mechanism of an Attention Head

•Q (Query) = “What am I looking for?”

•K (Key) = “What do I contain?”
•V (Value) = “What information will I pass?”

Asst. Prof. Aashitha L Shamma, CSE Department

Self Attention weights calculation

Asst. Prof. Aashitha L Shamma, CSE Department

Queries, Keys, and Values (Q, K, V)

Asst. Prof. Aashitha L Shamma, CSE Department

How Q, K, V are Generated

Asst. Prof. Aashitha L Shamma, CSE Department

Dot-Product Similarity (Q · Kᴛ)

Asst. Prof. Aashitha L Shamma, CSE Department

Scaled Dot-Product Attention

Asst. Prof. Aashitha L Shamma, CSE Department

Multi-Head Attention (MHA)

Asst. Prof. Aashitha L Shamma, CSE Department

Multi-Head Attention (MHA)

Asst. Prof. Aashitha L Shamma, CSE Department

Transformer
Block:

Asst. Prof. Aashitha L Shamma, CSE Department

Transformer
Block:

Asst. Prof. Aashitha L Shamma, CSE Department

1. Sublayers Defined in __init__(): Inside the initialization
function, all components of the Transformer block are created
2. Causal Mask Creation: is generated to prevent each query token
from attending to future key tokens.
3. Multi-Head Attention With Mask
• The MultiHeadAttention layer is applied to the input sequence.
• The previously created causal mask is passed to the attention
layer.
• The layer returns both the attention output and the attention
scores.

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Positional Encoding

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Simplified GPT model architecture

Asst. Prof. Aashitha L Shamma, CSE Department

An encoder-decoder Transformer model

Asst. Prof. Aashitha L Shamma, CSE Department

Problem: Self-Attention Computation in Transformers
Consider a simple sentence consisting of two tokens:
Token 1: “I”
Token 2: “AI”
• Assume a single-head self-attention mechanism.
• The query (Q), key (K), and value (V) matrices corresponding to
these two tokens are given as:
1 0 1 0 1 2
•𝑄= ,𝐾 = ,𝑉 =
0 1 1 1 3 4
• key dimension 𝑑𝑘 = 2.

Asst. Prof. Aashitha L Shamma, CSE Department

Tasks
• Compute the attention score matrix using the scaled dot-
product attention formula:
𝑄𝐾 𝑇
𝑑𝑘
• Apply the softmax function row-wise to obtain the attention
weight matrix.
• Compute the final attention output matrix by multiplying the
attention weights with the value matrix 𝑉.
• Interpret the result:
Explain how each token attends to itself and the other token
based on the computed attention weights.

Asst. Prof. Aashitha L Shamma, CSE Department

• Given
1 0 1 0 1 2
𝑄= ,𝐾 = ,𝑉 = , 𝑑𝑘 = 2, 𝑑𝑘 = 2 ≈ 1.414
0 1 1 1 3 4

1) Compute 𝑄𝐾 𝑇
First,
1 1
• 𝐾𝑇 =
0 1
Then,
𝑇 1 0 1 1 1 1
• 𝑄𝐾 = =
0 1 0 1 0 1
2) Scale the scores
1 1
𝑄𝐾 𝑇 1.414 1.414 0.707 0.707
•𝑆= = 0 1 =
2 0 0.707
1.414 1.414

Asst. Prof. Aashitha L Shamma, CSE Department

3) Apply softmax row-wise
• Softmax formula for a row 𝑎 𝑏 :
𝑒𝑎 𝑒𝑏
softmax 𝑎 𝑏 = 𝑎
𝑒 + 𝑒𝑏 𝑒𝑎 + 𝑒𝑏
Row 1: 0.707 0.707
• 𝑒 0.707 = 2.03
• Sum = 2.03 + 2.03 = 4.06
• ⇒ 2.03 /4.06 , 2.03/4.06 = 0.5 0.5

Row 2: 0 0.707
• 𝑒 0 = 1, 𝑒 0.707 = 2.03
• Sum = 1 + 2.03 = 3.03
• ⇒ 1 /3.03 , 2.03/3.03 = 0.33 0.67
• So attention weight matrix:
0.50 0.50
𝐴=
0.33 0.67 Asst. Prof. Aashitha L Shamma, CSE Department
4) Compute final attention output: 𝐴 × 𝑉
0.50 0.50 1 2
Output = 𝐴𝑉 =
0.33 0.67 3 4

Final:
2 3
Attention 𝑄 𝐾 𝑉 =
2.34 3.34
MEANS…….
Token 1 (“I”) has weights 0.5 0.5 →it uses both tokens equally.
Token 2 (“AI”) has weights 0.33 0.67 →it attends more to token 2 than
token 1.

Asst. Prof. Aashitha L Shamma, CSE Department

Problem:
Consider a sentence consisting of two tokens:
Token 1: “I”
Token 2: “AI”
• The embedding dimension is 4, and the number of attention heads is 2.
• Determine the dimension handled by each attention head.
• Assume that the input embedding matrix 𝑋 ∈ ℝ2×4 is split equally
across the two heads.
• For each head:
• Treat the split embeddings as the Query (Q), Key (K), and Value (V) matrices.
• Compute the scaled dot-product attention.
• Concatenate the outputs from both heads to obtain the final multi-
head attention output.

Asst. Prof. Aashitha L Shamma, CSE Department

Why must embedding dimension be divisible by heads?
• Because each head gets an equal slice
𝑑model
𝑑head =
num_heads
• Example:
𝑑model = 4, heads = 2 ⇒ 𝑑head = 2

• Each head works independently on 2 features

What happens inside multi-head attention?

For each token:
Head 1: 𝑥1 𝑥2
𝑥1 𝑥2 𝑥3 𝑥4 ⇒ቊ
Head 2: 𝑥3 𝑥4
embedding dim = 4
Asst. Prof. Aashitha L Shamma, CSE Department
Step 1: Input embedding matrix
1 0 1 0
•𝑋=
0 1 0 2
Rows = tokens
Columns = features

Step 2: Projection into two heads

Head 1 uses first 2 dimensions
1 1 0
𝑋 =
0 1
Head 2 uses last 2 dimensions
1 0
𝑋2 =
0 2

Asst. Prof. Aashitha L Shamma, CSE Department

Step 3: Self-Attention in Head 1 Step 4: Self-Attention in
Head 2
• Q₁ = K₁ = V₁ = X¹
• Q₂ = K₂ = V₂ = X²
1 0
𝑄1 𝐾1𝑇= 1 0
0 1 𝑄2 𝐾2𝑇 =
• Apply softmax (row-wise): 0 4
• Apply softmax:
0.73 0.27 0.73 0.27
𝐴1 = 𝐴2 =
0.27 0.73 0.02 0.98
• Output: • Output:

0.73 0.27 0.5 0.5

𝑂1 = 𝐴1 𝑉1 = 𝑂2 =
0.27 0.73 0.5 0.5
Asst. Prof. Aashitha L Shamma, CSE Department
Step 5: Concatenate heads
0.73 0.27 0.73 0.54
• Concat 𝑂1 𝑂2 =
0.27 0.73 0.02 1.96

Step 6: Final linear layer (optional)

• MultiHead 𝑋 = Concat 𝑂1 𝑂2 𝑊 𝑂

Note:
Single head = one way of understanding a sentence
Multi-head = multiple perspectives at the same time
“One head may focus on grammar, another on meaning, so on — all
in parallel.”
Asst. Prof. Aashitha L Shamma, CSE Department
Problem
If after the Multi-Head Attention + Add & Norm block of
a Transformer encoder, the normalized output is given as:
Z=

• Do the calculation for position-wise feed-forward

network (FFN) defined as:
where
1 0
𝑊1 = , 𝑏1 = 0 0
0 1
1 1
𝑊2 = , 𝑏2 = 0 0
1 1

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Sentence:
“I did not like the movie”
Attention mechanism:
“not” attends to “like”
without FFN:
model cannot invert sentiment
“like” still dominates
----------------------------------------------------------------------------------------
With FFN:
FFN learns:
not + like ⇒ negative meaning
Without FFN:
not + like ⇒ average of words

Asst. Prof. Aashitha L Shamma, CSE Department

Problem
Show the Masking in Decoder Self-Attention
• Sentence (Target sequence)
"I love AI"
• We have 3 tokens:
Token₁ = “I”
Token₂ = “love”
Token₃ = “AI”

Asst. Prof. Aashitha L Shamma, CSE Department

Step 1: Assume decoder self-attention scores (before masking)
• Assume the scaled dot-product attention scores
𝑄𝐾𝑇
are:
𝑑𝑘
1.0 2.0 3.0
𝑆 = 1.0 2.0 3.0
1.0 2.0 3.0
Each row = current token attending to all tokens
Step 2: Apply Look-Ahead Mask (decoder mask)
• Mask future tokens using −∞:
0 −∞ −∞
𝑀= 0 0 −∞
0 0 0

Asst. Prof. Aashitha L Shamma, CSE Department

• Step 3 : Add mask to attention scores
1.0 −∞ −∞
𝑆masked = 𝑆 + 𝑀 = 1.0 2.0 −∞
1.0 2.0 3.0
• Step 4 : Apply softmax row-wise
softmax 1.0 − ∞ − ∞ = 1 0 0
Can attend only to itself

softmax 1.0 2.0 −∞ ≈ 0.27 0.73 0

Can attend to “I” and “love” only

softmax 1.0 2.0 3.0 ≈ 0.09 0.24 0.67

Can attend to all previous tokens

Asst. Prof. Aashitha L Shamma, CSE Department

Step 5: Final attention matrix
1.00 0.00 0.00
𝐴 = 0.27 0.73 0.00
0.09 0.24 0.67
----------------------------------------------------------------------------------
WITHOUT MASKING WHAT WILL HAPPEN,

Token “I” attends 67% to “AI”

Token “love” attends 67% to “AI”
The model already knows the future word
This is information leakage
Model learns shortcuts other than modeling true causal relationships between tokens
Asst. Prof. Aashitha L Shamma, CSE Department
Why masking needed in decoders?
Without masking, the decoder performs very well during training
because it can see future words.
However, during testing or generation on a new sentence, future
words are not available, and the model struggles to generate the
next word.
Masking is therefore required to ensure that training and testing
conditions are the same.

Asst. Prof. Aashitha L Shamma, CSE Department

Encoder-Only Transformers

Asst. Prof. Aashitha L Shamma, CSE Department

Decoder-Only Transformers

Asst. Prof. Aashitha L Shamma, CSE Department

Encoder–Decoder (Seq2Seq) Transformers

Asst. Prof. Aashitha L Shamma, CSE Department

RECALL TOPICS ON NLP

Asst. Prof. Aashitha L Shamma, CSE Department

What is Attention Mechanism
Transformer architecture have huge breakthrough in AI.
A revolutionary phase for LLM.

Asst. Prof. Aashitha L Shamma, CSE Department

First Large Language Models do like this;

Asst. Prof. Aashitha L Shamma, CSE Department

• Embedding transforms the text into high dimensional dense vector
which captures the meaning of each tokens

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
• This embedding comes from an embedding matrix where,
No: of rows=vocabulary size
No: of columns=embedding dimension size

Asst. Prof. Aashitha L Shamma, CSE Department

• So each tokens will have its own embedding lookup positions

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
• However there is a problem: Embedding do not distinguish words
with multiple meanings

Asst. Prof. Aashitha L Shamma, CSE Department

• So single embedding value generated regardless of the context

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Meaning of the word differ and depends on other words or tokens.
So “Apple” meaning differs when checking with “Phones” and
“Juice” in the given examples .To get context info single token
interacts with other token to assign weight to each other .As
interacts each other it assign importance to each other. Attention
weight shows how much one token attends another.

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Why Positional Encoding Matters in Self-Attention

Asst. Prof. Aashitha L Shamma, CSE Department

• Lack of sequential understanding is a significant limitation of self
attention. Without taking care of this we cannot use self attention
in NLP tasks.

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
• Self attention is not bother about positions of the tokens. How to
overcome?→ Positional encoding
• Along with the word embedding encode the positional information
to self attention
•

PROBLEM
Cant use discrete number
Cant use large unbounded numbers

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
PROBLEM

The sine function provides a smooth

and continuous representation,
but its periodic nature causes
positions far apart to have similar
values

Asst. Prof. Aashitha L Shamma, CSE Department

Solution PROBLEM

positional encoding is
expressed as a vector using
both sine and cosine
to provide a more unique and
distinguishable
representation across
positions.

Their periodic nature still

causes repetition over long
distances.

Asst. Prof. Aashitha L Shamma, CSE Department

PROBLEM

Using the same frequency

for sine and cosine functions
makes different positions
look similar, especially over
long sequences. This limits
the model’s ability to
distinguish fine-grained
position differences
and capture both short-term
and long-term
dependencies.

Asst. Prof. Aashitha L Shamma, CSE Department

SOLUTION

Use multiple frequencies — each dimension pair (sin & cos) encodes
positions at a different scale.

•Low-frequency pairs capture global, broad positional trends.

•High-frequency pairs capture fine, local variations.

By using 512 dimensions (→ 256 sin + 256 cos pairs),

the model builds a rich, multi-resolution representation of position,
allowing it to understand both nearby and distant word relationships.

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Explanation for step 3(in previous slide)

Asst. Prof. Aashitha L Shamma, CSE Department

Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Asst. Prof. Aashitha L Shamma, CSE Department
Here's a brief overview of the generation order of some popular
Transformer models:
• Transformer (2017): The original Transformer model, introduced by
Vaswani et al., which revolutionized the field of NLP.
• BERT (2018): Bidirectional Encoder Representations from
Transformers, introduced by Google, which achieved state-of-the-art
results in many NLP tasks.
• GPT-1 (2018): The first version of the Generative Pre-trained
Transformer, introduced by OpenAI, which demonstrated impressive
language generation capabilities.
• GPT-2 (2019): An improved version of GPT-1, which showed even
more impressive language generation capabilities and sparked
discussions about AI safety.
• RoBERTa (2019): A variant of BERT that achieved state-of-the-art
results on many NLP benchmarks, introduced by Facebook AI.
Asst. Prof. Aashitha L Shamma, CSE Department
• DistilBERT (2019): A distilled version of BERT, which achieved
similar performance to BERT with fewer parameters, introduced by
Hugging Face.
• GPT-3 (2020): A massive version of GPT-2, which demonstrated
unprecedented language generation capabilities and sparked
widespread interest in large language models.
• ALBERT (2020): A Lite BERT model that achieved state-of-the-art
results on many NLP benchmarks with fewer parameters, introduced
by Google.
• ELECTRA (2020): A model that uses a generator-discriminator
framework to achieve state-of-the-art results on many NLP tasks,
introduced by Google.
• T5 (2020): A Text-to-Text Transfer Transformer model that achieved
state-of-the-art results on many NLP tasks, introduced by Google.
• GPT-4 (2023): The latest version of the GPT series, which has
demonstrated even more impressive language generation capabilities.
Asst. Prof. Aashitha L Shamma, CSE Department
 Key Innovation /
Model Architecture Type Directionality Training Objective Typical Use Case
Strength
Learns temporal
Basic sequence
RNN (Recurrent Sequential Predict next token in dependencies but
Unidirectional modeling (speech,
Neural Network) (recurrent) sequence struggles with long-
simple text)
term memory
Solves vanishing Speech recognition,
LSTM (Long Short- Sequential Unidirectional / Predict next token
gradient, captures time-series
Term Memory) (recurrent, gated) Bidirectional with gating
longer dependencies forecasting
Causal language Strong text Text generation,
GPT (Generative Pre- Decoder-only Unidirectional
modeling (next-token generation and dialogue,
trained Transformer) Transformer (left→right)
prediction) reasoning ability summarization
Masked Language
BERT (Bidirectional
Encoder-only Modeling (MLM), Next Deep contextual QA, NER, sentiment
Encoder Bidirectional
Transformer Sentence Prediction understanding analysis
Representations)
(NSP)
Better training
Encoder-only
RoBERTa Bidirectional MLM (without NSP) strategy and larger QA, classification
(improved BERT)
data
MLM + Sentence Parameter sharing → Same as BERT but
ALBERT Encoder-only Bidirectional
Order Prediction lightweight BERT faster
Replaced Token More efficient
ELECTRA Encoder-only Bidirectional Detection (generator– pretraining, learns Classification, QA
discriminator setup) faster than BERT
Combines BERT’s
Permutation-based Bidirectional (via Permutation QA, text
XLNet bidirectionality with
Transformer permutations) Language Modeling classification
GPT’s autoregression
T5 (Text-to-Text Text-to-text Converts every task
Bidirectional encoder Summarization,
Transfer Encoder–Decoder framework (unified into text-generation
+ decoder translation, QA
Transformer) for all NLP tasks) form
Asst. Prof. Aashitha L Shamma, CSE Department
Color Shade Model Family / Category Models Included Description / Focus
Earlier deep learning
architectures that process text
Pre-Transformer (Sequential token-by-token. Capture
Pink RNN, LSTM
Models) sequential dependencies but
struggle with long-term context.

Transformer decoder or
encoder-decoder models
GPT Family (Decoder-based focused on generation tasks —
Yellow-Green GPT, T5
Models) predicting next tokens or
transforming text-to-text.

Transformer encoder models for

understanding tasks like
BERT Family (Encoder-based BERT, RoBERTa, ALBERT, classification, NER, and QA.
Light Blue Shades
Models) ELECTRA Focus on deep contextual
representation.

Combines BERT’s
bidirectionality with GPT’s
Green-Blue Tint Hybrid / Bridge Model XLNet autoregressive generation —
achieves both understanding and
generation abilities.
Asst. Prof. Aashitha L Shamma, CSE Department
• Highlights

Asst. Prof. Aashitha L Shamma, CSE Department