Machine Learning and Deep Learning
(ITDLC6031)
Module 6:
Advanced Topics in Deep Learning
Course In-charge:
Dr. Radhika Kotecha
Professor and Head, Department of Information Technology
Contents
Total
COs Hours /
Module Subtopics Hours /
Mapped Subtopic
Module
Attention Mechanisms,
Autoencoders, Generative
Adversarial Networks (GANs), CO6 03
Comparison with Variational 06
Autoencoders (VAE).
Using GANs for Generating Image
CO6 03
Data, Evaluating Generative Models.
2
Transformers
• A transformer model is a neural network that learns the context of sequential
data and generates new data out of it.
• Originating from a 2017 research paper by Google "Attention is All You
Need“
• Generative Pre-trained Transformers (GPTs) are AI models that use
transformer architecture to process and generate human-like text, code, etc.
• Key examples of GPT include OpenAI's ChatGPT (GPT 5.4), Google
Gemini, and Claude, which are used for content creation, code generation,
chatbots, and language translation.
3
From LSTM to Transformers
• RNNs/LSTMs process input sequentially, one element at a time.
• This sequential nature limits parallel computation, so they do
not fully utilize GPUs, making training slow.
• They also struggle with long-range dependencies because
information is passed step by step, and important context may be
lost over long sequences.
• Transformers overcome these limitations using the Attention
mechanism.
• Attention helps the model focus on important words directly, no
matter how far apart they are in the sequence.
• Transformers also allow parallel processing, which significantly
improves training speed.
4
Transformers
• Originally devised for sequence transduction or neural machine translation,
transformers excel in converting input sequences into output sequences.
• Considering a Transformer for language translation as a simple black box, it
would take a sentence in one language, and output its translation in another.
• The encoder takes in our
input and outputs a matrix
representation of that input.
For instance, the English
sentence “How are you?”
• The decoder takes in that
encoded representation and
iteratively generates an
output. In our example, the
translated sentence “¿Cómo
estás?”
5
Transformers
6
Encoder Workflow
• Encoder: Fundamental
component of Transformer.
• Primary Function: To
transform the input tokens
into contextualized
representations.
• Difference: Unlike earlier
models that processed
tokens independently, the
Transformer encoder
captures the context of each
token with respect to the
entire sequence.
7
Encoder Step 1: Input Embeddings
8
Encoder Step 2: Positional Encoding
9
Encoder Step 3: Stack of Encoder Layers
The Transformer encoder consists of a stack of identical layers (6 in the
original Transformer model), comprising two sub-modules:
• A multi-headed attention mechanism
• A fully connected network
10
Attention Mechanism (Self, Multi-Headed)
Query (Q), Key (K), Value (V)
11
Attention Mechanism (Self, Multi-Headed)
12
Transformer Architecture
13
Autoencoders
• An Autoencoder is a type of neural network that can learn to
reconstruct images, text, and other data from compressed
versions of themselves.
• An Autoencoder neural network is an Unsupervised
Machine learning algorithm that applies backpropagation,
setting the target values to be equal to the inputs.
• Autoencoders are used to reduce the size of our inputs into
a smaller representation.
• If anyone needs the original data, they can reconstruct it
from the compressed data.
14
Autoencoders
15
Autoencoders: Step 1
16
Autoencoders: Step 2
17
Autoencoders: Step 3
18
Autoencoders: Summarized
19
Characteristics of Autoencoders
An autoencoder can learn non-linear transformations with
a non-linear activation function and multiple layers.
20
Layers in Autoencoders
• Encoder: The Encoder layer compresses the input image
into a latent space representation. It encodes the input
image as a compressed representation in a reduced
dimension. The compressed image is a distorted version of
the original image.
• Code: The Code layer represents the compressed input fed
to the decoder layer / used for further processing.
• Decoder: The decoder layer decodes the encoded image
back to the original dimension. The decoded image is
reconstructed from latent space representation, and it is a
lossy reconstruction of the original image.
21
Autoencoders
22
Autoencoders
23
Code Layer
• The layer between the encoder
and decoder, i.e. the code is also
known as Bottleneck.
• This is a well-designed approach to
decide which aspects of observed
data are relevant information and
what aspects can be discarded.
• It does this by balancing two
criteria :
• Compactness of
representation, measured as
the compressibility.
• Retains some behaviorally
relevant variables from the
input.
24
Applications of Autoencoders:
Image Coloring
25
Applications of Autoencoders:
Feature Variation
26
Applications of Autoencoders:
Dimensionality Reduction
27
Applications of Autoencoders:
Denoising Image
28
Types of Autoencoders
1. Denoising Autoencoders
• Denoising autoencoders create a corrupted copy of
the input by introducing some noise.
• This helps to avoid the autoencoders to copy the
input to the output without learning features about
the data.
• These autoencoders take a partially corrupted input
while training to recover the original undistorted
input.
29
Types of Autoencoders
2. Convolutional Autoencoders
• Convolutional auto-encoders are a type of
autoencoder that use convolutional neural networks
(CNNs) as their building blocks.
• The encoder consists of multiple layers that take a
image or a grid as input and pass it through different
convolution layers thus forming a compressed
representation of the input.
• The decoder is the mirror image of the encoder it
deconvolves the compressed representation and
tries to reconstruct the original image.
30
Types of Autoencoders
3. Variational Autoencoders
• This type of autoencoder can generate new images
just like GANs.
• After training, you can just sample from the
distribution followed by decoding and generating
new data.
• Variational autoencoder models tend to make strong
assumptions related to the distribution of latent
variables.
• As VAEs are much more flexible and customizable in
their generation behaviour than GANs, they are
suitable for art generation of any kind.
31
Types of Autoencoders
4. Deep Autoencoders
• Deep Autoencoders consist of two identical deep
belief networks: One network for encoding and
another for decoding.
• Typically deep autoencoders have 4 to 5 layers for
encoding and the next 4 to 5 layers for decoding.
• We use unsupervised layer by layer pre-training for
this model.
• Deep autoencoders are useful in topic modeling, or
statistically modeling abstract topics that are
distributed across a collection of documents.
32
Generative Adversarial Networks (GANs)
• Generative Adversarial Networks (GANs) are a
powerful class of neural networks that are used
for unsupervised learning.
• Developed & introduced by Ian J. Goodfellow in
2014.
• GANs are basically made up of a system of two
competing neural network models which compete
with each other and are able to analyze, capture
and copy the variations within a dataset.
• Major application in case of limited data
33
Discriminative versus Generative Models
Example of Discriminative Model Example of Generative Model
(Supervised Learning) (Unsupervised Learning)
Approaches that explicitly or implicitly model the distribution of inputs as
well as outputs are known as Generative Models, because by sampling from
them it is possible to generate synthetic data points in the input space
34
Generative Adversarial Networks (GANs)
• Generative: To learn a generative model, which
describes how data is generated.
• Adversarial: The training of a model is done in
an adversarial setting.
• Networks: Use Deep Neural Networks as the
Artificial Intelligence (AI) algorithm for training
purpose.
35
Generative Adversarial Networks (GANs)
36
Generative Adversarial Networks (GANs)
37
Generative Adversarial Networks (GANs)
38
Generative Adversarial Networks (GANs)
39
Types of GAN Models
• Vanilla GAN: Uses MLP.
• Conditional GAN: Some conditional parameters
are added to Generator and Discriminator.
• Deep Convolutional GAN: Uses CNN.
40
Evaluating Generative Models
• Goal: Assess how well a model generates realistic and
meaningful data
• Quality: Are outputs realistic? (e.g., human-like text, clear
images)
• Diversity: Are outputs varied or repetitive?
• Coherence: Is the content logically consistent?
• Relevance: Does it match the input/context?
• Common Metrics:
• BLEU / ROUGE: For text similarity
• Perplexity: Measures prediction confidence
• FID (Fréchet Inception Distance): For image quality
41
Applications of GAN Models
• Image Synthesis and Generation
• Image-to-Image Translation
• Text-to-Image Synthesis
• Data Generation for Training
• Data Augmentation
42
GANs versus VAEs
Aspect GAN (Generative Adversarial Network) VAE (Variational Autoencoder)
Uses two networks (Generator and Uses encoder-decoder architecture with
Basic Idea
Discriminator) in competition probabilistic approach
Architecture Generator + Discriminator Encoder + Latent Space + Decoder
Training
Adversarial training Optimizes reconstruction loss
Mechanism
Latent Space Not explicitly structured Well-defined, continuous latent space
Output Quality Produces sharp and realistic images Produces smoother but often blurry images
Training Difficult to train, may suffer from
Stable and easier to train
Stability instability
Mode Collapse Common issue Does not suffer from mode collapse
Interpretability Low interpretability More interpretable due to latent variables
Sampling Sampling is indirect via generator Direct sampling from latent distribution
Image generation, deepfakes, super- Data generation, anomaly detection,
Applications
resolution representation learning
43
Example Questions
1. Explain Attention Mechanism in Transformers, with / without application.
2. Explain Encoder workflow in Transformers.
3. Explain Q, K, V in Transformers with example.
4. Explain Autoencoders for any application (e.g. image denoising).
5. Explain types of Autoencoders.
6. Explain applications of Autoencoders.
7. Explain the role of Generator in GANs.
8. Explain the role of Discriminator in GANs.
9. Explain the generic architecture of GANs.
[Link] applications of GANs.
[Link] GANs for any real-world application (e.g. fake currency note
detection, in healthcare, etc.) and explain the architecture of GANs.
[Link] evaluation metrics for GANs.
[Link] GANs and VAEs.
44
References
1. S. Rose, L. Kumar, D Renuka, Deep Learning using Python, Wiley.
2. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT
Press.
3. T. Mitchell, Machine Learning, McGraw Hill.
4. C. Aggarwal, Neural Networks and Deep Learning: A Textbook,
Springer.
5. J. Han, M. Kamber, Data Mining Concepts and Techniques, Morgan
Kaufmann.
6. M. Nielsen, Neural Networks and Deep Learning, Determination
Press.
7. Online Blogs and Videos
45