Other Deep Learning Architectures

Dr. SALIM A.

Dean CSE/Research
MVJ College of Engineering

November 21, 2025

Encoder-Decoder Modeling
I A standard approach to handle applications such as image captioning or
machine translation is pass the inputs (video, text, audio etc) through the
encoder network, which gives you a representaion of the input. This
representation is called a Context Vector.
I The context vector is passed through a decoder network which gives you
the final text.

Figure: Encoder Decoder Modeling

Neural Machine Transation Problem
I Assume as an example, we want translate a sentence English to Hindi.
I The Encoder part would be an RNN, where each word of the input sentence is
given as input at one time step of the RNN.
I The final output of the RNN is context vector
I Context vector is fed into a decoder RNN, which gives you output in Hindi
I In Machine translation applications, it is wiser to read the full sentence and then
start giving the output of the translated sentence(since different languages have
different grammer, word positions in the output will be in different order)
Image Captioning Problem
I Image Captioning Problem: Encoder is a CNN followed by fully connected
network, which produces are a context vector.
I Decoder is a RNN, which produces the caption.

Figure: Image Captioning

RNN Issues?

I In an RNN, the hidden states are responsible for storing relevant input
information.
I A hidden state at time step t is a compressed form of all previous inputs
(x1 , x2 , ......., xn ).
I But if the input is very long, can hT encode all information without
forgetting? No. This is information bottleneck!

Figure: Hidden states in RNN

RNN Issues?

I We cannot guarantee that words seen at earlier time steps be reproduced

in later time steps of outputs

Figure: Issue in RNN: hT may fail to reproduce all words correctly in translated
output
Failure of Encoder Decoder modeling
I BLEU Score
I Bilingual Evaluation Understudy. It is a metric for evaluating quality
of machine translated output.
I When sentence length increases, encoder-decoder model using RNN
fails.

Figure: Sentence length vs Bleu Score

Attention Mechanism

Figure: What is the boy doing?

Figure: Summarize the text
I Human way: Identify the
artifacts in the image and then I Human way: Identify the
pay attention to relevant artifacts relevant parts of the text and
prepare summary
Attention Mechanism :Temporal data

Figure: Encoder Decoder Model

I Given an encoder, with hj as hidden state at time step j and decoder with
st as hidden state at time step t
Attention Mechanism :Temporal data

I Attention Mechanism
I Given an encoder, with hj
as hidden state at time step
j and decoder with st as
hidden state at time step t
I Instead of directly outputing
hT to the decoder,
Attention mechanism
creates a shortcut between
context vector( ct ) and
entire source of input (X )
I Decoder hidden state (st ) at
time t given by :

st = f (st−1 , yt−1 , ct )
Figure: Attention Models
Attention Mechanism :Temporal data

I Context Vector ct
T
X
ct = αt,j hj
j=1

I where αt,j gives degree of

alignment between st−1 and hj . It
is the softmax over some scoring
function which captures the score
between st−1 and each of the
0
hidden state (hj ) in the encoder.

e (score(st−1 ,hj ))
αt,j = P 0
T
0
j =1
e (score(st−1 ,hj ))
Figure: Attention Model
Attention Mechanism :How to calculate Score

I Using αt,j context vetors

(c1 , c2 , ...., ck )can be computed
and the corresponding ct given as
the input in each time step of
decoder network.
I Score.
I Content based attention :

score(st , hi ) = cosine(st , hi )
I Additive attention :
I Learning a set of
weights Wa and using
a vector V

score(st , hi ) = V T tanh(Wa [st , hi ])

I General attention : Figure: Attention Model
score(st , hi ) = stT (Wa hi )
Attention Mechanism :Spatial Data

Figure: Attenttion mechanism :Spatial Data

I A fully connected layer after the CNN, will cause the network to loose the
spatial information. So the output of CNN(output volume m x n x c), in
depthwise parts will provide certain patch of the original image and hence
give spatial information.
Motivation for Transformers

I Sequential computation prevents parallelization.

I Though LSTM and GRU are available, recurrent networks need attention
mechanism to deal with long range dependencies.
I If the data to be processsed is a very long time series data, for example, a
book chapter data, RNN, LSTM or GRU may not be capable to hold
data. Attention mechanism helps to focus on the relevant parts of
previous sequence.
I But if attention gives us access to any state, RNN iself may not be
required. For example, in machine translation, while generating a
particular word as output, we can decide on whcih part of the input
sequence should we focus.
Transformers

I Transformer model is an encoder decoder

network to perform a sequence to
sequence modeling without any RNN
I Entirely built on self attention mechanism
without sequence aligned recurrent units
I Key Components
I Self Attention
I Multi head Attention
I Positional Encoding
I Encoder Decoder Architecture.
Transformers

Figure: Components of s single encoder

unit

I Consider a machine translation task

I Each encoder has several
I In Encoder, there are several encoder
components:
modules, each of which feed into the I A self attention module which
encoder at the next level. outputs a vector for every input
I The encoder at the highest level feeds word
into each of the decoder modules I A feed forward network, which
Self Attention
I Consider two input sentences we want to
translate:
I The animal didn’t cross the street because
it was too tired.
I The animal didn’t cross the street because
it was too wide.
I In the first case, ”it” refers to animal, but in the
second case, ”it” refers to the street.
Identification of these differences is a hard task
for traditional ”Seq2Seq” models.
I Self attention: Given a sequence, when it process
one element of the sequence, make a copy of the
sequence, and find which all parts of the same
sequence are important to process this particular
element.
I Thus, it is not required to keep the history of
hidden states as in RNN to maintain history of
previous sequence inputs. Whenever we process
one element of the sequence, at that time, we
process an attention vector over the entire
Self Attention Implementation

I Self Attention
I Assume the input is ”Thinking Machine” two words. Embedd each
word using some text processing technque)
I Step 1: Create three vectors from embedded input vector xi
I Query Vector (qi ), Key Vector (ki ) and Value Vector (vi )
I These vectors are created by multiplying inputs with weight
matrices W Q , W K and W V , learned during the training.
Self Attention Implementation

I Step 2
I Calculate self attention scores of all words of the input sentence
against themselves.
I By taking the dot product of query vector with the key vector of the
respective words
Self Attention Implementation

I Step 2
I By taking the dot product of query vector with the key vector of the respective
words
I For E.g.:Input ”Thinking”, first score is q1 xk1 and the second score is q1 xk2
p
I Scores are divided by length(k). This is called scaled dot product attention
Self Attention Implementation

I Step 3
I Softmax is used to get normalized probability scores; determines how
much each word will be expressed at this position ( To get the distribution
of the attention of each of the words in the sequence with respect to the
word under consideration)
Self Attention Implementation

I Step 4
I Multiply each value vector by softmax score; This will the words
that we focus intact.
I Step 5
I Sum up weighted value vectors, which produces output of self
attention layer at this position.
I z1 is weighted sum of v1 xsoftmaxvalue + v2 xsoftmaxvalue + .....
Similarly z2 is found when the next word is processed.
Multihead Attention

I Multihead Attention: Instead of using single

W Q , W K and W V , transformers suggests to
have multiple W Q , W K and W V which give
multiple query vector, key vector and value vector
and thus have multiple self attention layers
I Expands model ability to focus on different
positions
I Gives attention layer multiple
”representation spaces” as there are
multiple sets of Query/Key/Value weight
matrices. Each set is used project input
embedding to different subspaces
Multihead Attention

I R
I In a transformer architecture, there are multiple layers, the first
encoder layer receives X as input and every other layer receives
output of previous encoder layer as input. R indicates output of
previous encoder layer.
Positional Encoding

I Positional Encoding
I Unlike RNN and CNN encoders, attention encoder outputs do
not depends on the order of inputs. (Since we can choose to
focus on what you want irrespective of the order of the inputs)
I But order of sequence conveys important information for
machine translation tasks and language modeling
I Idea: add positional information of input token in the sequence
into input embedding vectors.
I Final embedding are concatenation of learnable embedding and
positional encoding.
I The role of the positional encoding is to bring some value
where for a specific input which you are currently processing is
in the sequence.
Decoder

I Encoder
I The decoder is a stack of identical layers (for example 6)
I Each layer has a multihead attention layer and fully connected feed
forward [Link] to encoder, the decoder sublayer has a residual
connection and layer normalization
I Masked multihead attention : to prevent attending subsequent future
positions (not to look into future target sequence when predicting current
position)
Deep Generative Models

Figure: Discriminative vs Generative models

I Discriminative
I aims to learn differentiating features between various classes in a dataset.
For example Support Vector Machines
I Generative
I aims to learn underlying distribution of each class in a dataset. In
otherwords, the goal is to learn the parameterization of distributions of
each class.
I For every input data point, the generative model will find the probability
that the distribution of positive class(+1) generates the given data point
and the probability that the distribution of negativee class (−1) generates
the given data point.
Discriminative vs Generative Models

Figure: Discriminative vs Generative models

I Discriminative
I Directly models the posterior, p(y |x), where x is always given as input
I Generative
I Models the joint distribution, p(x, y ).
I Recall p(y |x) = p(x,y ) = p(x|y )p(y )
p(x) p(x)
I This joint distribution p(x, y ) says the probability that given data point x
generated with class label +1 and the probability that the data point x
generated with class label −1
Generative Models

Two types of Generative Models

I Fully Visible Models:
I Directly model observations without introducing
extra variables.
I For Eg:- Considering each pixel value of image
as an observation, and based on that the model
generate new images
I Latent Variable Models:
I Defining hidden variables which generate
observed data
I Explicit likelihood Estimation Models :
Explicitly define and learn likelihood of
data. Eg:- Variational Autoencoder
I Implicit Estimation Models :Learn to
directly generate samples from models
distribution, without explicitely defining
any density function. Eg:- Generative
Adversarial Networks
Taxonomy of Generative Models

Figure: Taxonomy of Generative models

I Explicit :Assume certain PDF(Prob. density function) and try to learn

parameterization.
I Implicit: No assumption of distribution, but it learn certain density function,
which can generate data that looks similar to what we have seen in our original
training data.
Generative Adversarial Networks(GANs)

I Goals
I Build a good sampler that
allows to draw high quality
samples from pmodel (x), where
pmodel (x) defines distribution of
samples from a model, which is
learned through the algorithm.
I The distribution of generated
samples, pmodel (x), should looks
similar to the original data
distribution, pdata (x)
I Ideally we want the output
samples to be similar but not
exactly same as train data,
because we want to generate
diverse samples beyond what we
already have.
Generative Adversarial Networks(GANs)

I Method
I Introduce a latent variable z
with simple prior p(z)(For eg:
Gaussian )
I Sample z ∼ p(z), pass it
through a Generator x̂ = G (z),
where x̂ ∼ pG . The output of
generator would be images itself
and pG indicates distribution of
images generated by the
generator G .
I Introduce a mechanism to
ensure pG ≈ pdata , without using
any particular parameterization
on pG or pdata .
How to ensure pG ≈ pdata

I Method
I Use a classifier called
discriminator that differentiates
between real samples
x ∼ pdata (class 1) and generated
samples x̂ ∼ pG (class 0).
I Train generator G such that the
discriminator misclassifies
generated sample x̂ into class 1
and eventually the discriminator
can no more differentiate
between x ∼ pdata and x̂ ∼ pG
I Thus, the job of the
discriminator is to separate the
fake or generated samples from
real samples and the job of
generator is to generate samples
that fool the discriminator.
Training of GAN

I Training Objective
:MinG MaxD (Ex∼pdata [logD(x)] + Ez∼pz [log (1 − D(G (z))])
I As far as the discriminator is concerned
I The first part of the objective function MaxD (Ex∼pdata [logD(x)]): for
data coming from the original distribution, the discriminator has to
maximize its log likelihood.
I The second part of the objective function
MaxD (Ez∼pz [log (1 − D(G (z))]): for samples from the Gaussian and
then inputing it to the generator,followed by the generator outputs
to the discriminator, we want ensure that D(G (z)) = 0 so that
Ez∼pz [log (1 − D(G (zid))] is maximized.
I As far as the generator is concerned
I The second part of the objective function
MinG (Ez∼pz [log (1 − D(G (x))]): for samples from the Gaussian, we
want ensure that D(G (x)) = 1 so that Ez∼pz [log (1 − D(G (x))] is
minimized.
I The discriminator has a Sigmoid activation in its output layer, since it has
to classify its inputs as real or fake
Training of GAN

I O1 : MaxD (Ex∼pdata [logD(x)]) I O2 : MinG (Ez∼pz [log (1 − D(G (z))])

I Train the discriminator, D, such I Train the Generator, G , such
that the sample belongs to pdata , that the sample belongs to pG ,
maximize the log probability of maximize its probability of it
it being real sample being real sample
I If we train the discriminator completely to optimize the objective O1 , all
samples from G are identified as fake by the discriminator.
I Solution : Alternate training between discriminator objective O1 and
Generator objective O2 .
Training of GAN
Instance Segmentation vs Semantic Segmentation

Image Segmentation

I Image segmentation is the task of

partitioning image into multiple
regions.
I Partitioning is done based on certain
characteristics of the input pixels.
Pixels from same partion show similar
characteristics. Can be divided into :

I Semantic Segmentation
I Instance Segmentation
Semantic Segmentation
I Semantic segmentation refers to the process of linking each pixel in an
image to a class label
I It is image classification at the pixel level (Each and every pixel is labelled
to a class)
I In an image having many cars, segmentation will label all the objects as
car objects.
I In the example image, all pixels belonging different classes like human,
car, house,sign boards and grass are labelled with different colors.
I Semantic segmentation identifies the class of each pixel, but it cannot
differentiate multiple instances of same object

Figure: Semantic Segmentation

Instance Segmentation
I instance Segmentation :Identification of boundaries of the objects at the
detailed pixel level.
I Example shows the difference between semantic segmentation and
instance segmentation. Left:All pixels belonging to the class persons have
been classified as person. Right : Classified to the classes and identifies
the boundary of each object.

Figure: Instance Segmentation

Use of Semantic Segmentation

I For Autonomous Driving I For Medical Applications

I To navigate through the road, the

car has to know which pixels I Figure shows the brain MRI.
belonging to the road and which Segmentation of white matter, grey
pixels belonging to its matter, Cerebrospinal fluid from brain
surroundings. image are very important in diagnosis
of diseases.
U-Net Architecture
I UNet was a ground breaking discovery in the realm of image segmentation
I Immense value in the analysis of biomedical images.
I A UNet is a special type of Convolutional Neural Network (CNN), composed of
two main components: a contracting path and an expanding path.
U-Net Architecture

I A UNet: Two main components

I Contracting path: aims to decrease the spatial dimensions of the
image, while also capturing relevant information about the image.
I Uses a combination of convolution and pooling layers to
extract and capture features within an image.
I Expanding path: aims to upsample the feature map
I Uses both convolution and up-convolution operations to
combine learnt features and upsample the input feature map
until it generates a segmentation map.
I Contracting path is a five block operation.
I Block1: 572x572 grayscale image is fed into the UNet followed two
3x3 convolutions, each with ReLU, then 2x2 Maxpooling with a
stride 2. No. of channels 64.
I Block2: two 3x3 convolutions, each with ReLU, then 2x2
Maxpooling. Channels :128.
I Block 3 and 4 are same as the previous bolock.
I Block 5: No. of channels increased to 1024. Two 3x3 convolutions,
but one is in expanding path.
U-Net Architecture
I The expanding path uses both convolution and upconvolution operations.

Figure: UNet Expanding path

U-Net Architecture
I Expanding Path
I Block 5:
I Continuing on from the contracting path, a second 3x3
convolution (unpadded) is applied with a ReLU layer after it.
Then 2x2 upconvolution and halving the number of channels
to 512
I Block 4:
I Using skip connections, the corresponding feature map from
the contracting path is then concatenated, doubling the
feature channels to 1024.
I Two 3x3 convolution, with ReLU, and Reducing the channels
to 512
I By 2x2 upconvolution spatial dimension become twofold, and
number of channel to 256.,
I Block 3: Procedure is same as block 4. Bloc 2: Same as block 3.
I Block 1
I 128 channels after concatenation
I Two 3x3 convolutions, ReLU. Channels to 64.
I Finally, a 1x1 convolution layer, followed by an activation layer
(sigmoid for binary classification) is used to reduce the number of
channels to the desired number of classes(here 2 classes).
U-Net Architecture
I UpConvolution
I Also known as a deconvolution or transpose convolution, is a
method used to upsample images and recover spatial information

Figure: UNet Expanding path

I Skip Connections
I Skip connections are used to send images directly from the
contracting path to the expanding path without them having to go
through all the blocks. This allows for both high and low level
features to be preserved and learnt, reducing any information loss
that occurs during the contracting path.
U-Net Architecture

I Image Example
I UNets are often used in medical imaging. They play crucial roles in
detecting and locating tumors, cysts and other abnormalities

Figure: UNet Segmented output