Deep Learning

BITS Pilani Dr. Pratik Narang

Pilani Campus Department of CSIS
 BITS Pilani
Pilani Campus

Dimensionality reduction
Dimensionality reduction

• In machine learning, dimensionality reduction is the process of

reducing the number of features that describe some data.

• This reduction is done either by selection (only some existing

features are conserved) or by extraction (a reduced number of new
features are created based on the old features)

• Useful in many situations that require low dimensional data (data

visualisation, data storage, heavy computation…).

• Commonly used approaches: PCA, ICA

BITS Pilani, Pilani Campus

Dimensionality reduction

Let’s call encoder the process that produce the “new features”
representation from the “old features” representation (by selection or
by extraction) and decoder the reverse process.
Dimensionality reduction can then be interpreted as data compression
where the encoder compress the data (from the initial space to the
encoded space, also called latent space) whereas the decoder
decompress them.

Source: [Link] BITS Pilani, Pilani Campus

Principal components analysis (PCA)

The idea of PCA is to build n_e new independent features

that are linear combinations of the n_d old features and
so that the projections of the data on the subspace
defined by these new features are as close as possible to
the initial data (in term of euclidean distance).
In other words, PCA is looking for the best linear subspace
of the initial space (described by an orthogonal basis of
new features) such that the error of approximating the
data by their projections on this subspace is as small as
possible.

BITS Pilani, Pilani Campus

 BITS Pilani
Pilani Campus

Latent variable models

What is a latent variable?

Myth of the Cave

Plato, Republic

BITS Pilani, Pilani Campus

 BITS Pilani
Pilani Campus

Autoencoders
Typical DNNs characteristics

So far, Deep Learning Models have things in common:

• Input Layer: (maybe vectorized), quantitative
representation
• Hidden Layer(s): Apply transformations with nonlinearity
• Output Layer: Result for classification, regression,
translation, segmentation, etc.
• Models used for supervised learning

BITS Pilani, Pilani Campus

Example

Source: [Link] BITS Pilani, Pilani Campus

Changing the objective!

Now we will talk about unsupervised learning with Deep Neural Networks

Source: [Link] BITS Pilani, Pilani Campus

 Autoencoders: definition

Autoencoders are neural networks that are trained to copy their inputs to
their outputs.

• Usually constrained in particular ways to make this task more difficult.

• They compress the input into a lower-dimensional code and then

reconstruct the output from this representation. The code is a compact
“summary” or “compression” of the input, also called the latent-space
representation.

• Structure is almost always organized into encoder network, f, and

decoder network, g : model = g(f(x))

BITS Pilani, Pilani Campus

Autoencoders

A special type of feed forward neural

network which does the following
Encodes its input xi into a hidden
representation h

Decodes the input again from this hidden representation

The model is trained to minimize a loss function which will

ensure that ˆxi is close to xi

BITS Pilani, Pilani Campus

Autoencoders for representation learning

BITS Pilani, Pilani Campus

Autoencoders

Autoencoders are mainly a dimensionality reduction (or compression)

algorithm with a couple of important properties:

• Data-specific: They are only able to meaningfully compress data

similar to what they have been trained on. Since they learn features
specific for the given training data, they are different than a standard
data compression algorithm like gzip.

• Lossy: The output of the autoencoder will not be exactly the same
as the input, it will be a close but degraded representation.

• Unsupervised: Autoencoders are considered an unsupervised

learning technique since they don’t need explicit labels to train on.

BITS Pilani, Pilani Campus

Reconstruction quality

BITS Pilani, Pilani Campus

Autoencoders

• Let us consider the case where

dim(h) < dim(xi)
• If we are still able to reconstruct ˆxi
perfectly from h, then what does it
say about h?
• h is a loss-free encoding of xi. It
captures all the important
characteristics of xi
• Do you see an analogy with PCA?

• An autoencoder where dim(h) < dim(xi) is called an

under-complete autoencoder
BITS Pilani, Pilani Campus
Autoencoders

• Let us consider the case when

dim(h) ≥ dim(xi)

• In such a case the autoencoder

could learn a trivial encoding by
simply copying xi into h and then
copying h into ˆxi

BITS Pilani, Pilani Campus

Autoencoders

• Let us consider the case when

dim(h) ≥ dim(xi)

• In such a case the autoencoder

could learn a trivial encoding by
simply copying xi into h and then
copying h into ˆxi

BITS Pilani, Pilani Campus

Autoencoders

• Let us consider the case when

dim(h) ≥ dim(xi)

• In such a case the autoencoder

could learn a trivial encoding by
simply copying xi into h and then
copying h into ˆxi

BITS Pilani, Pilani Campus

Autoencoders

• Let us consider the case when

dim(h) ≥ dim(xi)

• In such a case the autoencoder

could learn a trivial encoding by An autoencoder where
simply copying xi into h and then dim(h) ≥ dim(xi) is called an
copying h into ˆxi over-complete autoencoder

• Such an identity encoding is

useless in practice as it does not Where can they be useful?
really tell us anything about the
important characteristics of the
data
BITS Pilani, Pilani Campus
Autoencoders

Further ahead:

Choice of f(xi) and g(xi)

Choice of loss function

BITS Pilani, Pilani Campus

 BITS Pilani
Pilani Campus

Choice of f(xi) and g(xi)

Autoencoders

• Suppose all our inputs are binary (each xij ∈ {0, 1})
• Which function would be most apt for the decoder?
• ˆxi = tanh(W∗h + c)
• ˆxi = W∗h + c
• ˆxi = logistic(W∗h + c) g is typically chosen as the sigmoid function

BITS Pilani, Pilani Campus

 Autoencoders

• Suppose all our inputs are real (each xij ∈ R)

• Which function would be most apt for the decoder?
• ˆxi = tanh(W∗h + c)
• ˆxi = W∗h + c Again, g is typically chosen as the
• ˆxi = logistic(W∗h + c) sigmoid function

What will logistic and tanh functions do?

They will restrict the reconstructed ˆxi
to lie between [0,1] or [-1,1] whereas
we want ˆxi ∈ Rn

BITS Pilani, Pilani Campus

 BITS Pilani
Pilani Campus

Choice of loss function

 Autoencoders

• Consider the case when the inputs are

real valued
• The objective of the autoencoder is to
reconstruct ˆxi to be as close to xi as
possible
• We can formalize this using the
objective function:

We can train the autoencoder just like a regular feedforward network using
backpropagation. All we need is a formula for ∂L(θ)/ ∂W∗ and ∂L(θ) /∂W

BITS Pilani, Pilani Campus

 Autoencoders

• Consider the case when the

inputs are binary
• We use a sigmoid decoder
which will produce outputs
between 0 and 1, and can be
interpreted as probabilities.
• For a single n-dimensional ith Again, all we need is a formula for ∂L (θ)/ ∂W∗
input we can use the following and ∂L (θ)/ ∂W to use backpropagation
loss function

What value of ˆxij will minimize this function?

The above function will be minimized when ˆxij = xij

BITS Pilani, Pilani Campus
 BITS Pilani
Pilani Campus

Types of Autoencoders
Undercomplete Autoeconders

Undercomplete Autoeconders are defined to have a hidden layer h, with

smaller dimension than input layer.
• Network must model x in lower dimensional space + map latent space
accurately back to input space.
• Encoder network: function that returns a useful, compressed representation
of input.
• If network has only linear transformations, encoder learns PCA. With typical
nonlinearities, network learns generalized, more powerful version of PCA.

Source: [Link] BITS Pilani, Pilani Campus

Architecture

Source: [Link] BITS Pilani, Pilani Campus

Training

Four hyperparameters need to be set before training an autoencoder:

Code size: number of nodes in the middle layer. Smaller size results in
more compression.
Number of layers: the autoencoder can be as deep as we like
Number of nodes per layer: a stacked autoencoder is one where the
layers are stacked one after another. Usually stacked autoencoders look
like a “sandwich”. The number of nodes per layer decreases with each
subsequent layer of the encoder, and increases back in the decoder. Also
the decoder is symmetric to the encoder in terms of layer structure.
Loss function: we either use mean squared error (MSE) or binary
crossentropy. If the input values are in the range [0, 1] then we typically
use crossentropy, otherwise we use the mean squared error.

BITS Pilani, Pilani Campus

Training

• We can make the autoencoder very powerful by increasing the number

of layers, nodes per layer and most importantly the code size.
• Increasing these hyperparameters will let the autoencoder to learn
more complex codings.
• But we should be careful to not make it too powerful. Otherwise the
autoencoder will simply learn to copy its inputs to the output, without
learning any meaningful representation. It will just mimic the identity
function.
• This is why we prefer a “sandwich” architecture, and deliberately keep
the code size small.
• Since the coding layer has a lower dimensionality than the input data,
the autoencoder is said to be undercomplete. It won’t be able to directly
copy its inputs to the output, and will be forced to learn intelligent
features

BITS Pilani, Pilani Campus

 Denoising autoencoders

Another way to force the autoencoder to learn useful

features is by adding random noise to its inputs and
making it recover the original noise-free data.

• This way the autoencoder can’t simply copy the input to

its output because the input also contains random noise.
• We are asking it to subtract the noise and produce the
underlying meaningful data.
• This is called a denoising autoencoder.

• Robustness: Extends well to real-world tasks such as

removing background noise from audio, or cracks/stains
in images. BITS Pilani, Pilani Campus
Example

Source: [Link] BITS Pilani, Pilani Campus

Denoising autoencoders

• We introduce a corruption process C(˜x | x), which represents a

conditional distribution over corrupted samples ˜x, given a data
sample x. The autoencoder then learns a reconstruction
distribution preconstruct (x | ˜x) estimated from training pairs (x,˜x) as
follows:

• Sample a training example x from the training data.

• Sample a corrupted version ˜x from C(˜x | x=x),

• Use (x,˜x) as a training example for estimating the autoencoder

reconstruction distribution preconstruct(x |˜x) =pdecoder (x | h) with h the
output of encoder f(˜x) and pdecoder typically defined by a decoder g(h)

BITS Pilani, Pilani Campus

Sparse autoencoders

Force the autoencoder to learn useful features using regularization

We can regularize the autoencoder by using a sparsity constraint such
that only a fraction of the nodes would have nonzero values, called
active nodes.
Add a penalty term to the loss function such that only a fraction of the
nodes become active. This forces the autoencoder to represent each
input as a combination of small number of nodes, and demands it to
discover interesting structure in the data.
This method works even if the code size is large, since only a small
subset of the nodes will be active at any time.
Biological Inspiration: Only a small percentage of neurons fire
simultaneously in the brain.
Practical Effect: The model focuses on the most salient features in the
data, often improving interpretability.

BITS Pilani, Pilani Campus

Contractive autoencoders

Goal: Make learned features robust to small changes in the input.

• In standard AEs, small pixel noise can cause large changes in hidden
representation ℎ.
• CAEs penalize such sensitivity — encouraging the encoder to map nearby
inputs to nearby latent codes.
• This makes the latent space smoother and preserves the manifold structure
of the data.

BITS Pilani, Pilani Campus

Contractive autoencoders

The goal is to learn Stable and Smooth Representations

A Contractive Autoencoder adds a regularization term to make the hidden

representation less sensitive to small changes in the input.

This encourages the encoder to learn a smooth latent space — where nearby
inputs map to nearby hidden codes.
Loss function:
2
𝜕ℎ 2
𝐿 =∥ 𝑥 − 𝑥ො ∥ +𝜆 ∥ ∥
𝜕𝑥 𝐹
∥ 𝑥 − 𝑥ො ∥2 :the reconstruction loss
𝜕ℎ
∥ ∥2𝐹 :a sensitivity penalty — measures how much the hidden code ℎ changes if
𝜕𝑥
we slightly change the input 𝑥.
𝜆: a weight that controls how strongly we penalize this sensitivity.

BITS Pilani, Pilani Campus

Contractive autoencoders

Benefits and Uses of Contractive Autoencoders

Smooth latent space: Neighboring inputs → similar codes.

Noise robustness: Model resists input perturbations (even without explicit

noise like in Denoising AE).

Manifold learning: Captures local geometry of data distribution.

Applications:
– Feature extraction for clustering or classification
– Anomaly detection (stable representations highlight outliers)
– Pretraining step for deeper models

BITS Pilani, Pilani Campus

Convolutional autoencoders

• Standard autoencoders flatten images into vectors.

• Convolutional Autoencoders maintain the spatial layout, using
convolution/deconvolution (transposed convolution) layers.
• Better for image data, as the network can capture local patterns (edges,
textures).
• Often yields superior reconstruction quality vs. fully connected
autoencoders.
• Use Cases:
• Denoising of images (e.g., removing Gaussian noise, image inpainting).
• Feature Extraction: The encoder’s output can serve as a learned
representation for classification tasks.

BITS Pilani, Pilani Campus

Case study: MAEs

2022
BITS Pilani, Pilani Campus
MAEs

• Instead of feeding the whole image into the autoencoder, a large

random subset of patches is masked (removed) from the input.
• The autoencoder only “sees” a fraction (e.g., 25%) of the original
image patches.

• Encoder: Processes the visible patches only (the unmasked

subset), encoding them into latent embeddings.

• Decoder: Takes the latent embeddings (plus positional information

for where each patch belongs) and reconstructs the entire image,
including the masked patches.

BITS Pilani, Pilani Campus

MAEs

We drop most of the input patches before encoding, then

train the network to restore the missing content.

This increases the difficulty of the reconstruction task and

encourages the model to learn rich, high-level features
of the data.

BITS Pilani, Pilani Campus

 MAEs: architecture overview

• Patch Embedding: Split the image into patches (e.g., 16×16 pixels)
and embed them (like a small linear projection or convolution). Sample:
196 patches in a 224×224 image with 16×16 patches.
• Random Masking: Randomly select a large portion (e.g., 75%) of
these patches to remove. Remaining subset fed into the encoder.
• Encoder (Transformer or CNN): The encoder only processes the
visible (unmasked) tokens.
• Decoder: A smaller or simpler network that receives the latent
embeddings of the visible patches + learnable placeholders for the
masked patches. It reconstructs the full set of patches (both originally
visible and masked).
• Loss Function: Typically Mean Squared Error (MSE) between
reconstructed and original pixel patches (only for the masked patches,
to focus the learning on what was not seen).

BITS Pilani, Pilani Campus

 Key contributions

• High Masking Ratio

• By masking most of the image, the autoencoder is forced to learn
global structures and semantic context, not just local textures.
• Leads to representations that generalize well to downstream tasks
(e.g., classification, detection) once you fine-tune the encoder.

• Efficiency
• The encoder only processes a fraction of the tokens, making the
training more computationally feasible than if it had to process the
entire image.

• Self-Supervised
• No labels needed. The task is purely “reconstruct missing patches,”
which can be done with any unlabeled image dataset.

BITS Pilani, Pilani Campus

 Training MAEs

• Forward Pass
• Sample an image, mask out patches.
• Encode visible patches → produce latent vectors.
• Decode latent vectors + masked placeholders → reconstruct the missing
patches.

• Loss Computation
• Compare reconstructed masked patches vs. the original masked patches.
• Typical metric is MSE loss for pixel-wise difference.

• Backpropagation
• Update encoder + decoder weights to minimize reconstruction error.

• Representation Learning
• After training, we can discard the decoder and use the encoder’s latent
representation for downstream tasks (e.g., image classification). Often, we
just fine-tune the encoder on a small set of labeled images.
BITS Pilani, Pilani Campus
Visual results

Source: [Link]
BITS Pilani, Pilani Campus
Case study: U-net for image segmentation

BITS Pilani, Pilani Campus

U-net architecture

BITS Pilani, Pilani Campus

 U-net architecture

• Encoder (Contraction Path)

• A series of convolution + pooling layers that capture what is in the image
(context).
• Gradually reduces the spatial dimensions while increasing feature depth.

• Decoder (Expansion Path)

• A series of transposed convolutions (or up-convolutions) that upscale the
feature maps.
• Recovers the spatial resolution needed for a segmentation map (pixel-
level predictions).

• Skip Connections
• Directly connect encoder feature maps to decoder layers at the same
scale.
• Helps retain detailed spatial information lost in pooling.
• Greatly improves segmentation accuracy, especially for medical imaging.

BITS Pilani, Pilani Campus

 U-net architecture

Source: [Link] BITS Pilani, Pilani Campus

U-net is an autoencoder?

• Encoder-Decoder Concept:
• Like an autoencoder, U-Net “encodes” the input into a
lower-dimensional feature space and then “decodes” it
back to an image-sized output.

• Primary Difference:
• Instead of reconstructing the same input, U-Net is
reconstructing a segmentation mask (i.e., class labels for
each pixel).

BITS Pilani, Pilani Campus

Use cases

• Medical Image Segmentation: Segmenting organs or

tumors in MRI or CT scans (the original U-Net paper
focuses on biomedical images).

• Autonomous Driving: Segmenting roads, cars,

pedestrians in real-time from camera feeds.

• Satellite Image Analysis: Detecting buildings, roads,

vegetation from aerial imagery.

BITS Pilani, Pilani Campus

Encoder-Decoder Evolution

Autoencoders → Variational Autoencoders (probabilistic)

Denoising AEs → foundation for self-supervised learning

MAE (masked AE) → direct extension to transformer-based SSL

BITS Pilani, Pilani Campus

Applications of Autoencoders

Data denoising

Dimensionality reduction

Information retrieval

Anomaly detection

Content generation (Generative models) – VAEs

BITS Pilani, Pilani Campus

 BITS Pilani
Pilani Campus

Variational Autoencoders
Content generation

One could be tempted to think that, if the latent space is

regular enough (well “organized” by the encoder during
the training process), we could take a point randomly
from that latent space and decode it to get a new content.

Does this work?

BITS Pilani, Pilani Campus

Once the autoencoder is trained, can I remove the
encoder, feed a hidden representation h to the
decoder, and decode a Xˆ from it ?

BITS Pilani, Pilani Campus

Content generation

The regularity of the latent space for autoencoders is a difficult point that
depends on:
• the distribution of the data in the initial space,
• the dimension of the latent space,
• the architecture of the encoder.

So, it is pretty difficult (if not impossible) to ensure, a priori, that the
encoder will organize the latent space in a smart way compatible with
the generative process.
The autoencoder is solely trained to encode and decode with as few loss
as possible, no matter how the latent space is organised.
Thus, it is natural that, during the training, the network takes advantage of
any overfitting possibilities to achieve its task as well as it can.

BITS Pilani, Pilani Campus

Content generation

• Autoencoders learn a hidden representation h but not a

distribution P(h|X)
• Similarly the decoder is also deterministic and does not
learn a distribution over X (given a h, we can get a X but
not P(X|h) )
• We are interested in sampling from P(h|X)

BITS Pilani, Pilani Campus

Variational autoencoders

• To use the decoder of the autoencoder for generative

purpose, we have to be sure that the latent space is
regular enough.
• One possible solution to obtain such regularity is to
introduce explicit regularisation during the training
process.
• A variational autoencoder can be defined as being an
autoencoder whose training is regularised to avoid
overfitting and ensure that the latent space has good
properties that enable generative process.
• Instead of encoding an input as a single point, we encode
it as a distribution over the latent space

BITS Pilani, Pilani Campus

VAEs

Source: [Link] BITS Pilani, Pilani Campus

VAEs

As a standard autoencoder, a VAE is composed of both an encoder

and a decoder and that is trained to minimise the reconstruction error
between the encoded-decoded data and the initial data.
However, in order to introduce some regularisation of the latent space,
we proceed to a slight modification of the encoding-decoding
process: instead of encoding an input as a single point, we
encode it as a distribution over the latent space.
The model is then trained as follows:
• the input is encoded as distribution over the latent space
• a point from the latent space is sampled from that distribution
• the sampled point is decoded and the reconstruction error can be
computed
• the reconstruction error is backpropagated through the network

BITS Pilani, Pilani Campus

VAEs

Let {X = xi} (i=1 to N) be the training data

We can think of X as a random variable in Rn
Eg: X could be an image, the dimensions of X correspond to pixels
of the image
We are interested in learning an abstraction (i.e., given an X find the
hidden representation z)
We are also interested in generation (i.e., given a hidden
representation, generate an X)
In probabilistic terms we are interested in P(z|X) and P(X|z)

BITS Pilani, Pilani Campus

VAEs

Goal 1: Learn a distribution over the

latent variables (Q(z|X))
Goal 2: Learn a distribution over the
visible variables (P(X|z))
VAEs use a neural network based
encoder for Goal 1 and a neural
network based decoder for Goal 2

BITS Pilani, Pilani Campus

Encoder

What do we mean when we say we want to learn a distribution?

We mean that we want to learn the parameters of the distribution
But what are the parameters of Q(z|X)?
Well, it depends on our modeling assumption!
We assume that the latent variables come from a standard normal
distribution N (0, I) and the job of the encoder is to then predict the
parameters of this distribution

BITS Pilani, Pilani Campus

 Decoder

The job of the decoder is to predict a

probability distribution over X : P(X|z)
Again, we will assume a certain form for
this distribution
We could assume that P(X|z) is a
Gaussian distribution with unit variance
The job of the decoder f would then be to
predict the mean of this distribution as
fφ(z)

BITS Pilani, Pilani Campus

VAEs

In practice, the encoded distributions are

chosen to be normal so that the encoder can
be trained to return the mean and the
covariance matrix that describe these
Gaussians.

The reason why an input is encoded as a

distribution with some variance instead of a
single point is that it makes possible to
naturally express the latent space
regularisation: the distributions returned by
the encoder are enforced to be close to a
standard normal distribution.

Source: [Link] BITS Pilani, Pilani Campus

VAEs

Essentially, the latent spaces of VAEs

are, by design, continuous, allowing
easy random sampling and interpolation.

VAEs achieve this by: making its encoder

not output an encoding vector of
size n, rather, outputting two vectors of
size n: a vector of means, μ, and
another vector of standard deviations, σ.

Source: [Link] BITS Pilani, Pilani Campus

VAEs

However …

Source: [Link] BITS Pilani, Pilani Campus

VAEs

In order to achieve continuous latent space as desired, the Kullback–

Leibler divergence (KL divergence) is introduced into the loss
function as a regularization term.
The KL divergence between two probability distributions simply
measures how much they diverge from each other.
Minimizing the KL divergence here means optimizing the probability
distribution parameters (μ and σ) to closely resemble that of the
target distribution.
This loss encourages the encoder to distribute all encodings (for all
types of inputs, eg. all digits), evenly around the center of the latent
space.
If it tries to “cheat” by clustering them apart into specific regions, away
from the origin, it will be penalized.

BITS Pilani, Pilani Campus

VAEs

Using just the KL loss results in a latent space results in encodings

densely placed randomly, near the center of the latent space, with
little regard for similarity among nearby encodings.
The decoder finds it impossible to decode anything meaningful from
this space, simply because there really isn’t any meaning.
Optimizing the KL loss with the reconstruction loss: results in the
generation of a latent space which maintains the similarity of nearby
encodings on the local scale via clustering, yet globally, is very
densely packed near the latent space origin (compare the axes with
the original).
This is the equilibrium reached by the cluster-forming nature of the
reconstruction loss, and the dense packing nature of the KL loss,
forming distinct clusters the decoder can decode.

BITS Pilani, Pilani Campus

VAEs

The continuity and completeness obtained with

regularisation tend to create a “gradient” over the
information encoded in the latent space.

Source: [Link] BITS Pilani, Pilani Campus

VAEs

Interpolating between samples Adding new features to samples

Source: [Link] BITS Pilani, Pilani Campus

Beta-VAEs

• Introduce a hyperparameter β that scales the KL divergence term in

the VAE loss.

• Effect:
• Larger β ⇒ more emphasis on latent-space disentanglement (forces
independence between latent dimensions).
• Potentially better interpretability but might hurt reconstruction quality.

• Example: Used to learn features like shape, rotation, or style in

images, each dimension controlling a different factor of variation.

BITS Pilani, Pilani Campus

input image

New generated content!!

Source: [Link]

Reconstructed image BITS Pilani, Pilani Campus

 BITS Pilani
Pilani Campus

Thank you!