Module IV: Deep Unsupervised Learning
Unsupervised learning involves training models on data without labeled outputs. Deep
unsupervised learning leverages deep architectures to learn high-level representations or
generate data.
1. Autoencoders (AEs)
🔹 Definition:
An autoencoder is a type of neural network used to learn efficient representations (encoding)
of data, typically for dimensionality reduction or denoising.
🔹 Architecture:
Input Layer: Accepts raw input.
Encoder: Maps input to a hidden (latent) representation.
Latent Space (Bottleneck): Compressed knowledge representation.
Decoder: Attempts to reconstruct the original input from the latent space.
Output Layer: Reconstructed version of the input.
🔹 Mathematical Form:
🔹 Types of Autoencoders:
Vanilla Autoencoder
Sparse Autoencoder: Adds a sparsity constraint to encourage the model to activate
only a few neurons.
Denoising Autoencoder: Trains on noisy input but targets original clean input.
Contractive Autoencoder: Penalizes the Jacobian matrix of the encoder to encourage
robustness.
🔹 Applications:
Image compression
Denoising
Dimensionality reduction
Feature learning
2. Variational Autoencoders (VAEs)
🔹 Definition:
VAEs are probabilistic generative models that learn a distribution over the latent
space and allow for data generation.
🔹 Key Idea:
Instead of learning a single point in latent space, VAEs learn a distribution over
possible latent variables.
🔹 Components:
🔹 Applications:
Image generation
Semi-supervised learning
Representation learning
3. Generative Adversarial Networks (GANs)
🔹 Definition:
GANs are a class of generative models where two neural networks, a generator and a
discriminator, compete in a zero-sum game.
🔹 Components:
Generator (G): Generates fake samples from noise.
Discriminator (D): Classifies real vs fake samples.
🔹 Objective:
🔹 Intuition:
G tries to fool D with realistic samples.
D tries to correctly distinguish between real and generated data.
🔹 Variants:
DCGAN (Deep Convolutional GAN)
Conditional GAN
CycleGAN
StyleGAN
🔹 Applications:
Image synthesis
Art and design
Data augmentation
Super-resolution
4. Autoencoder + Deep Belief Networks (DBM/DBN)
🔹 Deep Belief Networks (DBN):
Composed of stacked Restricted Boltzmann Machines (RBMs). Each RBM is trained
unsupervised, and then the network is fine-tuned.
RBM Architecture:
o Input layer and hidden layer
o No intra-layer connections
Training: Contrastive Divergence
🔹 Deep Boltzmann Machine (DBM):
Unlike DBNs, all layers in DBMs are symmetrically connected and trained jointly. They are
energy-based models.
🔹 Use in Autoencoders:
Layer-wise pretraining of autoencoders.
Helps in initializing weights to avoid poor local minima.
5. Attention and Memory Models
🔹 Attention Mechanism:
Used to focus on relevant parts of the input for a specific task. Widely used in NLP and
vision.
🔹 Applications:
Machine translation
Image captioning
Speech recognition
🔹 Formula:
Where:
Q: Query
K: Key
V: Value
🔹 Types:
Soft Attention: Differentiable
Hard Attention: Non-differentiable (requires reinforcement learning)
6. Dynamic Memory Networks (DMNs)
🔹 Architecture:
DMNs integrate memory and attention mechanisms to process and reason over sequential
data like text and video.
🔹 Modules:
Input Module: Encodes input sequences.
Question Module: Encodes the query.
Episodic Memory Module: Uses attention to update memory.
Answer Module: Produces final output (e.g., answer to question).
🔹 Applications:
Visual question answering (VQA)
Dialogue systems
Text comprehension
Concept Purpose Key Feature
Dimensionality
Autoencoder Learns compressed representations
reduction
Learns distributions, probabilistic latent
Variational Autoencoder Generative modeling
space
GAN Data generation Generator + Discriminator adversarial
Concept Purpose Key Feature
game
Deep probabilistic
DBM/DBN Layer-wise pretraining
models
Attention Relevance focus Soft/hard focus on parts of input
Dynamic Memory
Sequential reasoning Memory + attention for reasoning
Network