23-204-0603 DEEP LEARNING

CUSAT Scheme 2023

MODULE II: Deep Learning — Complete Exam-Ready Notes

MODULE II: DEEP LEARNING

This module introduces deep learning as a paradigm, covers the architectural principles of deep
networks, explores key building blocks (RBMs, Autoencoders, VAEs), and explains how deep
networks differ from shallow ones.

1. WHAT IS DEEP LEARNING?

Definition
Deep Learning is a sub-field of Machine Learning that uses artificial neural networks with multiple
layers (hence 'deep') to automatically learn hierarchical representations of data. The term 'deep'
refers to the depth (many layers) of the neural network.
In contrast to traditional machine learning, where feature engineering is done manually, deep
learning learns features automatically from raw data through hierarchical abstraction.

Key Characteristics of Deep Learning

• Automatic Feature Learning: No need for manual feature engineering. The network learns
relevant features from raw data.
• Hierarchical Representations: Lower layers learn simple features (edges, colors), higher
layers learn complex abstractions (shapes, objects, concepts).
• Large Data Requirement: Deep networks typically require large amounts of labeled data to
train effectively.
• Computational Power: Requires GPUs for practical training.
• End-to-End Learning: Can learn directly from raw inputs (pixels, audio) to outputs (labels,
text).

Why Deep Learning Works

• Each additional layer allows the network to compose more complex representations from
simpler ones.
• Non-linear activation functions allow each layer to learn non-linear transformations.
• With enough data and computation, deep networks can learn extremely complex mappings.

Deep Learning vs Traditional Machine Learning

Traditional ML Deep Learning
Manual feature engineering required Automatic feature learning from raw data
Works well with small/medium data Requires large amounts of data
Interpretable models (e.g., decision trees) Often 'black box' models
Shallow architectures (few layers) Deep architectures (many layers)
CPU sufficient for most tasks Requires GPU acceleration
2. COMMON ARCHITECTURAL PRINCIPLES OF DEEP NETWORKS
Deep networks share common design principles regardless of whether they are CNNs, RNNs, or
plain MLPs. Understanding these principles is crucial for designing and analyzing architectures.

2.1 Parameters
Parameters are the learnable variables of a neural network — the weights and biases that are
adjusted during training through backpropagation.
• Weights (W): Determine the strength of connections between neurons.
• Biases (b): Allow shifting the activation function, giving the network more flexibility.
Total Parameters = Sum over all layers of (neurons_in * neurons_out +
neurons_out)
Example: A layer with 256 inputs and 128 outputs has 256*128 + 128 = 32,896 parameters.
• More parameters = more capacity to learn complex functions.
• Too many parameters + too little data = overfitting.
• Too few parameters = underfitting.
The number of parameters defines model capacity. Deeper and wider networks have
more parameters. Modern networks can have billions of parameters (GPT-3: 175 billion).

2.2 Layers in Deep Networks

Deep networks are organized into stacked layers. Each layer type performs a specific operation:
• Input Layer: Receives raw data. No computation. Acts as a pass-through.
• Hidden Layers: Perform feature extraction and transformation. The 'depth' comes from
multiple hidden layers. Each applies: z = W*x + b, then a = f(z).
• Output Layer: Produces the final prediction. Activation function depends on task (linear for
regression, sigmoid for binary classification, softmax for multi-class).
• Depth: The number of hidden layers. Deep = many hidden layers.
• Width: The number of neurons per layer. Wider = more features learned at each level.
Trade-off: Deeper networks learn more abstract features but are harder to train (vanishing
gradients). Wider networks increase capacity but may overfit.

2.3 Activation Functions (Deep Learning Perspective)

In deep networks, the choice of activation function is critical for training stability:
• ReLU is the default for hidden layers: Simple, avoids vanishing gradient for positive inputs,
promotes sparsity.
• Sigmoid/Tanh: Avoided in deep hidden layers due to vanishing gradients. Still useful in
output layers or RNNs.
• Softmax: Only in output layer for multi-class classification.
• Key principle: Must be non-linear. Linear activations collapse the deep network to a single
linear transformation.

2.4 Loss Functions (Deep Learning Perspective)

Loss functions quantify how far the model's predictions are from the truth. In deep learning:
• Task-dependent: Choose the right loss for each task (MSE for regression, cross-entropy for
classification).
• Differentiable: Loss must be differentiable with respect to network outputs so
backpropagation can compute gradients.
 • Numerically stable: Avoid computations that cause overflow/underflow (e.g., log(0)).

2.5 Optimization Algorithms

These algorithms minimize the loss function by iteratively updating weights based on gradients:
• SGD (Stochastic Gradient Descent): Basic optimizer. Simple but can be slow and gets stuck.
• SGD with Momentum: Adds momentum term to smooth updates. v = beta*v + alpha*grad; w
= w - v.
• RMSProp: Adaptively scales learning rate per parameter using exponential moving average
of squared gradients. w = w - (alpha/sqrt(v+eps)) * grad.
• Adam (Adaptive Moment Estimation): Combines momentum and RMSProp. Maintains first
moment (mean) and second moment (variance) of gradients. Most widely used optimizer.
Adam: m = beta1*m + (1-beta1)*grad (first moment)
v = beta2*v + (1-beta2)*grad^2 (second moment)
m_hat = m/(1-beta1^t); v_hat = v/(1-beta2^t)
w = w - alpha * m_hat / (sqrt(v_hat) + eps)
Adam defaults: beta1=0.9, beta2=0.999, eps=1e-8, alpha=0.001
EXAM: Adam is the go-to optimizer for most deep learning tasks. It is adaptive (different
lr per parameter) and combines the benefits of momentum and RMSProp.

2.6 Hyperparameters
Key hyperparameters for deep networks include:
• Learning rate: Step size for gradient descent.
• Number of layers (depth): Controls representation complexity.
• Number of neurons per layer (width): Controls capacity per layer.
• Batch size: Number of samples per gradient update.
• Number of epochs: Number of complete passes through training data.
• Dropout rate: Fraction of neurons to drop during training.
• Regularization coefficient (lambda): Strength of L1/L2 penalty.
• Activation function type: ReLU, tanh, sigmoid, etc.

3. BUILDING BLOCKS OF DEEP NETWORKS

Before deep learning became dominant with gradient-based end-to-end training, key unsupervised
building blocks were used to pre-train deep networks layer by layer. The main building blocks are
Restricted Boltzmann Machines (RBMs), Autoencoders, and Variational Autoencoders.

4. RESTRICTED BOLTZMANN MACHINES (RBMs)

Definition
A Restricted Boltzmann Machine (RBM) is a generative stochastic artificial neural network that can
learn a probability distribution over its inputs. It is a two-layer undirected graphical model consisting
of a visible layer and a hidden layer, with no connections within the same layer — hence
'Restricted'.
Architecture
• Visible Layer (v): Represents the observed data (inputs). Each visible unit encodes a feature
of the input (e.g., a pixel value).
• Hidden Layer (h): Represents latent features learned from the data. Captures higher-level
patterns.
• Connections: Every visible unit is connected to every hidden unit (fully connected between
layers). No connections within the same layer (the restriction).
• Weights (W): Matrix of weights between visible and hidden units.
• Biases: 'a' for visible units, 'b' for hidden units.

Diagram Description (Draw in Exam)

[DIAGRAM]: Draw two rows of circles:
• Top row: 3 circles labeled 'h1, h2, h3' — 'Hidden Layer'
• Bottom row: 4 circles labeled 'v1, v2, v3, v4' — 'Visible Layer'
• Draw lines from EVERY hidden circle to EVERY visible circle (fully connected between
layers)
• No lines within the same row
• Label connections as 'Weights W'
• Write 'No Intra-layer Connections' on the side

Energy Function
RBMs are energy-based models. The energy of a configuration (v, h) is:
E(v,h) = -a^T * v - b^T * h - v^T * W * h
The joint probability of (v, h) is:
P(v,h) = (1/Z) * exp(-E(v,h))
Where Z is the partition function (normalization constant). Lower energy = higher probability.

Conditional Probabilities
P(h_j = 1 | v) = sigmoid(b_j + Sum_i(v_i * W_ij))
P(v_i = 1 | h) = sigmoid(a_i + Sum_j(h_j * W_ij))

Training: Contrastive Divergence (CD)

Training RBMs is done using Contrastive Divergence (CD-k), an approximation of maximum
likelihood:
• Step 1 (Positive Phase): Clamp visible units to training data v. Compute hidden probabilities
P(h|v). Sample h.
• Step 2 (Negative Phase): Reconstruct visible units from h: compute P(v'|h). Sample v'.
Compute hidden from reconstructed: compute P(h'|v').
• Step 3 (Weight Update): DeltaW = alpha * (<v*h> - <v'*h'>), where <> denotes expected
values.
• Step 4: Repeat for all training examples.

Applications of RBMs
• Dimensionality reduction — similar to PCA but non-linear
• Feature learning and representation learning
• Pre-training layers of Deep Belief Networks (DBNs)
• Collaborative filtering (recommendation systems — Netflix Prize solution used RBMs)
• Image and speech recognition (as pre-training step)
Advantages
• Can model complex, multi-modal distributions of data.
• Generative model — can generate new samples similar to training data.
• Useful for unsupervised pre-training of deep networks.

Disadvantages
• Training is slow — Contrastive Divergence is only an approximation.
• Hard to scale to very high-dimensional data.
• Largely replaced by deep autoencoders and GANs in modern practice.

5. AUTOENCODERS
Definition
An autoencoder is an unsupervised neural network that learns to compress (encode) input data into
a compact latent representation and then reconstruct (decode) the original data from that
representation. The goal is to learn efficient data encodings automatically.
Key insight: The bottleneck in the middle forces the network to learn the most important features of
the data.

Architecture
• Encoder: Takes input x and compresses it to a lower-dimensional code z (latent
representation). Maps: x -> z
• Latent Space / Bottleneck / Code: Compressed representation z. Dimension of z <
dimension of x. This is the learned representation.
• Decoder: Takes latent code z and reconstructs the input x_hat. Maps: z -> x_hat

Diagram Description (Draw in Exam)

[DIAGRAM]: Draw an hourglass shape:
• Left: Wide column of circles = 'Input Layer (x)' — e.g., 4 circles
• Middle-left: Smaller column = 'Hidden Encoder Layer'
• Center: Very small column (1-2 circles) = 'Latent Code / Bottleneck (z)'
• Middle-right: Expanding column = 'Hidden Decoder Layer'
• Right: Wide column = 'Output Layer (x_hat)' — same size as input
• Arrows: Left to right
• Label: 'Encoder' on left half, 'Decoder' on right half
• Loss between x and x_hat: 'Reconstruction Loss'

Mathematical Formulation
Encoder: z = f_enc(x) = sigmoid(W_enc * x + b_enc)
Decoder: x_hat = f_dec(z) = sigmoid(W_dec * z + b_dec)
Loss: L = ||x - x_hat||^2 (MSE Reconstruction Loss)
Training: Minimize the reconstruction loss using backpropagation. The network learns to reproduce
its input at the output.

Types of Autoencoders
• Undercomplete Autoencoder: Latent dimension < input dimension. Standard type. Forces
compression and learning of essential features.
 • Sparse Autoencoder: Adds a sparsity penalty to the loss. Even if latent dim >= input dim,
sparsity forces meaningful feature selection. Loss = Reconstruction + lambda * Sparsity
penalty.
• Denoising Autoencoder: Input is corrupted with noise (random zeroing, Gaussian noise).
Network learns to reconstruct the CLEAN original. Forces more robust feature learning.
• Contractive Autoencoder: Adds penalty on the Jacobian of the encoder. Makes
representation robust to small input perturbations.
• Deep Autoencoder: Multiple hidden layers in both encoder and decoder. Can learn highly
non-linear representations.

Applications of Autoencoders
• Dimensionality Reduction: Alternative to PCA, but non-linear. Better for complex data.
• Feature Learning / Representation Learning: Unsupervised pre-training of features.
• Anomaly Detection: Train on normal data. High reconstruction error indicates anomaly.
• Data Denoising: Denoising autoencoders remove noise from images, audio.
• Image Compression: Encoder compresses, decoder decompresses.
• Generative Modeling: With some extensions (VAE), can generate new data samples.

Advantages
• Unsupervised — no labeled data needed.
• Non-linear dimensionality reduction.
• Can handle complex, high-dimensional data.

Disadvantages
• Reconstruction objective doesn't guarantee useful features for downstream tasks.
• Standard autoencoders can't generate new samples easily (VAEs fix this).
• Training can be unstable for very deep architectures.

6. VARIATIONAL AUTOENCODERS (VAEs)

Definition
A Variational Autoencoder (VAE) is a generative model that extends the standard autoencoder by
enforcing a probabilistic structure on the latent space. Instead of encoding input as a single point in
latent space, the encoder outputs a probability distribution (mean and variance) over the latent
space. This allows the model to generate new data samples.

Key Difference from Standard Autoencoders

• Standard AE: Encoder outputs z (a single point). Latent space is unstructured.
• VAE: Encoder outputs mean (mu) and variance (sigma^2). z is SAMPLED from N(mu,
sigma^2). Latent space is continuous and structured.

Architecture
• Encoder (Inference Network q(z|x)): Maps input x to distribution parameters: mu and
log(sigma^2). Outputs two vectors: mean vector and log-variance vector.
• Reparameterization Trick: Sample z = mu + sigma * epsilon, where epsilon ~ N(0, I). This
makes the sampling step differentiable, enabling backpropagation.
• Latent Space: z ~ N(mu, sigma^2). Enforced to be approximately N(0,1) via KL divergence
penalty.
 • Decoder (Generative Network p(x|z)): Maps z back to reconstructed input x_hat. Same
structure as standard autoencoder decoder.

Diagram Description (Draw in Exam)

[DIAGRAM]: Similar to autoencoder but with modifications at bottleneck:
• Left: Input x
• Encoder: Maps x to two outputs: 'mu' and 'log(sigma^2)'
• Sampling box: 'z = mu + sigma * epsilon, epsilon ~ N(0,1)' (Reparameterization)
• Decoder: Maps z to x_hat
• Right: Output x_hat
• Loss: 'Reconstruction Loss + KL Divergence'

VAE Loss Function (ELBO)

The VAE is trained by maximizing the Evidence Lower BOund (ELBO):
ELBO = E[log P(x|z)] - KL(q(z|x) || P(z))
Loss = Reconstruction Loss + KL Divergence Loss
Reconstruction Loss = -E[log P(x|z)] (e.g., MSE or Binary Cross-Entropy)
KL Loss = -0.5 * Sum(1 + log(sigma^2) - mu^2 - sigma^2)
Interpretation:
• Reconstruction Loss: Ensures the decoder can reconstruct inputs accurately.
• KL Divergence Loss: Regularizes the latent space to be close to a standard normal N(0,I).
This ensures the latent space is continuous and well-organized for generation.

Reparameterization Trick (Crucial for Training)

Problem: Sampling z from N(mu, sigma^2) is not differentiable — can't backpropagate through.
Solution: Instead of sampling z directly, compute:
z = mu + sigma * epsilon, where epsilon ~ N(0, I)
Now mu and sigma are deterministic functions of x, and epsilon is the random part. Gradients can
flow through mu and sigma. Only epsilon is random.
EXAM TIP: Reparameterization trick is THE key innovation in VAEs that enables training
with backpropagation. Without it, the sampling step blocks gradient flow.

Applications of VAEs
• Image Generation: Generate new, realistic images by sampling z ~ N(0,I) and decoding.
• Image Editing: Interpolate between two images in latent space.
• Representation Learning: Structured latent space useful for downstream tasks.
• Anomaly Detection: Samples with high reconstruction error or large KL are anomalies.
• Drug Discovery: Generate new molecular structures.

Comparison: AE vs VAE vs GAN

Property Autoencoder VAE GAN
Generative? No Yes Yes
Latent Space Unstructured point Continuous (Gaussian) N/A
Training Reconstruction loss ELBO (Recon + KL) Adversarial game
Sample Quality N/A Good Excellent
Training Stability Stable Stable Unstable
Mode Coverage N/A Good Mode collapse risk
MODULE II: KEY SUMMARY FOR EXAM
Topic Key Concept
Deep Learning Multiple layers, automatic feature learning, hierarchical representations
Parameters Learnable weights and biases; capacity scales with layers and width
Adam Optimizer Combines momentum + RMSProp; adaptive lr per parameter; most
popular
RBM 2-layer undirected graphical model; visible + hidden; trained via
Contrastive Divergence
RBM Energy E(v,h) = -a^T*v - b^T*h - v^T*W*h
Autoencoder Encoder-Bottleneck-Decoder; learns compressed representations;
unsupervised
AE Loss Reconstruction Loss = ||x - x_hat||^2 (MSE)
VAE Probabilistic AE; encoder outputs mu and sigma; reparameterization trick
VAE Loss ELBO = Reconstruction Loss + KL Divergence
KL Divergence Regularizes latent space to be N(0,1); enables generation
Reparameterization z = mu + sigma*epsilon; makes sampling differentiable

END OF MODULE II — DEEP LEARNING