Deep Learning & GenerativeAI Assignment

Name : Atharva Raibagi

SAP ID : 86092400006

Roll No: A006

1. Explain the difference between Hopfield networks and Boltzmann machines in how
their energy works and how they converge. Say where each is more useful.

Hopfield networks and Boltzmann machines are both classical neural models that rely on an
energy minimization perspective to perform computation. Even though they share the basic
idea that network dynamics are driven toward states with lower energy, the type of energy
function they use, the way neurons update their states, and the convergence properties of
the models differ significantly. These differences affect where each model is applied and
how effective they are in solving particular problems.

Hopfield networks are deterministic recurrent neural networks where every neuron is fully
connected to the others with symmetric weights. They operate on binary states like +1 or -1
and update their neurons asynchronously. The core principle behind Hopfield networks is
that they behave like a dynamical system with an energy function defined mathematically.
During operation, the network adjusts itself so that the total energy keeps decreasing after
each neuron update. Because the weights are symmetric and the update rule ensures that
energy reduces at every step, the network is guaranteed to converge to a stable state known
as an attractor. That state represents a stored memory pattern.

Hopfield networks are widely recognized for functioning as an associative memory system.
When given an incomplete or noisy input, they automatically relax into the closest stored
pattern in terms of energy. The reduction in energy corresponds to the model removing
noise and retrieving the correct memory. However, the capacity of Hopfield networks to
store patterns is limited. Typically, the maximum number of stable patterns is around 0.14
times the number of neurons. Beyond that, the energy landscape becomes cluttered with
spurious minima, causing errors in memory recall. Their deterministic nature also restricts
them from exploring beyond local minima, meaning they may converge to suboptimal
patterns if the initialization is poor.

Boltzmann machines extend Hopfield networks through the introduction of randomness

and hidden units. They are stochastic models guided by probabilities instead of strict
energy reduction rules. A Boltzmann machine contains visible units representing input data
and hidden units that discover deeper structures. Like Hopfield networks, the model defines
an energy function for each state. However, neurons update their states stochastically using
a probability determined by the Boltzmann distribution. This means that occasionally the
energy may increase during learning. That temporary increase provides the model with the
ability to escape shallow local minima and continue searching for better solutions. Over
repeated sampling, the network moves toward a thermodynamic equilibrium distribution
in which states with lower energy are more likely to occur.

This stochastic convergence behavior makes Boltzmann machines more powerful for
optimization problems and generative modeling. They can handle uncertainty and multiple
valid outcomes. However, training Boltzmann machines is computationally expensive
because calculating the exact energy distribution requires summing over all possible states.
This makes real-world training slow and limited to small architectures. To reduce
complexity, Restricted Boltzmann Machines remove intra-layer connections, simplifying
computation. RBMs later contributed significantly to early deep learning through Deep
Belief Networks.

In application areas, Hopfield networks are better suited for tasks involving fast retrieval of
stored patterns like memory completion and noise removal. They are simple, fully
deterministic, and converge quickly. Boltzmann machines, on the other hand, are more
useful when learning internal representations, generating new samples, or solving
combinatorial optimization problems where exploring alternative possibilities is essential.

In summary, Hopfield networks deterministically descend energy and converge to a fixed

attractor, making them ideal for associative memory. Boltzmann machines use stochastic
state changes, allowing them to explore the global energy landscape at the cost of
computation, making them suitable for generative modeling, feature learning, and
optimization. Both models are built around the idea that learning is equivalent to shaping
an energy landscape, but their differences in convergence behavior define their suitability
for different problem domains.

2. Define kernel, stride, padding, and receptive field in CNNs, and explain how they
affect the features a model learns.

Convolutional Neural Networks process visual and spatial information using layers that
capture patterns at different scales. Four essential concepts that define how a CNN extracts
features from images are kernel, stride, padding, and receptive field. These parameters
determine the resolution, coverage, and amount of detail the network learns across layers.

A kernel, also called a filter, is a small matrix of learnable weights that slides over the input
image to compute dot products with the underlying pixel region. This produces a feature
map highlighting specific visual structures such as edges, corners, or textures. Multiple
kernels in the same layer learn different feature types. The choice of kernel size affects the
kinds of information captured. A 3×3 kernel extracts finer local patterns. Larger kernels like
5×5 or 7×7 capture broader contextual information but increase computational cost.

Stride refers to how many pixels the kernel moves after each computation. If the stride is 1,
the kernel moves one pixel at a time, preserving more spatial detail and producing larger
feature maps. If stride is 2 or greater, the feature map becomes smaller due to more
aggressive downsampling. High stride values help reduce computation but may cause the
model to miss fine features.

Padding involves adding additional pixels, often zeros, around the border of an image
before convolution. Since a kernel cannot fully cover pixels near edges without additional
space, padding prevents shrinking of feature map dimensions through repeated
convolution. It also ensures that edge features are preserved rather than being lost. Same
padding maintains the original size, while valid padding reduces size but discards edge
information.

The receptive field refers to the area of the input image that influences a neuron’s activation
in deeper layers. As data moves through multiple convolutions and pooling operations, each
neuron ends up covering a larger region of the original input. Small receptive fields in early
layers detect low-level structures like curves or line orientations, while deeper layers
integrate multiple smaller features into meaningful shapes and objects. A properly designed
receptive field ensures that a network captures both detailed and global aspects of an
image.

These components work together to shape the learning behavior of CNNs. Larger kernels or
more pooling can increase the receptive field faster but may blur fine details. High stride
reduces feature map size, increasing speed but reducing resolution. Padding ensures edge
features are not ignored. Smaller kernels with deeper networks allow hierarchical learning:
simple features in early layers build into complex ones later. Therefore, kernel size, stride
value, padding strategy, and receptive field growth directly affect accuracy, computational
efficiency, and the richness of features learned in CNN-based models.

3. Compare VGG16, ResNet, and Inception in a simple way (depth, parameters, and
main idea). Suggest one for real-time mobile use and justify briefly.

VGG16, ResNet, and Inception are three influential CNN architectures that reflect different
strategies to build high-performing deep models. Their primary differences involve network
depth, parameter size, design pattern, and computational efficiency.

VGG16 consists of 16 layers and uses a straightforward architecture where multiple 3×3
convolution layers are stacked to gradually extract features. The simplicity of the design
made VGG16 a useful foundation for transfer learning. However, this simplicity results in a
large number of parameters, around 138 million, which requires significant memory and
computation. It performs well in accuracy but is slow and not suitable for devices with
limited resources.

ResNet introduced residual learning through skip or shortcut connections that allow
gradients to flow easily through deep layers. This eliminates the vanishing gradient
problem that prevented earlier networks from becoming very deep. ResNet architectures
can reach 50, 101, or even more layers. Despite the depth, they remain easier to train and
often have fewer parameters than VGG16 for similar accuracy. The main idea is that instead
of learning full transformations, the network learns residual differences, simplifying
optimization.

Inception networks take a different approach by using parallel convolutional operations

with different kernel sizes in a single block. An Inception block processes features at
multiple scales simultaneously. Bottleneck layers reduce parameter counts and
computation. This architectural efficiency enabled high accuracy with fewer parameters and
faster performance compared to conventional networks. Inception models like Inception V1
or V3 became popular in applications requiring a balance between speed and accuracy.

For real-time mobile use, Inception architectures or lightweight variants such as MobileNet
are preferred because they reduce computational load through efficient design. ResNet
variants like ResNet-18 may also be feasible for smartphones due to moderate size and
strong performance. VGG16 is typically avoided for mobile deployment due to its heavy
memory footprint and slower inference speed, which leads to practical challenges in
latency-sensitive applications.

4. In transfer learning, explain the difference between feature extraction and fine
tuning. Describe when you would freeze or unfreeze layers.

Transfer learning leverages knowledge from a model pretrained on a large dataset and
applies it to another task with limited data availability. Two common strategies in transfer
learning are feature extraction and fine tuning. The choice of method depends on how
similar the new task is to the original one and how much new data is available.

Feature extraction involves using the pretrained network as a fixed feature extractor.
Layers that learned to detect general visual structures such as edges, patterns, and shapes
are kept frozen. Only the final classification layers are replaced and trained on the new
dataset. Since the majority of the network remains unchanged, training becomes faster and
requires less data. This strategy is suitable when the new task is similar to the original one
and when the dataset is small, since the pretrained model already contains useful
knowledge that should not be altered.

Fine tuning refers to unfreezing some or all pretrained layers and retraining them on the
new data. This process allows the network to modify its internal feature representation to
better match the new task. Fine tuning is beneficial when the new dataset differs
substantially from the original or when more data is available to prevent overfitting.
However, it requires careful adjustment of learning rates, since changing too much can
damage previously learned general features and result in degraded performance.

Typically, a hybrid approach is used. Early layers that extract low-level features are frozen
because such features are universal across image types. Deeper layers are selectively
unfrozen when the new dataset contains distinct high-level patterns that the pretrained
model did not previously learn. Freezing or unfreezing layers thus depends on dataset size,
task similarity, computing resources, and the performance requirements of the final model.

In summary, feature extraction emphasizes using existing learned representations with

minimal modification, while fine tuning involves deeper adaptation of the pretrained
network. Knowing when to freeze or unfreeze allows the practitioner to balance learning
flexibility, accuracy, and efficiency in modern transfer learning workflows.

5. Write the basic RNN update and explain common

RNN setups (many-to-one and many-to-many) in simple terms.

A Recurrent Neural Network (RNN) processes sequential data by maintaining a hidden

state that carries information from previous time steps. This makes RNNs suitable for
tasks where inputs occur in order and where earlier information influences later
predictions, such as sentences, speech, or time-series patterns. The core mathematical
update that defines an RNN is expressed as:

ht = f(Whht-1 + Wxxt + b)

Here, xt represents the input at the current time step t, while ht-1 is the hidden state from
the previous time step. Wh is the weight matrix applied to the old hidden state, Wx is the
matrix applied to the input, b is the bias term, and f is a nonlinear activation function such
as tanh or ReLU. Through this update, the hidden state accumulates meaningful
information from the entire sequence, making ht a memory-like representation that
evolves as new inputs come in. The same weights Wh and Wx are reused at every time
step, which allows the model to learn patterns independent of sequence length.

RNNs can be configured in different architectural setups depending on how input

sequences are mapped to outputs. One common structure is the many-to-one setup. In this
form, the network takes multiple time-step inputs but produces only one final output. The
model reads the entire sequence, updates its hidden state repeatedly, and generates one
prediction after the last input has been processed. Tasks like sentiment analysis and
sequence classification use this configuration, since the goal is to understand the overall
meaning of the sequence instead of producing output at every step. The final hidden state
hT is treated as a summary of the full input sequence.

Another frequently used structure is the many-to-many setup. In this form, the network
takes a sequence as input and produces a sequence of outputs. There are two main
variations within this category. The first variation outputs one prediction at every time
step while reading inputs continuously. This is used in applications such as part-of-
speech tagging or video frame-by-frame action recognition where each input directly
corresponds to an output. The second variation delays the output until the full input
sequence has been read, then begins producing outputs in a sequential manner. This
structure is used in encoder-decoder models common in machine translation, where one
sentence is encoded first and then decoded into another language, generating output
words step by step. In this setup, the encoder compresses the input information into the
last hidden state, and the decoder unfolds that information into a new sequence.

These RNN setups allow the model to handle a wide range of temporal learning problems
because they do not assume equal lengths or one-to-one correspondence between input
and output. The many-to-one configuration focuses on producing a single interpretation
of a sequence, while many-to-many configurations focus on maintaining a sequence
relationship between inputs and outputs. Both rely on the idea that the hidden state acts as
a memory that changes with every new time step, enabling the network to associate past
information with present decisions.

In conclusion, the basic RNN update rule defines how information flows from one time
step to the next, allowing the model to learn temporal dependencies. The many-to-one
setup suits tasks where a single output is needed after reading a sequence, whereas the
many-to-many setup handles problems where sequential outputs are required. Together,
these configurations make RNNs versatile tools for processing sequential data in deep
learning.

6. Explain what Backpropagation Through Time (BPTT) is, why gradients can

vanish or explode, and give practical ways to reduce these problems

Recurrent neural networks (RNNs) are designed to process sequential data such as
speech, text, and time-series signals by maintaining a hidden state that carries information
across time steps. Training these networks requires a specialized version of
backpropagation called Backpropagation Through Time (BPTT). BPTT unfolds the RNN
across time, converting its recurrent structure into an equivalent deep feedforward
network where each layer represents the same RNN cell at a different time step. This
unfolding allows the calculation of gradients with respect to weights by applying the
chain rule over all time steps.

In a standard RNN, the hidden state update is expressed as:

ht = f(Whh ht-1 + Wxh xt + b)
where Whh represents recurrent weights, Wxh connects inputs to hidden states, xt is the
input at time t, and f is a nonlinear activation such as tanh or ReLU. During BPTT, errors
are propagated backward from the output at the final time step through earlier states. The
gradient contribution at each time depends on how much past information influenced the
current prediction.

However, BPTT faces two serious optimization problems known as the vanishing and
exploding gradient issues. These problems arise because the gradient gets multiplied
repeatedly by the recurrent weight matrix Whh and by derivatives of activation functions
during backward propagation across many time steps. If the eigenvalues of Whh are
smaller than one, repeated multiplication shrinks gradients exponentially, making them
approach zero. This is called the vanishing gradient problem. It prevents the model from
learning long-term dependencies because contributions from earlier steps vanish before
reaching the earlier layers. As a result, the network tends to focus only on recent
information, failing to capture long-range relationships such as grammar in language or
seasonal patterns in time series.

On the other hand, if the eigenvalues of Whh are greater than one, gradients grow
exponentially during backward propagation, causing the exploding gradient problem.
This leads to unstable training, very large weight updates, and sometimes numerical
overflow where weights become undefined. In practice, exploding gradients cause sudden
divergence in the loss function and failure of learning.

Several practical techniques have been developed to address these issues. One direct
method for controlling exploding gradients is gradient clipping. This approach limits the
maximum value of gradients during training by scaling them when they exceed a
threshold. Gradient clipping prevents instability while still allowing useful learning. For
the vanishing gradient problem, the most effective solution is the use of gated recurrent
architectures such as Long Short-Term Memory (LSTM) networks and Gated Recurrent
Units (GRUs). These architectures introduce memory cells and gating mechanisms that
allow error to flow more easily across long time intervals, maintaining gradient
magnitude through additive operations rather than repeated multiplication.

Another useful technique is using activation functions that do not saturate quickly. For
example, replacing tanh or sigmoid activation with ReLU can reduce vanishing gradients,
although ReLU-based RNNs may still struggle without additional architectural changes.
Orthogonal or unitary initialization of recurrent weight matrices helps maintain gradient
norm because orthogonal matrices preserve magnitude during multiplication. Truncated
BPTT is another widely used training method that limits backpropagation to only a fixed
number of time steps rather than propagating through the entire sequence. Though this
reduces the ability to learn extremely long dependencies, it keeps computation stable and
efficient.

Finally, regularization strategies such as dropout on recurrent connections, batch

normalization adapted for RNNs, and careful learning rate scheduling improve
optimization stability and help prevent gradient explosion from learning too aggressively.

In summary, BPTT is the primary method for training recurrent networks by unfolding
them across time and applying backpropagation. However, gradients can vanish or
explode due to repeated multiplication across long sequences. Practical solutions include
gradient clipping, gated architectures like LSTM and GRU, orthogonal initialization,
truncated BPTT, and improved activation functions. These techniques make training deep
recurrent models more stable while enabling them to retain information over long time
periods, which is crucial for many real-world sequence learning tasks.

7. Describe how LSTM gates work to keep long-term information. Explain when a

GRU might be better than an LSTM on small devices.

Long Short-Term Memory (LSTM) networks are a type of recurrent neural network
(RNN) specifically designed to address the problem of learning long-term dependencies
in sequential data. Traditional RNNs suffer from the vanishing and exploding gradient
problem, which makes it difficult for them to retain information over long sequences.
LSTMs overcome this limitation using a memory cell and a set of gates that regulate the
flow of information through the network. These gates allow LSTMs to selectively
remember or forget information, enabling the network to capture long-term dependencies
effectively.

An LSTM unit consists of three main gates: the input gate, the forget gate, and the output
gate, each controlling a different aspect of information flow. The input gate determines
how much of the new input information should be stored in the cell state. It receives the
current input and the previous hidden state, processes them through a sigmoid activation
function, and outputs a value between 0 and 1, where 0 means “completely ignore” and 1
means “fully accept.” This output is then multiplied with a candidate value, which is
computed from the input and previous hidden state via a tanh function, and added to the
existing cell state.

The forget gate is crucial for long-term memory management. It decides which
information in the cell state should be retained or discarded. Like the input gate, it uses a
sigmoid function to generate a mask between 0 and 1 for the previous cell state. By
controlling what to forget, the LSTM can prevent irrelevant or outdated information from
accumulating and interfering with future predictions, which is essential in tasks such as
time series forecasting or language modeling.

The output gate regulates what information from the cell state should be output at the
current time step. The cell state is passed through a tanh function to compress the values
between -1 and 1, and then multiplied by the sigmoid output of the output gate. This
generates the hidden state for the current time step, which is also passed to the next time
step in the sequence. Through this gating mechanism, LSTMs maintain a balance
between retaining long-term memory and incorporating new information, effectively
addressing the limitations of traditional RNNs.

Despite the effectiveness of LSTMs, they can be computationally expensive and have a
large number of parameters, which may not be suitable for small devices with limited
memory and processing power. In such cases, the Gated Recurrent Unit (GRU) offers a
lightweight alternative. GRUs simplify the LSTM architecture by combining the input
and forget gates into a single update gate and using a reset gate to control how much
past information should influence the current input. This results in fewer parameters and
a simpler computation, which translates into faster training and inference on devices with
constrained resources.

The update gate in a GRU functions similarly to the combination of the input and forget
gates in an LSTM. It decides how much of the past information needs to be retained
versus how much new information should be added. The reset gate determines the
contribution of previous hidden states when computing the new candidate hidden state.
By controlling the influence of past memory, GRUs can retain relevant information
without the overhead of multiple gates and separate cell states, making them particularly
efficient for small devices or real-time applications where memory and computation are
limited.

In practice, GRUs often perform comparably to LSTMs on many tasks, such as sequence
classification, time series prediction, and language modeling, while requiring less
computational power. They are especially preferred in mobile or embedded applications,
where hardware limitations demand more efficient models. However, for tasks that
require capturing very long-term dependencies or more complex sequential patterns,
LSTMs may still be advantageous due to their explicit cell state and more flexible gating
mechanism.

In summary, LSTMs use three gates—input, forget, and output—to maintain and update
long-term information in sequences effectively. These gates control the flow of
information into, out of, and within the cell state, preventing irrelevant information from
interfering with learning. GRUs, on the other hand, offer a simplified structure with fewer
gates and parameters, making them more suitable for small devices or scenarios where
computational efficiency is critical. Choosing between LSTM and GRU depends on the
specific requirements of the task and the constraints of the hardware environment.

8. Explain the difference between generative and discriminative models with clear

examples, and say when a generative approach can help with few labels.

Generative and discriminative models are two fundamental approaches in machine

learning for handling classification and prediction tasks. Although both aim to map inputs
to outputs, they differ significantly in their methodology, objectives, and applications.
Understanding these differences is crucial for selecting the right approach depending on
the data availability, task requirements, and problem complexity.

A discriminative model focuses on modeling the decision boundary between classes. In

other words, it directly learns the conditional probability (P(y|x)), where (x) is the input
data and (y) is the target label. The model does not attempt to understand the underlying
distribution of the input data; instead, it learns to distinguish between classes as
accurately as possible. Common examples of discriminative models include Logistic
Regression, Support Vector Machines (SVM), and Conditional Random Fields (CRF).
For instance, in email spam detection, a discriminative model like logistic regression
learns the relationship between features such as word frequencies and whether an email is
spam or not, without modeling how the features are generated in each class.
Discriminative models are often preferred when labeled data is abundant, as they
typically provide higher predictive accuracy and are simpler to train for classification
tasks.

In contrast, a generative model aims to model the joint probability distribution (P(x, y))
or equivalently (P(x|y)P(y)). This means the model attempts to understand how the data is
generated for each class. By learning this underlying distribution, generative models can
generate new samples that resemble the original data. Examples of generative models
include Naive Bayes, Gaussian Mixture Models, Hidden Markov Models, and modern
deep learning-based models such as Variational Autoencoders (VAEs) and Generative
Adversarial Networks (GANs). For example, in handwritten digit recognition using the
MNIST dataset, a generative model like a VAE can learn the distribution of images for
each digit and generate realistic new images of digits, in addition to classifying them.
Generative models provide more insight into the structure and variability of the data,
which can be advantageous for tasks beyond simple classification, such as data synthesis,
anomaly detection, or semi-supervised learning.

One key distinction between the two approaches is that discriminative models excel at
prediction but cannot generate new data, whereas generative models can both generate
data and perform classification by applying Bayes’ theorem to compute (P(y|x) = \
frac{P(x|y)P(y)}{P(x)}). While discriminative models usually achieve better performance
when sufficient labeled data is available, generative models can offer substantial benefits
in scenarios with limited labeled data. This is because generative models leverage the
distribution of input features to extract useful information, which can compensate for the
scarcity of labeled examples. For instance, in medical imaging where labeling requires
expert knowledge, a generative model can learn the distribution of healthy and diseased
tissues from a small set of labeled images and a larger set of unlabeled images. It can then
generate synthetic examples or refine the decision boundary, effectively improving
classification performance even when few labels are available.

Moreover, generative models are particularly useful in semi-supervised and unsupervised

learning. Techniques like semi-supervised GANs or VAEs allow models to exploit
unlabeled data to learn the underlying data structure. This makes them highly
advantageous in real-world scenarios where labeling data is expensive, time-consuming,
or prone to errors. In contrast, discriminative models require sufficient labeled data to
achieve comparable performance and do not benefit directly from unlabeled data.

In summary, discriminative models focus on modeling the boundary between classes and
are generally more accurate for standard supervised tasks with abundant labeled data.
Generative models, on the other hand, model how data is generated and can produce new
data, making them versatile for tasks beyond classification. Importantly, generative
models are valuable in situations with few labeled examples because they leverage
knowledge of the input distribution to improve learning. Choosing between these
approaches depends on the availability of labeled data, the need for data generation, and
the complexity of the underlying task.

9. Compare vanilla, denoising, sparse, and contractive autoencoders. Mention the

main regularization idea behind each and when you would choose one.

Autoencoders are a class of unsupervised neural networks designed to learn efficient

representations of data by encoding input into a lower-dimensional latent space and
reconstructing it. Variants such as vanilla, denoising, sparse, and contractive
autoencoders differ in their architecture, training objectives, and regularization strategies,
which affects their suitability for different tasks.

A vanilla autoencoder is the simplest form, consisting of an encoder that maps input to a
latent representation and a decoder that reconstructs the input from this representation.
The network is trained to minimize reconstruction loss, usually mean squared error for
continuous data or cross-entropy for binary data. Vanilla autoencoders do not impose
additional constraints on the latent representation beyond dimensionality reduction. They
are mainly used for data compression, dimensionality reduction, or as pretraining for
other neural networks. The key idea is learning an efficient encoding that preserves the
most important information about the input. However, vanilla autoencoders may overfit,
particularly if the hidden layer has the same or higher dimensionality than the input, as
the network can simply memorize the data.

Denoising autoencoders extend the vanilla autoencoder by introducing noise to the input
during training and learning to reconstruct the original, clean input. The input is
corrupted and the network is trained to minimize the reconstruction error between the
output and the original uncorrupted input. The main regularization idea is robustness,
forcing the network to capture meaningful features rather than memorizing the input.
Denoising autoencoders are useful when the data is noisy or when a more stable latent
representation is desired. They are commonly applied in image and speech denoising
tasks.

Sparse autoencoders incorporate a sparsity constraint on the activations of the hidden

layer. Even if the number of hidden units exceeds the input dimension, only a small
subset of neurons is allowed to activate for a given input. This is enforced using a
regularization term, often based on Kullback–Leibler divergence, that penalizes deviation
from a target activation rate. The main idea is feature selection and disentanglement. By
restricting the number of active neurons, the network learns distinct and interpretable
features. Sparse autoencoders are suitable for high-dimensional data where
interpretability is important, such as in text embeddings or gene expression analysis.

Contractive autoencoders introduce a penalty on the sensitivity of the hidden

representation with respect to the input, typically measured by the Frobenius norm of the
Jacobian of the hidden units. This penalization ensures that small changes in the input
produce minimal changes in the latent representation, making it robust to slight variations
in the input. The regularization idea is smoothness, which encourages the network to
learn features that capture the underlying manifold of the data. Contractive autoencoders
are particularly useful when learning invariant features is important, for instance in image
recognition or manifold learning tasks, where local variations in the input should not
affect the learned representation.

In summary, vanilla autoencoders focus on simple reconstruction without additional

regularization and are suitable for dimensionality reduction. Denoising autoencoders use
input corruption to encourage robust representations. Sparse autoencoders enforce
sparsity in hidden activations to promote feature disentanglement and interpretability.
Contractive autoencoders penalize sensitivity to input variations to learn smooth,
invariant representations. The choice among these depends on the data characteristics and
the application, with vanilla suitable for basic encoding, denoising for noisy inputs,
sparse for interpretability, and contractive for invariant feature extraction.

10. Explain how GANs train (the min–max idea) and what “mode collapse” means.

Say why WGAN with gradient penalty can be more stable than DCGAN.

Generative adversarial networks, or GANs, are a class of deep learning models designed
to generate realistic data by training two neural networks in opposition. The two networks
are called the generator and the discriminator. The generator learns to produce synthetic
data that resembles the real data, while the discriminator learns to distinguish between
real and generated data. The training process is formulated as a min–max optimization
problem. The generator aims to minimize the probability that the discriminator correctly
identifies its outputs as fake, while the discriminator aims to maximize this probability.
Mathematically, this is expressed as a two-player game in which the discriminator tries to
maximize the function of real and generated data, and the generator tries to minimize the
same function, creating a dynamic in which both networks iteratively improve. The
generator improves by producing more realistic data, and the discriminator improves by
becoming better at detecting fakes. Ideally, this process continues until the generator
produces data indistinguishable from real data, at which point the discriminator’s
accuracy converges to random guessing.

One common challenge in training GANs is mode collapse. Mode collapse occurs when
the generator learns to produce a limited variety of outputs that fool the discriminator but
do not represent the full diversity of the real data distribution. For example, if the
generator is trained on handwritten digits, it might produce only a few digits repeatedly,
ignoring other classes. Mode collapse reduces the utility of the generator, as it fails to
capture all modes of the data distribution, and is often caused by the instability of the
adversarial training, where the generator finds a narrow set of outputs that consistently
trick the discriminator.

Deep convolutional GANs, or DCGANs, improved stability compared to earlier GANs

by using convolutional architectures and specific training heuristics, but they can still
suffer from mode collapse and unstable convergence. One way to address these issues is
the Wasserstein GAN with gradient penalty, or WGAN-GP. The WGAN framework
replaces the original GAN loss based on Jensen–Shannon divergence with the
Wasserstein distance, which measures the cost of transporting probability mass between
distributions. This change provides smoother gradients for the generator, making the
optimization more stable. The gradient penalty further enforces the Lipschitz constraint,
which ensures that the discriminator (also called the critic in WGAN terminology) has
gradients with bounded norm. This prevents the critic from becoming too sharp or
producing vanishing or exploding gradients, which can destabilize training. By
combining the Wasserstein distance with a gradient penalty, WGAN-GP reduces the
likelihood of mode collapse and allows the generator to learn a more diverse set of
outputs, producing samples that better cover the full data distribution.

In practice, WGAN-GP tends to be more stable than DCGANs, especially for high-
dimensional or complex data such as images, because it provides consistent gradient
information to the generator even when the distributions of real and generated data are far
apart. While DCGANs rely on heuristic techniques like batch normalization, careful
learning rate schedules, and architectural choices to improve stability, WGAN-GP has a
principled loss function that directly addresses the limitations of original GAN training.
As a result, WGAN-GP can achieve higher-quality and more diverse outputs with fewer
training difficulties, making it a preferred choice for many generative tasks in computer
vision and other domains.

In summary, GANs train through a min–max game in which the generator tries to fool the
discriminator while the discriminator tries to detect fake samples. Mode collapse occurs
when the generator produces limited outputs, reducing diversity. DCGANs improve
training stability with convolutional architectures but can still face mode collapse.
WGAN with gradient penalty stabilizes training by using the Wasserstein distance and
enforcing a Lipschitz constraint, resulting in smoother gradients, reduced mode collapse,
and more diverse generated samples.

11. Briefly differentiate Vanilla GAN, DCGAN, cGAN, WGAN, Progressive GAN,

and conditional CVAE, and match each to a suitable use case.

Vanilla GAN is the basic form of generative adversarial network where a generator
network produces synthetic data and a discriminator network tries to distinguish it from
real data. It uses fully connected layers and is trained using the min–max adversarial loss.
Vanilla GANs are suitable for simple data generation tasks, such as low-dimensional
synthetic datasets or toy image generation, but can be unstable for high-dimensional
images.

DCGAN, or deep convolutional GAN, extends the vanilla GAN by using convolutional
and transposed convolutional layers in the generator and discriminator. This architecture
is designed to capture spatial hierarchies in images, improving stability and quality of
generated outputs. DCGANs are commonly used for generating realistic images, such as
human faces, artwork, or object images in computer vision tasks.

Conditional GAN, or cGAN, introduces conditional information such as labels or

attributes into both the generator and discriminator. This allows the generator to produce
outputs corresponding to specific classes or characteristics. For example, a cGAN can
generate images of a specific type of clothing given a class label or produce digits
corresponding to a particular number. cGANs are useful in scenarios where controlled or
guided generation is required.
Wasserstein GAN, or WGAN, modifies the GAN loss function by replacing the original
divergence measure with the Wasserstein distance. This provides smoother gradients and
more stable training, especially when the real and generated distributions are very
different. WGAN with gradient penalty further improves stability by enforcing a
Lipschitz constraint on the discriminator. WGANs are appropriate for high-dimensional
data or complex image synthesis where training stability is crucial, such as in high-
resolution photo generation.

Progressive GAN generates images in a multi-stage process, starting from low resolution
and progressively increasing the resolution as training progresses. This gradual learning
approach improves stability and allows the generator to produce high-resolution images
without training difficulties associated with starting at full resolution. Progressive GANs
are particularly suitable for tasks like generating photorealistic images, including human
faces, landscapes, or medical images at high resolution.

Conditional variational autoencoder, or conditional CVAE, is a generative model that

combines the principles of variational autoencoders with conditioning on additional
information such as class labels or attributes. The encoder learns a latent representation of
the input conditioned on this information, and the decoder reconstructs the data
accordingly. Conditional CVAEs are useful when generating structured outputs with
variability while conditioning on specific factors, such as generating images of a certain
object category, producing diverse sentence paraphrases, or simulating medical images
conditioned on disease type.

In summary, vanilla GAN is the simplest GAN and is suitable for low-dimensional or toy
datasets. DCGAN is designed for image generation with convolutional architectures and
performs well on natural images. cGAN adds conditioning information, making it
suitable for controlled or class-specific generation tasks. WGAN stabilizes training for
high-dimensional or complex datasets and reduces mode collapse. Progressive GAN
improves high-resolution image generation by growing the network progressively.
Conditional CVAE is a variational approach that generates diverse outputs conditioned
on labels or attributes and is suitable for structured data generation where variability and
control are required. Choosing among these models depends on the complexity of the
data, the need for high-resolution outputs, and whether conditional generation is desired.

12. Explain neural style transfer (content and style losses) and the CycleGAN idea

of cycle consistency. Say when you would pick one over the other for unpaired

data.

Neural style transfer is a technique that synthesizes a new image by combining the
content of one image with the style of another. It relies on deep convolutional neural
networks to separate and manipulate content and style representations. In this approach, a
pretrained network such as VGG is used to extract features from the content image and
the style image. The content loss measures the difference between the features of the
generated image and the content image, ensuring that the generated image preserves the
structure and semantic information of the original content. The style loss, on the other
hand, is computed using the correlations between feature maps of the style image, often
through Gram matrices, which capture the texture, color patterns, and stylistic elements.
By minimizing a weighted combination of content and style losses, the network generates
an image that retains the content of the original image while adopting the visual style of
the reference image. Neural style transfer is suitable when paired input images are
available, or when the goal is to explicitly combine content and style in a controlled
manner.

CycleGAN extends the idea of generative adversarial networks to the task of image-to-
image translation without requiring paired examples. In many real-world applications,
obtaining paired data is difficult or impossible, such as translating photographs to
paintings, horses to zebras, or summer scenes to winter landscapes. CycleGAN
introduces the concept of cycle consistency to address this challenge. It consists of two
generator networks and two discriminator networks. One generator maps images from
domain A to domain B, and the other maps from domain B back to domain A. Cycle
consistency loss ensures that if an image is transformed from A to B and then back to A,
the reconstructed image should be similar to the original input. This encourages the
generators to learn meaningful mappings between domains even when explicit pairs are
not available. In addition to the adversarial loss, which ensures that generated images are
realistic within each domain, the cycle consistency loss prevents arbitrary transformations
that do not preserve the underlying content.

The key difference between neural style transfer and CycleGAN lies in their data
requirements and application scenarios. Neural style transfer typically requires a content
image and a style reference, making it most suitable for tasks where a direct stylistic
transformation is desired and both images are available. It focuses on combining features
from specific images and produces a single output per content-style pair. CycleGAN, on
the other hand, is designed for unpaired datasets and learns a general mapping between
two domains. It is particularly useful when large collections of images exist for each
domain but there is no one-to-one correspondence between them. For instance, if one has
a dataset of horse images and a dataset of zebra images without exact pairings,
CycleGAN can learn to convert any horse image to a zebra image while preserving the
original structure.

Choosing between the two methods depends on the problem and data availability. Neural
style transfer is preferred when the task involves artistic stylization of individual images
and the style reference is known. It allows fine control over content and style
contributions. CycleGAN is preferable for unpaired data scenarios where the goal is to
learn a domain-to-domain mapping and generate multiple realistic outputs without paired
examples. In some cases, CycleGAN can also be adapted for style transfer between
unpaired datasets, but it generally requires more data and training time compared to direct
neural style transfer.
In summary, neural style transfer uses content and style losses to combine the structure of
one image with the artistic style of another, producing visually stylized outputs when
both images are available. CycleGAN leverages cycle consistency to learn mappings
between two unpaired domains, ensuring that images transformed from one domain to
another and back remain consistent. Neural style transfer is suitable for controlled,
paired-style transformations, while CycleGAN is ideal for unpaired datasets where
domain-level translation is required.

13. Describe simple, real-world uses of Generative AI in music and art (for example,

style transfer in images and multi-track music generation like MuseGAN).

Generative AI has found numerous applications in music and art, transforming creative
workflows and enabling new forms of expression. In art, one of the most widely
recognized applications is neural style transfer, which allows the combination of content
from one image with the artistic style of another. For example, a photograph of a
cityscape can be transformed to resemble the painting style of Van Gogh or Picasso by
minimizing content and style losses in a neural network. This approach provides artists
and designers with the ability to experiment with styles quickly and generate visually
compelling outputs without manually recreating the scene in a particular artistic form.
Beyond style transfer, generative models such as Generative Adversarial Networks
(GANs) can create entirely new images, illustrations, or textures that do not exist in the
real world, which can be used for concept art, game design, and advertising. These
models can also assist in upscaling, inpainting, or colorizing images, enabling restoration
of old or incomplete artworks.

In music, generative AI can produce compositions and multi-track arrangements that

emulate human creativity. Models like MuseGAN employ GAN architectures to generate
polyphonic music by learning patterns in existing musical datasets. MuseGAN can
produce multiple tracks, such as drums, bass, piano, and guitar, ensuring harmony and
temporal coherence between them. This allows composers, hobbyists, and producers to
generate background music, experiment with new melodies, or obtain creative inspiration
without extensive musical training. Another approach is using recurrent neural networks
or transformer-based models to generate sequences of notes, chords, or lyrics, which can
then be refined or combined with human creativity. Generative AI can also assist in
music style transfer, where a piece composed in one style can be reinterpreted in the style
of a different genre or artist, similar to visual style transfer in images.

These applications of generative AI provide practical benefits in real-world scenarios. For

visual art, style transfer can be used in marketing and social media to quickly produce
unique visuals that capture attention, while AI-generated art can reduce the time and
effort needed for content creation. Artists and designers can explore variations of their
work or produce concept sketches before committing to final versions. In music, AI-
generated compositions can be used in video games, films, or advertisements where
background scores are needed at scale and in diverse styles. Musicians can also
collaborate with AI systems to experiment with unconventional arrangements or develop
new musical motifs that might not have been conceived manually.

Generative AI also enables accessibility in creative domains. Individuals with limited

artistic or musical training can leverage these models to produce content that aligns with
professional quality, democratizing the creative process. For example, AI-assisted
platforms allow users to generate illustrations or music tracks by providing simple
prompts or inputs, lowering barriers to entry and enhancing productivity. Additionally,
AI can help preserve and extend cultural artifacts, such as recreating lost musical
compositions or simulating artistic styles from historical periods, thereby providing
educational and archival value.

Despite these advantages, it is important to consider ethical implications and originality.

AI-generated music and art often rely on learning patterns from existing works, which
raises questions about intellectual property and the balance between human creativity and
machine assistance. Nonetheless, the practical applications of generative AI in music and
art demonstrate its capacity to enhance creativity, accelerate production, and enable
innovative experiences that were previously difficult or impossible to achieve manually.

In summary, generative AI is applied in art through techniques like style transfer and
image generation, enabling the creation of visually compelling works, restoration of
images, and exploration of new artistic styles. In music, models like MuseGAN generate
multi-track compositions, assist in style transfer, and provide tools for experimentation
and creative inspiration. These technologies facilitate real-world uses in media,
entertainment, and personal creative projects while democratizing access to artistic and
musical creation.