Deep Learning Assignment Answers

1. Explain the difference between Hopfield networks and Boltzmann machines in how their
energy works and how they converge. Say where each is more useful.

Hopfield networks and Boltzmann machines are both energy-based neural network models
used mainly for associative memory and optimization tasks, but they differ significantly in
how they define energy, how they update states, and how they converge to a solution. Their
learning rules and practical applications also vary. Understanding these differences
requires looking closely at their structure, dynamics, and computational characteristics.

A Hopfield network is a recurrent neural network where each neuron is connected to every
other neuron in a symmetric manner, meaning the weight from neuron i to neuron j is the
same as from neuron j to neuron i. The network uses binary states, typically +1 and −1, and
deterministic update rules. Its main concept is based on energy minimization. Each state of
the network has an energy value computed using a well-defined energy function. When the
network is presented with an input, it updates the neurons one by one (asynchronously) in
a way that the energy continuously decreases. Because of the symmetric weights and
deterministic update scheme, the Hopfield network always moves downhill in the energy
landscape until it reaches a stable point. This stable point is called an attractor. Therefore,
Hopfield networks converge to local minima in a predictable manner. They are typically
used for storing and retrieving known patterns, also called associative memory. If a noisy or
incomplete version of a stored pattern is given as input, the network corrects it by
relaxation to the closest stored pattern in terms of energy reduction. Thus, Hopfield
networks behave like a content-addressable memory system.

The downside of Hopfield networks is that they can store only a limited number of patterns
reliably, approximately 0.14 times the number of neurons. If too many patterns are stored,
interference increases and the network retrieves incorrect patterns or gets stuck in
spurious minima. Moreover, they are not suitable for modeling uncertainty, since they use
deterministic updates and cannot properly explore multiple possible solutions. Their
convergence is fast and guaranteed, but often limited to the nearest local minimum rather
than a globally optimal pattern.

Boltzmann machines extend the idea of Hopfield networks and overcome some of their
limitations by introducing probabilistic neuron activation and hidden units. A Boltzmann
machine also uses an energy function defined over binary units. However, in contrast to
Hopfield networks, the neurons update stochastically based on probabilities derived from a
Gibbs distribution. The update rule allows a neuron to flip its state even if that temporarily
increases the energy. This randomness gives Boltzmann machines the capability to escape
from shallow local minima and explore the energy landscape more widely. Over time,
through repeated random sampling, the network converges toward a thermal equilibrium
distribution where lower energy states occur more frequently. Instead of converging strictly
to a single attractor, it reaches a probabilistic equilibrium that represents the likelihood of
different states. The global minimum has the highest probability, making Boltzmann
machines more powerful for optimization tasks.

Boltzmann machines include two kinds of units: visible units that interact with data and
hidden units that capture complex internal features. They learn through a training process
called stochastic gradient descent using contrastive divergence or related methods.
Learning is computationally expensive because computing the exact distribution requires
summation over many possible network states. Restricted Boltzmann Machines (RBMs)
simplify training by removing visible-visible and hidden-hidden connections, making them
widely used in deep learning models such as Deep Belief Networks.

Whereas Hopfield networks are mainly used for storing and recalling known patterns,
Boltzmann machines are more useful for discovering internal structure in data, learning
representations, and solving harder optimization or reasoning problems. They can work
better for pattern completion, feature learning, combinatorial optimization, and modeling
uncertainty. Their probabilistic nature makes them suitable for generative modeling where
multiple solutions may exist.

In summary, Hopfield networks use deterministic dynamics, converge to fixed point

attractors, and are suitable for fast associative memory with limited capacity. Their energy
always decreases as the network evolves. Boltzmann machines use stochastic updates,
sample from an equilibrium distribution, and can escape local minima, enabling them to
approximate globally optimal states. However, they require more computation and training
complexity. Both models rely on energy minimization principles, but their convergence
behavior and application areas differ significantly.

2. Define kernel, stride, padding, and receptive field in CNNs, and explain how they affect the
features a model learns.

A kernel in a convolutional neural network is a small matrix of learnable weights used to

scan or convolve over an input image or feature map. As the kernel slides across the image,
it multiplies with local pixel regions to extract features like edges, textures, and shapes. The
size of the kernel affects the level of detail captured. Smaller kernels capture fine features
such as edges, while larger kernels capture more global visual patterns.

Stride refers to how many pixels the kernel moves at each step during convolution. A stride
of 1 means the kernel moves one pixel at a time, producing a large output feature map with
detailed information. Higher stride values reduce the spatial resolution of the output,
making the network faster but possibly losing some fine-grained details. Stride controls the
amount of downsampling and affects how densely features are captured.

Padding refers to adding extra pixels, usually zeros, around the input before applying the
convolution. There are two common approaches: valid padding and same padding. Without
padding, the output feature map becomes smaller after each convolution, limiting the ability
to detect patterns near borders. Padding helps preserve spatial dimensions and improves
feature extraction at edges by allowing the kernel to properly process boundary regions.

The receptive field represents the region of the input image that a neuron in deeper layers
can see or is influenced by. As convolutions stack, the receptive field grows. A larger
receptive field allows neurons to capture more global and complex structures, such as
shapes or object context. However, if the receptive field grows too fast without enough
intermediate processing, the model might overlook important small-scale details. All four
concepts interact to shape what kind of patterns the model can learn, ranging from tiny
textures to high level shapes.

Together these parameters control how much detail is preserved, how hierarchical patterns
are formed, how efficiently the network runs, and how well the CNN detects structure
across different spatial scales.

3. Compare VGG16, ResNet, and Inception in a simple way (depth, parameters, and main
idea). Suggest one for real-time mobile use and justify briefly.

VGG16 has 16 layers and uses a very simple design where the architecture mainly stacks
3x3 convolutions. It has a large number of parameters, around 138 million, which makes it
slow and memory heavy. The strength of VGG16 is its simplicity and strong feature
extraction ability, but the drawback is high computation cost.

ResNet is deeper and can go up to 50, 101, or even more layers using residual connections.
These skip connections allow the network to learn identity mappings and avoid vanishing
gradients, enabling training of very deep networks. ResNet has fewer parameters compared
to VGG while being more accurate. Its main idea is learning residuals instead of direct
mappings for easier optimization.

Inception networks, like Inception V1 or V3, use parallel convolution paths in a block. They
combine kernels of different sizes at the same layer to extract multiple scales of features.
They are very efficient and use fewer parameters for the same accuracy by using bottleneck
layers. The architecture is more complex to design but highly optimized for performance.

For real-time mobile use, Inception or a lightweight ResNet variant is better because they
are computationally efficient with fewer parameters and faster inference speed. VGG16 is
not preferred on mobile because it is too heavy and consumes more memory and computing
power.

4. In transfer learning, explain the difference between feature extraction and fine tuning.
Describe when you would freeze or unfreeze layers.
In transfer learning, feature extraction means using a pretrained model to extract useful
visual or semantic features from input data, while keeping the pretrained layers fixed and
only training a new classifier on top. The pretrained network already learned general
features like edges, shapes, and textures from a large dataset. Freezing layers keeps these
learned features unchanged so the training becomes faster and requires less data. Feature
extraction is used when the new task is similar to the original one and data is limited.

Fine tuning means unfreezing some or all of the pretrained layers and training them again
along with new layers. This allows the network to adjust previously learned features to
match the new dataset more precisely. Fine tuning is helpful when the new dataset is
different from the one used in pretraining or when more training examples are available. It
gives more flexibility but requires careful tuning to avoid overfitting or damaging useful
features.

In summary, freezing layers is preferred when data is small and tasks are similar, while
unfreezing more layers is better when the new task requires the model to adapt its feature
representations more specifically.