Deep Learning Assignment #02

Question 1: Learning Rate Scheduling in Gradient Descent

1. Learning Rate Decay
Overview
Learning Rate Decay is a strategy where the initial learning rate is gradually reduced over
time during training. The primary idea is to use a relatively high learning rate at the start so
that the optimizer makes rapid progress toward a good region of the loss landscape and
then to reduce the rate later on to enable finer adjustments (or “fine-tuning”) of the model
parameters.
How It Adjusts the Learning Rate
• Step Decay (Geometric Decay):
With step decay, the learning rate is reduced by a fixed factor (for example, halved)
after every predetermined number of epochs. Mathematically, if α0\alpha_0α0 is
the initial learning rate, FFF is the decay factor (e.g., 0.5), and DDD is the number of
epochs after which the decay occurs, then the learning rate at epoch EEE can be
expressed as:
αE=α0×F⌊DE⌋

Exponential or Polynomial Decay:

Alternatively, the decay can be smooth (exponential decay) or follow a polynomial schedule. In
exponential decay, the learning rate at iteration n might be computed as:

αn=α0⋅e−dn

Benefits
• Initial Exploration: A high learning rate early on allows the model to make large
updates and quickly navigate through the loss surface.
• Fine-Tuning: As training proceeds, reducing the learning rate helps the model settle
into a region of low loss by making smaller, more precise updates.
• Stability: Lowering the learning rate toward the end of training often results in more
stable convergence and can reduce the chance of overshooting minima.

Scenarios Where It’s Most Effective

• Large-Scale Training: When training deep networks on large datasets, the initial
rapid progress followed by fine-tuning can lead to better convergence.
 • Well-Defined Convergence Phases: When you expect that most of the useful
learning happens early on and that fine adjustments are needed later, decay
schedules can help optimize performance.
• When Hyperparameter Tuning is Feasible: Decay strategies typically involve one or
two extra hyperparameters (decay factor and drop interval) that can be tuned based
on validation performance.

2. Cyclical Learning Rates

Overview

Cyclical Learning Rates (CLR) offer an alternative to monotonically decaying schedules. Instead of
simply reducing the learning rate over time, CLR periodically cycles the learning rate between a
lower bound and an upper bound. This cyclical pattern can help the optimizer escape local minima
and saddle points.

How It Adjusts the Learning Rate

• Triangular Policy:
The simplest form of CLR is the triangular policy. The learning rate linearly increases from a
base value to a maximum value over a certain number of iterations (half cycle), then
decreases back to the base value for the next half cycle. This creates a repeating triangular
waveform.

Triangular2 and Exponential Variants:

Variants include the “triangular2” policy, where the maximum learning rate is reduced after each
cycle, and the “exp_range” policy, where the cycle’s amplitude decays exponentially over time.

Benefits
• Escaping Local Minima:
By periodically increasing the learning rate, CLR can help the optimizer jump out of narrow
local minima or saddle points that might trap standard decaying schedules.

• Reduced Hyperparameter Tuning:

CLR can make the initial choice of the learning rate less critical since the rate continuously
varies, allowing the network to benefit from both high and low learning rates during
training.

• Faster Convergence:
Empirical studies have shown that cyclical schedules can sometimes lead to faster
convergence by balancing exploration (with higher rates) and fine-tuning (with lower rates).
Scenarios Where It’s Most Effective
• Non-Convex Optimization Problems:
Deep neural networks often have loss landscapes with many local minima and saddle points.
CLR’s dynamic nature is particularly useful in such complex landscapes.

• Limited Tuning Resources:

If you wish to reduce the need for exhaustive tuning of the learning rate, CLR’s built-in
cyclicity can be an advantage.

• When Training is Sufficiently Long:

Cyclical patterns require enough iterations or epochs to complete several cycles in order to
demonstrate their benefits.

• Modern Deep Learning Architectures:

In many state-of-the-art applications (e.g., computer vision, natural language processing),
CLR has been shown to improve both training speed and final performance.

Question 2: The Role of Computation Graphs in Backpropagation

Solution:
A computation graph is a visual representation of the sequence of operations used to compute a
function. In this graph:

• Nodes represent inputs, intermediate values, or outputs.

• Edges represent operations that transform one or more inputs into an output.

Deep learning frameworks (such as TensorFlow and PyTorch) build these graphs automatically. They
store intermediate results during the forward pass so that, during the backward pass, they can
efficiently apply the chain rule to compute gradients for each parameter.

Backpropagation Uses Computation Graphs

During the forward pass, the network computes the output by following the graph from inputs to
output, storing all intermediate values. In the backward pass, the chain rule is applied in reverse
order (hence "backpropagation"):

1. Initialize the gradient at the output (usually 1).

2. Propagate gradients backwards: For each node, multiply the incoming gradient by the
derivative of the operation at that node (using the chain rule).

3. Cache reuse: Since intermediate values were stored, the derivatives can be computed
without recalculating the forward pass.
 1. Output Node:
Compute the final function:

f= m+zf

This node takes m and z as inputs.

The graph can be visualized as:

This approach scales to much more complex neural networks, which is why computation graphs are
central to modern deep learning frameworks.

Question 3: Applications of Fully Connected Neural Networks

Solution:
1. Medical Diagnosis

Overview:
In medical diagnosis, FCNNs are employed to analyze structured patient data (such as demographics,
lab test results, genetic information, and clinical histories) for disease prediction and risk
stratification. While imaging tasks often favor convolutional architectures, many diagnostic tasks rely
on tabular or multimodal data where FCNNs excel.

Adaptation to Challenges:

Handling Heterogeneous Data:

➢ Patient data can be highly heterogeneous and noisy. FCNNs are typically designed with
several hidden layers that capture complex nonlinear interactions between diverse features.
Preprocessing steps like normalization and feature scaling are critical, and embedding layers
may be used for categorical variables.
➢ Regularization and Interpretability:
Medical datasets are often limited in size and susceptible to overfitting. Techniques such as
dropout, batch normalization, and L2 regularization are incorporated into FCNNs to improve
generalization. Moreover, integrating interpretability methods (e.g., SHAP or LIME) is vital
for clinical acceptance, as clinicians need to understand the factors driving predictions.
 ➢ Risk Stratification:
FCNNs can output probability scores for different disease classes or risk levels, supporting
decision-making processes in diagnosis. The network’s architecture is often fine-tuned to
ensure sensitivity and specificity align with clinical requirements.

Why FCNNs Are Suitable:

Their ability to model complex relationships in structured data and to combine various data types
makes FCNNs particularly useful for diagnosis tasks that rely on integrating multiple sources of
patient information.

2. Recommendation Systems

Overview:
In recommendation systems, FCNNs are used to predict user preferences based on historical
interactions, user profiles, and item features. These systems aim to suggest items (movies, products,
or services) that a user is likely to enjoy, making them a backbone of many e-commerce and
entertainment platforms.

Adaptation to Challenges:

➢ High-Dimensional and Sparse Data:

Recommendation tasks often start with sparse input data (e.g., user-item interaction
matrices). FCNNs are usually preceded by an embedding layer that transforms high-
dimensional, sparse representations into dense, continuous vectors. This step reduces
dimensionality while preserving meaningful relationships.

➢ Modeling Nonlinear Interactions:

After embedding, the dense vectors representing users and items are passed through
multiple fully connected layers. These layers learn complex, nonlinear interactions between
features that simpler linear models may miss, which is essential for accurate preference
prediction.
➢ Scalability and Personalization:
FCNNs can be designed to scale by adjusting the network depth and width to match the
complexity of user behaviors. In many implementations, the network is trained jointly with
additional components (e.g., autoencoders or factorization machines) to improve
personalization and adapt recommendations to changing user preferences.

Why FCNNs Are Suitable:

The flexibility of FCNNs in learning high-level abstract features and capturing intricate user–item
interactions makes them well-suited for recommendation tasks. Their capacity to integrate various
data sources (e.g., demographic data, browsing history, and explicit ratings) contributes to more
accurate and personalized recommendations.

Conclusion

Both medical diagnosis and recommendation systems are advanced real-world applications that
benefit from the powerful function-approximation capabilities of Fully Connected Neural Networks.
In medical diagnosis, FCNNs are adapted to handle heterogeneous, structured data with robust
regularization and interpretability measures—key for reliable clinical decision-making. In
recommendation systems, FCNNs, often combined with embedding layers, excel at modeling
complex nonlinear interactions in high-dimensional and sparse data, thereby enabling effective
personalization.