Deep Learning Assignment 01

Exploring Neural Network Architecture

1. Convolutional Neural Networks (CNNs)

A Convolutional Neural Network (CNN) is a type of artificial neural network designed for image processing,
computer vision, and pattern recognition. CNNs is a type of Feed-Forward Neural Networks used in tasks like image
analysis, natural language processing, and other complex image classification problems.

A CNN has hidden layers of convolutional layers that form the base of ConvNets.
CNNs automatically learn features from images instead of requiring manual feature extraction. They are widely
used in image classification, object detection, facial recognition, medical imaging, and more.
At a higher level, convolutional layers detect these patterns in the image data with the help of filters. The higher-
level details are taken care of by the first few convolutional layers.
The deeper the network goes, the more sophisticated the pattern searching becomes.

How Does a CNN Work?

A CNN processes an image through different layers to extract meaningful features. The key layers are:
a. Convolution Layer (Feature Extraction)
• This layer detects patterns like edges, textures, or objects in an image.
• Uses small filters (kernels) to scan the image and extract features.
• Example: A filter might detect vertical edges in one layer and circles in another.
b. Activation Function (ReLU)
• Introduces non-linearity to the model, making it capable of learning complex patterns.
• ReLU (Rectified Linear Unit) is commonly used because it speeds up training.
c. Pooling Layer (Size Reduction)
• Reduces the size of feature maps while preserving key information.
• Max Pooling picks the most important values, helping to remove noise. Example: If you take a high-
resolution image and shrink it, but still recognize the main object, that's similar to pooling. (d)
d. Fully Connected Layer (Classification)
• After extracting features, the final layers classify the image.
• The outputs are probabilities for each class (e.g., 90% Cat, 10% Dog).

Difference from Fully Connected Networks:

CNN: CNN architectures make the explicit assumption that the inputs are images, which allows encoding
certain properties into the model architecture. A simple CNN is a sequence of layers, and every layer of a CNN
transforms one volume of activations to another through a differentiable function.
A fully connected neural network consists of a series of fully connected layers that connect every neuron in one
layer to every neuron in the other layer.

FCN: The major advantage of fully connected networks is that they are “structure agnostic” i.e. there are
no special assumptions needed to be made about the input. While being structure agnostic makes fully connected
networks very broadly applicable, such networks do tend to have weaker performance than special-purpose
networks tuned to the structure of a problem space.

Real world Application.

1. Image and Video Recognition
Image and video recognition is the most well-known application of convolutional neural networks.
Enterprises use CNNs to improve their security systems with real-time facial recognition and automated content
curation. Retailers, for example, use CNNs to analyze customer interactions in-store, transforming visual data into
actionable marketing and product placement insights.
2. Natural Language Processing (NLP)
CNNs enhance NLP applications like chatbots and virtual assistants, improving customer
interactions and enabling sentiment analysis for better service optimization.
3. Autonomous Vehicles
CNNs enable road sign recognition and obstacle detection, enhancing safety and efficiency in self-
driving cars and logistics automation.
4. Healthcare Imaging
Used for early disease detection (e.g., cancer diagnosis) by analyzing medical images, improving
diagnostic accuracy and patient outcomes.
5. Financial Services
CNNs aid in fraud detection, risk management, and investment analysis by identifying hidden
patterns in financial data.
6. Retail & E-commerce
Power recommendation systems by analyzing customer behavior, optimizing inventory
management, and boosting sales & engagement.
7. Industrial Automation
Used for quality control and predictive maintenance, reducing downtime and operational costs in
manufacturing.

2. Recurrent Neural Networks (RNNs)

A Recurrent Neural Networks work very well with sequences of data as input. Its functionality can be seen in
solving NLP problems like sentiment analysis, spam filters, time series problems like sales forecasting, stock market
prediction, etc.
Recurrent Neural Networks have the power to remember what it has learned in the past and apply it in future
predictions.
The input is in the form of sequential data that is fed into the RNN, which has a hidden internal state that gets
updated every time it reads the following sequence of data in the input.
The internal hidden state will be fed back to the model. The RNN produces some output at every timestamp.

How does RNNs work:

1. Structure:
RNNs consist of input, hidden, and output layers. The input layer receives data, the hidden layer
processes it while maintaining memory, and the output layer gives the final result.

2. Hidden Layer & Memory:

RNNs process sequential data step by step. They use short-term memory by storing past inputs and
utilizing them for future predictions. Example: Given "Apple is", RNN recalls "Apple" and predicts "red"
accurately.

3. Training & Backpropagation:

RNNs are trained using Backpropagation Through Time (BPTT), which adjusts weights by calculating
errors across previous time steps. This improves model accuracy for tasks like speech recognition, translation,
and stock prediction.

Difference from Fully Connected Networks:

Feature RNNs FNNs

Primary Use Sequential data, tasks with temporal General-purpose supervised learning, tasks
dependencies where input order is irrelevant

Memory Maintains a hidden state or memory of past No memory of past inputs

inputs

Architecture Recurrent connections, feedback loops Feedforward, no feedback loops

Strengths Captures temporal dependencies, handles Simple to implement, learns complex

variable-length inputs relationships

Limitations Vanishing gradient problem, Struggles with sequential data

computationally expensive

Application of RNNs:
1. NLP & Text Processing

• Language Modeling: Predicts the next word (e.g., autocomplete).

• Machine Translation: Translates text between languages.
• Sentiment Analysis: Understands emotions in text.

2. Speech Processing

• Speech-to-Text: Converts spoken words into text.

• Text-to-Speech: Generates human-like speech.

3. Time-Series Forecasting
 • Financial Predictions: Stock prices, currency trends.
• Weather Forecasting: Predicts future climate patterns.

4. Music & Video Analysis

• Music Generation: Creates new compositions.

• Video Understanding: Detects actions & gestures in videos.

5. Robotics & Automation

• Path Prediction: Helps robots navigate.

• Gesture Control: Enables interaction using movements.

Sd

Activation Functions in Neural Networks

1. Rectified Linear Unit (ReLU)
The ReLU layer introduces non linearity to the network by applying the Rectified Linear Unit activation function.
Operation: f(x) = max(0, x), meaning it outputs the input directly if positive and zero otherwise, introducing non-
linearity and computational efficiency.

How it Works:
The ReLU function can be described mathematically as follows:
f(x)=max(0,x)f(x)=max(0,x)
Where:
• x is the input to the neuron.
• The function returns x if x is greater than 0.
• If x is less than or equal to 0, the function returns 0.
 This simplicity is what makes ReLU so effective in training deep neural networks, as it helps to maintain non-
linearity without complicated transformations, allowing models to learn more efficiently.

Formula can be written as,

f(x) = [ {x if x>0
{0 if x≤0 ]

Advantages:
One of the most significant advantages of ReLu is that it helps mitigate the vanishing gradient problem. The
vanishing gradient problem is a challenge that occurs when training deep neural networks using back propagation.
• ReLU helps in overcoming the vanishing gradient problem and accelerates convergence during training.
• Relu : More computationally efficient to compute than Sigmoid like functions since Relu just needs to pick
max(0, x) and not perform expensive exponential operations as in Sigmoids
• Relu : In practice, networks with Relu tend to show better convergence performance than sigmoid.

Common usage:
• Computational Efficiency: ReLU is simple to compute, making it fast to train neural networks.
• Overcomes Vanishing Gradients: Unlike sigmoid or tanh, ReLU doesn't suffer from the vanishing gradient
problem as much, especially for positive inputs. This means that the network can learn more effectively,
even in deeper layers.
• Sparse Activation: ReLU introduces sparsity into the network, meaning that many neurons will have zero
output, which can help with generalization and reduce overfitting.
• Widely Used in CNNs: ReLU is a standard choice for activation functions in Convolutional Neural Networks
(CNNs) due to its efficiency and effectiveness.
• Good for Computer Vision and NLP: ReLU has been shown to perform well in various computer vision and
natural language processing tasks.

Limitation:
"Unfortunately, ReLU units can be fragile during training and can "die". For example, a large gradient flowing
through a ReLU neuron could cause the weights to update in such a way that the neuron will never activate on any
datapoint again. If this happens, then the gradient flowing through the unit will forever be zero from that point
on. That is, the ReLU units can irreversibly die during training since they can get knocked off the data manifold. For
example, you may find that as much as 40% of your network can be "dead" (i.e. neurons that never activate across
the entire training dataset) if the learning rate is set too high. With a proper setting of the learning rate this is less
frequently an issue."
 2. Hyperbolic Tangent (Tanh)

Tanh Activation is an activation function used for neural networks:

Historically, the tanh function became preferred over the sigmoid
function as it gave better performance for multi-layer neural networks.
But it did not solve the vanishing gradient problem that sigmoids
suffered, which was tackled more effectively with the introduction
of ReLU activations.

Tanh Activation Works:

The tanh function outputs values in the range of -1 to +1. This means that it can deal with negative values more
effectively than the sigmoid function, which has a range of 0 to 1.
Unlike the sigmoid function, tanh is zero-centered, which means that its output is symmetric around the origin
of the coordinate system. This is often considered an advantage because it can help the learning algorithm
converge faster.
Because the output of tanh ranges between -1 and +1, it has stronger gradients than the sigmoid function.
Stronger gradients often result in faster learning and convergence during training because they tend to be more
resilient against the problem of vanishing gradients when compared to the gradients of the sigmoid function.

Advantages:
The tanh function still suffers from the vanishing gradient problem. During backpropagation, the gradients of
the tanh function can become very small (close to zero). This issue is particularly problematic for deep networks
with many layers; the gradients of the loss function may become too small to make significant changes in the
weights during training as they propagate back to the initial layers. This can drastically slow down the training
process and can lead to poor convergence properties.
The tanh function is frequently used in the hidden layers of a neural network. Because of its zero-centered
nature, when the data is also normalized to have mean zero, it can result in more efficient training.

Common use of Tanh:

The Tanh (hyperbolic tangent) function is commonly used as an activation function in neural networks,
particularly in hidden layers of recurrent neural networks (RNNs) and certain layers of feedforward neural
networks, where its output range of -1 to 1 and zero-centered activations are beneficia

Limitation of the tanh function:

The limitation of the tanh function is that it suffers from the vanishing gradient problem. This simply means that
as the input function becomes very small or very large depending on the case, the gradient of the function
approaches zero, thus making it difficult for the network to update the weights of the earlier layers to learn from
the input data. This is usually a big problem in deep neural networks having many layers, since the gradients can
become extremely small by the time they reach the earlier layers. This leads to slow convergence and poor
performance. Let's illustrate this using a code example.
Question 3: Exploring Loss Functions
The loss function is a measurable way to gauge the performance and accuracy of a machine learning model. In this
case, the loss function acts as a guide for the learning process within a model or machine learning algorithm.
The role of the loss function is crucial in the training of machine learning models and includes the following:
• Performance measurement: Loss functions offer a clear metric to evaluate a model's performance by
quantifying the difference between predictions and actual results.
• Direction for improvement: Loss functions guide model improvement by directing the algorithm to adjust
parameters(weights) iteratively to reduce loss and improve predictions.
• Balancing bias and variance: Effective loss functions help balance model bias (oversimplification) and
variance (overfitting), essential for the model's generalization to new data.
• Influencing model behavior: Certain loss functions can affect the model's behavior, such as being more
robust against data outliers or prioritizing specific types of errors.

1. Mean Squared Error (MSE) Mean Absolute Error

Mean Absolute Error (MAE), also known as L1 Loss, is a loss function used in
regression tasks that calculates the average absolute differences between predicted
values from a machine learning model and the actual target values. Unlike Mean
Squared Error (MSE), MAE does not square the differences, treating all errors with
equal weight regardless of their magnitude.
The mathematical equation for Mean Absolute Error (MAE) or L1 Loss is:
MAE = (1/n) * Σ|yᵢ - ȳ|

When to use MAE

MAE measures the average absolute difference between the predicted and actual values. Unlike MSE, MAE
does not square the differences, which makes it less sensitive to outliers. Compared to Mean Squared Error (MSE),
Mean Absolute Error (MAE) is inherently less sensitive to outliers because it assigns an equal weight to all errors,
regardless of their magnitude.
This means that while an outlier can significantly skew the MSE by contributing a disproportionately large error
when squared, its impact on MAE is much more contained. An outlier's influence on the overall error metric is
minimal when using MAE as a loss function. In contrast, MSE amplifies the effect of outliers due to the squaring
of error terms, affecting the model's error estimation more substantially.

Why It’s Suitable:

The Huber Loss function / Smooth Mean Absolute Error effectively combines two components for handling
errors differently, with the transition point between these components determined by the threshold δ:
Quadratic Component for Small Errors: For errors smaller than δ, it uses the quadratic component (1/2) * (f(x) - y)^2
Linear Component for Large Errors: For errors larger than δ, it applies the linear component δ * |f(x) - y| - (1/2) * δ^2
Huber loss operates in two modes that are switched based on the size of the calculated difference between the
actual target value and the prediction of the machine learning algorithm. The key term within Huber Loss is delta
(δ). Delta is a threshold that determines the numerical boundary at which the Huber Loss utilizes the quadratic
application of loss or linear calculation.
 The quadratic component of Huber Loss characterizes the advantages of MSE that penalize outliers; within
Huber Loss, this is applied to errors smaller than delta, which ensures a more accurate prediction from the model.

Real world Application:

• Predicting Delivery Times:
A food delivery company might use MAE to train a model that estimates delivery times, as it's less
sensitive to outliers caused by rare events like traffic incidents or roadworks.
• House Price Prediction:
MAE can be used in models that predict house prices, as it treats all errors equally, making it robust
to outliers.
• Demand Forecasting:
MAE can be used in models that predict demand for a product or service, as it treats all errors
equally, making it robust to outliers.
• Stock Price Forecasting:
MAE can be used in models that predict stock prices, as it treats all errors equally, making it robust
to outliers.
• Other Regression Tasks:
MAE can be used in any regression task where the goal is to predict a continuous variable, and
where all errors are equally important.

2. Cross-Entropy Loss (for Multi-Class Classification)

Cross-Entropy Loss(BCE) is a performance measure for
classification models that outputs a prediction with a probability
value typically between 0 and 1, and this prediction value
corresponds to the likelihood of a data sample belonging to a class
or category. In the case of Binary Cross-Entropy Loss, there are two
distinct classes. But notably, a variant of cross-entropy
loss, Categorical Cross-Entropy applies to multiclass classification scenarios.

• Loss: This is a mathematical quantification of the margin/difference between the prediction of a machine
learning algorithm and the actual target value.
• Entropy: A simple definition of entropy is that it is a calculation of the degree of randomness or disorder
within a system
• Cross Entropy: This is a term commonly utilised in information theory, and it measures the differences
between two probability distributions that can be used to identify an observation.
• Binary: This is an expression of numerical digits using either of two states, 0 or 1. This is extended to the
definition of Binary Classification where we ditingus=ish two classes(A and B) using binary representation,
where class A is assigned the numerical representation of 0 and class B is assigned 1.
Usage:
The equation above specifically applies to a scenario where the machine learning algorithm will make a
classification between two classes. This is a binary classification scenario.
 As noted in the equation by the negative symbol: ‘-’ BCE calculates the loss by determining the negative of two
terms, and for several predictions or data samples, the average of the negative of the following two terms:
1. The logarithm of the model’s predicted probability went the positive class is present y * log(f(x))
2. The logarithm of 1 minus the predicted probability of the negative class: (1 - y) * log(1 - f(x))
The BCE loss function penalizes inaccurate predictions, which are predictions that have a significant difference
from the positive class or, in other words, have a high quantification of entropy. When BCE is utilized as a
component within learning algorithms, this encourages the model to refine its predictions, which are probabilities
for the appropriate class during its training.

Why It’s Suitable:

Selecting the appropriate loss function to apply to a machine learning algorithm is essential, as the model's
performance heavily depends on the algorithm's ability to learn or adapt its internal weights to fit a dataset.
A machine learning model or algorithm's performance is defined by the loss function utilized, mainly because the
loss function component affects the learning algorithm used to minimize the model's error loss or cost function
value. Essentially, the loss function impacts the model's ability to learn and adapt the value of its internal weights
to fit the patterns within a dataset.
When appropriately selected, the loss function enables the learning algorithm to effectively converge to an
optimal loss during its training phase and generalize well to unseen data samples. An appropriately selected loss
function acts as a guide, steering the learning algorithm towards accuracy and reliability, ensuring that it captures
the underlying patterns in the data while avoiding overfitting or underfitting.

Applications:
MAE is used in various regression tasks, including:
• Demand Forecasting: Estimating the average error in predicting sales or customer demand.
• Housing Price Estimation: Assessing the accuracy of models that predict house prices.
• Sentiment Analysis: Evaluating the accuracy of models that predict sentiment scores.
• Wind Speed Forecasting: Measuring how close forecast values are to observed wind speed outcomes.