Deep Neural Networks

Syllabus
● Review of perceptron and multilayer perceptron
● Activation functions: Leaky ReLU, Softplus, Softmax
● Deep feedforward neural networks: architecture and components
● Backpropagation in Deep Neural Networks
● Loss functions for classification
● Overfitting and underfitting in deep neural networks

1. Review of perceptron and multilayer perceptron

Artificial Neural Networks (ANNs) are computational models inspired by the
structure and functioning of the human brain. They consist of interconnected processing
units called neurons, which work collectively to learn patterns and relationships from data.
Unlike traditional rule-based algorithms, ANNs learn automatically through exposure to
examples, making them highly effective for complex tasks where explicit programming is
difficult. Their ability to approximate nonlinear functions allows them to model intricate
real-world phenomena across diverse domains.
At the core of an ANN is the artificial neuron, which receives inputs, applies weights,
performs a weighted summation, and passes the result through an activation function.
Activation functions such as Sigmoid, ReLU, or Tanh introduce nonlinearity, enabling the
network to learn complex mappings. Neurons are arranged in layers: an input layer that
receives data, one or more hidden layers that extract features or patterns, and an output
layer that produces the final prediction. The structure and depth of these layers determine
the network’s learning capacity.

ANNs learn through a process called training, where they iteratively adjust weights to
minimize prediction errors. This is typically achieved using optimization algorithms such as
gradient descent and the backpropagation technique. During training, the network assesses
how far its output deviates from the expected result and updates the weights accordingly.
This iterative process continues until the network achieves a desired level of accuracy.
Training may involve large datasets, and the network’s performance is influenced by
hyperparameters such as learning rate, batch size, and number of epochs.

Artificial Neural Networks have numerous advantages, including adaptability,

robustness to noisy data, and strong generalization capabilities. They are widely used in
tasks such as image classification, speech recognition, medical diagnosis, financial
forecasting, and control systems. Despite their strengths, ANNs also present challenges such
as high computational requirements, long training times, and difficulty in interpreting how
decisions are made. Nonetheless, they remain a foundational component of modern
machine learning and form the basis for more advanced architectures such as Convolutional
Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and deep learning systems.

1.1 Key Characteristics

1. Nonlinear Processing: Neural networks can model highly complex and nonlinear
relationships due to nonlinear activation functions in hidden layers.
2. Learning from Data: ANNs improve performance by learning patterns directly from
training data rather than relying on explicitly programmed rules.
3. Generalization Ability: Once trained, neural networks can generalize from previously
seen data to new, unseen inputs, enabling reliable predictions.
4. Adaptive Nature: Weights and biases are continuously adjusted during training, allowing
the network to adapt to changing input patterns.

1.2 Why Neural Networks are needed

 Can model complex and nonlinear relationships that traditional algorithms cannot.
 Do not require strict assumptions like linearity or normality.
 Learn patterns directly from data without explicit human-defined rules.
 Handle high-dimensional, unstructured, or noisy data effectively.
 Useful in tasks such as image recognition, speech processing, NLP, and biomedical
analysis.
 Can generalize to new, unseen data, making them suitable for dynamic
environments.
 Adaptable and robust to noise and incomplete data.
 1.3 Perceptron
The Perceptron is the simplest type of artificial neural network and serves as the
fundamental building block for more complex neural networks. It was introduced by Frank
Rosenblatt in 1958 and is designed to perform binary classification tasks. The perceptron
mimics the behavior of a biological neuron: it receives multiple inputs, processes them using
weights, applies a bias, and passes the result through an activation function to produce an
output. This output typically represents one of two classes in a classification problem.

Mathematically, the perceptron computes a weighted sum of its inputs and adds a bias
term. The result is then passed through a step (threshold) activation function, which
determines whether the neuron “fires” (output = 1) or not (output = 0). The model can be
represented as:

( ) (∑ )
Where wᵢ are weights, xᵢ are inputs, b is bias, and f is the activation function. The step
function converts the continuous weighted sum into a binary output, making the perceptron
suitable for linearly separable data.

The learning process of a perceptron involves adjusting the weights and bias based
on the error between predicted and actual outputs. The Perceptron Learning Rule updates
weights iteratively using gradient-free methods: if the output is correct, no change is made;
if incorrect, weights are adjusted to reduce the error. Training continues until all training
samples are classified correctly or a maximum number of iterations is reached. This
supervised learning approach enables the perceptron to learn simple patterns and decision
boundaries.

Despite its historical importance, the perceptron has limitations. It can only solve
linearly separable problems; it fails for problems like XOR, which require non-linear decision
boundaries. However, multilayer perceptrons (MLPs) with hidden layers overcome this
limitation by using nonlinear activation functions. The perceptron remains a key concept in
understanding neural networks, serving as a foundation for more advanced architectures in
machine learning and deep learning.

1.4 Multi-layer Perceptron

A Multi-Layer Perceptron (MLP) is a class of feedforward artificial neural network
composed of multiple layers of interconnected neurons. Unlike a single-layer perceptron, an
MLP contains one input layer, one or more hidden layers, and an output layer, allowing it to
learn complex, nonlinear relationships in data. Each neuron performs a weighted sum of its
inputs and passes the result through a nonlinear activation function such as sigmoid, tanh,
or ReLU. This nonlinear transformation enables the MLP to approximate any continuous
function, making it a universal function approximator. The network's learning process
typically involves backpropagation, where errors at the output layer are propagated
backward to adjust the weights and biases throughout the network.

MLPs are widely used in supervised learning tasks such as classification, regression,
pattern recognition, and time-series prediction. The hidden layers extract intricate patterns
and higher-level features from the input data, while the output layer produces the final
prediction. The capacity of an MLP depends on the number of hidden layers, the number of
neurons in each layer, and the choice of activation functions. Because of their ability to
model complex mappings, MLPs serve as the foundation for many advanced neural
architectures and remain one of the most commonly used neural networks in practical
applications.
Types and Roles of Various Layers in an MLP

A Multi-Layer Perceptron (MLP) consists of three primary types of layers: the input
layer, one or more hidden layers, and the output layer. Each layer has a distinct functional
role in processing information and transforming it into meaningful predictions. The input
layer serves as the entry point of the network and holds the feature values from the dataset.
Although it does not perform any computation, it structures the data in a form that the
subsequent layers can interpret. The neurons in the input layer simply pass the input vector
to the first hidden layer without applying any activation or weight transformations.

The hidden layers are the computational core of the MLP and are responsible for
extracting patterns, learning relationships, and performing nonlinear transformations. Each
neuron in a hidden layer computes a weighted sum of its inputs and passes the result
through an activation function, enabling the network to model complex nonlinear functions.
Multiple hidden layers allow the MLP to build hierarchical representations of data, where
earlier layers capture simple features and deeper layers learn more abstract concepts. The
number of hidden layers and neurons determines the network’s learning capacity and
directly influences model performance.

Finally, the output layer generates the final predictions of the model. Its structure
and activation function depend on the type of task being performed. For classification tasks,
the output layer may use Softmax or sigmoid activations to produce probabilities, whereas
regression tasks typically use a linear activation to generate continuous numeric values. The
output layer converts the learned internal representations into meaningful outputs that
align with the target variable. Together, the three types of layers form a complete MLP
architecture capable of learning and generalizing from data through supervised training.

2. Activation functions
An activation function is a mathematical transformation applied to the output of a
neuron in a neural network. After computing a weighted sum of inputs, the neuron passes
this value through the activation function, which determines whether the neuron should be
activated and how strongly it should contribute to the next layer. Activation functions
introduce nonlinearity into the neural network, allowing it to learn and represent complex
patterns in data. Without these functions, the entire neural network would behave like a
simple linear model and would not be able to model real-world problems effectively.

The importance of activation functions lies in their ability to enable deep neural
architectures to extract high-level features and perform complex decision-making. They
allow neural networks to approximate nonlinear relationships such as image features,
speech patterns, medical signals, and other intricate datasets. Activation functions also play
a critical role in controlling gradient flow during training. Proper activation selection helps
avoid issues such as vanishing or exploding gradients, ensuring stable and efficient learning.
Overall, activation functions form the core mechanism that gives neural networks their
expressive power and distinguishes them from traditional linear models.

2.2 Rectified Linear Unit (ReLU)

The Rectified Linear Unit (ReLU) is a widely used activation function in deep neural
networks, especially in convolutional neural networks (CNNs). It introduces non-linearity
into the model by mapping negative input values to zero while keeping positive values
unchanged. The ReLU function is a piecewise linear function that outputs the input directly
if it is positive; otherwise, it outputs zero. In simpler terms, ReLU allows positive values to
pass through unchanged while setting all negative values to zero. This helps the neural
network maintain the necessary complexity to learn patterns while avoiding some of the
pitfalls associated with other activation functions, like the vanishing gradient problem. The
ReLU function can be described mathematically as follows:

( ) ( )
Where:

 x is the input to the neuron.

 The function returns x if x is greater than 0.
 If x is less than or equal to 0, the function returns 0.

This simplicity is what makes ReLU so effective in training deep neural networks, as it
helps to maintain non-linearity without complicated transformations, allowing models to
learn more efficiently. If we plot the graph of ReLU activation function, it will appear like
this:
Advantages of ReLU
 Computational Simplicity
 Mitigates Vanishing Gradient Problem
 Sparse Activation
 Improved Convergence

Drawbacks of ReLU
 Dying ReLU Problem
 Unbounded Output
 Noisy Gradients

2.2 Leaky Rectified Linear Unit (Leaky ReLU)

The primary motivation for using Leaky ReLU is to mitigate the "dying ReLU"
problem. In a standard ReLU function, any negative input is instantly converted to zero.
While this creates computational efficiency through sparsity, it can lead to a state where a
neuron effectively dies—it stops outputting anything other than zero and, crucially, its
gradient becomes zero. When the gradient is zero, the optimization algorithm responsible
for reducing error cannot adjust the weights connected to that neuron, rendering it useless
for the remainder of model training.

Leaky ReLU solves this by allowing a small, non-zero gradient when the unit is not
active. Instead of a flat line at zero for negative inputs, it introduces a small slope (typically
0.01). This slight "leak" ensures that the neuron retains some influence on the network's
output, preventing the vanishing gradient problem on a local scale. By keeping the gradients
alive, Leaky ReLU helps build more robust feature extractors, especially in deep networks
used for computer vision (CV). The mathematical representation of Leaky ReLU is given
below:
 ( ) {

where α is a small constant (often set to 0.01).

Advantages of Leaky ReLU

 Prevents Dying ReLU Problem
 Improved Gradient Flow
 Faster Convergence
 Better Feature Representation
 Simple and Computationally Efficient

Limitations of Leaky ReLU

 Fixed Negative Slope (α)
 Not Zero-Centered
 Unbounded Positive Output
 No Strong Theoretical Guarantee
 Slight Bias in Negative Region

2.3 Softplus Activation Function

The Softplus activation function is a smooth approximation of the Rectified Linear

Unit (ReLU). While ReLU introduces a sharp discontinuity at zero, Softplus replaces this
abrupt transition with a continuous and differentiable curve. This smooth behavior is
beneficial in optimization because gradient-based learning algorithms operate more reliably
when activation functions are differentiable everywhere. Unlike ReLU, which outputs
exactly zero for negative inputs, Softplus produces small positive outputs, ensuring that
gradients never become exactly zero. This helps maintain stable weight updates during
training and reduces the risk of inactive neurons.

Softplus is particularly useful in probabilistic and deep learning models where

smoothness is important for numerical stability. Because the function grows logarithmically
for large negative inputs and behaves approximately linear for large positive inputs, it
preserves the beneficial properties of ReLU while avoiding its discontinuity. This makes
Softplus attractive in models requiring continuous gradient flow, such as energy-based
models and Bayesian neural networks. The mathematical representation of Softplus is given
below:

( ) ( )
The derivative of Softplus is the sigmoid function, which guarantees smooth gradient
transitions:

( )

This connection links Softplus to probabilistic interpretations and stabilizes

backpropagation.
Why Use Softplus in Neural Networks?
The Softplus activation function is particularly useful for the following reasons:
 Smooth Approximation of ReLU: The Softplus function is often seen as a smoother
version of the ReLU function. While ReLU is simple and effective, it can have issues, such
as causing neurons to "die" if they always output zero for negative inputs. Softplus
avoids this issue by providing a smooth, continuous output for both positive and
negative inputs.
 Differentiability: Softplus is a differentiable function, unlike ReLU, which has a
discontinuity at zero. The continuous and differentiable nature of Softplus makes it
easier for gradient-based optimization algorithms to work effectively, ensuring smooth
learning during training.
 Preventing Dying Neurons: In the case of ReLU, when the input is negative, the output is
exactly zero, which can lead to dead neurons that do not contribute to learning. Softplus
gradually approaches zero for negative values, ensuring that neurons always produce
some non-zero output and continue contributing to the learning process.
 Numerical Stability: The Softplus function has better numerical stability than some other
activation functions because it avoids the issues that arise from very large or very small
values. It has a smooth output, and for very large or very small inputs, the function
behaves predictably, reducing the risk of overflow or underflow in computations.

Advantages of Softplus
 Smooth and fully differentiable everywhere
 Prevents zero gradients (no dead neurons)
 Stable gradient flow during training
 Reduces sharp transitions seen in ReLU
 Useful in probabilistic and continuous models

Limitations of Softplus
 Computationally more expensive than ReLU
 Slower convergence in some deep networks
 Does not create sparse activations
 Can cause small gradient saturation for large negative inputs
 Less commonly used in modern CNNs

2.4 Softmax Activation Function

The Softmax activation function is used primarily in the output layer of multi-class
classification neural networks. Its purpose is to convert raw model outputs (logits) into a
probability distribution across multiple classes. Unlike activation functions that operate
independently on each neuron, Softmax considers all outputs simultaneously and
normalizes them so that their sum equals one. This makes it ideal for classification problems
where the model must choose one class out of many possible categories.
 Softmax enhances interpretability because its outputs can be directly treated as class
probabilities. During training, it works closely with cross-entropy loss to measure prediction
error and guide gradient updates. The exponential component of Softmax amplifies
differences between logits, making the largest value dominate the distribution. This
encourages confident predictions while still preserving gradient information for all classes.
The mathematical representation of Softmax is:

( )
∑

where is the input to the i-th neuron and the denominator sums overall output neurons.
This normalization ensures a valid probability distribution .

Why Use Softmax in the Last Layer

The Softmax Activation function is typically used in the final layer of a classification neural
network because:
 It transforms the model raw output into interpretable probabilities.
 It ensures the outputs are mutually exclusive suitable for problems where each sample
belongs to exactly one class.
 It works seamlessly with the Cross Entropy Loss Function which measures the difference
between predicted and actual probabilities.

Advantages of Softmax
 Produces interpretable probability outputs
 Suitable for multi-class classification
 Works well with cross-entropy loss
 Encourages confident predictions
 Preserves gradient information for all classes
Limitations of Softmax
 Sensitive to large input values (numerical instability)
 Can produce overconfident predictions
 Assumes mutually exclusive classes
 Not suitable for multi-label classification
 Computationally expensive for very large class counts

3. Deep feedforward neural networks

A deep feedforward neural network (DNN) is a fundamental model in deep learning

in which information flows strictly in one direction, from the input layer through multiple
hidden layers to the output layer. Unlike recurrent or feedback networks, there are no
cycles or loops, making the structure mathematically straightforward and efficient for
supervised learning tasks. The term deep refers to the presence of several hidden layers that
allow the network to learn increasingly abstract representations of data. Because of this
layered representation capability, DNNs form the backbone of many modern AI systems
used in image recognition, speech processing, natural language tasks, and predictive
analytics.

The architecture of a deep feedforward network consists of stacked layers of

artificial neurons connected by learnable weights and biases. Each neuron computes a
weighted sum of its inputs and applies a nonlinear activation function, enabling the network
to model complex, non-linear relationships. As data propagates through the layers, early
layers tend to learn simple features, while deeper layers combine these into higher-level
abstractions. This hierarchical feature learning is what allows deep networks to outperform
shallow models on complex datasets.

Training a DNN involves adjusting weights to reduce prediction error using

backpropagation and gradient-based optimization algorithms such as stochastic gradient
descent or Adam. Backpropagation calculates how much each parameter contributes to the
error and updates it accordingly. The effectiveness of training depends on proper activation
functions, loss functions, and optimization strategies. With sufficient data and careful
tuning, deep feedforward networks can approximate highly complex functions, making
them powerful universal learners for a wide range of real-world applications.

Core Architectural Components

1. Input Layer

The input layer receives raw data features.

 Represents feature vector

 No computation beyond passing values
  Dimension equals number of input variables

Example: pixel values, sensor data, tabular features

2. Hidden Layers

Hidden layers perform nonlinear transformations.

 Multiple layers enable deep representation learning

 Each neuron computes:
weighted sum + bias → activation function
 Learn intermediate features automatically

Depth increases model expressiveness.

3. Output Layer

The output layer produces final predictions.

 Activation depends on task:

o Softmax → multi-class classification
o Sigmoid → binary classification
o Linear → regression
 Converts learned representation into decision

4. Weights and Biases

 Weights control strength of connections

 Bias shifts activation threshold
 Learned during training
 Define model capacity

5. Activation Functions

 Introduce nonlinearity.
 Without activation → network becomes linear.
 Nonlinearity enables learning complex mappings.
4. Backpropagation in Deep Neural Networks
Backpropagation is the fundamental algorithm that enables deep neural networks to
learn from data by systematically reducing prediction error. The training process begins
with forward propagation, where input data passes through each layer of the network to
produce an output. A loss function then measures the difference between predicted and
actual values. Backpropagation computes how sensitive this loss is to each parameter in
the network and sends error signals backward from the output layer to earlier layers.
This backward flow of information allows the network to assign responsibility for the
error to individual weights and biases.
The algorithm relies on the chain rule of calculus to efficiently compute gradients in
multi-layer systems. Instead of recalculating derivatives from scratch for each
parameter, backpropagation reuses intermediate results, making training
computationally feasible even for networks with millions of parameters. Once gradients
are computed, an optimization algorithm updates parameters in small steps to minimize
loss. This process is repeated across many training iterations, allowing the network to
gradually refine its internal representations. The combination of gradient computation
and iterative optimization is what enables deep models to approximate highly complex
functions.
In deep architectures, the behavior of gradients is critical to successful learning. If
gradients shrink exponentially as they move backward, earlier layers learn very slowly —
a phenomenon known as the vanishing gradient problem. Conversely, extremely large
gradients can destabilize training, causing oscillations or divergence. Architectural
choices such as ReLU-family activations, residual connections, and normalization layers
help maintain stable gradient flow. Proper weight initialization and adaptive optimizers
like Adam also contribute to training stability. These improvements allow modern deep
networks to train efficiently despite their depth.
 Backpropagation also plays a key role in regularization and generalization. During
training, techniques such as dropout, batch normalization, and early stopping influence
how gradients propagate and how the model distributes learning across layers. These
strategies prevent the network from memorizing training data and encourage it to learn
robust features that generalize to unseen inputs. Monitoring gradient behavior helps
diagnose training issues, such as overfitting, underfitting, or stalled convergence.

Advantages of Backpropagation
 Efficient computation of gradients using the chain rule
 Enables training of very deep neural networks
 Scales to millions of parameters
 Works with many optimization algorithms (SGD, Adam, RMSProp)
 Supports complex nonlinear function learning
 Improves accuracy through iterative error minimization
 Forms the foundation of modern deep learning systems

Limitations of Backpropagation
 Suffers from vanishing and exploding gradient problems
 Requires differentiable activation functions
 Computationally expensive for large networks
 Needs large labelled datasets for effective training
 Sensitive to hyperparameter tuning
 Can converge to local minima or saddle points
 Training may be slow without hardware acceleration