Fundamental of Deep
learning
Presented by:
Dr . Mona Hussein Alnaggar
2023-2024
1st term
Lecture 3
Agenda
• Neural network Model building steps
• List of Deep Learning Layers
• Activation function definition
• Activation function types
• Types of Artificial Neural Networks
Neural network Model building steps
Algorithm:
1. Visualizing the input data
2. Deciding the shapes of Weight and bias matrix
3. Initializing matrix, function to be used
4. Implementing the forward propagation method
5. Implementing the cost calculation
6. Backpropagation and optimizing
7. prediction and visualizing the output
Deep Neural net with forward and back
propagation
Weights and bias:
The weights and the bias that is going to be used for both the layers have to be declared initially and also
among them the weights will be declared randomly in order to avoid the same output of all units, while the
bias will be initialized to zero.
A bias value (b) is important to full control of the activation function (i.e., the output) for successful learning. This is
a sort of regularization
Multi-Layer Feed Forward Networks
Let’s understand how errors are calculated and weights are updated in backpropagation networks(BPNs).
Consider the following network in the below figure.
Multi-Layer Feed Forward Networks
There are three steps to solve the problem:
1. Computing the output, y.
2. Backpropagation of errors, i.e., between output and hidden layer, hidden and input layer.
3. Updating weights.
List of Deep Learning Layers
To specify the architecture of a neural network with all layers connected sequentially, create an array of
layers directly. To specify the architecture of a network where layers can have multiple inputs or outputs,
use a LayerGraph object. Use the following functions to create different layer types.
Input Layers:
FUNCTION DESCRIPTION
• Inputs images to a network
imageInputLayer
• Applies data normalization.
sequenceInputLayer • Inputs sequence data to a network.
List of Deep Learning Layers
Learnable Layers:
FUNCTION DESCRIPTION
• Applies sliding filters to the input .
convolution2dLayer • It convolves the input by moving the filters along the input vertically and horizontally and computing
the dot product of the weights and the input, and then adding a bias term.
transposedConv2dLayer • It upsamples feature maps.
fullyConnectedLayer • Multiplies the input by a weight matrix and then adds a bias vector.
• It is a recurrent neural network (RNN) layer that enables support for time series and sequence data in
a network.
lstmLayer • It performs additive interactions, which can help improve gradient flow over long sequences during
training .
• They are best suited for learning long-term dependencies.
List of Deep Learning Layers
Activation Layers:
FUNCTION DESCRIPTION
It performs a threshold operation to each element of the input, where any
reluLayer
value less than zero is set to zero.
It performs a simple threshold operation, where any input value less than
leakyReluLayer
zero is multiplied by a fixed scalar
It performs a simple threshold operation, where any input value less than
clippedReluLayer zero is set to zero . Any value above the clipping ceiling is set to that clipping
ceiling.
List of Deep Learning Layers
Normalization and Dropout Layers:
FUNCTION DESCRIPTION
• It normalizes each input channel across a mini-batch.
• The layer first normalizes the activations of each channel by subtracting the
mini-batch mean and dividing by the mini-batch standard deviation .
• Then, the layer shifts the input by a learnable offset and scales it by a
batchNormalizationLayer
learnable scale factor .
• Use batch normalization layers between convolutional layers and
nonlinearities, such as ReLU layers, to speed up training of convolutional
neural networks and reduce the sensitivity to network initialization.
crossChannelNormalizationLayer • It carries out channel-wise normalization.
dropoutLayer • It randomly sets input elements to zero with a given probability.
List of Deep Learning Layers
Pooling Layers:
FUNCTION DESCRIPTION
• It performs down sampling by dividing the input into rectangular
averagePooling2dLayer
pooling regions and computing the average values of each region.
• It performs down sampling by dividing the input into rectangular
maxPooling2dLayer
pooling regions, and computing the maximum of each region.
maxUnpooling2dLayer • It unpools the output of a max pooling layer.
List of Deep Learning Layers
Combination Layers:
FUNCTION DESCRIPTION
• It adds multiple inputs element-wise .
• Specify the number of inputs to the layer when you create it.
• The inputs have names ‘in1’, ‘in2’, …, ‘inN’, where N is the number of inputs.
additionLayer
• Use the input names when connecting or disconnecting the layer to other
layers using connectLayers or disconnectLayers.
• All inputs to an addition layer must have the same dimension.
• It takes multiple inputs that have the same height and width .
depthConcatenationLayer
• It concatenates them along the third dimension.
ML – List of Deep Learning Layers
Output Layers:
FUNCTION DESCRIPTION
softmaxLayer • It applies a softmax function to the input.
• It holds the name of the loss function the software uses for training the
classificationLayer
network for multiclass classification.
• It holds the name of the loss function the software uses for training the
regressionLayer
network for regression, and the response names.
Activation Functions
Activation Functions
What is an Activation function ?
In artificial neural networks, the activation function of a node defines the output of that node or neuron
for a given input or set of inputs. This output is then used as input for the next node and so on until a
desired solution to the original problem is found.
It maps the resulting values into the desired range such as between 0 to 1 or -1 to 1 etc. It depends upon
the choice of the activation function. For example, the use of the logistic activation function would map all
inputs in the real number domain into the range of 0 to 1.
Activation Functions
when we multiply each of them features with a weight (w1, w2, …, wm) and sum them all together, node output =
activation(weighted sum of inputs).
Activation Functions cont.
Mathematically,
Now the value of net input can be any anything from -inf to +inf. The neuron doesn’t really know how to
bound to value and thus is not able to decide the firing pattern. Thus, the activation function is an
important part of an artificial neural network. They basically decide whether a neuron should be activated
or not. Thus, it bounds the value of the net input.
The activation function is a non-linear transformation that we do over the input before sending it to the
next layer of neurons or finalizing it as output.
Activation Function
The activation function is a function that performs calculations to provide an output that may act as
input for the next neurons. An ideal activation function should handle non-linear relationships by
using the linear concepts and it should be differentiable so as to reduce the errors and adjust the
weights accordingly.
Types of Pytorch Activation Function
1. Step Activation Function
2. Sigmoid Activation Function
3. ReLU Activation Function
4. Leaky ReLU Activation Function
5. Tanh Activation Function
6. Softmax Activation Function
Activation Functions cont.
Several different types of activation functions are used in Deep
Learning. Some of them are explained below:
1. Step Function:
Step Function is one of the simplest kind of activation functions. In this,
we consider a threshold value and if the value of net input say y is
greater than the threshold then the neuron is activated.
Mathematically,
Activation Functions cont.
2. Sigmoid Function:
Sigmoid function is a widely used activation function. It is defined as:
Graphically,
The biggest advantage that it has overstep and linear function is that it
is non-linear.
This is an incredibly cool feature of the sigmoid function.
This essentially means that when I have multiple neurons having
sigmoid function as their activation function –
the output is nonlinear as well. The function ranges from 0-1 having an
S shape.
Activation Functions in Pytorch
Sigmoid Activation Function:
Sigmoid Function is a non-linear and differentiable activation function. It is an S-shaped curve that does
not pass through the origin. It produces an output that lies between 0 and 1. The output values are often
treated as a probability. It is often used for binary classification. It is slow in computation and, graphically
Sigmoid has the following transformative behavior:
Activation Functions cont.
Sigmoid Function: It is by far the most commonly used activation function in neural networks. The need
for sigmoid function stems from the fact that many learning algorithms require the activation function to
be differentiable and hence continuous. There are two types of sigmoid function:
1. Binary Sigmoid Function 2. Bipolar Sigmoid Function
Activation Functions cont.
3. ReLU:
The ReLU function is the Rectified linear unit. It is the most widely
used activation function. It is defined as:
The main advantage of using the ReLU function over other activation
functions is that it does not activate all the neurons at the same time.
What does this mean ? If you look at the ReLU function if the input is
negative, it will convert it to zero and the neuron does not get activated.
Activation Functions in Pytorch
ReLU Activation Function:
ReLU stands for Rectified Linear Activation function. It is a non-linear function and, graphically ReLU has
the following transformative behavior:
O
Activation Functions cont.
4. Leaky ReLU:
Leaky ReLU function is nothing but an improved version of the ReLU
function .
Instead of defining the Relu function as 0 for x less than 0, we define
it as a small linear component of x. It can be defined as:
Activation Functions in Pytorch
Leaky ReLU Activation Function:
Leaky ReLU Activation Function or LReLU is another type of activation function which is similar to ReLU
but solves the problem of ‘dying’ neurons and, graphically Leaky ReLU has the following transformative
behavior:ion Function:
5- Tanh Activation Function:
5- Tanh function is a non-linear and differentiable function like the sigmoid function, but
output values range from -1 to +1. It is an S-shaped curve that passes through the origin
and, graphically Tanh has the following transformative behavior:
The problem with the Tanh Activation function is it is slow, and the vanishing gradient
problem persists. Let us illustrate the use of the Tanh function with the help of a Python
Program.
Tanh Activation Function cont.:
Tanh function is a non-linear and differentiable function similar to the sigmoid function, but output values
range from -1 to +1. It is an S-shaped curve that passes through the origin and, graphically Tanh has the
following transformative behavior:
Tanh Activation Function cont. :
Hyperbolic Tangent Function: It is bipolar in nature. It is a widely adopted activation function for a special
type of neural network known as Backpropagation Network. The hyperbolic tangent function is of the
form
6- Softmax Activation Function:
•The softmax function is different from other activation functions as it is
placed at the last to normalize the output.
•activation functions can be used in combination with Softmax to produce
the output in probabilistic form.
• It is used in multiclass classification and generates an output of
probabilities whose sum is 1. The range of output lies between 0 and 1.
•Softmax has the following transformative behavior:
Some Important terminologies and mathematical concept s
• Propagation is a procedure to repeatedly adjust the weights so as to minimize the difference between
actual output and desired output.
• Hidden Layers is which are neuron nodes stacked in between inputs and outputs, allowing neural
networks to learn more complicated features (such as XOR logic).
• Backpropagation is a procedure to repeatedly adjust the weights so as to minimize the difference
between actual output and desired output.
• Gradient Descent is used while training a machine learning model. It is an optimization algorithm, based
on a convex function, that tweaks its parameters iteratively to minimize a given function to its local
minimum. A gradient measures how much the output of a function changes if you change the inputs a
little bit.
What are the types of Artificial Neural Networks?
Feedforward Neural Network:
• The feedforward neural network is one of the most basic artificial neural networks.
• In this ANN, the data or the input provided travels in a single direction.
• It enters into the ANN through the input layer and exits through the output layer while hidden layers
may or may not exist.
• feedforward neural network has a front-propagated wave only and usually does not have
backpropagation.
Convolutional Neural Network:
•A Convolutional neural network has some similarities to the feed-forward neural network,
•the connections between units have weights that determine the influence of one unit on
another unit.
•CNN has one or more than one convolutional layer that uses a convolution operation on the
input and then passes the result obtained in the form of output to the next layer.
•CNN has applications in speech and image processing which is particularly useful in computer
vision.
Modular Neural Network:
• A Modular Neural Network contains a collection of different neural networks that work independently
towards obtaining the output with no interaction between them.
• Each of the different neural networks performs a different sub-task by obtaining unique inputs
compared to other networks.
• The advantage of this modular neural network is that it breaks down a large and complex
computational process into smaller components, thus decreasing its complexity while still obtaining
the required output.
Radial basis function Neural Network:
•Radial basis functions are those functions that consider the distance of a point concerning
the center.
• RBF functions have two layers.
•In the first layer, the input is mapped into all the Radial basis functions in the hidden layer
and then the output layer computes the output in the next step.
• Radial basis function nets are normally used to model the data that represents any
underlying trend or function.
Recurrent Neural Network:
• The Recurrent Neural Network saves the output of a layer and feeds this output back to the input to
better predict the outcome of the layer.
• The first layer in the RNN is quite similar to the feed-forward neural network and the recurrent neural
network starts once the output of the first layer is computed.
• After this layer, each unit will remember some information from the previous step so that it can act as
a memory cell in performing computations.