Practical No:1
Title: Write a program to scheme a few activation functions that are used in
neural networks
Objective: To Study Use of Activation Functions
Theory:
An activation function determines if a neuron should be activated or not
activated. This implies that it will use some simple mathematical operations to
determine if the neuron’s input to the network is relevant or not relevant in the
prediction process.
The ability to introduce non-linearity to an artificial neural network and generate
output from a collection of input values fed to a layer is the purpose of the
activation function.
Types of Activation functions
Activation functions can be divided into three types:
1. Linear Activation Function
2. Binary Step Function
3. Non-linear Activation Functions
These activation functions are mainly divided basis on their range and curves. The
remainder of this article will outline the major non-linear activation functions
used in neural networks.
1. Sigmoid
Sigmoid accepts a number as input and returns a number between 0 and 1. It’s
simple to use and has all the desirable qualities of activation functions:
nonlinearity, continuous differentiation, monotonicity, and a set output range.
This is mainly used in binary classification problems. This sigmoid function gives
the probability of an existence of a particular class.
Sigmoid Activation Function — Graph
Mathematically, it can be represented as:
Sigmoid Activation Function — Equation
Pros and Cons
● It is non-linear in nature. Combinations of this function are also non-linear, and
it will give an analogue activation, unlike binary step activation function. It has a
smooth gradient too, and It’s good for a classifier type problem.
● The output of the activation function is always going to be in the range (0,1)
compared to (-∞, ∞) of linear activation function. As a result, we’ve defined a
range for our activations.
● Sigmoid function gives rise to a problem of “Vanishing gradients” and Sigmoid
saturate and kill gradients.
● Its output isn’t zero centered, and it makes the gradient updates go too far in
different directions. The output value is between zero and one, so it makes
optimization harder.
● The network either refuses to learn more or is extremely slow.
[Link] (Hyperbolic Tangent)
TanH compress a real-valued number to the range [-1, 1]. It’s non-linear, But it’s
different from Sigmoid, and its output is zero-centered. The main advantage of
this is that the negative inputs will be mapped strongly to the negative and zero
inputs will be mapped to almost zero in the graph of TanH.
TanH Activation Function — Graph
Mathematically, TanH function can be represented as:
TanH Activation Function — Equation
Pros and Cons
● TanH also has the vanishing gradient problem, but the gradient is stronger for
TanH than sigmoid (derivatives are steeper).
● TanH is zero-centered, and gradients do not have to move in a specific direction.
3. ReLU (Rectified Linear Unit)
ReLU stands for Rectified Linear Unit and is one of the most commonly used
activation function in the applications. It’s solved the problem of vanishing
gradient because the maximum value of the gradient of ReLU function is one. It
also solved the problem of saturating neuron, since the slope is never zero for
ReLU function. The range of ReLU is between 0 and infinity.
ReLU Activation Function — Graph
Mathematically, it can be represented as:
ReLU Activation Function — Equation
Pros and Cons
● Since only a certain number of neurons are activated, the ReLU function is far
more computationally efficient when compared to the sigmoid and TanH
functions.
● ReLU accelerates the convergence of gradient descent towards the global
minimum of the loss function due to its linear, non-saturating property.
● One of its limitations is that it should only be used within hidden layers of an
artificial neural network model.
● Some gradients can be fragile during training.
● In other words, For activations in the region (x<0) of ReLu, the gradient will be 0
because of which the weights will not get adjusted during descent. That means,
those neurons, which go into that state will stop responding to variations in
input (simply because the gradient is 0, nothing changes.) This is called
the dying ReLu problem.
Python Code:
***the sigmoid function from the Keras library as follows***
import tensorflow as tf
from [Link] import sigmoid
input_array = [Link]([-1, 0, 1], dtype=tf.float32)
print (sigmoid(input_array))
Output: [Link]([0.26894143 0.5 0.7310586 ], shape=(3,), dtype=float32)
***tanh function***
import tensorflow as tf
from [Link] import tanh
input_array = [Link]([-1, 0, 1], dtype=tf.float32)
print (tanh(input_array))
output: [Link]([-0.7615942 0. 0.7615942], shape=(3,), dtype=float32)
***To use the ReLU activation in TensorFlow****
import tensorflow as tf
from [Link] import relu
input_array = [Link]([-1, 0, 1], dtype=tf.float32)
print (relu(input_array))
output: [Link]([0. 0. 1.], shape=(3,), dtype=float32)
Conclusion: Thus we studied different activation functions.