Overview of Activation Functions
Uploaded by
Rifat ahmmedOverview of Activation Functions
Uploaded by
Rifat ahmmedCommon questions
Powered by AIThe ReLU activation function offers several advantages, including computational efficiency and the acceleration of gradient descent convergence due to its linear, non-saturating properties. ReLU selectively activates neurons, which improves computational efficiency compared to sigmoid or tanh functions, as not all neurons need to be active at the same time. However, the main disadvantage is the 'Dying ReLU' problem, where some neurons may never activate if they keep receiving negative inputs, leading to zero gradients and thus inactive neurons which don't update during backpropagation .
The softmax activation function handles multiple output classes by converting raw class scores into probability distributions, ensuring that the probabilities sum to 1 across all output classes. This is done by exponentiating each class score and normalizing them by dividing by the sum of exponentiated scores. This ensures predictions are always mutually exclusive and the model output can be interpreted as a clear probability across multiple classes, which is crucial for multi-class classification tasks .
Leaky ReLU preserves the ability for backpropagation even when input values are negative by maintaining a small, non-zero slope (usually 0.01) in the negative region of the input. This non-zero slope ensures that the gradient isn’t zero for negative inputs, allowing for weight updates to be propagated even when neurons receive negative inputs, thus mitigating the Dying ReLU problem .
In a neural network using the sigmoid activation function, the backpropagation algorithm adjusts weights by computing the gradient of the error with respect to each weight. This involves calculating the error at the output layer, then propagating the error backward through the network. The sigmoid function's derivative (sensitive to changes in input), given by σ'(x) = σ(x)(1 - σ(x)), is used to scale the error gradient at each neuron. The weight adjustment is calculated with the delta rule: Δw = -η * (error gradient) * input, where η is the learning rate, allowing the network to reduce error iteratively .
The 'Dying ReLU' problem occurs when ReLU neurons become inactive and always output zero for any input. This happens because the derivative of ReLU is zero for negative input values, causing the gradients to also become zero, thus preventing any backpropagation or updates to the neuron weights. Leaky ReLU addresses this by allowing a small, non-zero, constant gradient for negative input values, which means that some gradient is allowed to pass through even when the input is negative, thereby keeping the neurons active and updating .
In the perceptron learning algorithm, when an instance is misclassified (i.e., the actual output does not match the target output), the weights are updated by adjusting them in the direction that reduces the error. This is done by using the rule: w_new = w_old + η(T - O)x, where η is the learning rate, T is the target output, O is the actual output, and x is the input. The term (T - O) determines the direction of the weight change, ensuring weights are nudged to more accurately classify the data .
The tanh activation function improves upon the sigmoid function by outputting values in the range of -1 to 1, which ensures that the data is zero-centered, allowing for easier mapping of output values as strongly negative, neutral, or strongly positive. This helps mitigate issues related to gradient descent, as it allows for faster convergence compared to the non-zero-centered output of the sigmoid function. The primary drawback of the tanh function is the vanishing gradient problem, where gradients become extremely small during backpropagation in deep neural networks, slowing down or even stopping the learning process .
The sigmoid activation function faces challenges in deep networks due to the vanishing gradient problem. As the function's output saturates at its extremities (close to 0 or 1), the derivative becomes very small, causing gradients calculated during backpropagation to also become tiny. This makes it difficult for gradient-based optimization algorithms to update the weights effectively, slowing down or halting the learning process in deeper layers .
The ReLU activation function is computationally more efficient than sigmoid and tanh because it does not activate all neurons simultaneously and requires only a simple thresholding at zero as opposed to more complex exponential functions in sigmoid and tanh. This simplicity allows for faster computations and reduces the likelihood of vanishing gradients, particularly in deep networks, thereby speeding up convergence and reducing training times significantly compared to sigmoid and tanh functions, which can slow down training due to their saturation and complex derivative calculations .
The sigmoid function is limited in handling multi-class classification problems because it outputs a probability in the range of 0 to 1 for each class independently without considering mutual exclusivity. This means that the sum of the output probabilities may not equal 1 across all classes, which is necessary for a consistent interpretation of probabilities across multiple classes. In contrast, the softmax function outputs probabilities for each class in such a way that they all sum to 1, thus effectively handling the multi-class problem by ensuring that the predictions are mutually exclusive and collectively exhaustive .