Deep Learning-4
Hidden Units
Deep Learning Srihari
Topics in Hidden Units
1. ReLU and their generalizations
2. Logistic sigmoid and Hyperbolic tangent
3. Other hidden units
3
Deep Learning Srihari
Choice of hidden unit
• Previously discussed design choices for neural
networks that are common to most parametric
learning models trained with gradient
optimization
• We now look at how to choose the type of
hidden unit in the hidden layers of the
model
• Design of hidden units is an active research
area that does not have many definitive
guiding theoretical principles 4
Deep Learning
Choice of hidden Srihari
unit
• ReLU is an excellent default choice
• But there are many other types of hidden
units available
• When to use which kind (though ReLU is
usually an acceptable choice)?
• We discuss motivations behind choice of
hidden unit
– Impossible to predict in advance which will work
best
– Design process is trial and error
5
• Evaluate performance on a validation set
Differentiability
Deep Learning Srihari
•
ignored
Neural network training
– not usually arrives at a local
minimum of cost function
– Instead reduces value significantly
• Not expecting training to reach a
point where gradient is 0,
– Accept minima to correspond to
points of undefined gradient
• Hidden units not differentiable
are usually non-differentiable at
only a small no. of points 7
Left and Right
Deep Learning Srihari
Differentiability
• A function g(z) has a left derivative defined by
the slope immediately to the left of z
• A right derivative defined by the slope of the
function immediately to the right of z
• A function is differentiable at z = a only if both
• If a ∈ I is a limit point of I ∩ [a,∞) and the left derivative
• If a ∈ I is a limit point of I ∩ (–∞,a] and the right derivative
• are equal
Function is not continuous: No derivative at marked point
However it has a right derivative at all points with δ+f(a)=0 at all points 8
Deep Learning Srihari
What a Hidden unit does
• Accepts a vector of inputs x and computes an
affine transformation* z = WTx+b
• Computes an element-wise non-linear function
g(z)
• Most hidden units are distinguished from each
other by the choice of activation function g(z)
– We look at: ReLU, Sigmoid and tanh, and other
hidden units
*A geometric transformation that preserves lines and parallelism (but not
10
necessarily distances and angles)
Deep Learning Srihari
Rectified Linear Unit & Generalizations
• Rectified linear units use the activation function
g(z)=max{0,z}
– They are easy to optimize due to similarity with
linear units
• Only difference with linear units that they output 0 across
half its domain
• Derivative is 1 everywhere that the unit is active
• Thus gradient direction is far more useful than with
activation functions with second-order effects
Deep Learning Srihari
Use of ReLU
• Usually used on top of an affine transformation
h=g(WTx+b)
• Good practice to set all elements of b to a
small value such as 0.1
– This makes it likely that ReLU will be initially active
for most training samples and allow derivatives to
pass through
Deep Learning Srihari
ReLU vs other activations
• Sigmoid and tanh activation functions cannot be
with many layers due to the vanishing gradient
problem.
• ReLU overcomes the vanishing gradient
problem, allowing models to learn faster and
perform better
• ReLU is the default activation function with MLP
and CNN
11
Deep Learning Srihari
Generalizations of ReLU
• Perform comparably to ReLU and
occasionally perform better
• ReLU cannot learn on examples for which the
activation is zero
• Generalizations guarantee that they
receive gradient everywhere
14
Deep Learning Srihari
Three generalizations of ReLU
• ReLU has the activation function g(z)=max{0,z}
• Three generalizations of ReLU based on
using a non-zero slope αi when zi<0:
hi=g(z,α)i=max(0,zi)+αi min(0,zi)
1. Absolute-value rectification:
• fixes αi=-1 to obtain g(z)=|z|
2. Leaky ReLU:
• fixes αi to a small value like 0.01
3. Parametric ReLU or PReLU:
• treats αi as a parameter 15
• The function is defined as f(x) = αx, where α is a small constant (e.g.,
0.01). This gives a slight upward slope for negative inputs, preventing
the "dying neuron" problem.
ELU (Exponential Linear Units)
Swish
• SiLU was first proposed alongside the GELU in 2016,[4] then again proposed
in 2017 as the Sigmoid-weighted Linear Unit (SiL) in reinforcement
learning. The SiLU/SiL was then again proposed as the SWISH over a year
after its initial discovery, originally proposed without the learnable
parameter β, so that β implicitly equaled 1. The swish paper was then
updated to propose the activation with the learnable parameter β.
• In 2017, after performing analysis on ImageNet data, researchers
from Google indicated that using this function as an activation
function in artificial neural networks improves the performance, compared
to ReLU and sigmoid functions. It is believed that one reason for the
improvement is that the swish function helps alleviate the vanishing
gradient problem during backpropagation.
Source:[Link]
• For β = 0, the function is linear: f(x) = x/2.
• For β = 1, the function is the Sigmoid Linear Unit (SiLU).
• With β → ∞, the function converges to ReLU.
• Thus, the swish family smoothly interpolates between a linear
function and the ReLU function
Deep Learning Srihari
Logistic Sigmoid
• Prior to introduction of ReLU, most neural
networks used logistic sigmoid activation
g(z)=σ(z)
• Or the hyperbolic tangent
g(z)=tanh(z)
• These activation functions are closely related
because
tanh(z)=2σ(2z)-1
• Sigmoid units are used to predict probability
that a binary variable is 1
Deep Learning
Sigmoid Srihari
•
Saturation
Sigmoidals saturate across most of domain
– Saturate to 1 when z is very positive and 0 when z is
very negative
– Strongly sensitive to input when z is near 0
– Saturation makes gradient-learning difficult
• ReLU and Softplus increase for input >0
Sigmoid can still be used
When cost function undoes the
Sigmoid in the output layer
Deep Learning Srihari
Sigmoid vs tanh Activation
• Hyperbolic tangent typically performs better
than logistic sigmoid
• It resembles the identity function more closely
tanh(0)=0 while σ(0)=½
• Because tanh is similar to identity near 0,
training a deep neural network ŷ = w tanh (U tanh (V x ))
T T T
resembles training a linear model ŷ = w T U T V T x
so long as the activations can be kept small
Deep Learning Srihari
Sigmoidal units still useful
• Sigmoidal more common in settings other than
feed-forward networks
• Recurrent networks, many probabilistic models
and autoencoders have additional
requirements that rule out piecewise linear
activation functions
• They make sigmoid units appealing
despite saturation
Deep LearningOther Hidden Srihari
•
Units
Many other types of hidden units possible, but
used less frequently
– Feed-forward network using h = cos(Wx + b)
• on MNIST obtained error rate of less than 1%
– Radial Basis ⎛ 1
2
⎞
hi = exp⎜− 2 ||W:,i − x || ⎟
⎜
⎝ σ ⎟
⎠
• Becomes more active as x approaches a template W:,i
– Softplus g(a) = ζ(a) = log(1+ e a )
• Smooth version of the rectifier
– Hard tanh
• Shaped similar to tanh and the rectifier but it is bounded
g(a) = max(−1, min(1,a)) 26