Softmax
Topics in Digital Media
Faisal Qureshi
1 / 17
Softmax
Our goal is to extend ideas first explored in logistic regression, which is a
binary classifier, to multi-class problems.
2 / 17
Multinomial distribution
Multinomial distribution can be used to model a random variable X that
takes values in {1, · · · , k}.
Pr(X = i) = φi
Pk
Since probabilities all sum to 1, i=1 φi = 1. Therefore,
Pk−1
φk = 1 − i=1 φi .
Parameters of a multinomial distribution are φ1 , · · · , φk−1 .
Example
I Classification in 3 or more classes
I Which of the k diseases does a patient have?
3 / 17
Multinomial distribution
Using indicators variables introduced previously, we can write the the
probability of a multinomial random variable X as follows:
I (x) I2 (x) I (x)
Pr(X) = φ11 φ2 · · · φkk
K
I (x)
Y
= φii
i=1
Pk−1
φk = 1 − i=1 φi
Pk−1
Ik (x) = i=1 1 − Ii (x)
4 / 17
Indicator variable
(
(i) 1 if y (i) = c
Ic (y ) =
0 otherwise
5 / 17
Multiclass classification
The goal of multiclass classification is to learn hθ (x), which can be used
to assign a label y ∈ {1, · · · , K} to the input x. Label y takes values in
{1, · · · , K}, so we can use multinomial distribution to specify its
probability distribution.
Under the assumption that data is i.i.d.
N
Y K
Y Ij (y(i) )
Pr(y|X, θ) = hθj (x(i) )
i=1 j=1
Change of notation: θi , where i ∈ 1, · · · , K now refers to an
(M + 1)-dimensional vector. Previously θi referred to the ith element of
the (M + 1)-dimensional vector θ.
6 / 17
Likelihood for multiclass classification
Likelihood for ith example
L(θ) = Pr(y (i) |X, θ)
K
Y Ij (y(i) )
= hθj (x(i) )
j=1
7 / 17
Negative log likelihood for ith example
Define y(i) , a K-dimensional vector as follows:
(
(i) 1 if Ij (y (i) )
yj =
0 otherwise
Here i ∈ [1, N ] and j ∈ [1, K].
We can now write negative log likelihood as follows
K
(i)
X
l(θ) = − yj log hθj (x(i) )
j=1
8 / 17
Softmax function
Softmax function or normalized exponential function “squashes” a
K-dimensional vector z of arbitrary real values to a K-dimensional vector
S(z) of real values in the range [0, 1] that add up to 1.
ezi
S(z)i = P z
ke
k
Softmax function is often used to highlight the largest values and suppress
values which are significantly below the maximum value.
Code example (from Wikipedia)
>>> import math
>>> z = [1.0, 2.0, 3.0, 4.0, 1.0, 2.0, 3.0]
>>> z_exp = [[Link](i) for i in z]
>>> print([round(i, 2) for i in z_exp])
[2.72, 7.39, 20.09, 54.6, 2.72, 7.39, 20.09]
>>> sum_z_exp = sum(z_exp)
>>> print(round(sum_z_exp, 2))
114.98
>>> softmax = [round(i / sum_z_exp, 3) for i in z_exp]
>>> print(softmax)
[0.024, 0.064, 0.175, 0.475, 0.024, 0.064, 0.175]
9 / 17
Derivative of softmax function
Case 1
ezi ( k ezk ) − ezi ezi
P
∂
S(z)i = P
∂zi ( k ezk )2
P zk
ezi e − ezi ezi ez i
k
= P z P z = P z 1− P z
ke ke ke ke
k k k k
= S(z)i (1 − S(z)i )
Case 2
∂ −ezi ezj
S(z)i = P z 2
∂zj ( k e k)
ezi
zj
ezi ezi
e
=− P z P z =− P z P z
ke ke ke ke
k k k k
= −S(z)i S(z)j
10 / 17
Derivative of softmax function
We can use Kronecker’s delta function δij to represent the derivative of a
softmax function in terms of itself as follows
∂
S(z)i = S(z)i (δij − S(z)j )
∂zj
Here (
1 if i = j
δij =
0 otherwise
11 / 17
Softmax classifier
Probability distribution of label y is given by softmax function.
hθi (x) = S(x(i) )j
T
ex θi
= PK Tθ
j=1 ex j
Negative log likelihood for softmax classifier
K
(i)
X
l(θ) = − yj log S(x(i) )j (For ith example)
j=1
12 / 17
Softmax classifier (K-classes)
✓10 X x T ✓1 x T ✓1
1 PK T
i=1 x ✓i
✓11
1 1
x1
✓12 x T ✓2
X x T ✓2
x2 PK T
i=1 x ✓i
2 2
✓1M x T ✓K
X x T ✓K
xM PK T
i=1 x ✓i
K K
13 / 17
Softmax classifier derviation
Notation change: drop superscript (i) and let S(x(i) )j = πj for simplicity.
K
∂ X 1 ∂
=− yj πj l ∈ {1, · · · , K}
∂θl j=1
π j ∂θl
yl (πl (1 − πl )x) X yj (−πl πj x)
=− −
πl πj
j6=l
X
= −yl + yl πl + yj πl x
j6=l
K
X
= −yl + πl yj x
j=1
K
X
= (−yl + πl ) x Because yj = 1
j=1
14 / 17
Softmax classifier gradient descent
Notation: k here refers to the iteration number for gradient descent. η is
the learning rate. l ∈ {1, · · · , K}, where K is the number of classes or
distinct values labels can take.
Stochatic gradient descent
(k+1) (k)
θl = θl − η∇l l(θ)
T
!
(k) ex θl
= θl +η PK Tθ
− yl x
j=1 ex j
15 / 17
Cross Entropy
Figure 1:
16 / 17
Summary
I Softmax classifier
I Multinomial distribution
17 / 17