Scribd
Upload
27 views5 pages

Softmax Function and Temperature Scaling

The document explains the Softmax function, which converts class probabilities into values that sum to 1, and is commonly used with Cross Entropy loss in neural networks. It also discusses how to implement the Softmax function in Python and the concept of temperature scaling to adjust the confidence of the output distribution. The temperature parameter allows for control over the sharpness of the probability distribution, affecting the model's confidence in its predictions.

Uploaded by

Cát Lăng
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
27 views5 pages

Softmax Function and Temperature Scaling

The document explains the Softmax function, which converts class probabilities into values that sum to 1, and is commonly used with Cross Entropy loss in neural networks. It also discusses how to implement the Softmax function in Python and the concept of temperature scaling to adjust the confidence of the output distribution. The temperature parameter allows for control over the sharpness of the probability distribution, affecting the model's confidence in its predictions.

Uploaded by

Cát Lăng
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Softmax Function

• Cross Entropy loss comes in hand with hand with the Softmax layer.
• The Softmax layer turns all the class probabilities to values that sum up to 1.
𝑒 𝑧𝑗 K = Number of classes
𝜎(𝑧)𝑗 = 𝐾 Do this for j = 1…..K (for all output neurons)
σ𝑘=1 𝑒 𝑧𝑘

Final Layer
Outputs 𝐾

෍ 𝑒 𝑧𝑘 = 0.058 + 2.36 + 1.32 = 3.73


𝑘=1

+ =1 𝑒 −2.85 = 0.058
0.058
3.73
= 0.016

2.36
𝑒 0.86 = 2.36 = 0.631
3.73
[Link] 0.016 + 0.631 + 0.353 = 1
𝑒 0.28 = 1.32 1.32
= 0.353
3.73
Softmax Function

maximum

[Link]
Softmax from Scratch in Python
From Scratch

Using PyTorch to Verify

0.3479

print(tens[0]/[Link]())
Softmax in Python
list = []
for pred in predictions:
[Link]([Link](pred))
sum = sum(list)
outputs = []
for value in list:
[Link](value/sum)
lower temp (< 1)  more confident (sharper)
Temperature Softmax higher temp (> 1)  less confident

• Sometimes, we may want our distribution to be more or less confident


• We can control this by dividing the logits (pre-softmax) by a temperature
temp = 1 temp = 0.5 temp = 0.1
1 1 1
0.8

0.15
0.05

Dog Cat Horse Dog Cat Horse Dog Cat Horse

temp = 5 temp = 10 temp = 100


1 1 1

uniform

Dog Cat Horse Dog Cat Horse Dog Cat Horse

You might also like