Understanding and Improving Knowledge Distillation

Jiaxi Tang∗, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed [Link], Sagar Jain
Google, Inc
{jiaxit,rakeshshivanna,zhezhao,dongl,animasingh,edchi,sagarj}@[Link]
arXiv:2002.03532v2 [[Link]] 28 Feb 2021

Abstract
Knowledge Distillation (KD) is a model-agnostic technique to improve model
quality while having a fixed capacity budget. It is a commonly used technique
for model compression, where a larger capacity teacher model with better quality
is used to train a more compact student model with better inference efficiency.
Through distillation, one hopes to benefit from student’s compactness, without
sacrificing too much on model quality. Despite the large success of knowledge
distillation, better understanding of how it benefits student model’s training dy-
namics remains under-explored. In this paper, we categorize teacher’s knowledge
into three hierarchical levels and study its effects on knowledge distillation: (1)
knowledge of the ‘universe’, where KD brings a regularization effect through label
smoothing; (2) domain knowledge, where teacher injects class relationships prior to
student’s logit layer geometry; and (3) instance specific knowledge, where teacher
rescales student model’s per-instance gradients based on its measurement on the
event difficulty. Using systematic analyses and extensive empirical studies on both
synthetic and real-world datasets, we confirm that the aforementioned three factors
play a major role in knowledge distillation. Furthermore, based on our findings,
we diagnose some of the failure cases of applying KD from recent studies.

1 Introduction
Recent advances in artificial intelligence have largely been driven by learning deep neural networks,
and thus, current state-of-the-art models typically require a high inference cost in computation and
memory. Therefore, several works have been devoted to find a better quality and computation
trade-off, such as pruning (Han et al., 2015b) and quantization (Han et al., 2015a; Jacob et al., 2018).
One promising and commonly used method for addressing this computational burden is Knowledge
Distillation (KD), proposed by Hinton et al. (2015), which uses a larger capacity teacher model
(ensembles) to transfer its ‘dark knowledge’ to a more compact student model. Through distillation,
one hopes to achieve a student model that not only inherits better quality from the teacher, but is also
more efficient for inference due to its compactness. Recently, we have witnessed a huge success of
knowledge distillation, irrespective of the model architecture and application domain (Kim & Rush,
2016; Chen et al., 2017; Tang & Wang, 2018; Anil et al., 2018; He et al., 2019).
Despite the large success of KD, surprisingly sparse research has been done to better understand the
mechanism of how it works, which could limit the applications of KD and also raise unexpected or
unexplainable results. For example, to successfully ‘distill’ a better student, one common practice
is to have a teacher model with as good quality as possible. However, recently Mirzadeh et al.
(2019) and Müller et al. (2019) have found this intuition would fail under certain circumstances.
Furthermore, Anil et al. (2018) and Furlanello et al. (2018) have analyzed that even without using a
powerful teacher, distilling a student model to itself using mutual or self-distillation also improves
quality. To this end, some researchers have made attempts on understanding the mechanism of KD.
∗
Work done when was a student at Simon Fraser University and while interning at Google.

Preprint.
For example, Yuan et al. (2019) connects label smoothing to KD. Furlanello et al. (2018) conjectures
KD’s effect on re-weighting training examples. In this work, we found that the benefits of KD comes
from a combination of multiple effects, and propose partial KD methods to dissect each of the effects.
This work is an attempt to shed light upon the ‘dark knowledge’ distillation, making this technique
less mysterious. More specifically, we make the following contributions:
• For KD on multi-class classification task, we systematically break down its effects into:
(1) label smoothing from universal knowledge, (3) injecting domain knowledge of class
relationships to student’s output logit layer geometry, and (2) gradient rescaling based on
teacher’s measurement of instance difficulty. We provide theoretical analyses on how KD
exhibits these effects, and improves student model’s quality (Section 3).
• We propose partial-distillation techniques using hand-crafted teacher’s output distribution
(Section 4) to simulate and validate different effects of knowledge distillation.
• We empirically demonstrate and confirm our hypothesis on the effects of KD on both
synthetic and real-world datasets. Furthermore, using our understanding, we diagnose some
recent failures of applying KD (Section 5).

2 Related Work
In the context of deep learning, knowledge transfer has been successfully used to effectively compress
the power of a larger capacity model (a teacher) to a smaller neural network (a student). Adopting
this teacher-student learning paradigm, many forms of knowledge have been investigated: layer
activations (Romero et al., 2014), auxiliary information (Vapnik & Izmailov, 2015), Jacobian matrix
of the model parameters (Czarnecki et al., 2017; Srinivas & Fleuret, 2018), Gram matrix derived from
pairs of layers (Yim et al., 2017), activation boundary (Heo et al., 2019), etc. Among these, the original
(or vanilla) KD – learning from teacher’s output distribution (Hinton et al., 2015) is the most popular.
Besides compression, KD has also been successfully applied to improve generalization (Furlanello
et al., 2018), reproducibility (Anil et al., 2018), defend adversarial attacks (Papernot et al., 2016), etc.
Though KD has been successfully applied in various domains, there has been very few attempts on
understanding how and why it helps neural network training. Hinton et al. (2015) argued that the
success of KD could be attributed to the output distribution of the incorrect classes, which provides
information on class relationships. From learning theory perspective, Vapnik & Izmailov (2015)
studied the effectiveness of knowledge transfer using auxiliary information, known as Privileged
Information. Following which, Lopez-Paz et al. (2015) established the connection between KD and
privileged information. Recently, Phuong & Lampert (2019) showed a faster convergence rate from
distillation. However, most of the existing theoretical results rely on strong assumptions (e.g., linear
model, or discarding ground-truth when training the student), and also fails to explain the recent
failure cases of distilling from a better quality teacher (Mirzadeh et al., 2019; Müller et al., 2019).
The most relevant work to our own is (Furlanello et al., 2018). Though the main focus of their
work is to propose KD techniques to boost quality, they also provide intuitions for the effectiveness
of KD. In our work, we offer theoretical analysis on some of their conjectures, and improve on
erroneous assumptions. Furthermore, we systematically investigate the mechanism behind knowledge
distillation by decomposing its effects, and analyzing how each of these effects helps with student
model’s training using our proposed partial-distillation methods.

3 Analyzing Mechanisms of Knowledge Distillation

In this section, we provide a systematic analyses for the mechanisms behind KD based on theoretical
and empirical results. We start by introducing essential background, dissect distillation benefits from
three main effects, and conclude by connecting and summarizing these effects.
Background. Consider the task of classification over [K] := {1 . . . K} classes, given (x, y) ∈
X × Y, with y ∈ {0, 1}K denoting one-hot encoded label, and t ∈ [K] denoting the ground-truth
class. The goal is to learn a parametric mapping function f (x; θ) : X 7→ Y where θ ∈ Θ can be
characterized by a neural network. We learn the parameters θ via Empirical Risk Minimization
of the surrogate loss function, typically optimized using a variant of Stochastic Gradient Descent:
PK
θ∗ = arg minθ∈Θ L(y, f (x; θ)), where L is the cross-entropy loss H(y, q) = i=1 −yi log qi , and

2
q = f (x; θ) is the network’s output distribution computed by applying softmax over the logits z:
qi = softmax(zi ) = PKexp(z i)
exp(z )
. We could also scale the logits by temperature T > 1 to get a
j=1 j

smoother distribution q̃i = softmax(zi /T ). Gradient of a single-sample w.r.t. logit zi is given by:
∂L/∂zi = qi − yi . Lets denote ∂i = ∂L/∂zi . (1)

3.1 Knowledge of the universe – benefits from label smoothing

Label Smoothing (LS) (Szegedy et al., 2016) is a technique to soften one-hot label y by a factor
of , such that the modified label becomes: ỹiLS = (1 − )yi + /K. Label smoothing mitigates
the over-confidence issue of neural networks, and improves model calibration (Müller et al., 2019).
Knowledge Distillation (KD) on the other hand, uses an additional teacher model’s predictions p for
training:
∗
= arg min LKD (y, p, f (x; θ), λ, T ) = (1 − λ)H(y, q) + λH(p̃, q̃) ,

θKD
θ∈Θ

where λ ∈ [0, 1] is a hyper-parameter; and q̃ and p̃ are temperature softened student and teacher’s
predictions. Logits gradient for KD is given by:
∂LKD /∂zi = (1 − λ)(qi − yi ) + (λ/T )(q̃i − p̃i ). Lets denote ∂iKD = ∂LKD /∂zi . (2)
Yuan et al. (2019) established the connection between KD and LS: In terms of gradient propagation,
KD is equivalent to LS, when T = 1, and teacher’s probability distribution p follows a uniform
distribution, i.e., a Bayesian prior of the universe. In other words, we can view KD as an adaptive
version of label smoothing, suggesting it should inherit most of the regularization benefits from label
smoothing, such as model regularization and better calibration, not being over-confident (Müller
et al., 2019).
In the next two subsections, we analyze the unique characteristics of real teacher’s distribution over
uniform distribution, and demonstrate how they could potentially facilitate student model’s training.

3.2 Domain knowledge – teacher injects class relationships prior

KD leverages class relationships as captured by the teacher’s probability distribution p over the
incorrect classes. As argued by Hinton et al. (2015) on MNIST dataset, model assigns relatively
high probability for class ‘7’, when the ground-truth class is ‘2’. In this section, we first confirm
their hypothesis using empirical studies. Then, we provide new insights to explain how the teacher
informs the class relationships to its student at optimality, and improves model quality.
To illustrate that the teacher’s distribution p captures class relationships, we train ResNet-56 on
CIFAR-100 dataset. CIFAR-100 contains 100 classes over 20 super-classes, with each super-class
containing 5 sub-classes. Figures 1a and 1b show the heatmap for Pearson correlation coefficient
on teacher’s distribution p at different temperatures. We sort the class indexes to ensure that the 5
classes from the same super-class appear next to each other. With a lower temperature in Figure 1a,
there’s no pattern on the heatmap showing class relationships. But as we increase the temperature in
Figure 1b, classes within the same super-class clearly have a high correlation with each other, as seen
in the block diagonal structure. This observation verifies that teacher’s distribution p indeed reveals
class relationships, with proper tuning on the softmax temperature.
In this work, we found that the teacher’s predictions on incorrect classes also provides a prior for
student model training. Before diving into the details, we recall the case of label smoothing (Szegedy
et al., 2016):
• From an optimization point of view, He et al. (2019) showed that there is an optimal constant
margin log(K(1 − )/ + 1), between the logit of the ground-truth zt , and all other logits
z−t , using a label smoothing factor of . For fixed number of classes K, the margin is a
monotonically decreasing function of .
• From geometry perspective, Müller et al. (2019) showed the logit zk = h> wk for any
class k is a measure of squared Euclidean distance kh − wk k2 on latent space between the
activations of the penultimate layer2 h, and weights wk for class k in the last logit layer.
2
Here h can be concatenated with a “1” to account for the bias.

3
 (a) (b) (c) (d)
Figure 1: Using 10K samples from CIFAR-100 for ResNet-56, we plot Pearson correlations of output probability
p with varying softmax temperature (a) T = 3, (b) T = 10, and (c) T = 10 where only top-10 largest values
in p are preserved. In (d), we show cosine similarities computed from the weights of the final logits layer. Since
classes within the super-class are grouped together, we see a block diagonal class correlation structure.

Above findings suggest that label smoothing encourages kh − wt k2 ≥ kh − w−t k2 and pushes all
the other incorrect classes equally apart. Following a similar proof technique, we extend to KD:
Proposition 1. With KD, the optimal solution of student’s final logit layer weights {wk∗ , ∀k ∈ [K]}
enforces different inter-class distances based on teacher’s probability distribution p:
kh − wi∗ k2 < kh − wj∗ k2 iff pi > pj , ∀i, j ∈ [K]\t,
where h is the activations of the penultimate layer. (See proof in Suppl. Section 7.1)
From Figure 1b, teacher assigns higher probability to the classes within the same super-class, and
hence KD encourages hierarchical clustering of logit layer weights based on the class relationships.

3.3 Instance specific knowledge – teacher rescales gradients based on event difficulty

Another important characteristic of the teacher distribution p is that the prediction (confidence) pt on
the ground-truth class is different across instances. Comparing ratio of gradients (eqs. (1) and (2)):
∂ KD
 
λ q̃i − p̃i
ωi = i = (1 − λ) + , (3)
∂i T qi − yi
we find that KD performs gradient rescaling in the logits space based on teacher model’s prediction
confidence pt on the ground-truth class t. The gradient rescaling factor ωi is larger on average, when
teacher is more confident on making the right prediction. More specifically, we state the following:
Proposition 2 (Gradient Rescaling). Given any example (x, y) ∈ X × Y, let p̃t = q̃t + c̃t + η,
where c̃t > 0 is teacher’s relative prediction confidence on the ground-truth class t ∈ [K] and η is a
zero-mean random noise. Then the logit’s gradient rescaling factor by applying KD is given by:
"P KD
#
i∈[K]\t ∂i
 KD   
∂t λ c̃t
Eη = Eη P = (1 − λ) + .
∂t i∈[K]\t ∂i T 1 − qt

See proof in Suppl. Section 7.1. At a given snapshot during training, we could assume c̃t to be a
constant for all examples. Then for any pairs of examples (x, y), (x0 , y 0 ) ∈ X × Y, if the teacher is
more confident on one of them, i.e., p > p0 , then the average ω for all classes will be greater than ω 0 .
To validate our claim, in Figure 2, we plot the relationship between ωt and pt at the end of training.
On CIFAR-100 (Krizhevsky et al., 2009), we use ResNet (He et al., 2016) with depth 20 as the
student model, and depth 56 as the teacher (see Suppl. Section 7.2 for more details). The plot shows
a clear positive correlation between the two. Notably, the correlation will be even stronger when
closer to the beginning of training.
In (Furlanello et al., 2018), the authors conjecture that per-example weight is associated with the
largest value in p. In Proposition 2, we show an alternative gradient rescaling effect. It’s important
to distinguish that the weight is associated with teacher’s confidence on ground-truth, instead of
the largest value. Once the teacher makes a wrong prediction, using the largest value would yield
contradictory result. It is also trivial to show that when we have two classes, ωi6=t = ωt , the primary
effect of KD is gradient rescaling. So we can regard the use of KD on binary classification (Anil
et al., 2018) as taking the binary log-loss, and multiply with the weight ωt .

4
Altogether, we show knowledge distillation has an effect
of gradient rescaling with its factor associated with the
teacher’s prediction on ground-truth. Weight will be higher
when pt is larger. Alternatively, this suggests that KD
would magnify the gradients (w.r.t. logits), for training
examples that are considered easier from teacher’s per-
spective, and vice versa; which has a similar flavor to
Curriculum Learning. Bengio et al. (2009) suggested that
this may speedup training convergence and helps optimiza-
tion to reach a better local minima. This is also related
to Roux (2016), which shows that re-weighting examples Figure 2: Applying KD for ResNet-20 stu-
during training using model’s prediction confidence leads dent model with ResNet-56 as the teacher on
to a tighter bound on the classification error and leads to CIFAR-100, we plot pt vs. ωt (in log scale)
better generalization. with 10K samples at the end of training.

3.4 Summary on primary effects of KD

We conclude the section by summarizing KD’s primary effects for classification task from three
hierarchical levels of knowledge. From the most general perspective, KD brings a regularization
effect by introducing smoothened teacher distribution (i.e., a Bayesian prior of the universe). Then
from domain knowledge, teacher’s probability mass on the incorrect classes reflect class relationships,
therefore providing more guidance to the student. We showed the guidance is through influencing
student’s final logit layer geometry: instead of pushing penultimate layer activations equally apart
from the incorrect classes’ weights as in label smoothing, KD, as an adaptive label smoothing,
encourages differences in inter-class distances. Finally, for instance specific knowledge, the teacher
model rescales student models’s gradients with its measurement on event difficulty, i.e., confidence
on the ground-truth class. As a result, all the three levels of knowledge complement each other, which
could potentially facilitate student model’s training process and further improve model generalization.

4 Isolating Effects by Partial Knowledge Distillation Methods

To further dissect the different effects of KD, in this section, we synthesize hand-crafted teacher
distributions, denoted by ρ. Each synthetic teacher distribution ρ contains partial information
from the real teacher’s distribution p, enabling us to isolate and study the effects of KD (namely,
gradient rescaling and prior on optimal geometry). We propose KD-pt and KD-sim – former only
incorporates the gradient rescaling effect and excludes class relationship information, and the latter
only incorporates class relationships but not gradient rescaling. We then try to combine the two
effects together in an attempt to approximate the performance of vanilla KD.
Examine gradient rescaling effect by KD-pt. As discussed, label smoothing neither has gradient
rescaling effect, nor information about class relationships, due to its uniform teacher distribution.
However, if we borrow pt (prediction on ground truth class t ∈ [K]) from the real teacher’s
probability distribution p, we can synthesize a partial teacher distribution that is able to incorporate
gradient rescaling effect. More specifically, we craft teacher’s probability distribution ρpt as follows:
ρpt
i = pt if i = t, (1 − pt )/(K − 1) otherwise. From Proposition 2, it is trivial to see that KD-pt is
capable of rescaling gradients for different examples. However, it does not capture class relationships.
Examine optimal geometry prior by KD-sim. Following a similar methodology, we synthesize a
teacher distribution that only captures class relationships, and ignores gradient rescaling. To achieve
this, we use the weights of the last logit layer W ∈ RK×d from the teacher model to obtain class
relationships. We believe the teacher, due to its larger capacity is able to encode class semantics in
the weights of the last logit layer. Thus, we create a distribution ρsim as the softmax over cosine
similarity3 of the weights: ρsim = softmax(ŵt Ŵ > ), where Ŵ ∈ RK×d is the `2 -normalized logit
layer weights, and ŵt = wt /kwt k is the t-th row of Ŵ corresponding to the ground truth. Though
other distance metrics could also be used as a measure of class similarity, we leave the discussion of
analysing the different choices as future work. To verify our assumption, we check the heatmap of
3
In practice, besides tuning the temperature of the softmax, one could also raise the similarities to a power
< 1 to amplify the resolution of cosine similarities. Please refer to Section 7.2 in Suppl. for more details.

5
cosine similarities in Figure 1d, which clearly shows a similar pattern as the Pearson correlation of
the teacher’s distribution p in Figure 1b. From Propositions 2 and 1, our proposed method, though
simple and straightforward, can preserve class relationships only, and therefore achieve our purpose.
Note that KD-sim doesn’t require a prior knowledge of class hierarchy, but if available (as in CIFAR-
100), we could also synthesize a teacher’s probability distribution apriori. In Suppl. Section C, we
synthesize ρ by setting different values for (1) ground-truth class t, (2) classes within the same
super-class of t, and (3) other incorrect classes. The quality of the resulting method is slightly poor
compared to KD-sim, but still improves student model’s generalization.
Compounded effects. We also explore a combination of the above proposed orthogonal partial KD
techniques to validate if the resulting method can approximate vanilla KD. We study a simple linear
combination of synthetic teacher’s probability distribution, that is, (1 − α)ρpt + αρsim and name
the method KD-pt+sim. It is easy to verify that this compounded method smoothens out the label
distribution, rescales gradients, and also injects optimal prior geometry through class relationships.

5 Empirical Studies
In this section, we evaluate the effectiveness of our proposed partial-distillation methods, to better
understand how much each of these effects benefits the student model, and how the improvements are
associated with the dataset properties. With our understandings, we propose a simple way to improve
distillation quality and we diagnose the recent failures of KD.

5.1 How does class correlations influence distillation?

Performance of KD is dependent on the dataset properties. A natural question is – Does KD perform

only gradient rescaling when all the classes are uncorrelated to each other? We showed this to be
true for binary classification (Section 3.3). To answer the same for multi-class classification task, we
generate synthetic dataset, where we can control the class similarities within the same super-class.
Setup. Inspired by (Ma et al., 2018), we synthesize a classification dataset with K classes and C
super-classes, such that each super-class has K/C classes, and each class will be assigned with
a carefully generated basis vector, so that we could control the class correlations within the same
super-class. Also, data points can be generated to be linearly non-separable to control for task
difficulty. See Suppl. Section 7.2 for more details. In our experiments, we set input dimension
d = 500 with K = 50 and C = 10. We use |D| = 500k data-points for training, and |Dvalid | = 50k
for validation. We use a simple 2-layer fully-connected neural network with tanh activation, and
hidden layer dimensions 64 for the student, and 128 for the teacher. By injecting non-linearities when
generating the synthetic data, we are able to control the task difficulty trade-off (i.e., not too easy, but
hard enough to have a large margin between the two models for KD). Figure 6 in Suppl. shows a
visualization of a toy dataset.
Results and analysis. Table 1 shows the classification accuracy on the validation set when varying
class similarity within each super-class (denote as τ ). We notice a large margin between the teacher
and student, and Knowledge Distillation (KD) benefits the student significantly. Interestingly, when
all classes are uncorrelated (τ = 0.0), we notice KD-pt even outperform KD, verifying our claim of
gradient scaling effect of KD. When increasing τ , we see a significant improvement in performance
of KD-sim, suggesting that the injected prior knowledge of class relationships can also aid student
model in generalization. Note that for this task, the data points that are close to the decision boundary
are harder to classify, and can be regarded as difficult examples. It is worth mentioning that the
performance of both the student and teacher drastically drop when having larger τ , as the classes
within the same super-class will be too similar and difficult to distinguish.

5.2 How effective are the partial-distillation methods?

We next use two popular image classification datasets – CIFAR-100 (Krizhevsky et al., 2009) and
ImageNet (Russakovsky et al., 2015) to analyze the quality of our proposed partial-distillation
methods, and also to verify if we could approximate the performance of KD by compounding effects.
Setup. On CIFAR-100 we use ResNet-20 as the student, and ResNet-56 as the teacher. On ImageNet
with 1000 classes, we use ResNet-50 as the student, and ResNet-152 as the teacher. For more

6
Table 1: Accuracy (%) on synthetic dataset with Table 3: Best validation and test Perplexity (lower
different class similarities within each super-class. is better) over 4 individual runs on PTB language
Method τ = 0.0 τ = 0.3 τ = 0.4 modeling. Best k value for KD-topk is 100.
Teacher 54.82 60.60 61.53 Method #Params Validation Test
Student 38.45 42.81 47.10 Teacher 24.2M 60.90 58.58
KD 55.97 56.95 57.70 Student 9.1M 64.17 61.55
KD-pt 57.05 56.99 56.25 KD 9.1M 64.04 61.33
KD-sim 51.90 53.69 57.20 KD-topk 9.1M 63.59 60.85
Table 2: Mean and Std. for top-1 accuracy (%) Val. accuracy on CIFAR-100
over 4 individual runs. Best k for KD-topk is 25
and 500 for CIFAR-100 and ImageNet, resp. 0.7625
0.7600
0.7575
Method CIFAR-100 ImageNet
k=5 k=10 k=25 k=50 k=100
Teacher 75.68 ± 0.42 77.98 ± 0.12 (5%) (10%) (25%) (50%) (vanilla KD)
Student 72.51 ± 0.27 76.34 ± 0.11
0.780
Val. accuracy on ImageNet
LS 73.87 ± 0.16 76.83 ± 0.07
KD 75.94 ± 0.26 77.49 ± 0.07 0.775

KD-pt 75.08 ± 0.16 77.00 ± 0.08 0.770

k=10 k=50 k=100 k=500 k=1000
KD-sim 74.30 ± 0.17 76.95 ± 0.07 (1%) (5%) (10%) (50%) (vanilla KD)
Figure 3: Top-1 accuracy vs. k for KD-topk from
KD-pt+sim 75.24 ± 0.17 77.17 ± 0.08 4 individual runs on CIFAR-100 and ImageNet.
KD-topk 76.17 ± 0.25 77.85 ± 0.03

details, please refer to Section 7.2 in Suppl. Note that instead of using different model families as
in (Furlanello et al., 2018; Yuan et al., 2019), we use the same model architecture (i.e., ResNet)
with different depths for the student and teacher to isolate any unknown effects introduced by model
family discrepancy.
Results and analysis. Table 2 shows the overall performance with the best hyper-parameters for each
of the methods. On both datasets, teacher model is much better than the student, and label smoothing
(LS) improves student model’s generalization. KD can further boost student model’s quality by
a large margin, especially on CIFAR-100, where KD even outperforms the teacher. We try to
uncover the different benefits from distillation using partial-KD methods. Both KD-pt, and KD-sim
outperforms LS; especially KD-pt on CIFAR-100. This suggests that the different effects from KD
benefits the student in different aspects depending on the dataset. Furthermore, by combining the two
effects together in KD-pt+sim (using α = 0.5), we see a further improvement in quality.

5.3 Regulated knowledge sharing improves distillation

Following our understanding, any methods that can enhance and balance knowledge sharing at the
three granular levels could potentially improve knowledge distillation. Extending KD-pt, we take a
step forward to use the top-k largest values of teacher’s probability p, and uniformly distributes the
rest of the probability mass to the other classes, we name this method as KD-topk. For better intuition,
from Figure 1c we observe that only preserving top-10 largest values could closely approximate the
class correlations as in the full teacher’s distribution p, and is also less noisy. This finding shows that
only a few incorrect classes that are strongly correlated with the ground-truth class are useful for KD
to boost the domain knowledge, and the probability mass on other classes are random noise (which is
not negligible under high temperature T ), and only has the effect of label smoothing in expectation.
Furthermore, KD-topk can better utilize instance specific class relationships, since in KD-sim, all
examples from the same class will have the same relationships to the other classes, which is restricted.
For example, on MNIST dataset, only some versions of ‘2’ looks similar to ‘7’.
Using the above intuition, we test KD-topk for image classification on CIFAR-100 and ImageNet, and
language modeling on Penn Tree Bank (PTB) dataset. We apply state-of-the-art LSTM model (Merity
et al., 2017) with different capacities for the teacher and student. Details of PTB dataset and model
specifications are in Section 7.2 of Suppl. For image classification, the performance of KD-topk
is shown in the last row of Table 2. We see that KD-topk outperforms KD on both datasets. For
language modeling, the results are shown in Table 3, which suggests a similar trend for KD-topk. We

7
Table 4: On CIFAR-100, Top-1 accuracy
(%) of various KD methods using teach-
ers with or without label smoothing.
Teacher: ResNet-56
Method
 = 0.0  = 0.1
Teacher 75.39 76.69 ↑
KD 76.00 75.02 ↓ (a) (b)

KD-pt 74.81 74.13 ↓ Figure 4: On CIFAR-100, for teacher with LS ( = 0.1),

we plot (a) Pearson correlations with T = 10, and (b) cosine
KD-sim 74.40 74.01 ↓
similarities computed from the weights of the final logits layer.

(a) (b) (c) (d)

Figure 5: On CIFAR-100, we plot the histogram of ResNet-56 teacher’s confidence on ground-truth pt , with
different levels of label smoothing factor : (a)  = 0.0 and T = 1; (b)  = 0.0 and T = 5; (c)  = 0.1 and
T = 1 and (d)  = 0.1 and T = 3. The distribution of pt becomes skewed after enabling label smoothing.

plot the performance uplift of KD-topk along with k in Figure 3. As shown, the best performance is
achieved with a proper tuning of k < K, which captures class relationships and also reduces noise.
The results align with our understanding of KD, and also suggest a way to achieve better distillation
quality.

5.4 Diagnosis of failure cases

Having a good understanding of KD enables us to diagnose failure cases. Müller et al. (2019)
observed that although label smoothing (LS) improves teacher model’s quality, it results in a worse
student model when applying KD. Verified on CIFAR-100 in Table 4, we found that the unfavorable
distillation performance could be attributed to two factors – Firstly, as argued by the Müller et al.
(2019) and illustrated in Figure 4, LS destroys class relationship information. Secondly, we found
that the skewed teacher’s prediction distribution on the ground-truth (see Figure 5) also hinders the
effectiveness of KD, especially that of KD-pt, since gradient rescaling will be less effective. Results
of KD-sim and KD-pt from last two columns of Table 4 verifies our hypothesis.
For another failure case, Mirzadeh et al. (2019) showed that the ‘distilled’ student model’s quality
gets worse as we continue to increase teacher model’s capacity. Larger capacity teacher might overfit,
and predict high (uniform) confidence on the ground truth on all the examples; and thereby hindering
the effectiveness of gradient rescaling. Another explanation could be that there exists an optimal
model capacity gap between the teacher and student, which could otherwise result in an inconsistency
between teacher’s prediction confidence on the ground-truth, and the desired example difficulty for
the student. Perhaps, an ‘easy’ example for larger capacity teacher is overly difficult for the student.

6 Conclusion and Future Work

We provide novel techniques to better understand the mechanism of knowledge distillation (KD).
Through systematic analyses, we uncover two key beneficial effects of KD over label smoothing.
Firstly, supervision from teacher’s prediction on the ground-truth rescales student’s gradients for
different training examples. Secondly, teacher’s probability mass on the incorrect classes reveals
class relationships by injecting prior knowledge of the optimal geometry of student’s output layer.
These effects also explain why sometimes a better teacher may not be suitable for distillation, and
self-distillation gives quality gains. To have a closer look at these two effects, we proposed partial-
distillation methods, and evaluated their performance on both synthetic and real-world datasets.
Experimental results support our claims, help diagnose unpleasent results, and inspire ways to
improve KD. In future work, we would like to extend our understanding of knowledge distillation
under different data distributions, e.g., uniform vs long-tail distribution; and also consider the effect

8
of noisy inputs and labels. We would like to also investigate other cheaper and effective ways of
distillation e.g., looking into approximate versions of KD-topk.

7 Appendix
7.1 Analyzing Mechanisms of Knowledge Distillation

Proposition 1. (Paper) With KD, the optimal solution of student’s final logit layer weights {wk∗ , ∀k ∈
[K]} enforces different inter-class distances based on teacher’s probability distribution p:
kh − wi∗ k2 < kh − wj∗ k2 iff pi > pj , ∀i, j ∈ [K]\t,
where h is the activations of the penultimate layer.

Proof. At the optimal solution of the student, equating gradient in Equation (2) to 0, we get:
λ λ λ
(1 − λ)(qk∗ − yk ) + (q̃k∗ − p̃k ) = 0 =⇒ (1 − λ)qk∗ + q̃k∗ = (1 − λ)yk + p̃k (4)
T T T
Using a similar proof technique as Müller et al. (2019), kh − wk∗ k2 = khk2 + kwk∗ k2 − 2h> wk∗ ,
where h is the penultimate layer activations, and wk∗ are the weights of the last logits layer for
class k ∈ [K]. Note that khk2 is factored out when computing the softmax, and kwk∗ k2 is usually a
(regularized) constant across all classes. Equating zk∗ = h> wk∗ , and using the property softmax(z) =
softmax(z + c), ∀c ∈ R, we get:
 1 
qk∗ = softmax(zk∗ ) = softmax(h> wk∗ ) = softmax − kh − wk∗ k2
2
Plugging the above in equation 4, we get:
 1  λ  1  λ
(1 − λ)softmax − kh − wk∗ k2 + softmax − kh − wk∗ k2 = (1 − λ)yk + p̃k
2 T 2T T
Note that softmax is a monotonically increasing function, and we can rewrite LHS as:
 1  λ
g − kh − wk∗ k2 ; λ, T = (1 − λ)yk + p̃k ,
2 T
where g(x; λ, T ) : R → R is a monotonically increasing function, parameterized by λ and T . Now
for the incorrect classes, equating yk = 0, and noting that softmax temperature scaling preserves
relative ordering of teacher’s probabilities p̃k proves the claim.

Proposition 2. [Gradient Rescaling] (Paper) Given any example (x, y) ∈ X ×Y, let p̃t = q̃t +c̃t +η,
where c̃t > 0 is teacher’s relative prediction confidence on the ground-truth class t ∈ [K] and η is a
zero-mean random noise. Then the logit’s gradient rescaling factor by applying KD is given by:
"P KD
#
i∈[K]\t ∂i
 KD   
∂t λ c̃t
Eη = Eη P = (1 − λ) + .
∂t i∈[K]\t ∂i T 1 − qt

Proof. We first consider the ground-truth class t ∈ [K]. Using yt = 1, p̃t = q̃t + c̃t + η and E[η] = 0
in equation 3, we get:
 
λ c̃t
Eη ∂tKD /∂t = (1 − λ) +
 
T 1 − qt
Now, sum of the incorrect class gradients is given by:

X X  λ
∂iKD =

(1 − λ)qi + (q̃i − p̃i )
T
i∈[K]\t i∈[K]\t
λ
= (1 − λ)(1 − qt ) + (p̃t − q̃t ) = −∂tKD
T
Penultimate equality follows from q, p̃ and q̃ being probability masses. Similarly applies for ∂i , and
hence the proof.

9
 (a) (b) (c)
Figure 6: Visualization of 5K synthetic data points (with input dimensionality d = 2) on 2-D plane.
We use K = 4, C = 2, means there are two super-classes, one associate with label {0,1} and the
other one associate with label {2,3}. We vary τ and M and produce 3 plots: (a) τ = 0.0, no sine
function is used; (b) τ = 0.9, no sine function is used and (c) τ = 0.9, M = 2.

7.2 Experimental details

Implementation of KD. In practice, the gradients from the RHS of Equation (2) are much smaller
compare to the gradients from LHS when temperature T is large. Thus, it makes tuning the balancing
hyper-parameter λ become non-trivial. To mitigate this and make the gradients from two parts in the
similar scale, we multiply T 2 to the RHS of Equation (2), as suggested in (Hinton et al., 2015).

Implementation of KD-sim. When synthesizing the teacher distribution for KD-sim, we use
ρsim = softmax(ŵt Ŵ > ), where Ŵ is the l2 -normalized logit layer weights and ŵt is the t-th row
of Ŵ . However, the cosine similarities computed for softmax are limited in the range of [0, 1].
Therefore the resulting distribution is highly likely to be uniform. To mitigate this and bring more
resolution to be cosine similarities, we use the following:
ρsim = softmax((ŵt Ŵ > )α /β).
Here α < 1 is a hyper-parameter to amplify the resolution of cosine similarities, β is another
hyper-parameter indicating the temperature for softmax.

Synthetic dataset. For the synthetic dataset, we generate a single data-point as follows:

1. Randomly sample C orthonormal basis vectors, denoted by ui ∈ Rd ∀i ∈ [C].

2. For each orthonormal basis ui , we sample (K/C − 1) unit vectors uj ∈ Rd that are τ
cosine similar to ui .
3. Randomly sample an input data point in d-dimensional feature space x ∼ Nd (0, I).
4. Generate one-hot encoded label y ∈ Y with target: t = arg maxk∈[K] u> k x̂ +
PM >

M
m=1 sin(am uk x̂ + bm ) , where x̂ is the l2 -normalized x; a, b ∈ R are arbitrary
constants; and we refer to the controlled sin complexity term M ∈ Z+ as task difficulty.

After producing basis vectors with procedure (1) and (2), we run procedure (3) and (4) for |D| times
with fixed basis vectors to generate a synthetic dataset D = {(x, y)}. By tuning the cosine similarity
parameter τ , we can control the classes correlations within the same super-class. Setting task-difficulty
M = 0 generates a linearly separable dataset, and M > 0 generates more non-linearities by the sin
function (see Figure 6 in Suppl. for visualization on a toy example).
Following the procedure showed above, we get a toy synthetic dataset where we only have input
dimensionality d = 2 with K = 4 classes and C = 2 super-classes. Figure 6 shows a series of
scatter plots with different settings of class similarity τ and task difficulty M . This visualization
gives a better understanding of the synthetic dataset and helps us imagine what it will look like in
high-dimensional setting that used in our experiments. For the model used in our experiments, besides
they are 2-layer network activated by tanh, we use residual connection (He et al., 2016) and and
batch normalization (Ioffe & Szegedy, 2015) for each layer. Following (Ranjan et al., 2017; Zhang

10
 Method Hyper-parameter setting
LS  = 0.3 for any τ .
λ = 0.7, T = 3 when τ = 0.0
λ = 0.7, T = 5 when τ = 0.1
λ = 0.7, T = 2 when τ = 0.2
KD Method Hyper-parameter setting
λ = 0.7, T = 3 when τ = 0.3
LS  = 0.1
λ = 0.7, T = 10 when τ = 0.4
KD λ = 0.3, T = 5
λ = 0.7, T = 5 when τ = 0.5;
KD-pt λ = 0.3, T = 5
λ = 0.7, T = 3 when τ = 0.0
KD-sim λ = 0.3, α = 0.3, β = 0.3
λ = 0.7, T = 5 when τ = 0.1
KD-topk k = 25, λ = 0.5, T = 5
λ = 0.7, T = 2 when τ = 0.2
KD-pt (b) CIFAR-100
λ = 0.7, T = 3 when τ = 0.3
λ = 0.7, T = 10 when τ = 0.4
λ = 0.7, T = 5 when τ = 0.5
KD-sim λ = 0.7, α = 0.5, β = 0.5 for any τ
(a) Synthetic

Method Hyper-parameter setting

LS  = 0.1
Method Hyper-parameter setting
KD λ = 0.7, T = 20
KD λ = 0.1, T = 50
KD-pt λ = 0.2, T = 25
KD-topk k = 100, λ = 0.1, T = 50
KD-sim λ = 0.3, α = 0.5, β = 0.3
KD-topk k = 500, λ = 0.5, T = 3 (d) Penn Tree Bank (PTB)

(c) ImageNet
Table 5: Hyper-parameter settings for different methods on different datasets.

et al., 2018), we found using l2 -normalized logits layer weight Ŵ and penultimate layer ĥ provides
more stable results. The model is optmized by Adam (Kingma & Ba, 2014) for a total of 3 million
steps without weight decay and we report the best accuracy. Finally, Nvidia V100 GPU is used as the
accelerator hardware. Please refer to Table 5a for the best setting of hyper-parameters.

CIFAR-100 dataset. CIFAR-100 is a relatively small dataset with low-resolution (32 × 32) images,
containing 50k training images and 10k validation images, covering 100 classes and 20 super-classes.
It is a perfectly balanced dataset – we have the same number of images per class (i.e., each class
contains 500 training set images) and 5 classes per super-class. To process the CIFAR-100 dataset,
we use the official split from Tensorflow Dataset4 . Both data augmentation 56 for CIFAR-100 and the
ResNet model7 are based on Tensorflow official implementations. Also, following the conventions,
we train all models from scrach using Stochastic Gradient Descent (SGD) with a weight decay of
1e-3 and a Nesterov momentum of 0.9 for a total of 10K steps. The initial learning rate is 1e-1, it will
become 1e-2 after 40K steps and become 1e-3 after 60K steps. We report the best accuracy for each
model. All experiments on CIFAR-100 are conducted by using Nvidia V100 GPU as the accelerator
hardware. Please refer to Table 5b for the best setting of hyper-parameters.

ImageNet dataset. ImageNet contains about 1.3M training images and 50k test images, all of
which are high-resolution (224 × 224), covering 1000 classes. The distribution over the classes is
approximately uniform in the training set, and strictly uniform in the test set. Our data preprocessing
4
[Link]
5
[Link]
6
We turn on the random brightness/saturation/constrast for better model performance.
7
[Link]

11
 Method % top-1 accuracy
Student 72.51
KD 75.94
KD-rel 74.14
KD-sim 74.30
KD-pt+rel 75.07
KD-pt+sim 75.24
Table 6: Performance of KD-rel on CIFAR-100. We report the mean result for 4 individual runs with
different initializations. We use β1 = 0.6, β2 = 0.1 0.3
4 , β3 = 95 .

and model on ImageNet dataset are follow Tensorflow TPU official implementations8 . The Stochastic
Gradient Descent (SGD) with a weight decay of 1e-4 and a Nesterov momentumof 0.9 is used. We
train each model for 120 epochs, the mini-batch size is fixed to be 1024 and low precision (FP16) of
model parameters is adopted. We didn’t change the learning rate schedule scheme from the original
implementation. Please refer to Table 5c for the best setting of hyper-parameters. We used TPU-v3
as the accelerator hardware.

Penn Tree Bank dataset. We use Penn Tree Bank (PTB) dataset for word-level language modeling
task using the standard train/validation/test split by (Mikolov et al., 2010). The vocabulary is capped
at 10K unique words. We consider the state-of-the-art LSTM model called AWD-LSTM proposed
by Merity et al. (2017). The model used several regularization tricks on top of a 3-layer LSTM,
including DropConnect, embedding dropout, tied weight, etc. We use different capacity (indicated by
hidden size and embedding size) as our Teacher and Student. To be specific, Teacher has a hidden size
of 1150 and an embedding size of 400, while Student has a smaller hidden size of 600 and a smaller
embedding size of 300. We follow the official implementation9 with simple changes for KD-topk.
Besides capacity, we keep the default hyper-parameter as in the official implementation to train our
Teacher. However, when training smaller Student model, we follow another implementation10 to: (1)
lower the learning rate to 0.2, (2) increase training epochs to 1000, (3) use 0.4 for embedding dropout
rate and (4) use 0.225 for RNN layer dropout rate. We used Nvidia P100 GPU as the accelerator
hardware.

8 Additional Experiments
Examine optimal geometry prior effect with class hierarchy. In section 4, we mentioned the
optimal geometry prior effects of KD can also be examined using existing class hierarchy. Suppose
our data has a pre-defined class hierarchy (e.g., on CIFAR-100), we can also use it to examine the
optimal geometry prior effects of KD. To be specific, let St ⊂ [K]\t denote the other classes that
share same parent of t. We simply assign different probability masses to different types of classes:

β1 if i = t,
ρrel = β if i ∈ St , (5)
i
 2
β3 otherwise,
where β1 > β2 > β3 are a hyper-parameters we could search and optimize, and we name this
method as KD-rel. As shown in Table 6, we found KD-rel performs slightly worse than KD-sim on
CIFAR-100. The trend is still hold when we compound each effect with KD-pt.

References
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormándi, George E. Dahl, and Geoffrey
E. Hinton. Large scale distributed neural network training through online distillation. arXiv
preprint arXiv:1804.03235, 2018.
8
[Link]
9
[Link]
10
[Link]

12
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In
Proceedings of the 26th annual international conference on machine learning, pp. 41–48. ACM,
2009.
Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Darkrank: Accelerating deep metric learning via
cross sample similarities transfer. arXiv preprint arXiv:1707.01220, 2017.
Wojciech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, and Razvan Pascanu.
Sobolev training for neural networks. In Advances in Neural Information Processing Systems, pp.
4278–4287, 2017.
Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.
Born again neural networks. arXiv preprint arXiv:1805.04770, 2018.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks
with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for
efficient neural network. In Advances in neural information processing systems, pp. 1135–1143,
2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pp. 770–778, 2016.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for
image classification with convolutional neural networks. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pp. 558–567, 2019.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge distillation with
adversarial samples supporting decision boundary. In Proceedings of the AAAI Conference on
Artificial Intelligence, volume 33, pp. 3771–3778, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv
preprint arXiv:1503.02531, 2015.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig
Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient
integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pp. 2704–2713, 2018.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. arXiv preprint
arXiv:1606.07947, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report,
Citeseer, 2009.
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying Distillation and
Privileged Information. arXiv preprint arXiv:1511.03643, 2015.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships
in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD
International Conference on Knowledge Discovery & Data Mining, pp. 1930–1939. ACM, 2018.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing lstm
language models. arXiv preprint arXiv:1708.02182, 2017.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. Recurrent
neural network based language model. In Interspeech, 2010.

13
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh. Improved knowledge
distillation via teacher assistant: Bridging the gap between student and teacher. arXiv preprint
arXiv:1902.03393, 2019.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv
preprint arXiv:1906.02629, 2019.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a
defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on
Security and Privacy (SP), pp. 582–597. IEEE, 2016.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International
Conference on Machine Learning, pp. 5142–5151, 2019.
Rajeev Ranjan, Carlos D Castillo, and Rama Chellappa. L2-constrained softmax loss for discrimina-
tive face verification. arXiv preprint arXiv:1703.09507, 2017.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Nicolas Le Roux. Tighter bounds lead to improved classifiers. arXiv preprint arXiv:1606.09202,
2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang,
Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition
challenge. International journal of computer vision, 115(3):211–252, 2015.
Suraj Srinivas and François Fleuret. Knowledge transfer with jacobian matching. arXiv preprint
arXiv:1803.00443, 2018.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking
the inception architecture for computer vision. In Proceedings of the IEEE conference on computer
vision and pattern recognition, pp. 2818–2826, 2016.
Jiaxi Tang and Ke Wang. Ranking distillation: Learning compact ranking models with high per-
formance for recommender system. In Proceedings of the 24th ACM SIGKDD International
Conference on Knowledge Discovery & Data Mining, pp. 2289–2298. ACM, 2018.
Vladimir Vapnik and Rauf Izmailov. Learning Using Privileged Information: Similarity Control and
Knowledge Transfer. Journal of machine learning research, 16(2023-2049):2, 2015.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast
optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pp. 4133–4141, 2017.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisit knowledge distillation: a
teacher-free framework. arXiv preprint arXiv:1909.11723, 2019.
Xu Zhang, Felix Xinnan Yu, Svebor Karaman, Wei Zhang, and Shih-Fu Chang. Heated-up softmax
embedding. arXiv preprint arXiv:1809.04157, 2018.

14