========================================
KNOWLEDGE DISTILLATION (KD) TECHNIQUES
========================================
This file lists important research papers for Knowledge Distillation techniques,
including offline, online, deep mutual learning, self-distillation, and ensemble
methods.
------------------------------------------------
1. OFFLINE / CLASSIC KNOWLEDGE DISTILLATION
------------------------------------------------
1. Distilling the Knowledge in a Neural Network
Authors: Geoffrey Hinton, Oriol Vinyals, Jeff Dean
Year: 2015
Summary: Introduced teacher-student knowledge distillation using soft
probabilities.
Link: [Link]
2. Knowledge Distillation: A Survey
Authors: Jianping Gou et al.
Year: 2020
Summary: Comprehensive survey on KD, covering offline methods.
Link: [Link]
------------------------------------------------
2. ONLINE KNOWLEDGE DISTILLATION
------------------------------------------------
3. Knowledge Distillation by On-the-Fly Native Ensemble (ONE)
Authors: Xu Lan, Xiatian Zhu, Shaogang Gong
Year: 2018
Summary: Online KD method with on-the-fly teacher ensemble.
Link: [Link]
4. Peer Collaborative Learning for Online KD
Authors: Guile Wu & Shaogang Gong
Year: 2020
Summary: Peer-based training and ensembling for online KD.
Link: [Link]
5. Decoupled Knowledge with Ensemble Learning for Online Distillation
Authors: Baitan Shao & Ying Chen
Year: 2023
Summary: Improved online distillation using ensemble predictions.
Link: [Link]
------------------------------------------------
3. DEEP MUTUAL LEARNING (DML)
------------------------------------------------
6. Deep Mutual Learning
Authors: Ying Zhang, Tao Xiang, Timothy Hospedales, Huchuan Lu
Year: 2018
Summary: Peer-based online distillation where multiple models learn from each
other.
Link: [Link]
------------------------------------------------
4. SELF-DISTILLATION
------------------------------------------------
7. Be Your Own Teacher: Self Distillation
Authors: Linfeng Zhang et al.
Year: 2019
Summary: Distilling knowledge within the same model.
Link: [Link]
8. Towards Understanding Ensemble, KD & Self-Distillation
Authors: Zeyuan Allen-Zhu & Yuanzhi Li
Year: 2020
Summary: Theoretical analysis connecting KD, self-distillation, and ensemble
effects.
Link: [Link]
------------------------------------------------
5. ENSEMBLE DISTILLATION
------------------------------------------------
9. Online Ensemble Model Compression Using KD
Authors: Devesh Walawalkar et al.
Year: 2020
Summary: Uses ensemble predictions to improve student models.
Link: [Link]
10. Learn from the Past: Experience Ensemble KD
Authors: Chaofei Wang et al.
Year: 2022
Summary: Ensembles intermediate models to improve distillation.
Link: [Link]
------------------------------------------------
6. SURVEYS & PAPER COLLECTIONS
------------------------------------------------
11. Knowledge Distillation: A Survey
Authors: Jianping Gou et al.
Year: 2020
Summary: Comprehensive survey of KD methods including DML, online, self-
distillation, feature-based, and ensemble KD.
Link: [Link]
12. Awesome Knowledge Distillation Paper Collection (GitHub)
Maintainer: Defang Chen
Summary: Curated list of KD papers, including DML, online KD, self-
distillation, adversarial distillation, and feature-based KD.
Link: [Link]
------------------------------------------------
END OF FILE
------------------------------------------------