self-distillation

This repository collects papers for "A Survey on Knowledge Distillation of Large Language Models". We break down KD into Knowledge Elicitation and Distillation Algorithms, and explore the Skill & Vertical Distillation of LLMs.

A curated collection of papers, technical reports, frameworks, and tools for on-policy distillation (OPD) of large language models

Calligrapher: Freestyle Text Image Customization

Enhancing LLMs with LoRA

A pytorch implementation of paper 'Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation', https://arxiv.org/abs/1905.08094

[ACL 2024] The official codebase for the paper "Self-Distillation Bridges Distribution Gap in Language Model Fine-tuning".

Source code of paper "RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation"

Deep Hash Distillation for Image Retrieval - ECCV 2022

(NeurIPS 2025 🔥) Official implementation for "Efficient Multi-modal Large Language Models via Progressive Consistency Distillation"

[CVPR 2025] Official PyTorch implementation of MaskSub "Masking meets Supervision: A Strong Learning Alliance"

Unified reproductions, benchmarking & evaluation for perception-aware RLVR and OPD in VLMs. 19 methods · 30+ benchmarks · Official code of CGPO (ACM MM 2026 Oral).

Official code for "Vision Transformers with Self-Distilled Registers" (NeurIPS 2025 Spotlight)

Aligning Language Models from User Interactions via Self-Distillation

Official implementation of iSDFT: information-proximal on-policy self-distillation for continual learning in LLMs. Training code, checkpoints, and evaluation results.

Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization. [arXiv 2026]

Official implementation of “ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation”

Self-Distillation with weighted ground-truth targets; ResNet and Kernel Ridge Regression

Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes. Official implementation.

(Unofficial) Data-Distortion Guided Self-Distillation for Deep Neural Networks (AAAI 2019)

Self-Distillation and Knowledge Distillation Experiments with PyTorch.