Multi-stage knowledge distillation for sequential

recommendation with interest knowledge

[Link]
25523014263
INTRODUCTION:
 Deep learning models can understand complex user behaviors but require a lot of computing power,
making them slow and hard to use in real-time applications.
 Knowledge distillation is a technique that helps transfer knowledge from a large, complex model
(the teacher) to a smaller, simpler model (the student). This helps balance performance and
efficiency.
 Existing methods in knowledge distillation for recommendations often only focus on whether a user
likes an item, missing out on why they like it. This can lead to problems, especially with changing
user interests.
 The authors apply curriculum learning, where the model learns in stages from easier to harder
tasks. This helps the student model absorb knowledge more effectively.
 The framework introduced is called Multi-Stage Knowledge Distillation with Interest Knowledge
(MSKDIK). It allows the student model to better utilize the knowledge from the teacher model to
improve its performance.

RELATED WORK

 Early sequential recommendation methods used techniques like Markov chains and matrix
factorization. These methods focus on user interactions but often struggle with understanding long-
term dependencies in user behavior.
 Recurrent Neural Networks (RNNs), such as Gated Recurrent Units (GRUs) and Long Short-Term
Memory (LSTM) networks, have been applied successfully to model user behavior.
 RNN-based models can have issues with noise in the data and may not always capture user interests
effectively.
 Newer models, like SASRec, use attention mechanisms to better focus on important parts of the
user’s historical interactions, improving prediction accuracy.
 Many of these advanced models suffer from high inference latency, making them slow and difficult
to use in real-time applications. Some solutions have been proposed to address this, like candidate
generation and re-ranking methods to speed up recommendations.
 Knowledge distillation is a technique that helps compress large models (teachers) into smaller ones
(students) by transferring knowledge. This has been effective in image classification and is now
being applied to recommendation systems.
 However, many existing methods focus only on the final predictions and do not consider the ranking
nature of recommendations, which is different from simple classification tasks.
 New methods have been developed that focus on ranking, like Ranking Distillation (RD), which uses
the top predicted items to guide the smaller model. Others, like Collaborative Distillation (CD),
sample knowledge based on user interactions.
 As the student model becomes more similar to the teacher model, it becomes harder to improve
performance further. New strategies are needed to extract different types of knowledge effectively
 Recent research has explored “hint regression,” which helps the student model learn from
intermediate layers of the teacher model. However, many of these approaches do not focus on the
relationships between different pieces of knowledge.
 Various new techniques have emerged to enhance performance in sequential recommendations,
such as using multiple interest embedding or adaptive knowledge extraction methods.

METHOD:

Behavior sequence Bu = [b1u , b2u, ⋯, bNu ] with ascending time order is given to predict the next
 |U| represents the number of users, |V| represents the number of items.

behavior bun+1.
 U: A set of users.
 V: A set of items.
 Each user's behavior is captured in a sequence, which is used to predict future interactions.
 Types of knowledge teacher model transfer to student model in this paper
 Interest Representation Knowledge: Refers to the potential representations of user interests at
specific time steps.
 Interest Drift Knowledge: Concerns changes in user interests over time, indicating shifts in
preferences.
 Three stages of knowledge distillations
 Response Based Knowledge Distillation: Utilizes the probability distribution from the teacher
model’s prediction layer, employing a probabilistic sampling method for effective knowledge
transfer.
 Interest Representation Distillation: Groups similar interest representations and uses an interest
expert module to refine the student model’s understanding of these interests.
 Interest Drift Distillation: Analyzes the representations of adjacent time steps to identify any shifts
in user interest, constructing a graph to represent these relationships for more targeted knowledge
transfer.

RELU:

 The ReLU function outputs the input directly if it is positive; otherwise, it outputs zero.
  This means it introduces non-linearity into the model while maintaining a simple and efficient
computation.
 By outputting zero for negative values, ReLU promotes sparsity in the activations. This can lead
to more efficient models as fewer neurons are activated.
 ReLU is commonly used as an activation function in hidden layers of deep learning models.
 It is particularly popular in convolutional neural networks (CNNs) and deep feedforward
networks.
 ReLU is typically applied after the linear transformation (dot product) in the layers of the neural
network.

EXPERIMENTS AND DATASETS

Experiments:

 Details about how the datasets were processed before feeding them into the model. This could
include normalization, encoding categorical variables, or splitting the data into training, validation,
and test sets.
 We evaluate the proposed knowledge distillation method on GRU4Rec+, SASRec
 the training epoch is set to 200
 Adam (Adaptive momentum) = RMSprob+Momentum gradient descending method is adopted to
optimize the loss function.
 The word embedding dimension dS of the student model is 10 and the word embedding dimension
dT of the teacher model is 50
 The evaluation metric used are NDCG@N and MRR@N. HR@N measures whether the target item
exists in the top-N prediction list
 The distillation framework is implemented in Python 3.6
 The models are compiled using the PyTorch1.7 package
 The operating system is Ubuntu 18.04 64-bit
 The training server is equipped with NVIDIA RTX3090 GPU, with 10,752 CUDA cores and 24 GB
memory. Also, with 10th Generation Intel i7 processors, total cores 8, total threads 16 and base
frequency 2.90 GHz.