Improving Language Understanding by Generative Pre-Training

[GPT-1]
This paper highlights the benefits of utilizing a large corpus of unlabeled data for pre-training,
and comparatively smaller labeled dataset for task-specific finetuning. As opposed to initializing
with random weights, this paper showed results that pre-training with a large unlabeled corpus
enabled the underlying model to capture more than just word-level knowledge, and learn rich
linguistic information that can be used to train a general purpose language model(Transformer
decoder architecture used here) that can be further adapted to specific tasks.
Its major contribution is a two-stage learning process:

1. Unsupervised Pre-training
Objective: A language model is trained on a large, diverse corpus of unlabeled text (BooksCor-
pus). Given an unsupervised corpus of tokens U = {u1 , . . . , un }, the objective is to predict
the next token in a sequence given the preceding tokens.
X
L1 (U ) = log P (ui |ui−k , . . . , ui−1 ; Θ)
i
Purpose: The main idea is that while learning to predict the next word, the model will
inherently have to learn grammar, semantics, world knowledge, and some reasoning capa-
bilities, that form the basis of many Natural Language Understanding tasks. The use of
BookCorpus, with its long stretches of continuous text, enables the model to learn beyond
just single sentences.
Model Architecture: The model uses a 12-layer Transformer decoder-only architecture.
This choice is due to the Transformer’s self-attention mechanism, which effectively captures
long-range dependencies in text better RNN/LSTM-based approaches. Each layer consists of
masked multi-head self-attention and a position-wise feed-forward network. The decoder is
chosen because of the nature of the objective function, i.e. the next word is to be predicted
using predicted using previous tokens only.
2. Supervised fine-tuning
The pre-trained model is then adapted to specific tasks using smaller, labeled datasets. This
requires minimal changes to the model architecture. Input data for various tasks (e.g.,
classification, entailment, similarity, multiple-choice QnA) is transformed into a single or-
dered token sequence, using special delimiter tokens as required. A single new linear output
layer(Wy ) is added on top of the final Transformer layer’s output to produce task-specific
predictions(P (y|x) = sof tmax(hl Wy ))
P
The fine-tuning objective L2 (C) = (x,y) logP (y|x) maximizes the likelihood of the correct
labels. Prior work shows that including language modeling as an auxiliary objective improves
generalization and accelerates convergence.
L3 (C) = L2 (C) + λ ∗ L1 (C)

Analysis

The authors observe that transferring each transformer layer(from the unsupervised pre-training
task) to the supervised target tasks improves performance, indicating that each layer in the pre-
trained model contains functionality for solving the target tasks.

1
The effectiveness of pretraining is further emphasized when target tasks are performed on the
pre-trained model without supervised fine-tuning(zero-shot performance).

BERT: Pre-training of Deep Bidirectional Transformers for Lan-

guage Understanding
BERT’s architecture is a multi-layer bidirectional Transformer encoder. It builds upon the success
of the Transformer model and unsupervised pre-training for NLU tasks. The results achieved from
BERT demonstrated the importance of bidirectional pre-training for language modeling. It uses
masked language model objective(MLM) and next sentence prediction tasks(NSP) for pre-training,
thus obtaining substantial results for both token-level and sentence-level tasks. Ablation studies
show that BERT proved effective not only as a fine-tuning model but also as a feature extractor,
where its contextual embeddings could be fed into separate task-specific models.

Input Representation

The input representation for BERT is designed to handle both single sentences and sentence pairs
(A,B). For a given token, its input representation is constructed by summing the corresponding
token embedding (from a WordPiece vocabulary of size 30K), segment embedding (indicating if the
token belongs to sentence A or B), and position embedding. Special tokens [CLS] and [SEP] are
used. The [CLS] token is always the first token of every sequence and the [SEP] token is used to
separate sentences in a pair.

Pre-training Tasks

BERT is pre-trained using two unsupervised tasks on a large corpus comprising BooksCorpus(700M
words) and English Wikipedia(2500M words).
This first objective is called as Masked Language Modeling(MLM), also referred as Cloze task
in prior work. To do deep bidirectional pre-training without allowing tokens to ”see themselves”
indirectly, 15% of the input WordPiece tokens in each sequence are masked at random. This
results in a mismatch between the pre-training and finetuning, as [MASK] does not appear during
finetuning. To address this, only 80% of the chosen tokens are replaced with a [MASK] token, 10%
are replaced with a random token, and 10% are kept unchanged. A transformer encoder is then
trained to predict the actual tokens where [MASK] is present.
The second task is Next Sentence Prediction (NSP). This is a binary classification task that
trains the model to understand sentence relationships. For each pre-training example, two sentences
A and B are chosen. 50% of the time, B is the actual next sentence following A (labeled IsNext),
and 50% of the time, B is a random sentence from the corpus (labeled NotNext). During NSP
pre-training, the model is trained to make a binary classification (IsNext vs. NotNext) based only
on the representation of the [CLS] token. To be successful at this task, the model must learn to
condense the relevant information from the entire pair of input sentences into the [CLS] token’s
representation. If it didn’t, it wouldn’t be able to accurately predict the relationship between the
sentences. So, the NSP task effectively ”teaches” the [CLS] token to become a good sentence-pair
summarizer for classification purpose.

2
Fine-tuning BERT

Fine-tuning BERT for specific downstream tasks is straightforward and requires minimal architec-
tural modification. The pre-trained BERT model is initialized with its learned parameters, and an
additional output layer specific to the task is added. All parameters of BERT, along with the new
layer, are then fine-tuned end-to-end using labeled data for the specific task.

GLUE Benchmark Evaluation

To assess its general language understanding capabilities, BERT was fine-tuned and evaluated on
the General Language Understanding Evaluation (GLUE) benchmark. This benchmark comprises
a diverse set of tasks such as natural language inference, sentence similarity, sentiment analysis,
question answering, and linguistic acceptability. For these tasks, the input was formatted as single
sentences or sentence pairs using the [CLS] and [SEP] tokens, and the final hidden state of the
[CLS] token was used for classification.

SQuAD v1.1

In this task, given a question and a passage, the model should predict the text span in the passage
that answers the question. Start and end vectors were introduced during fine-tuning to predict the
answer span boundaries based on the token representations.

SQuAD v2.0

The SQuAD v2.0 task extends SQuAD v1.1 by allowing questions to be unanswerable from the
given passage. BERT was adapted by treating unanswerable questions as having an answer span
at the [CLS] token. A threshold was used to decide between predicting a no-answer response or a
text span.

SWAG

Given a sentence, the task is to choose the most plausible continuation from four choices. This task
was chosen to evaluate BERT on commonsense inference. Each sentence-choice pair was formed
into an input sequence, and a score was computed using the [CLS] token’s representation.

Attention is All You Need

The paper introduces the Transformer, a new network architecture for sequence-to-sequence tasks.
Earlier models relied on complex RNNs or CNNs(often in an encoder-decoder configuration) which,
although effective, had many drawbacks such as: parallelization was difficult, exploding/vanishing
gradients, and difficulty in capturing long term dependencies- even with the effectiveness of atten-
tion mechanism. The Transformer model focuses only on attention mechanism to capture dependen-
cies between input and output. This approach allows for much more parallelization during training
and, as demonstrated by experiments, can lead to superior translation quality with substantially
less training time.

3
Model Architecture

The Transformer follows an encoder-decoder structure. The encoder maps an input sequence of
symbol representations (x1 , ..., xn ) to a sequence of continuous representations z = (z1 , ..., zn ).
Given z, the decoder then generates an output sequence (y1 , ..., ym ) one symbol at a time, consuming
previously generated symbols as additional input (auto-regressive).
The encoder is composed of a stack of N identical layers (N=6 in the base model). Each encoder
layer has two main sub-layers: a multi-head self-attention mechanism and a point-wise fully con-
nected feed-forward network. Residual connection and then layer normalization is applied around
each of the sub layer output . So output of each sub-layer is LayerNorm(x + Sublayer(x)), where
Sublayer(x) is the function implemented by the sub-layer.
The decoder is also composed of a stack of N identical layers. In addition to the two sub-layers in
encoder, the decoder adds a third sub-layer, which performs multi-head attention over the output
of the encoder stack. Similar to the encoder, residual connections and layer normalization are used
around each sub-layer. The self-attention sub-layer in the decoder is modified to prevent positions
from attending to subsequent positions, ensuring the auto-regressive property by masking future
tokens.

Attention Mechanism

An attention function maps a query and a set of key-value pairs to an output, where the query, keys,
values, and output are all vectors. The output is a weighted sum of the values, where the weight
assigned to each value is computed by a compatibility function of the query with the corresponding
key.
The paper introduces Scaled Dot-Product Attention. The input consists of query and key
vectors of dimension dk , and
√ values of dimension dv . The dot products of the query with all keys
are computed, divided by dk , and softmax function is applied to obtain the weights on the values.
In practice, this is computed on a set of queries simultaneously(present in matrix Q), and keys and
values present in matrices K and V. The matrix of outputs is computed as:
 Q ∗ KT 
Attention(Q, K, V ) = sof tmax √ ∗V
dk
√
The scaling factor 1/ dk is used because for large values of dk , the dot products can grow large in
magnitude.
Instead of performing a single attention function with dmodel -dimensional keys, values, and queries,
the authors found it better to use Multi-Head Attention. This involves linearly projecting
the queries, keys, and values h times with different, learned linear projections to dk , dk , and dv
dimensions respectively. On each of these projected versions, the attention function is performed
in parallel, yielding dv -dimensional output values. These are then concatenated and once again
projected, resulting in the final values. The formula for Multi-Head Attention is:

M ultiHead(Q, K, V ) = Concat(head1 , ..., headh ) ∗ W O

In the paper they use h=8, with dk = dv = dmodel /h = 64. This multi-head approach allows the
model to jointly attend to information from the same input but with focus on different aspects.
Attention is used in three ways in the Transformer:

4
 1. In encoder-decoder attention layers, the queries come from the previous decoder layer, and
the keys and values come from the output of the encoder. This allows every position in the
decoder to attend over all positions in the input sequence.
2. The encoder contains self-attention layers, where all keys, values, and queries come from the
output of the previous layer in the encoder.
3. Similarly, self-attention layers in the decoder allow each position in the decoder to attend to
all positions in the decoder till that position only, implemented via masking.

Position-wise Feed-Forward Networks

Along with attention sub-layer, each layer in the encoder and decoder contains a fully connected
feed-forward network (FFN), which is applied to each position separately and identically. It consists
of two linear transformations with a ReLU activation in between:
F F N (x) = max(0, x ∗ W1 + b1 ) ∗ W2 + b2
The linear transformations are same across different positions, they use different parameters from
layer to layer. The dimensionality of input and output is dmodel = 512, and the inner-layer has
dimensionality df f = 2048.

Embeddings and Softmax

A learned embedding matrix is used to convert input and output tokens to vectors of dimension
dmodel . A linear transformation and softmax function convert the decoder output √ to predicted
next-token probabilities. In the embedding layer, these weights are multiplied by dmodel .

Positional Encoding

Since this model does not have recurrence or convolution, positional encodings are added to input
embeddings to add information about the position of the vector. The paper uses sine and cosine
functions of different frequencies, but learned positional embeddings achieve similar results.

BART: Denoising Sequence-to-Sequence Pre-training for Natural

Language Generation, Translation, and Comprehension
BART (Bidirectional and Auto-Regressive Transformers) is a pre-training model designed for a
wide range of natural language processing tasks such as generation, translation, and comprehension.
It has a denoising encoder built with a Transformer-based sequence-to-sequence architecture. It
studies different approaches of corrupting the input(including comparison with different pre-training
objectives of prior work) and how the results obtained from pre-training can be finetuned for various
tasks, thus offering a generalized architecture.

Model Architecture

BART has a standard Transformer encoder-decoder architecture. The encoder processes corrupted
input bidirectionally, while the decoder learns to generate the original text autoregressively.

5
Pre-Training

Pre-training BART involves two steps: corrupting an input document using various noising func-
tions, and then training the model to reconstruct the original document. The paper evaluates
several noising schemes:

• Token Masking: Random tokens are replaced with a [MASK] symbol.

• Token Deletion: Random tokens are deleted.

• Text Infilling: Spans of text are replaced with a single [MASK] token, forcing the model to
predict the number of missing tokens and their content.

• Sentence Permutation: Sentences in a document are randomly shuffled.

• Document Rotation: A document is rotated to start at a random token.

The combination of text infilling and sentence permutation was found to be effective in the exper-
iments.

Fine-tuning BART

BART can be fine-tuned for various tasks as follows:

• For sequence classification tasks, the same input is fed to both the encoder and decoder, and
the final decoder hidden state is used for classification.

• For token classification tasks also, the same input is fed to both the encoder and decoder,
and the final hidden state output of the decoder for each token is used for classification.

• For sequence generation tasks, the encoder processes the input, and the decoder generates the
output autoregressively, making it suitable for abstractive summarization, question answering,
and dialogue.

• For machine translation (into English), BART’s pre-trained decoder is used, and a new source
language encoder is trained to map foreign text into a representation that BART can decode
into English.

Large-Scale Pretraining
To evaluate its performance on a large scale, BART was scaled up(similar to RoBERTa) and
pretrained on the same large corpus.

• On discriminative tasks like GLUE and SQuAD, BART performed comparably to RoBERTa.

• On generative tasks, BART achieved new state-of-the-art results, especially on abstractive

summarization. Dialogue response generation and abstractive question answering also had
significant improvement.

6
Sequential Recommendation: Problem Definition
Let U = {u1 , u2 , . . . , u|U | } be a set of users and V = {v1 , v2 , . . . , v|V | } be a set of items. For each
user u ∈ U , their interaction history is represented as a chronologically ordered sequence of items
(u) (u) (u) (u)
Su = [v1 , v2 , . . . , vnu ], where vt ∈ V is the item user u interacted with at time step t, and
nu = |Su | is the length of this historical sequence.
(u)
The problem of sequential recommendation is to predict the next item vnu +1 that user u is most
likely to interact with at the next time step nu + 1, given their historical interaction sequence Su .
This involves learning a model that can estimate the probability distribution over all possible next
items:
(u)
P (vnu +1 = v | Su ) for all v ∈ V.
The goal is to recommend the item(s) with the highest predicted probability.

Pre-processed Dataset Format: The dataset consists of a .txt file containing lines of the form
(user id item id). Items for a user are present in chronological order of their interactions.

SASRec: Self-Attentive Sequential Recommendation

The SASRec model adapts the Transformer model for the problem of sequential recommendation,
which demonstrated good results for machine translation task and has a parallelizable architecture,
which was difficult to achieve in RNNs. The main architecture of SASRec consists of masked self-
attention blocks and a feed-forward network. The use of masked self-attention blocks is chosen
because the authors assume that the model should train on predicting the next interaction based
on previous interactions only.
Previous approaches to sequential recommendation were Markov Chain-based and RNN-based.
MC-based approaches gave good results on sparse datasets, whereas RNN-based approaches re-
quired dense datasets to perform well. Since SASRec is attention-based, it was able to capture
long-range dependencies on dense datasets and focused on recent interactions for sparse datasets.
So SASRec is able to outperform all existing models for both sparse and dense datasets.

Data Partitioning

The data partition function reads the preprocessed dataset file. The function iterates through these
interactions, grouping items by user. For each user, their sequence of item interactions is split into
three parts:

• user train: Contains all items except the last two

• user valid: Contains the second last item from the user’s sequence

• user test: Contains the last item from the user’s sequence

If a user has less than three interactions, all their items are allocated to user train.

7
Training Batch Sampling

The WarpSampler class is used for generating batches for training. For each training instance in a
batch, the sample function does the following:
-It randomly selects a user from the user train set.
-It constructs three sequences of length maxlen (the maximum sequence length):

• seq: This is the input sequence to the model. it has all the items except the last. If this
sequence is shorter than maxlen, it is padded with zeros at the beginning. If it is longer, only
the most recent maxlen items are kept.

• pos: This sequence contains the positive target items. For each item seq[i], pos[i] is the item
that chronologically followed seq[i] in the user’s original training sequence.

• neg: This sequence contains the negative target items. For each pos[i](non padding term),
a corresponding neg[i] is generated by randomly sampling an item ID that the user has not
interacted with in their entire history (user train[user]).

Thus we generate (user, seq, pos, neg) tuples for the training loop to consume.

Embedding Layer

The function of this layer is to convert item id’s and positions to d-dimensional vector representa-
tion.
Item Embeddings: Here, an item emb table of size |itemnum+1|×d is created, that includes the
representation for padding term at the start. The representation for padding term is all zeroes.
This item emb table is stored and reused for the prediction layer as well. L2 regularization can be
applied to this table to prevent a particular feature value from becoming too large.
Positional Embeddings: A separate learnable positional embedding table (pos emb table) is
created. Its dimensions are same as item emb table. These are learnable positional encodings, as
the paper found them to perform better than fixed sinusoidal encodings for their task.
The item embeddings and positional embeddings are then added element-wise. A dropout is applied
to these combined embeddings, followed by masking to make embeddings at padding positions zero.

Self-Attention Block

This is the core of SASRec model. Each block has two main sub-layers:
Multi-Head Self-Attention: The input seq is first layer-normalized and then passed as queries
to the multihead attention function, while the original (before normalization) seq is passed as keys
(from which K and V are derived internally). Casual masking is implemented to prevent the model
from attending to future items during sequential prediction. The default no. of heads is 1. The
multihead attention function itself incorporates the residual connection (adding its input queries
to its output).
Point-Wise Feed-Forward Network (FFN): The output from the self-attention sub-layer is
again layer-normalized and then passed to the FFN. This FFN consists of two linear operations with

8
a ReLU activation in between. The hidden dimension of both FFN layers is d. The feedforward
function also includes a residual connection.
After each complete self-attention block (self-attention + FFN), the padding mask is re-applied to
the output to ensure padding positions remain zeroed out. Finally, after all blocks, one last layer
normalization is applied.

Prediction Layer

For Training: The final processed sequence embeddings are used to predict the next item at each
time step. Embeddings for positive (pos emb) and negative (neg emb) target items are looked up
from the item emb table. The logits (relevance scores) are then computed as the dot product of
processed sequence embedding and corresponding positive item embedding pos emb, and similarly
for neg emb logits.
For Evaluation: To predict the item after an entire input sequence, the model uses the output
embedding from the last time step of the processed sequence . The test logits are calculated by
taking the dot product of this final sequence embedding with the embeddings of candidate test
items (1 true item + 100 negative items, looked up from item emb table).

Training Process

The model is trained to minimize a binary cross-entropy loss. For each time step t in a training
sequence, the loss considers the model’s prediction for the actual next item (pos[t]) and a sampled
negative item (neg[t]). If y is the true label (1 for positive, 0 for negative) and p is the predicted
probability σ(logit),the loss is calculated as:

BCE loss = −[y log(p) + (1 − y) log(1 − p)]

Masking Strategies

SASRec uses two masking strategies:

Padding Mask: This is to ensure that padding terms in the input sequences do not influence the
computations in the self-attention mechanism or contribute to the loss. A mask tensor is created
based on non-zero items in input sequence. In the mask, positions corresponding to padding
term are 0, and otherwise 1. This mask is multiplied with the sequence embeddings after the
initial embedding layer and after each self-attention block to zero out contributions from padding
positions. Within the multihead attention function, attention scores corresponding to padding keys
or queries are made negligible by setting them to a very large negative number before the softmax
operation, resulting in almost-zero attention weights.
Causal Mask: This is to ensure that when predicting the item at time step t+1, the model should
only attend to items at time steps 1 to t and not to items at t+1, t+2,... (future items). For this
mask, a lower triangular matrix is created. This matrix has ones on and below the diagonal and
zeros above. This mask is applied to the attention scores before the softmax. Scores at positions
(i, j) where j > i (query i attending to a future key j) are set to a very large negative number. This
ensures that an item can only attend to previous items and itself.

9
Evaluation Metrics

Hit Rate@10 and NDCG@10 are used to evaluate recommendation performance. Rank starts from
0. HR@10 counts the fraction of times the ground-truth next item is present among the top 10 most
probable items. If the rank of the true item is less than 10 (i.e., from 0 to 9), it’s a hit. NDCG@10
gives higher scores if the true item is ranked higher in the top-10 list, it is a position-aware metric.
(
1
1 X
log2 (ranku +2) if ranku < 10
NDCG@K =
|Users| 0 otherwise
u∈Users

To avoid heavy computation, for each user 100 items are negatively sampled, and these are ranked
along with the ground-truth item. HR@10 and NDCG@10 can be calculated using the rank of
ground truth item among these 101 items.

Datasets Used

The model is evaluated on datasets from different domains and varying sparsity.

Amazon Beauty and Amazon Games are known for their high sparsity and variability. Steam can
also be considered sparse but less variable user interactions. MovieLens is a dense and widely used
benchmark.

BERT4Rec: Sequential Recommendation with Bidirectional En-

coder Representations from Transformer
BERT4Rec aims to capture the dynamic and evolving preferences of users based on their historical
interactions more accurately, rather than just assuming a fixed pattern. Traditional sequential
models often process user behavior from left to right, which can limit the richness of item represen-
tations within a sequence and enforces a strict chronological order. BERT4Rec proposes using a
bidirectional self-attention mechanism, inspired by the successful BERT model in NLP. This bidi-
rectional architecture allows each item in a user’s history to gather contextual information from
both preceding and succeeding interactions, thus enabling more powerful representations.
To train such a bidirectional model without the prediction position seeing the ground truth item it is
supposed to predict, BERT4Rec uses the Cloze task. During training, a fraction of items within an
input sequence are randomly masked, and the model’s objective is to predict these masked items
based on their surrounding unmasked context. This strategy enables the model to learn robust
bidirectional representations. For making predictions at inference time, a special [mask] token is
appended to the end of a user’s known sequence, and the model predicts the item that best fits
this masked position. BERT4Rec demonstrates significantly improved performance over SASRec,
highlighting the advantages of its bidirectional representations.

10
Data Partitioning

The dataset files are in the same format as in SASRec. the raw data points are then partitioned
into user train, user val and user test in the same manner as well. The provided code then merges
the validation items back into the training sequence for each user before testing the model.

Vocabulary Creation

A vocabulary is created to map the item id’s to integer based on the frequency of their occur-
rence(meaning more frequent items are assigned lower integer ID’s), starting from 1. After this
the special tokens [PAD],[MASK] and [NO USE] are assigned int ID’s in continuation. The total
vocabulary size that is used for the model’s embedding layer, accounts for all unique items, special
tokens, and the padding term 0.

Training Instance Creation and Masking Strategies

Each training instance has a masked sequence, the positions of masked items, and their true labels.
Following masking strategies are used:
Cloze Task: For each input sequence (can be reused dupe factor times and can be further processed
by a sliding window if longer than max seq length):

• masked lm prob fraction of items (e.g., 0.15 or 0.2) are randomly selected as candidates for
masking.

• Each selected candidate item is then modified. With a high probability (mask prob, e.g., 0.8),
it is replaced by the [MASK] token. Otherwise with probability 1 - mask prob, it is either
kept as the original item (50% chance) or replaced by a random item from the vocabulary
(50% chance).

• The function returns the modified token sequence, the positions of the masked items, and
their original item IDs as labels.

Mask Last: In this masking strategy only the last valid item in a given sequence is replaced by
”[MASK]”. To better align the training objective with the final recommendation task, samples are
also created where only the last item in the input sequence is masked. for each input sequence,
dupe factor times standard masking(cloze task) is done, and one time only last item is masked.
This prepares the model to directly predict the next item. This strategy is also applied during
testing, since the final goal of the model is to predict the last item only.

Embedding Layer

Item Embeddings: The embedding lookup function creates an embedding table) for all items
and special tokens in the vocabulary. The size of this table is vocab size x d(hidden size).
Positional Embeddings: The embedding postprocessor function adds learnable positional em-
beddings to the item embeddings. The summed item and positional embeddings then pass through
layer normalization and dropout before they are input to the encoder layers.

11
Transformer Encoder Blocks

Transformer encoder layers form the core architecture. Each layer consists of:
Multi-Head Self-Attention: This allows each item in the sequence to attend to all other items
(including itself) in both forward and backward directions, capturing bidirectional context. The
the input sequences are projected into queries, keys, and values for num attention heads parallel
attention computation. The outputs are concatenated and linearly projected.
A residual connection adds the input of the attention sub-layer to its output, followed by layer
normalization. Dropout is applied before the residual addition.
Position-wise Feed-Forward Network: This is a two-layer fully connected network with a
GELU activation function in between. It is applied independently to each position. The inter-
mediate size is 4 * d. Another residual connection and layer normalization are applied after the
FFN.
The final output of the encoder blocks is sequence output, representing the contextualized hidden
states for each item in the input sequence, including the masked items.

Training Process

From the sequence output, the hidden state vectors corresponding to the masked lm positions (the
positions where items were masked during data generation) are taken. These selected hidden states
are passed through an additional transformation layer (a dense layer with GELU activation and
layer normalization). The output logits are computed by performing a matrix multiplication be-
tween these transformed hidden states and the shared item embedding table . An output bias
term is also added. The loss function is the negative log-likelihood of predicting the correct
masked lm ids (true labels of the masked items) given these logits. The loss is averaged over
the actual (non-padded) masked predictions.
The goal is to train the model such that the probability for the true original items are high.
next-item prediction: The [MASK] token is appended to the end of the user’s known historical
sequence. The model’s task is to predict the item that should replace this final [MASK] token,
based on its output hidden state.

Evaluation Metrics

Similar to evaluation strategy in SASRec, for each user in the evaluation batch, a list of 101 candi-
date items is constructed: the true next item plus 100 negative items. Negatives may be sampled
uniformly at random or based on item popularity (using probabilities derived from [Link]).
These 101 items are ranked based on their predicted scores. The rank of the true positive item is
used to calculate standard recommendation metrics:

• Hit Ratio (HR@k): Whether the true item is within the top-k ranked items (calculated
for k=1, 5, 10).

• Normalized Discounted Cumulative Gain (NDCG@k): A position aware metric that

gives higher scores for ranking the true item higher (calculated for k=1, 5, 10).

12
 • Mean Reciprocal Rank (MRR): The average of the reciprocal of the rank of the true
item.

Datasets Used

The model is evaluated on four datasets, from different domains and varying sparsity. Three of
them are the same as used in SASRec.

Analysis

• Sequential recommendation models (like FPMC, GRU4Rec+, Caser) generally performed

better than non-sequential methods (like BPR-MF, NCF), highlighting the importance of the
order of user interactions in capturing user preferences.

• Self-attention based models(SASRec and BERT4Rec) genrerally outperformed RNN-based

(GRU4Rec, GRU4Rec+) and CNN-based (Caser) sequential models, indicating that the self-
attention mechanism is a more powerful tool for capturing dependencies in user sequences
compared to recurrent or convolutional approaches.

• BERT4Rec, which uses a bidirectional self-attention mechanism, significantly outperformed

SASRec, which uses a unidirectional (left-to-right) self-attention model.

• Mask proportion is an important factor during training. Datasets with short sequence length
prefer a larger mask proportion, while datasets with longer sequences prefer a smaller pro-
portion

• Optimal max seq length is highly dependent on avg sequence length of dataset. Amazon
Beauty prefers a smaller max seq length=20, and MovieLens-1m achieves best performance
on max seq length=200.

• It is observed that long sequence datasets prefer a larger h, while short sequence datasets
prefer smaller h, indicating more heads are better at capturing long-range dependencies.

13
Code Implementation

Figure 1: HR@10

Figure 2: NDCG@10

Training and Test Set Generation [SASRec]

The initial division of the raw sequential data into training, validation, and test sets is done by the
data partition function in [Link].
In that function, the raw data file is parsed, and a dictionary named User is created where keys
are user IDs and values are lists of item IDs representing their interaction history in chronological
order. The code then iterates through each user in the User dictionary, to split the data into
train, test and validation sets. This creates three dictionaries: user train, user valid, and user test.
For each user, their sequence of interactions is split based on its length: If a user has less than 3
interactions, all its interactions go to the training dictionary(user train).
user train stores the initial part of each user’s sequence, used for training the model.
user valid stores the second last item of the user, it is used for validation (to tune hyperparame-
ters).
user test stores the last item of the user, it is used for final testing.

Generation of training tuples

Once the user train dictionary is created, the WarpSampler class in [Link] generates the actual
training tuples that are fed into the model. These tuples consist of an input sequence, corresponding

14
positive next items, and corresponding negative next items. [user, seq, pos, neg]
A user is chosen randomly that has more than 2 items in their history.
seq[i]: current item that the model sees.
pos[i]: the item next to seq[i]. The model is trained to predict pos[i] with a high probability.
neg[i]: a randomly sampled item that is not present in complete user history. The model is trained
to predict neg[i] with a low probability.
Padding and Truncation: If the user’s sequence user train[user][:-1] is shorter than maxlen, the
left part of these arrays will remain zeros (padding). If it’s longer, only the most recent maxlen
interactions will be included.

Training and Test Set Generation [BERT4Rec]

The initial user train, user valid and user test split is the same as in SASREc.
The training sequences contain all the items of the user history except the last item, stored in
user train data str. For test data sequences (user test data str), for each user their entire historical
sequence is taken.

Generating Individual Training Instances

If a user’s sequence is longer than max seq length, it can be broken down into multiple overlapping
subsequences of length max seq length. If its shorter, it is padded.
Masked Language Model (MLM) Objective Application: For each sequence (now of length
max seq length after padding/truncation):
A fraction (masked lm prob, e.g., 15-20%) of the items in the sequence are randomly chosen to be
masked.
With a high probability (mask prob=1 in code, 0.8 in BERT paper), the chosen item is replaced
with a special [MASK] token. With a small probability (e.g., 0.1), the chosen item is left unchanged
(the model has to predict the original item even if it sees it). With another small probability (e.g.,
0.1), the chosen item is replaced with a random other item from the vocabulary.
The original item IDs of the masked positions are stored as the targets for prediction.
Mask Last Instances (for training): Along with random masking, gen data [Link] also creates
specific training instances where only the very last actual item in each sequence is masked, to
align the training objective with the final evaluation task (predicting the next item, which is
unidirectional task)
Duplication (dupe factor): The entire process of creating masked instances from the training
sequences can be repeated dupe factor times. Each time, a different random set of items will be
masked in each sequence, thus increasing training data.
A Single Training Instance Format: a single training instance (defined by the TrainingInstance
class and then written to TFRecord) contains the following info:
info: user id
input ids: list of tokens of length max seq length. Some of these will be the [MASK], some might

15
be IDs of random replacement items, and the rest are original item IDs, padded with [PAD].
input mask: list of 1s and 0s of length max seq length. 1 for real tokens (items, [MASK]), 0 for
[PAD] tokens.
masked lm positions: list of indices (0 to max seq length - 1) indicating which positions in
input ids were masked.
masked lm ids: list of the original item IDs that were at the masked lm positions before masking.
These are the target items.
masked lm weights: a list of 1s for actual masked predictions and 0s for padding in masked lm ids
or positions.

How the MLM Objective is Used for Training for a Sequential Task

A batch of training instances to the Bert4RecModel. The model processes the input ids (with
[MASK] tokens) through its bidirectional Transformer layers.
For each position specified in masked lm positions, the model takes the final hidden state output
by the Transformer. This hidden state is then passed through a feed-forward network (MLM
head in Bert4RecModel) and projected to the vocabulary size to produce logits (scores) for all
possible items. The loss function is calculated by comparing these predicted logits with the true
masked lm ids (the original items that were masked). Gradients are backpropagated to update all
model weights.
The training instances with only the last item masked specifically help align the model to sequential
recommendation, while taking bidirectional representations into account.

16
GRU4Rec Training Process
Dataset: The input data consists of sequences of user interactions, where each sequence is a
”session.” A session is a series of item IDs that a user interacted with in chronological order during
a single visit. In this dataset, explicit user IDs across sessions are not used, and each session is
treated independently.
Data Splitting: In this model, entire sessions are assigned to one split according to their ending
times. A single session is not broken across train/test set.

Model Architecture(GRU4RecModel class)

Figure 3: General Architecture of the network

1. Embedding Layer
The item ID is first converted into an embedding vector. The input embedding for an item is shared
with its output embedding. Specifically, [Link] is used to look up the embedding for the input
item. Dropout (dropout p embed) is applied to the embeddings during training.
2. GRU Layers
The item embedding is fed into one or more GRU layers. Each GRU layer maintains a hidden state.
This state is updated at each step, capturing sequential information from the items seen so far in
the session. Dropout (dropout p hidden) is applied to the output of each GRU layer (except the
last) if training=True.
3. Output Layer and Score Calculation
The final hidden state (Xh) from the last GRU layer represents the current session’s context. To
predict the next item, this session context vector Xh is used to compute scores for all candidate
items (or a sampled subset). Scores are computed via a dot product between the session vector Xh
and the output embeddings of the candidate items, plus a bias term.

Data Preparation and Iteration (SessionDataIterator class)

Input Data: The training data has SessionId, ItemId, and Time columns, sorted by SessionId and
then Time.

17
Item ID Mapping: Item IDs (can be integers/strings) are mapped to contiguous integer indices
(0-indexed). This is stored in [Link].
Session-Parallel Mini-batches: The iterator creates mini-batches by taking batch size number
of active sessions and processing them in parallel.
For each step in the training loop: in idx is a tensor of shape (current batch size) that contains
the item IDs of the current event for each active session in the batch. out idx (or y in the iterator)
serves as the set of ”target” items for which scores will be computed. It contains the actual next
item for each of the current batch size active sessions (these are the positive samples), and negative
samples also.
The actual next items from other sessions in the current mini-batch inherently act as negative
samples for a given session’s prediction. The loss functions are designed to handle this. self.n sample
additional items are drawn from the overall item distribution (weighted by sample alpha which
controls popularity bias.) A sample alpha closer to 1 makes the negative samples heavily biased
towards popular items. A sample alpha closer to 0 makes the negative samples more uniformly
random.
Handling Variable Session Lengths: Sessions in a mini-batch have different lengths. The
iterator processes them step-by-step up to the length of the shortest session in the current batch.
When a session ends, its replaced by a new session and the corresponding hidden state in H for
that slot in the batch is reset to zero.

Training Loop ([Link] method)

Hidden State Initialization: At the beginning of each epoch, when new sessions fill batch slots,
the hidden states H for the GRU layers are initialized to zeros. H is a list of tensors, one for each
GRU layer, each of shape (batch size, layer size).
Iterating Through Sessions: The SessionDataIterator generates in idx (current items) and
out idx (target items: true next + negatives).
Forward Pass:
R = [Link](in idx, H, out idx, training=True)
This computes the scores R for each item in out idx with respect to each session context derived
from in idx and current H. R will have shape (current batch size, len(out idx)).
Loss Calculation: The loss is computed based on the scores R and the true next items. Cross
entropy Loss is used in the code. Scores R are passed through a softmax function to get probabilities.
The loss is the negative log probability of the true next item.

What the Model Learns to Predict

For a given input sequence (session prefix), the model learns to output a vector of scores, one score
for each potential next item (from a sampled subset). For cross-entropy loss, it learns to maximize
the softmax probability of the true next item during training.

18
Code Implementation Results

Table 1: Performance comparison of different models on next-item prediction.

Datasets Metric BERT4Rec SASRec GRU4Rec GRU4Rec+SR-emb biGRU

HR@1 0.0953 0.1962 0.1219 0.2978 0.0607
HR@5 0.2207 0.3677 0.3831 0.5379 0.0838
HR@10 0.3025 0.4677 0.4941 0.6095 0.0947
Beauty
NDCG@5 0.1599 0.2850 0.2553 0.3440 0.0726
NDCG@10 0.1862 0.3171 0.2914 0.3825 0.0762
MRR 0.1701 0.2879 0.2283 0.3139 0.0704
HR@1 0.0957 0.3679 0.0655 0.0833 0.0791
HR@5 0.2710 0.7303 0.1067 0.1273 0.1031
HR@10 0.4013 0.8521 0.1679 0.1903 0.1556
Steam
NDCG@5 0.1842 0.5603 0.0558 0.0706 0.0993
NDCG@10 0.2261 0.5998 0.0712 0.0865 0.1134
MRR 0.1949 0.5279 0.0405 0.0578 0.0956
HR@1 0.3478 0.1167 0.1905 0.0618
HR@5 0.6166 0.3435 0.4571 0.0990
HR@10 0.7118 0.4548 0.5765 0.1327
Games
NDCG@5 0.4917 0.2328 0.3290 0.0802
NDCG@10 0.5227 0.2689 0.3677 0.0911
MRR 0.4734 0.2113 0.3027 0.0786
HR@1 0.2863 0.3679 0.0601 0.0798 0.0983
HR@5 0.5876 0.7103 0.1988 0.2241 0.1342
HR@10 0.6970 0.8185 0.2974 0.3209 0.1957
ML-1m
NDCG@5 0.4454 0.5510 0.0924 0.1531 0.1062
NDCG@10 0.4818 0.5863 0.1272 0.1843 0.1396
MRR 0.4254 0.5210 0.1081 0.1426 0.1046
HR@1 0.3440 0.6158 0.1459 0.2021 0.1163
HR@5 0.6323 0.9266 0.4657 0.5118 0.1676
HR@10 0.7473 0.9732 0.5844 0.6524 0.2318
ML-20m
NDCG@5 0.4967 0.7888 0.3090 0.3630 0.1271
NDCG@10 0.5340 0.8042 0.3637 0.4087 0.1477
MRR 0.4785 0.7504 0.2967 0.3476 0.1223

Adapting for Sequential Recommendation[GRU4Rec]

Since this model was originally designed for session-based recommendation which have SessionID,
ItemID and Time columns, to adapt it to the sequential recommendation I replaced the UserID
column name with SessionID, and added increasing integers for Time column, thus modifying the
datasets for GRU4Rec model. I also used a different data split, where all sessions are used for
training(containing all but the last item) and the last item prediction is used for testing.

19
Initializing with SASRec Embeddings[GRU4Rec+SR-emb]
Upon initializing the item embedding matrix with item embeddings from SASRec, improvement is
more pronounced in Beauty, Games and MovieLens-20M datasets.

BiGRU Training Process

Figure 4: BiGRU Training Process

1. Building vocabulary

First, the entire dataset is scanned to identify all the unique ItemIDs, then they are mapped
to unique integer IDs starting from 2 because 0 is reserved for [PAD] token and 1 is reserved for
[MASK] token. Each users item interaction is now a sequence of integer IDs, these integer sequences
are what the model will actually see.

2. Masked Language Model Objective

Before a batch of sequences is fed to the model, some tokens are masked using the masking strategy
that is used in BERT. If an item is selected to be masked, 80% of the time the item ID is replaced
with the [MASK] token’s ID, 10% of the time: The item ID is left unchanged and 10% of the time
the item ID is replaced with a different random item’s ID.
The code keeps a separate record of the original, true item IDs at the masked positions. These are
the labels the model will be graded against.
Since user sequences have different lengths, the shorter sequences in the batch are padded with the
[PAD] token’s ID (0) until all sequences have the same length.

20
3. BiGRU Model

Given a sequence of n items (Item A, Item B, [Mask], ... , Item D).

Forward GRU Pass: This process starts from the left. Item A is processed first, producing a
hidden state hf 1 . This state is then passed to the next GRU unit along with Item B to produce
hf 2 , and so on. It only contains information about the past.
Backward GRU Pass: This process starts from the right. The n’th item is processed first,
producing a hidden state (hbn ). This state is then passed to the previous GRU unit (moving left)
along with the (n-1)’th token to produce hb(n−1) . It only contains information about the future.
Concatenation: To get a complete understanding of the [MASK] token’s context, the model takes
the final state from the forward pass at that position hf 2 and the final state from the backward
pass at that same position hb2 and combines them. This concatenated vector, [hf 2 , hb2 ], now
understands the sequence both before and after the masked item.

Figure 5: BiGRU Model

4. Prediction Layer

The model then gathers the final Bi-GRU outputs only at the positions that were chosen to be
masked(including those that were left unchanged or replaced by a random token). This final,
context-rich vector is fed into a prediction layer to guess the original item.
These specific outputs are passed through a final transformation layer (Linear, GELU, LayerNorm)
to produce logits—a score for every possible item in the vocabulary.
The model’s task is to predict the original item ID at each of these masked positions, regardless of
whether it was replaced by [MASK], kept the same, or replaced by a random item.
Loss Calculation: The cross-entropy loss between the predicted logits and the true labels for the
masked items is computed.

21
5. Backpropagation and Optimization

The calculated loss is backpropagated through the network, and the optimizer updates the model’s
weights (embeddings, GRU weights, etc.) to improve its predictions.
Repeat: This process repeats for all batches in one epoch and for all subsequent epochs, allowing
the model to progressively learn the complex relationships between items within sessions.

22