Retrieval-Augmented Code Review Comment

Generation
Hyunsun Hong Jongmoon Baik
School of Computing School of Computing
KAIST KAIST
Daejeon, Republic of Korea Daejeon, Republic of Korea
[Link]@[Link] jbaik@[Link]

Abstract—Automated Code review comment generation (RCG) review-related activities, including reading, understanding, and
arXiv:2506.11591v1 [[Link]] 13 Jun 2025

aims to assist developers by automatically producing natural providing feedback on code changes [4]. Moreover, authors
language feedback for code changes. Existing approaches are often face delays of 15 to 64 hours before receiving review
primarily either generation-based, using pretrained language
models, or information retrieval-based (IR), reusing comments comments, which can hinder development velocity [5]. As
from similar past examples. While generation-based methods software systems grow, the volume of code review requests
leverage code-specific pretraining on large code–natural language increases significantly; for example, Microsoft Bing reportedly
corpora to learn semantic relationships between code and natural conducts approximately 3,000 code reviews per month [6].
language, they often struggle to generate low-frequency but se- These challenges have led researchers to explore automating
mantically important tokens due to their probabilistic nature. In
contrast, IR-based methods excel at recovering such rare tokens the review process. Review Comment Generation (RCG) has
by copying from existing examples but lack flexibility in adapting emerged as a promising approach, aiming to ease the burden of
to new code contexts—for example, when input code contains manual review comment writing by automatically generating
identifiers or structures not found in the retrieval database. To relevant, context-aware feedback for code changes.
bridge the gap between generation-based and IR-based methods, RCG has been researched in two directions: Generation-
this work proposes to leverage retrieval-augmented generation
(RAG) for RCG by conditioning pretrained language models on based RCG and Information-Retrieval (IR)-based RCG.
retrieved code–review exemplars. By providing relevant examples Generation-based RCG utilizes a generative language model.
that illustrate how similar code has been previously reviewed, the With the hypothesis that software exhibits repetitive and pre-
model is better guided to generate accurate review comments. dictable property [7], generation-based RCG takes advantage
Our evaluation on the Tufano et al. benchmark shows that RAG- of probabilistic language models trained on the large code-
based RCG outperforms both generation-based and IR-based
RCG. It achieves up to +1.67% higher exact match and +4.25% review corpus to generate most probable code review words.
higher BLEU scores compared to generation-based RCG. It also Tufano et al. [8], [9] were first to apply pre-trained language
improves the generation of low-frequency ground-truth tokens by model (PLM) to code review automation and Li et al. [10] later
up to 24.01%. We additionally find that performance improves proposed a code review-specific pre-training methodology.
as the number of retrieved exemplars increases. Both studies contributed benchmark datasets that have since
Index Terms—Code Review Automation, Deep Learning, Soft-
ware Quality Assurance, Retrieval-Augmented Generation become foundational for evaluating RCG methodology [9],
[10]. Based on these benchmarks, subsequent works have
focused on adapting advanced natural language processing
I. I NTRODUCTION
(NLP) techniques to enhance generation-based RCG [11]–
Code review is a systematic and collaborative process that [13]. IR-based RCG, on the other hand, utilizes information
involves examining code written by others to assess its quality retrieval. It finds the most probable code reviews from a
and readiness for integration. If a code change is deemed large code review history database and regards these retrieved
unsatisfactory, the reviewer may reject the merge request and exemplars as code reviews for the input code [14]–[16]. Rather
provide feedback in the form of review comments, enabling than generating new text from generative language model,
the author to revise the code accordingly. This process plays a IR-based RCG reuses exemplar mined from a code review
key role in identifying latent issues early in the development history database. It is more efficient than generation-based
cycle, preventing potential defects, and improving maintain- RCG because it does not require any training [14]. Notably,
ability through refactoring [1]. For this reason, code review is CommentFinder [14], an IR-based RCG approach, reports
widely adopted and strongly recommended practice as a core a 32% performance improvement over the generation-based
component of software quality assurance [2]. However, code method proposed by Tufano et al. [9] in RCG. This enhance-
review is inherently labor-intensive and time-consuming, re- ment is achieved with a 49× speedup over the generation-
quiring experienced developers to thoroughly inspect changes based method [9], by leveraging simple Bag-of-Words lexical
and deliver precise, constructive feedback [3]. Studies report similarity to retrieve similar code examples from the training
that developers spend an average of six hours per week on dataset. Although both generation-based RCG and IR-based
 produce LFGTs that are absent from the input. Conversely,
IR-based RCG can retrieve LFGTs from similar examples in
the retrieval corpus even when they are not present in the
input code (e.g., “IIRC” or “Java”), but it struggles to capture
identifier tokens that appear in the input but are rare in the
retrieval set. This observation motivates our investigation into
a hybrid approach that integrates both paradigms to leverage
their respective strengths.
To bridge the complementary gap between generation-based
and IR-based RCG, we propose to leverage Retrieval Aug-
mented Generation (RAG) [20] to RCG by augmenting input
code query with top-k most similar code’s review exemplars.
RAG enhances generative language model’s performance on
Fig. 1. A motivating example of generation-based and IR-based code review knowledge-intensive tasks by augmenting input queries with
comment generation. Words in ground truth were underlined. relevant knowledge passages [20]. By incorporating retrieved
exemplars, RAG enables generative language models to access
domain-specific knowledge and refer to LFGTs without requir-
RCG have been studied independently with its own advan- ing additional training [21]. Building on this principle, RAG
tages, their integration has not been explored. However, there has been effectively employed in diverse software engineering
is room for two methods to be integrated harmoniously. tasks, including code summarization [22], code completion
As generative language models innately tend to generate [23], code generation [24], and test case generation [25]. Given
high-frequency words, it has trouble with generating low- that code review requires extensive knowledge of project-
frequency but semantically important tokens—referred to as specific practices—such as coding conventions, review focus
Low-Frequency Ground-Truth Tokens (LFGTs) [17], [18]. areas, and stylistic guidelines [26]—RAG can be beneficial, as
Accurately generating LFGTs is critical for improving the retrieved exemplars help convey this domain-specific knowl-
overall quality and usefulness of generated review comments. edge to the generation model. However, the use of RAG
We hypothesize that incorporating retrieval candidates from for generating review comments has not been explored. To
IR-based RCG can expose the generative model to informative address this gap, we propose RAG-Reviewer: RAG-based
exemplars containing LFGTs, thereby guiding the language RCG framework that integrates IR-based and generation-based
model to generate contextually appropriate LFGTs. approaches.
Figure 1 shows a motivating example. This is an example The RAG-Reviewer framework comprises two primary
result of generation-based RCG (CodeT5 [19]) and IR-based components: a retrieval module and a generation module. The
(CommentFinder [14]) RCG on the Tufano et al.’s benchmark retrieval module retrieves top-k similar review exemplars that
dataset [9]. Given an input code snippet, the generation-based are most similar to the input code snippet from a large his-
approach produces review comments using its parametric torical code review database. The generation module utilizes
knowledge acquired during training. In contrast, the IR-based a PLM to generate a review comments for the input code
method identifies lexically similar code from the training snippet with the retrieved exemplars. Given an input code
dataset and directly reuses the associated review comment for review, the retrieval module first identifies the top-k most
as the output. The generation-based model tends to produce similar code examples and extracts their corresponding review
high-frequency tokens such as “it” (46,654 occurrences in the comments. These retrieved exemplars are then concatenated
training corpus), “should” (26,721), and “return” (142,643), with the input code and passed to the generation module. To
but it fails to generate LFGTs like “Also” (7,476), “IIRC” adapt the generator for this retrieval-augmented input format,
(123), “internal” (1,621), and “Java” (7,016). Although the we fine-tune the language model to produce review comments
model successfully generate “ReadOnlyArray” (1), a token that jointly reflect information from both the input and the
that appears only once in the training corpus, this is primarily retrieved examples. Empirical evaluations demonstrate that
due to its presence in the input code itself, rather than the RAG-Reviewer outperforms both generation-only and IR-only
model’s ability to generalize to rare vocabulary. Conversely, baselines. It achieves consistent improvements in Exact Match
the IR-based method is capable of retrieving review comments and BLEU scores across all PLMs and enhances the generation
that contain LFGTs such as “Also,” “IIRC,” “internal,” and of LFGTs. The key contributions of this work are as follows:
“Java” by leveraging similar examples in the retrieval set. • We propose RAG-Reviewer, a RAG framework for RCG
However, it fails to retrieve identifier-specific tokens like that unifies generation-based and IR-based approaches.
“ReadOnlyArray” when such tokens are rare in the retrieval To the best of our knowledge, this is the first study to
corpus. This example illustrates a complementary gap between apply the RAG paradigm to the RCG task.
generation-based RCG and IR-based RCG. While generation- • We conduct comprehensive experiments across multiple
based RCG can effectively generate LFGTs that are present PLMs, showing that RAG-Reviewer improves generation
in the input code (e.g., “ReadOnlyArray”), it struggles to accuracy, better captures LFGTs, and exhibits perfor-
 Fig. 3. An example of code review comment generation

Within this process, code review process involves three tasks:

code repair (code-to-code), review comment generation (code-
to-comment) and code refinement (code&comment-to-code) as
illustrated in Figure 2. The field of Code Review Automation
(CRA) aims to automate these three tasks [9] and we focus
on automated review comment generation in this work.
B. Code Review Comment Generation
Fig. 2. Code review process workflow
Code Review Comment Generation (RCG) automates the
generation of review comments in the code review process,
mance improvements as the number of retrieved exem- as shown in Figure 3. RCG assists code reviewers by au-
plars increases. tomatically generating review comments, which can serve
• To support reproducibility and encourage future research, as references or starting points for writing their own final
we publicly release our implementation on GitHub: feedback. Formally, given an input code sequence to be re-
[Link] viewed, Cinput = {c1 , c2 , . . . , cn }, and its corresponding code
review comments, NReview = {n1 , n2 , . . . , nm }, generation-
II. BACKGROUND based RCG is a sequence-to-sequence probabilistic mapping
function G : Cinput → NReview . In a probabilistic sequence
In this section, we provide a brief overview of code review modeling framework, the probability of generating NReview
process and code review comment generation. given Cinput is defined as:
A. Code Review Process m
Y
PG (NReview | Cinput ) = PG (ni | Cinput , n1 , . . . , ni−1 )
Code review is the one of the widely adapted software qual-
i=1
ity assurance practices in both open-source projects and indus-
tries [27], [28]. Through code review process, various aspects where PG denotes a probabilistic autoregressive model param-
of code quality (e.g., readability, maintainability, testability, eterized by G.
scalability, secure and more) can be inspected by collective III. A PPROACH
expertise of reviewers. Figure 2 shows code review process
scenario in debugging the faulty code. Suppose that an issue A. Overall Approach
occurred in main branch due to a bug. Then, (1) the author who In this work, we propose RAG-Reviewer, a RAG-based
is responsible for this issue repairs the buggy code and requests code Review comment generation framework. RAG-Reviewer
the code reviewer to pull repaired code into main branch. integrates the strengths of both IR-based and generation-based
Later, (2) the code reviewer evaluates the proposed code to methods. It consists of two core components: (a) Retrieval
determine whether the revised code is suitable for integration Module that retrieves review comments corresponding to code
into the main branch. If the code still contains defects, the snippets that are most similar to the given input code and (b)
reviewer rejects the pull request and provides feedback through Generator Module that produces review comments for the
a review comment. (3) The author subsequently revises the input code by conditioning on both the input and the retrieved
code based on the reviewer’s suggestions and resubmits the exemplars.
pull request. (4) Upon re-evaluation, if the changes address the By augmenting the input code with exemplars retrieved
identified issues, the reviewer approves the code for merging via the retrieval module, the generative language model can
into the main branch. Through this code review process, all leverage the outputs of the IR-based approach to enhance the
codes can be inspected by peer reviewers before it integrated to quality of generated review comments. Figure 4 provides an
the main codebase, thereby enhancing overall software quality. overview of RAG-Reviewer, which operates in two phases:
 Fig. 4. Overall framework of RAG-Reviewer

model training and model application. During the training where hi is the vectorized representation of ICi . We use
phase, RAG-Reviewer constructs a retrieval database from the the UniXcoder [29] as dc , because it can effectively pre-
training dataset, comprising code snippets, their corresponding serve code’s semantic and syntactic information. Especially,
review comments, and their encoded vector representations. UniXcoder showed state-of-the-art performance in IR-based
Each code snippet is encoded into a dense vector space to RCG [16]. Based on this, we construct the retrieval database
facilitate similarity computation using Maximum Inner Prod- Dreview as follows:
uct Search (MIPS). The retrieval module then identifies the
Dreview = {(hi , ICi , CRi )|i = 1, . . . , M } (2)
top-k most similar code snippets from this database using the
same encoding mechanism. To prevent data leakage, the top- To find the most similar exemplars corresponding to a given
1 retrieved exemplar—which is identical to the input training input code Cinput , we encode Cinput into hx using dc :
instance—is excluded during training. The generator is fine-
hx = dc (Cinput ) (3)
tuned on these augmented inputs, learning to condition on
both the input code and the retrieved exemplars. During model Note that we used same encoder dc for both Cinput and ICi
application, the same retrieval procedure is applied to obtain because they are both forms of code, making them unimodal.
relevant exemplars for a new input code snippet. Using the same encoder ensures consistency in the dense
vector representation. After encoding, we compute similarity
B. Retrieval Module
score si between Cinput and ICi s using the inner dot product:
The retrieval module is responsible for identifying the
most relevant exemplars from the retrieval database given si = ⟨hx , hi ⟩, ∀i ∈ {1, . . . , M } (4)
an input code snippet. These retrieved exemplars provide where ⟨·, ·⟩ denotes the inner product. Based on the computed
contextual guidance that the generator can leverage to produce similarity scores, we then select the top-k most similar exem-
more accurate and informative review comments. The retrieval plars with respect to the input code:
database is constructed from the training dataset. Formally, the 
M

training dataset is defined as a set Treview = {(ICi , CRi )|i = CR = TopK {si }i=1 , Dreview , K = {(IC1 , CR1 ) , . . . , (ICk , CRk )} (5)
1, . . . , M } where ICi denotes the input code snippet, CRi where TopK(S, D, K) returns top-k most similar code re-
represents its corresponding ground-truth review comment and view exemplars based on the similarity score set S, retrieval
M is the total number of training instances. This dataset serves database D and specified value of K. The resulting top-k
as the basis for both retrieval and generator training within the retrieval candidates CR are then used to augment the input
RAG-Reviewer framework. To enable similarity computation in the generation module.
between the input code and the code snippets in the retrieval
database, all input code instances ICi s must be encoded into C. Generation Module
dense vector representations. This vectorization allows for Given the output (Equation 5) of retrieval module, the list
the efficient measurement of semantic similarity—typically of top-k most similar exemplars with respect to the input code,
using cosine similarity—between code snippets. Formally, we we construct the input to the generator by concatenating the
employ an encoder dc : input code sequence X with the retrieved candidates CR :
hi = dc (ICi ), ∀(ICi , CRi ) ∈ Treview (1) X ′ = X ⊕ C1 ⊕ C2 · · · ⊕ Ck , Ci ∈ CR (6)
 TABLE I
F REQUENCY DISTRIBUTION OF REVIEW COMMENTS IN T UFANO ET AL .’ S TRAINING DATASET

Token Freq. (≤ x) ≤1 ≤5 ≤ 10 ≤ 20 ≤ 50 ≤ 100 All (≥ 1)

Unique Tokens Count 2,592 (10.08%) 8,827 (34.23%) 12,817 (49.84%) 16,558 (64.39%) 20,544 (79.88%) 22,507 (87.52%) 25,717 (100%)
# Review Comments Containing Tokens 2,486 (1.85%) 16,944 (12.62%) 32,768 (24.41%) 52,597 (39.18%) 80,620 (60.06%) 98,550 (73.41%) 134,239 (100%)

TABLE II
S TATISTICS OF T UFANO ET AL .’ S DATASET

Dataset Train Valid Test

Count 134,239 16,780 16,780
Avg. Code Length 167 168 171
Avg. Review Length 26 26 26 (a) Code length distribution (b) Review length distribution

Fig. 5. Token length distribution of Tufano et al.’s test dataset

where Ci denotes ith most similar code’s exemplar. Each

Ci consists of a code snippet ICi and its associated review (Equation 1) at every gradient step, as changes to dc would al-
comment CRi . Thus, we have to decide what to retrieve for ter hi , making recalculations computationally expensive [20].
each Ci . Following methodology of Parvez et al. [24], we ex-
plore two input augmentation strategies, pairs and singleton. IV. E XPERIMENT S ETUP
The pair setting incorporates both the exemplar code and its A. Research Question
corresponding review comment, whereas the singleton setting
leverages only the review comment: We formulate the following research questions to evaluate
the effectiveness of RAG-Reviewer:
• RQ1: Does RAG-Reviewer outperform existing baseline
Xp′ = X ⊕ CR1 ⊕ IC1 · · · , (ICi , CRi ) ∈ CR (7) methods in terms of review comment generation quality?
Xs′ = X ⊕ CR1 ⊕ CR2 · · · , (ICi , CRi ) ∈ CR (8) • RQ2: Does RAG-Reviewer mitigate the challenges faced
by language models in generating low-frequency tokens?
where ICi means ith most similar code and CRi means • RQ3: What is the impact of the number of retrieved
its corresponding review comment retrieved from database exemplars on the performance of RAG-Reviewer?
Dreview . To clearly delineate the boundaries between code
These research questions are empirically investigated in Sec-
and review comments within each exemplar, we introduce
tion V.
special delimiter tokens, [csep] and [nsep], following the input
formatting strategy proposed by Parvez et al. [24]. B. Dataset
Given the augmented input X ′ , RAG-Reviewer generate
the output sequence Y = (y1 , . . . , yN ) using a generative We use the dataset introduced by Tufano et al. [9]. which
language model under an autoregressive formulation: was constructed from large-scale Java open-source projects
on GitHub and Gerrit. Each code is function-level granularity
N
Y written in Java and paired with its corresponding review com-
′
Pθ (Y | X ) = Pθ (yi | X ′ , y1:i−1 ) (9) ment. Detailed statistics are shown in Table II, and Figure 5
i=1 illustrates the token length distribution in the test set. All
where Pθ is a language model parameterized by θ, which lengths were calculated using the CodeT5 [19] tokenizer. As
generates each token conditioned on the input X ′ and previous shown in Table I, 87.52% of unique tokens in the review
i − 1 tokens. We experiment with various types of pre- comment training corpus occur no more than 100 times,
trained language models as the generator and compare their and 73.41% of comments contain at least one such token.
performance within the RAG-Reviewer framework. Following prior work [22], [30], we define these as low-
To train the generator, RAG-Reviewer minimizes the fol- frequency tokens. This high prevalence makes the Tufano et
lowing negative log-likelihood loss function over the target al. dataset a suitable benchmark for evaluating our approach
sequence Y true = (y1true , . . . , yN
true
): to improving low-frequency token generation.

N
X C. Evaluation Metrics
L(θ) = − log Pθ (yitrue | X ′ , y1:i−1
true
) (10) Following the previous code review comment generation
i=1
work [9]–[11], [14], we evaluate the quality of generated
where θ denotes the trainable parameters of the generator. We review comments using the metric BLEU [31] and Exact
fine-tuned only the generator while keeping the code encoder Match (EM) score. These metrics are widely used in other
dc fixed because training dc would necessitate updating hi automated software engineering work [22]–[24], [30].
 BLEU measures the frequency of overlapped n-grams be- tagging and cross pre-training. It is trained on a curated
tween the generated comment and the reference comment dataset of over 10,000 real-world code change and review
using n-gram precision. We use BLEU-4, which considers up comment pairs from popular Java projects. AUGER incor-
to 4-gram matches. The BLEU-N score is computed as: porates review-specific tags and demonstrates significant
XN
! improvements in review comment quality.
BLEU-N = BP · exp wn log pn
E. Implementation Details
n=1
where pn is the n-gram precision, wn is the weight for each All experiments were conducted on a NVIDIA Tesla
n-gram (uniformly set to 1/N ), and BP is a brevity penalty V100-SXM2-32GB GPU. For CodeT5, CodeT5+ and
that penalizes overly short generated sequences. Higher BLEU CodeReviewer, we used the AdamW optimizer (learning
scores indicate greater similarity to the ground truth review rate = 3e-5, weight decay = 0.01) with a linear learning
comment, ranging from 0% to 100%. rate schedule and 10% warm-up steps. To address memory
EM measures the ratio of generated outputs that are exactly constraints, we applied gradient accumulation over 3 steps
identical to the ground truth review comment. It is a stricter with a batch size of 12 (effective batch size = 36). We used
metric than BLEU because it requires a character-wise match. gradient clipping with a maximum norm of 1.0 and enabled
The EM score also ranges from 0% to 100%, where 100% mixed-precision training. The input and output lengths were
indicates that all the generated outputs for the test dataset are capped at 512 tokens for code and 128 tokens for review com-
character-wise identical to the ground truth. ments, based on the dataset’s token distribution (see Table II
and Figure 5). Longer input sequences were truncated to fit
D. Baselines within token limits. For RAG-based models, input prompts
To evaluate the effectiveness of RAG-Reviewer, we com- were augmented with as many retrieved exemplars as would
pare it against a range of baselines, including both generation- fit within the input token budget. We used a beam size of 10 for
based and IR-based methods. We reproduced all the baselines. evaluation. Training was conducted for up to 20 epochs, with
• CommentFinder: A first IR-based RCG approach that early stopping applied after 3 epochs of no improvement on the
uses Bag-of-Words (sparse vector) for code vectorization validation set. The best model checkpoint was selected based
and cosine similarity for candidate ranking [14]. on validation performance. For Tufano T5 and AUGER, we
• UniXCoder-IR: Proposed by Kartal et al. [16], this used the Adam optimizer (learning rate = 3e-4) with a batch
method uses UniXCoder (dense vector) [29] for code size of 12 and no gradient accumulation, based on preliminary
vectorization. Retrieval is based on similarity metrics tuning results. All baseline models were implemented using
such as cosine or Euclidean distance. publicly available replication packages [36]–[42].
• Tufano T5: A small version of the T5 [32] architecture
(6-layer encoder and 6-layer decoder) pre-trained for code V. E XPERIMENTAL R ESULT
review automation. It was pre-trained on code and natural In this section, we present experimental result of our work
language pairs collected from Stack Overflow and Code- following three research questions.
SearchNet [33] using masked language modeling [9].
• CodeT5: A pre-trained encoder-decoder model that sup-
A. (RQ1) Does RAG-Reviewer outperform existing baseline
ports both code understanding and generation. Based methods in terms of review comment generation quality?
on the T5 [32] architecture, CodeT5 [19] introduces Singleton vs. Pair Retrieval Strategy. Before comparing
identifier-aware objectives and bimodal dual generation RAG-Reviewer with existing baselines, we first evaluate which
tasks. It was pre-trained on CodeSearchNet [33] and ad- retrieval strategy achieves superior performance within RAG-
ditional GitHub repositories. We use the CodeT5-base Reviewer. Following Parvez et al. [24], we consider two strate-
variant, which has 12 encoder and 12 decoder layers. gies: (1) Singleton Retrieval, which retrieves a larger number
• CodeT5+: An extended version of CodeT5 that incorpo- of review comments without their associated code snippets
rates a mixture of pre-training objectives including span and (2) Pair Retrieval, which retrieves fewer exemplars but
denoising, contrastive learning, text–code matching, and preserves both the code snippets and their corresponding
causal language modeling. CodeT5+ [34] is trained on review comments.
both unimodal code and bimodal code–text corpora. We As shown in Table III, the pair retrieval strategy consistently
use the CodeT5p-220m version. outperforms singleton retrieval in terms of both EM and BLEU
• CodeReviewer: Specifically pre-trained for code review scores across all PLMs. The EM gains range from +0.14%
automation tasks, CodeReviewer [10] leverages a mul- (AUGER: 2.32% → 2.46%) to +0.53% (Tufano T5: 2.01%
tilingual dataset of code diffs and review comments → 2.54%). The BLEU gains range from +0.53% (CodeT5:
from GitHub. It introduces four novel pre-training tasks 12.45% → 12.98%) to +1.76% (CodeReviewer: 11.76% →
tailored for code review, such as diff tag prediction and 13.52%). These results suggest that providing both code and its
denoising objectives for both code and review comments. review comment in retrieved exemplars helps the model better
• AUGER: AUGER [35] is a T5-based [32] model that learn code–comment relationships. Given the inherent input
enhances review comment generation using review-line token limitations of language models, it is crucial to include
 TABLE III TABLE IV
P ERFORMANCE COMPARISON OF RAG-R EVIEWER WITH T HE NUMBER OF CORRECTLY GENERATED LOW- FREQUENCY REVIEW
GENERATION - BASED AND IR- BASED BASELINES TOKENS

Name EM (%) BLEU (%) Token Freq. (≤ x) ≤ 20 ≤ 40 ≤ 60 ≤ 80 ≤ 100

Information Retrieval Vanilla Tufano T5 1102 1891 2450 2866 3238

CommentFinder 2.80 12.41 Pair Tufano T5 1342 2282 3030 3508 3917
% Improvement 21.78% 20.68% 23.67% 22.40% 20.97%
UniXCoder-IR 2.79 12.80
Vanilla CodeReviewer 867 1525 2007 2405 2703
Generation Pair CodeReviewer 1043 1869 2473 2936 3352
Tufano T5 0.87 9.32 % Improvement 20.30% 22.56% 23.22% 22.08% 24.01%
CodeReviewer 1.23 9.27
Vanilla CodeT5 975 1770 2294 2711 3080
CodeT5 1.39 9.91 Pair CodeT5 1018 1835 2429 2876 3247
CodeT5+ 1.71 11.59 % Improvement 4.41% 3.67% 5.88% 6.09% 5.42%
AUGER 1.10 9.78
Vanilla CodeT5+ 1142 1956 2557 3089 3489
Retrieval Augmented Generation (Singleton) Pair CodeT5+ 1229 2185 2909 3426 3856
Singleton Tufano T5 2.01 11.82 % Improvement 7.62% 11.71% 13.77% 10.91% 10.52%
Singleton CodeReviewer 2.32 11.76 Vanilla AUGER 807 1583 2232 2708 3073
Singleton CodeT5 2.40 12.45 Pair AUGER 879 1751 2509 3043 3472
Singleton CodeT5+ 2.53 11.97 % Improvement 8.92% 10.61% 12.41% 12.37% 12.98%
Singleton AUGER 2.32 11.81
Retrieval Augmented Generation (Pair)
Pair Tufano T5 2.54 12.54 semantic context that IR methods may overlook. Moreover, as
Pair CodeReviewer 2.76 13.52
Pair CodeT5 2.90 12.98 shown in Table III, RAG-Reviewer’s performance generally
Pair CodeT5+ 3.01 12.39 improves with stronger PLMs, suggesting that its effectiveness
Pair AUGER 2.46 12.93 scales with the capacity of the underlying generator.

Answering RQ1
helpful information—both in quantity and quality—within Pair retrieval consistently outperforms singleton re-
the available budget. Our experiments show that rather than trieval in both EM and BLEU, and RAG-Reviewer
allocating tokens to more review-only exemplars, it is more achieves higher performance than generation-based
effective to include both the review comments and their and IR-based methods.
corresponding code. Notably, since code snippets are typically
much longer than comments (see Table II), pair retrieval fits
much fewer exemplars within the token limit. Nevertheless, B. (RQ2) Does RAG-Reviewer mitigate the challenges faced
providing richer contextual information through paired code- by language models in generating low-frequency tokens?
comment exemplars yields better performance than using a Effect on Generating Low-frequency Tokens. A well-
larger number of comment-only examples. known challenge for language models in generation-based
Generation-based vs. RAG-Reviewer. RAG-Reviewer methods is their difficulty in generating low-frequency ground-
consistently outperforms generation-based models across all truth tokens (LFGTs), due to a bias toward high-frequency
PLMs in both EM and BLEU scores. Notably, even models patterns learned during training [17], [18]. In contrast, IR-
with modest baseline performance, such as Tufano T5 and based approaches can help expose such tokens by retrieving
CodeReviewer, show the most significant gains—Tufano T5 similar examples that contain them. While we have observed
improves from 0.87% to 2.54% in EM (+1.67%), while that RAG-Reviewer improves generation over standard base-
CodeReviewer improves from 9.27% to 13.52% in BLEU lines in RQ1, we further examine whether it specifically helps
(+4.25%). These results suggest that retrieval augmentation mitigate this limitation by evaluating its ability to generate
benefits not only strong backbones but also weaker ones, LFGTs compared to generation-based methods. We define
enhancing both precision and fluency by grounding generation LFGTs as tokens that appear fewer than 100 times in the
in relevant contextual exemplars. training review comment corpus following previous work [22],
IR-based vs. RAG-Reviewer. Compared to IR-based meth- [30]. For each generated review comment, we count a token
ods, RAG-Reviewer achieves comparable or slightly higher as correctly generated if it appears in both the output and the
performance in most cases. For example, Pair CodeT5 ground truth. We then group these correctly generated tokens
(2.90%) and Pair CodeT5+ (3.01%) slightly outperform Com- by their frequency in the training set using thresholds of ≤20,
mentFinder (2.80%) and UniXCoder-IR (2.79%) in EM. ≤40, ≤60, ≤80, and ≤100. The improvement is measured as
BLEU scores show modest gains as well, with Pair CodeRe- the relative increase over the generation-based baseline.
viewer reaching 13.52% versus 12.41% and 12.80% for the IR As shown in Table IV, RAG-Reviewer improves the gen-
baselines. While the margins are small, RAG-Reviewer offers eration of LFGTs across all evaluated models. Tufano T5 ex-
the added benefit of generating comments conditioned on both hibits the most consistent relative gains, ranging from 20.68%
the input code and retrieved exemplars, allowing it to leverage to 23.67%. And CodeReviewer shows similar improvements
Fig. 6. Exact match performance of Pair CodeT5 with varying numbers of
retrieved exemplars on Tufano et al. benchmark

(ranging from 20.30% to 24.01%), while AUGER demon-

strates increases between 8.92% and 12.98%. CodeT5+ shows
improvements of 7.62% to 13.77%, and CodeT5 achieves more
modest gains between 3.67% and 6.09%. These results suggest Fig. 7. BLEU-4 scores across different code and review comment lengths
that retrieval augmentation can help generation models recover
rarer tokens by grounding generation in retrieved exemplars.
While the level of improvement varies by model, the overall mation can be included, thereby reducing the benefit of adding
result indicates that RAG-Reviewer helps language model more examples.
generate LFGTs more effectively.
Answering RQ3
Answering RQ2
Including more retrieved exemplars in the input con-
RAG-Reviewer enhances the generation of low- sistently improves RAG-Reviewer’s generation qual-
frequency tokens across all models, alleviating innate ity. However, performance gains diminish with higher
limitation of generation-based methods. counts due to input length constraints of the language
model.

C. (RQ3) What is the impact of the number of retrieved

exemplars on the performance of RAG-Reviewer? VI. D ISCUSSION
This discussion covers three parts: how RAG-Reviewer
Impact of Retrieval Quantity on Generation Quality.
performs with different code and review comment lengths,
While RQ1 showed that exemplar quality (Pair) is more
findings from manual analysis, and directions for future work.
important than quantity (Singleton), we further investigate
whether increasing the number of high-quality exemplars still A. Performance with Different Length
improves performance. To explore this, we evaluated the Pair To evaluate how RAG-Reviewer performs across varying
CodeT5 model using retrieval sizes ranging from 0 (vanilla) input and output lengths, we compare BLEU-4 scores of
to 8. This setup was chosen based on findings from RQ1: pair Pair CodeReviewer and Pair CodeT5 with their vanilla coun-
retrieval consistently outperformed singleton retrieval across terparts. Test samples are bucketed by token length—20-
all backbones, and CodeT5 showed mid-range performance token intervals for code and 10-token intervals for review
among the RAG-Reviewer variants—making it a representa- comments—and average BLEU-4 is computed per bucket. As
tive case for analysis. shown in Figure 7, both RAG models consistently outperform
As shown in Figure 6, exact match (EM) improves steadily their vanilla versions across all length intervals. Although
from 1.39% with no retrieval (vanilla CodeT5) to 2.90% with BLEU-4 scores decrease with longer sequences, the RAG
eight exemplars. The biggest jump occurs when using just one models maintain relatively higher performance.
exemplar (+1.17%), suggesting that even a small amount of
retrieval helps the model generate more accurate comments. B. Manual Analysis
Additional exemplars continue to improve performance, but While RQ1 showed that RAG-Reviewer outperforms both
the gains become smaller. This is likely due to the model’s IR-based and generation-based methods in terms of EM and
input token limit, which restricts how much exemplar infor- BLUE, these metrics does not fully reflect the quality of
 TABLE V
M ANUAL ANALYSIS OF REVIEW COMMENTS ON 100 INPUT CODE FROM
THE T UFANO ET AL . MANUAL ANALYSIS SAMPLES

Category RAG-Reviewer CommentFinder Tufano T5

Exact Match 2 4 0
Semantically Equivalent 39 30 36
Alternative Solution 9 9 10
Incorrect 50 57 54

TABLE VI
R ETRIEVED EXEMPLARS USED FOR INPUT AUGMENTATION IN F IGURE 8
(K EY LFGT ‘ TRY- WITH - RESOURCES ‘ UNDERLINED )

Top-K Retrieved Review Comment

1 STDERR as exit() follows
10 Should we use try-with-resources here to close it later? Test only so I think it may not matter much.
16 This should be part of the try-with-resources block.
20 The try-with-resources takes care of the closing, right?

generated review comments. To evaluate outputs beyond these

metrics, we conducted a manual analysis of 100 review com- Fig. 8. Example of a Semantically Equivalent review comment from the
ments generated by the best-performing RAG-Reviewer (Pair Tufano et al. manual analysis samples (Key LFGT ‘try-with-resources‘
underlined)
CodeT5+), using the same 100 samples manually analyzed
in Tufano et al. [9], following the human evaluation method
of prior work [9], [14]. Each generated review comment was
classified into one of four categories based on the labeling
scheme introduced in Tufano et al. [9]: (a) Exact Match –
the generated review comment is character-wise identical to
the ground-truth comment; (b) Semantically Equivalent – the
generated review comment differs in wording but conveys the
same meaning as the ground truth’s intention; (c) Alternative
Solution – the generated review comment is neither identical
nor semantically equivalent but still provides a meaningful
and useful suggestion; (d) Incorrect – the generated review
comment is irrelevant and unhelpful for the given input code.
Table V summarizes the manual classification results.
While RAG-Reviewer made fewer exact matches than Com-
mentFinder, it produced more semantically equivalent com-
ments (39 vs. 30) and fewer incorrect ones (50 vs. 57), indi-
cating stronger relevance even when not perfectly matching
the ground truth. Compared to Tufano T5, RAG-Reviewer
yielded slightly fewer alternative solutions (9 vs. 10) but
more semantically equivalent comments (39 vs. 36) and fewer
incorrect outputs (50 vs. 54). Figures 8 and 9 show concrete
Fig. 9. Example of a Alternative Solution review comment from the Tufano
examples. In Figure 8, RAG-Reviewer generates a semanti- et al. manual analysis samples
cally correct suggestion regarding try-with-resources,
unlike the unrelated outputs from Tufano T5 and Com-
mentFinder. Notably, RAG-Reviewer successfully generates
the key LFGT try-with-resources (279 occurrences C. Future Work
in the training corpus), which neither Tufano T5 nor Com-
mentFinder produced. Although the input code does not con- A key direction for future work is jointly training the
tain this token, RAG-Reviewer can infer it from retrieved retriever and generator. While this study fine-tuned only the
exemplars containing relevant context. As shown in Table VI, generator, prior work [21] showed that training both compo-
several retrieved examples (e.g., top-10, 16, and 20) include nents can improve retrieval quality and overall generation.
try-with-resources, allowing the model to refer to low- Another direction is integrating RAG with large language
frequency but semantically important tokens. This demon- models (LLMs). Although recent studies [12], [13] focused
strates how retrieval augmentation provides contextual cues on efficient fine-tuning LLMs for RCG, combining them with
that help the model generalize beyond the input. RAG may further enhance their performance.
 VII. R ELATED W ORK retrieved code snippets are returned as the predicted output.
In this section, we review related work on Code Review CommentFinder achieved a 32% improvement in exact match
Automation (CRA) from two main perspectives: generation- accuracy and was 49× faster than a generation-based baseline
based and information retrieval-based (IR-based) approaches. [9]. Building on this, Shuvo et al. [15] incorporated structural
We also discuss prior applications of retrieval-augmented features such as class and library names to improve retrieval
generation (RAG) in software engineering. quality. More recently, Kartal et al. [16] demonstrated that
dense representations from transformer-based models (e.g.,
A. Generation-based Code Review Automation UniXCoder [29]) outperform sparse representation (e.g., BoW)
With the advancement of pretrained language models in CRA retrieval tasks.
(PLMs) [43]–[47], there has been growing interest in applying
them to various software engineering tasks [19], [29], [48], C. Retrieval-Augmented Generation in Software Engineering
[49], including CRA. Tufano et al. [8], [9] were among the RAG combines the strengths of retrieval- and generation-
first to demonstrate the use of PLMs for CRA by curating a based methods by incorporating retrieved exemplars into the
dataset of code–review comment pairs and framing the task as input of a generative model. In software engineering, RAG
a sequence-to-sequence translation problem. Building on this, has shown promise across several tasks, such as code summa-
Li et al. [10] proposed specialized pretraining tasks designed rization [21], [22], [24], [30], [52], code completion [23], and
to better capture the relationship between code changes (diffs) automated program repair [53].
and corresponding review comments, further improving per-
Zhang et al. [22] were among the first to apply RAG to code
formance on CRA.
summarization, using LSTM-based models. Parvez et al. [24]
Recent works have continued to explore the use of PLMs
extended this idea by using transformer-based PLMs as both
on standard CRA benchmarks. For example, Zhou et al. [11]
retriever and generator. Lu et al. [23] applied RAG to code
conducted a comparative study and found that CodeT5 [19]
completion by retrieving similar completed code examples to
consistently outperformed other code-specific PLMs. Beyond
guide the model. Jin et al. [53] introduced RAG into LLM-
PLMs, LLMs have also been adapted to CRA. Lu et al. [12]
based program repair by augmenting prompts with retrieved
fine-tuned LLaMA [50] using Parameter-Efficient Fine-Tuning
bug–fix pairs to improve repair quality.
(PEFT) techniques and demonstrated competitive performance
on the benchmark from Li et al. [10], though they reported
VIII. T HREATS TO VALIDITY
that Tufano T5 [9] still outperformed LLaMA on the Tufano
benchmark. Based on this finding, we opted to use PLMs Internal validity. We reimplemented Tufano T5 in PyTorch
rather than LLMs in our study, as they align better with the due to compatibility issues with its original TensorFlow 2.6.0
Tufano benchmark. code. Pretrained weights were converted accordingly, and the
Expanding on Lu et al., Nashaat et al. [13] introduced a training setup followed prior RCG work [10], [12]. While care
framework that incorporates organizational data with few-shot was taken to align with the original setup, minor discrepancies
learning and reinforcement learning from human feedback. may exist. Hyperparameters were selected based on prior work
More recently, Li et al. [51] proposed CodeDoctor, a category- and GPU memory limits but may not be optimal.
aware review comment generator that utilizes LLM-guided External validity. Our evaluation is based solely on the Java
classification and a category-specific decoder. While Code- dataset from Tufano et al. [9]. To improve generalizability,
Doctor showed strong performance in multi-category review future work should evaluate across more languages and diverse
comment generation, it assumes predefined categories are software projects.
available and constrains output accordingly. In contrast, our
work focuses on general (non-categorical) review comment IX. C ONCLUSION
generation. This direction remains important for building fully
automated and flexible CRA systems, as real-world code In this paper, we presented RAG-Reviewer, a RAG-based
reviews often include issues that span or fall outside fixed code Review comment generation framework. RAG-Reviewer
categories. conditions pretrained language models on both input code
and retrieved code–review exemplars to improve generation
B. Information Retrieval-based Code Review Automation quality. We evaluated the approach on the Tufano et al.
While generation-based CRA shows strong performance, benchmark and observed consistent gains over generation-
it often incurs high computational cost during training and based and IR-based baselines, particularly in generating low-
inference. IR-based CRA offers a more efficient alternative by frequency tokens and handling varying input lengths. Our
bypassing language generation and instead reusing comments analysis further reveals that using fewer but more informative
from similar examples in a retrieval corpus. exemplars—including relevant code—outperforms comment-
Hong et al. [14] proposed CommentFinder, an IR-based only augmentation, and that increasing the number of exem-
CRA system that uses sparse Bag-of-Words (BoW) vector plars within the same retrieval strategy leads to additional
representations to retrieve top-k lexically similar code snippets improvements. These results demonstrate the effectiveness of
based on cosine similarity. The paired review comments from retrieval augmentation in enhancing code review automation.
 R EFERENCES [21] H. Lu and Z. Liu, “Improving retrieval-augmented code comment
generation by retrieving for generation,” in 2024 IEEE International
[1] S. McIntosh, Y. Kamei, B. Adams, and A. E. Hassan, “An empirical Conference on Software Maintenance and Evolution (ICSME). IEEE,
study of the impact of modern code review practices on software 2024, pp. 350–362.
quality,” Empirical Software Engineering, vol. 21, pp. 2146–2189, 2016. [22] J. Zhang, X. Wang, H. Zhang, H. Sun, and X. Liu, “Retrieval-based
[2] E. Fregnan, F. Petrulio, L. Di Geronimo, and A. Bacchelli, “What hap- neural source code summarization,” in Proceedings of the ACM/IEEE
pens in my code reviews? an investigation on automatically classifying 42nd International Conference on Software Engineering, 2020, pp.
review changes,” Empirical Software Engineering, vol. 27, no. 4, p. 89, 1385–1397.
2022. [23] S. Lu, N. Duan, H. Han, D. Guo, S.-w. Hwang, and A. Svyatkovskiy,
[3] Y. Huang, X. Liang, Z. Chen, N. Jia, X. Luo, X. Chen, Z. Zheng, and “Reacc: A retrieval-augmented code completion framework,” arXiv
X. Zhou, “Reviewing rounds prediction for code patches,” Empirical preprint arXiv:2203.07722, 2022.
Software Engineering, vol. 27, pp. 1–40, 2022. [24] M. R. Parvez, W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W.
[4] A. Bosu and J. C. Carver, “Impact of peer code review on peer impres- Chang, “Retrieval augmented code generation and summarization,”
sion formation: A survey,” in 2013 ACM/IEEE International Symposium arXiv preprint arXiv:2108.11601, 2021.
on Empirical Software Engineering and Measurement. IEEE, 2013, pp. [25] J. Shin, R. Aleithan, H. Hemmati, and S. Wang, “Retrieval-augmented
133–142. test generation: How far are we?” arXiv preprint arXiv:2409.12682,
[5] Y. Hong, C. K. Tantithamthavorn, and P. P. Thongtanunam, “Where 2024.
should i look at? recommending lines that reviewers should pay atten- [26] X. Han, A. Tahir, P. Liang, S. Counsell, and Y. Luo, “Understanding
tion to,” in 2022 IEEE international conference on software analysis, code smell detection via code review: A study of the openstack com-
evolution and reengineering (SANER). IEEE, 2022, pp. 1034–1045. munity,” in 2021 IEEE/ACM 29th International Conference on Program
[6] P. C. Rigby and C. Bird, “Convergent contemporary software peer review Comprehension (ICPC). IEEE, 2021, pp. 323–334.
practices,” in Proceedings of the 2013 9th joint meeting on foundations [27] A. Alami, M. L. Cohn, and A. Wasowski,˛ “Why does code review
of software engineering, 2013, pp. 202–212. work for open source software communities?” in 2019 IEEE/ACM 41st
[7] A. Hindle, E. T. Barr, M. Gabel, Z. Su, and P. Devanbu, “On the International Conference on Software Engineering (ICSE). IEEE, 2019,
naturalness of software,” Communications of the ACM, vol. 59, no. 5, pp. 1073–1083.
pp. 122–131, 2016. [28] C. Sadowski, E. Söderberg, L. Church, M. Sipko, and A. Bacchelli,
[8] R. Tufano, L. Pascarella, M. Tufano, D. Poshyvanyk, and G. Bavota, “Modern code review: a case study at google,” in Proceedings of the 40th
“Towards automating code review activities,” in 2021 IEEE/ACM 43rd international conference on software engineering: Software engineering
International Conference on Software Engineering (ICSE). IEEE, 2021, in practice, 2018, pp. 181–190.
pp. 163–174. [29] D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, “Unixcoder:
[9] R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, Unified cross-modal pre-training for code representation,” arXiv preprint
and G. Bavota, “Using pre-trained models to boost code review automa- arXiv:2203.03850, 2022.
tion,” in Proceedings of the 44th international conference on software [30] B. Wei, Y. Li, G. Li, X. Xia, and Z. Jin, “Retrieve and refine:
engineering, 2022, pp. 2291–2302. exemplar-based neural comment generation,” in Proceedings of the 35th
[10] Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, IEEE/ACM International Conference on Automated Software Engineer-
J. Green, A. Svyatkovskiy, S. Fu et al., “Codereviewer: Pre-training for ing, 2020, pp. 349–360.
automating code review activities,” arXiv preprint arXiv:2203.09095, [31] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for
2022. automatic evaluation of machine translation,” in Proceedings of the 40th
[11] X. Zhou, K. Kim, B. Xu, D. Han, J. He, and D. Lo, “Generation-based annual meeting of the Association for Computational Linguistics, 2002,
code review automation: How far are we?” in 2023 IEEE/ACM 31st pp. 311–318.
International Conference on Program Comprehension (ICPC). IEEE, [32] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena,
2023, pp. 215–226. Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning
[12] J. Lu, L. Yu, X. Li, L. Yang, and C. Zuo, “Llama-reviewer: Advancing with a unified text-to-text transformer,” Journal of machine learning
code review automation with large language models through parameter- research, vol. 21, no. 140, pp. 1–67, 2020.
efficient fine-tuning,” in 2023 IEEE 34th International Symposium on [33] H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt,
Software Reliability Engineering (ISSRE). IEEE, 2023, pp. 647–658. “Codesearchnet challenge: Evaluating the state of semantic code search,”
[13] M. Nashaat and J. Miller, “Towards efficient fine-tuning of language arXiv preprint arXiv:1909.09436, 2019.
models with organizational data for automated software review,” IEEE [34] Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi,
Transactions on Software Engineering, 2024. “Codet5+: Open code large language models for code understanding
[14] Y. Hong, C. Tantithamthavorn, P. Thongtanunam, and A. Aleti, “Com- and generation,” arXiv preprint arXiv:2305.07922, 2023.
mentfinder: a simpler, faster, more accurate code review comments [35] L. Li, L. Yang, H. Jiang, J. Yan, T. Luo, Z. Hua, G. Liang, and C. Zuo,
recommendation,” in Proceedings of the 30th ACM joint European “Auger: automatically generating review comments with pre-training
software engineering conference and symposium on the foundations of models,” in Proceedings of the 30th ACM Joint European Software
software engineering, 2022, pp. 507–519. Engineering Conference and Symposium on the Foundations of Software
[15] O. Shuvo, P. Mahbub, and M. M. Rahman, “Recommending code Engineering, 2022, pp. 1009–1021.
reviews leveraging code changes with structured information retrieval,” [36] R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk,
in 2023 IEEE International Conference on Software Maintenance and and G. Bavota, “Tufano t5 replication package,” [Link]
Evolution (ICSME). IEEE, 2023, pp. 194–206. RosaliaTufano/code_review_automation, 2021, accessed: 2025-05-24.
[16] Y. Kartal, E. K. Akdeniz, and K. Özkan, “Automating modern code [37] Y. Wang, W. Wang, S. Joty, and S. C. H. Hoi, “Codet5-base repli-
review processes with code similarity measurement,” Information and cation package,” [Link] 2021,
Software Technology, vol. 173, p. 107490, 2024. accessed: 2025-05-24.
[17] P. Arthur, G. Neubig, and S. Nakamura, “Incorporating discrete [38] Y. Wang, H. Le, A. D. Gotmare, N. D. Q. Bui, J. Li, and S. C. H. Hoi,
translation lexicons into neural machine translation,” arXiv preprint “Codet5p-220m replication package,” [Link]
arXiv:1606.02006, 2016. codet5p-220m, 2023, accessed: 2025-05-24.
[18] J. Zhang, M. Utiyama, E. Sumita, G. Neubig, and S. Nakamura, [39] Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder,
“Guiding neural machine translation with retrieved translation pieces,” J. Green, A. Svyatkovskiy, S. Fu et al., “Codereviewer replica-
arXiv preprint arXiv:1804.02559, 2018. tion package,” [Link] 2022, ac-
[19] Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware cessed: 2025-05-24.
unified pre-trained encoder-decoder models for code understanding and [40] L. Li, L. Yang, H. Jiang, J. Yan, T. Luo, Z. Hua, G. Liang, and C. Zuo,
generation,” arXiv preprint arXiv:2109.00859, 2021. “Auger replication package,” [Link]
[20] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, auger-fse-2022, 2022, accessed: 2025-05-24.
H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., “Retrieval- [41] Y. Hong, C. Tantithamthavorn, P. Thongtanunam, and A. Aleti, “Com-
augmented generation for knowledge-intensive nlp tasks,” Advances in mentfinder replication package,” [Link]
neural information processing systems, vol. 33, pp. 9459–9474, 2020. CommentFinder, 2022, accessed: 2025-05-24.
[42] D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin,
“Unixcoder replication package,” [Link]
unixcoder-base-nine, 2021, accessed: 2025-05-24.
[43] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning
with neural networks,” Advances in neural information processing
systems, vol. 27, 2014.
[44] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by
jointly learning to align and translate,” arXiv preprint arXiv:1409.0473,
2014.
[45] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in
neural information processing systems, vol. 30, 2017.
[46] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training
of deep bidirectional transformers for language understanding,” in Pro-
ceedings of the 2019 conference of the North American chapter of the
association for computational linguistics: human language technologies,
volume 1 (long and short papers), 2019, pp. 4171–4186.
[47] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal,
A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod-
els are few-shot learners,” Advances in neural information processing
systems, vol. 33, pp. 1877–1901, 2020.
[48] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin,
T. Liu, D. Jiang et al., “Codebert: A pre-trained model for programming
and natural languages,” arXiv preprint arXiv:2002.08155, 2020.
[49] D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan,
A. Svyatkovskiy, S. Fu et al., “Graphcodebert: Pre-training code repre-
sentations with data flow,” arXiv preprint arXiv:2009.08366, 2020.
[50] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux,
T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al.,
“Llama: Open and efficient foundation language models,” arXiv preprint
arXiv:2302.13971, 2023.
[51] Y. Li, Y. Wu, Z. Wang, L. Huang, J. Wang, J. Li, and M. Huang,
“Codedoctor: multi-category code review comment generation,” Auto-
mated Software Engineering, vol. 32, no. 1, p. 25, 2025.
[52] J. A. Li, Y. Li, G. Li, X. Hu, X. Xia, and Z. Jin, “Editsum: A
retrieve-and-edit framework for source code summarization,” in 2021
36th IEEE/ACM International Conference on Automated Software En-
gineering (ASE). IEEE, 2021, pp. 155–166.
[53] M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, and
A. Svyatkovskiy, “Inferfix: End-to-end program repair with llms,” in
Proceedings of the 31st ACM Joint European Software Engineering
Conference and Symposium on the Foundations of Software Engineering,
2023, pp. 1646–1656.