Defects4Ruby: Benchmarking and Analyzing Bug

Detection and Repair for Ruby Using Language

Models
Meghdad Dehghan* Mohammadreza Saeidi* Rohit Dandamudi*
University of British Columbia University of British Columbia University of British Columbia
Kelowna, BC, Canada Kelowna, BC, Canada Kelowna, BC, Canada
[Link]@[Link] [Link]@[Link] [Link]@[Link]

Jie JW Wu Fatemeh H. Fard Gema Rodrı́guez-Pérez

University of British Columbia University of British Columbia University of British Columbia
Kelowna, BC, Canada Kelowna, BC, Canada Kelowna, BC, Canada
[Link]@[Link] [Link]@[Link] [Link]@[Link]

Abstract—Effective methods for detecting and repairing soft- capabilities [4], have shown promise in various programming-
ware bugs are essential to ensuring the stability and security of related tasks, including code generation [5, 6], code summa-
software systems. While Pre-trained Language Models (PLMs) rization [7], bug detection [8], and Automated Program Repair
and Large Language Models (LLMs) are used for bug detection
and Automated Program Repair (APR), the literature mainly (APR) [9]. Despite impressive results in tasks such as code
studied a few programming languages, leaving behind low- generation, the success rate of PLMs in complex programming
resource languages such as Ruby. Low-resource programming tasks like bug detection and APR remains relatively low
languages are languages with limited training data available, [4]. Little is known about their performance for low-resource
however, they can be widely used for several applications and programming languages, such as Ruby, where limited training
have a large community of developers. In this study, we provide
the first insights for bug detection and automatic program repair samples are available [10, 11].
for Ruby language using PLMs and LLMs. In addition to Ruby is a widely used programming language with appli-
reporting results, we contribute to the research field by open- cations ranging from web development to game design and
sourcing our collected dataset, named Defects4Ruby, to study is utilized by companies like Airbnb, Shopify, and GitHub
these tasks in Ruby. Our results on Defects4Ruby show that [12, 13]. The Rails framework, one of Ruby’s most popular
PLMs and LLMs underperform in bug detection compared to
traditional machine learning models; while PLMs outperform tools, is extensively adopted for web server development.
LLMs for APR. Both bug detection and APR results indicate a With over two million projects hosted on GitHub and a large
need to develop new techniques to detect and fix bugs in Ruby. community of developers, Ruby is among the top 10 program-
Index Terms—Bug Detection, Automated Program Repair, ming languages used on GitHub [14]. Despite its popularity,
Language Models, Mining Software Repositories, Ruby. automated techniques such as bug detection and APR remain
underexplored for Ruby, particularly with the advent of PLMs
I. I NTRODUCTION and Large Language Models (LLMs). While research exists
In the rapidly evolving field of software development, for high-resource languages like Java [9] and C [15], and
bugs profoundly impact software systems, leading to crashes, even for Ruby in traditional machine learning contexts [16],
data corruption, and security vulnerabilities [1]. Bugs dis- the potential of PLMs and LLMs in automating software
rupt software functionality and pose significant financial and engineering tasks for Ruby remains largely unexplored.
reputational risks for companies [2]. Addressing bugs incurs To address this gap, we curate a dataset of real-world
substantial costs, with companies allocating considerable re- Ruby bugs by mining commits from GitHub repositories,
sources to bug detection, localization, and repair [3]. This following methodologies employed in prior studies [17, 18].
highlights the urgent need for techniques to detect and fix This approach enables us to collect diverse bugs at the method
bugs automatically. level while mitigating data leakage issues. We publish this
Recent advancements in artificial intelligence, particularly open-source dataset to support open science initiatives (see
in Pre-trained Language Models (PLMs), have opened new Section IX for details).
avenues for automating software development tasks. PLMs, Using the curated dataset, we evaluate the capabilities of
with their remarkable ability to simulate human linguistic PLMs, such as CodeBERT and CodeT5, and LLMs, including
GPT-4o and CodeLlama, for bug detection and APR tasks
*These authors contributed equally to this work. in Ruby. Additionally, we compare their performance with

1
traditional machine learning models. Our experiments are of buggy and fixed code. The core idea is to treat the code
conducted in both intra-project and cross-project settings, before a bug-fixing commit as the buggy version and the
providing insights into the generalizability of these models code after the commit as the potentially fixed version. This
across different domains. methodology relies on bug-fixing commits being submitted to
We aim to answer the following research questions in this address identified bugs in the codebase. Consequently, we can
study: reasonably conclude that the source code before such a commit
• RQ1: How do PLMs and LLMs perform in bug contains a bug, and the changes introduced by the commit are
detection and APR tasks for Ruby? While prior intended to fix it. However, as suggested by Chowdhury et
studies have explored PLMs’ performance for various al. [17], certain additional criteria should be met to ensure
code-related tasks [19, 20], little is known about their that the code after a bug-fixing commit is free of bugs. For
effectiveness in bug detection and APR for Ruby. This instance, the code should exhibit a two-year stability period
question evaluates the models’ efficiency in fine-tuning and lack subsequent bug-fixing commits on the same method.
and zero-shot learning for these tasks. The results show The details of the data collection process are as follows:
that machine learning algorithms slightly perform better 1) Repository Selection: The first step of collecting the
in bug detection as a classification task while the PLM required dataset is finding repositories that will be mined
and LLMs fail to effectively classify the buggy and non- to collect bug-fixing commits. To do so, GST [21], we use
buggy codes. In the APR task, the results of the PLM a dataset containing 25 characteristics of 735, 669 GitHub
outperforms the LLMs indicating the potential superior repositories. With various inclusion and exclusion criteria, the
of fine-tuning over zero-shot prompting of larger models. GST dataset can be queried using a web application1 .
• RQ2: How do PLMs perform in bug detection and Inclusion Criteria. In this paper, we select repositories
APR in cross-project settings? Real-world software with the main programming language being Ruby. In addition,
projects often require models to generalize across do- selected repositories should:
mains. This question investigates the impact of cross- 1) Have at least 1,000 commits to ensure that the final
project settings on the performance of PLMs and tra- dataset is not biased towards specific projects in terms
ditional machine learning models for bug detection and of having bug-fix pairs. Without this criterion, several
APR tasks in Ruby. The machine learning methods repositories may have significantly fewer bug-fix pairs
consistently outperform the PLM in all cross-project than other projects (e.g., more enormous repositories).
experiments for the bug detection task. The results of the 2) Be created before June 3, 2022, to ensure that each
APR task for this RQ indicate that the PLM performs selected repository contains at least one file that has
better than the Naive Copy method. remained unchanged for at least two years before the
Our contributions are summarized as follows: data collection date (i.e., June 3, 2024). This criterion is
1) Dataset Curation: We curate a diverse dataset of real- inspired by Chowdhury et al.’s [17] methodology.
world Ruby bugs by mining GitHub repositories. This Exclusion Criteria. Repositories that are forked are ex-
dataset is made publicly available to encourage further cluded. A fork is a new repository that shares code with the
research in this domain. original repository [22]. If forked repositories are included in
2) Model Evaluation: We systematically evaluate the per- the data collection process, it can cause duplicate samples in
formance of PLMs, LLMs, and traditional machine the final dataset.
learning models for bug detection and APR tasks in By applying these criteria, 2, 114 Ruby repositories were
Ruby, in both intra- and cross-project settings. collected to extract bug-fixing commits in the next step.
3) Insights: We provide actionable insights for researchers, 2) Bug-Fixing Commits Identification: Bug-fixing commits
with results indicating close-to-random performance in are identified by traversing all commits in each selected
some scenarios, which we report as negative results. repository. PyDriller [23], a tool that clones a project and
The rest of this paper is structured as follows: Section iterates through its commits on a specific branch, is utilized for
II and Section III cover methodology, detailing the data this purpose. Two Regular Expressions (RegEx) are applied to
collection process and chosen models with evaluation metrics, the commit messages: The first RegEx, as shown in Listing 1,
respectively. Section IV reports the findings of our study. excludes commits that are cherry-picking or revert commits,
Section V discusses the results and their implications. Section or related to documentation, test files, formatting, or linting.
VI reviews related work. Section VII examines the threats to Cherry-picking involves copying a commit from one branch
validity. Finally, Section VIII concludes the study and outlines to another [24], potentially leading to duplicate samples in
directions for future work. the dataset. Revert commits indicate that the changes are no
longer valid. Commits related to documentation, formatting,
II. DATASET
and linting aim to improve code structure rather than address
A. Data Collection logical bugs. Commits related to tests are also excluded
In this paper, inspired by Chowdhury et al. [17] and Tufano
et al.’s [18] approaches, we mine GitHub data to collect pairs 1 [Link]

2
because changes in test files often reflect changes in the repositories, only 1, 289 contained at least one valid bug-fix
business logic. pair, with the remaining 829 repositories excluded due to the
absence of qualifying bug-fix pairs. This study will use this
Listing 1: Exclusion RegEx.
dataset for both bug detection and APR tasks.
\\ b ( c h e r r y | r e v e r t s ? | t e s t s ? | r u b o c o p | 4) Manual Evaluation: A manual analysis was performed
documentations ?| b e a u t i f ( ? : y | ying | i e s ) ? | on a randomly selected sample of 100 commits to ensure the
t ( r e ) ? format ( ? : i n g | t i n g ) ? | l i n t ( ? : e r | i n g ) ) quality of the collected bug-fixing commits. The first three
\\ b authors individually reviewed these commits and determined
The second RegEx, inclusion RegEx, identifies bug-fixing whether they were submitted to fix reported bugs. After re-
commits based on keywords found in their messages, follow- solving seven conflicts, it was found that eight of the commits
ing Chowdhury et al. [17] and Tufano et al.’s [18] studies. As were not bug-fixing commits. This results in a precision of
shown in Listing 2, it looks for messages containing words 92% for the dataset.
like fix OR solve OR repair OR address AND bug As this precision was above 91% and is accepted as a valid
OR problem OR error OR exception with variations accuracy in similar studies [25], we used our collected dataset
in these words considered. for the experiments in this work. Additionally, we discuss the
related threats in Section VII.
Listing 2: Inclusion RegEx. 5) Dataset Preparation: Following the described process,
\\ b ( ? : f i x ( ? : s | e s | ed | i n g ) ? | we collected 10, 702 bug-fix pairs, each consisting of a buggy
( r e ) ? s o l v ( ? : e | e s | ed | i n g ) ? | r e p a i r ( ? : ed | s | method and its corresponding fixed method. These pairs are
i n g ) ? | a d d r e s s ( ? : e s | ed | i n g ) ? ) \ \ b . * ? \ \ b ( ? : directly used in the APR task, where the buggy code serves
bug ( ? : s | gy | y ) ? | p r o b l e m s ? | e r r o r s ? | as the input to the models, and the fixed code is treated as the
exceptions ?)\\ b ground truth for the models’ output.
3) Bug-Fix Pairs Extraction: For each bug-fixing commit, We process each pair separately for the bug detection task
changed files and methods are analyzed to isolate single- by treating the buggy and fixed code as individual records. A
method bug-fix pairs, focusing exclusively on commits where binary label is assigned to each record to distinguish between
exactly one method has been modified. Also, new methods them, indicating whether the code is buggy or not. This process
are excluded as they lack a previous counterpart to be con- results in a dataset containing 21, 404 records, evenly split
sidered buggy code. Each commit’s current and previous code between buggy and non-buggy code.
snapshots are extracted to represent the buggy and fixed code As the total number of records in the dataset is approx-
versions, respectively. Several additional criteria, inspired by imately 10K for within-project settings and fewer than 5K
the methods in previous works [17, 18], are applied to ensure for cross-project settings, we opted to divide the dataset into
data quality, as described below. training and test splits, with the ratio of 8:2. This decision
Reviewing Future Commits: To ensure each extracted is motivated by the observation that the models show no
bug-fix pair accurately represents buggy and fixed code, the overfitting issues, as evidenced by the low changes in training
method’s subsequent commits are examined to confirm that loss during the training phase. Even with an increased number
no later bug-fixing commits modify the current version of the of epochs, the models remain underfitted. The absence of
technique. If a future bug-fixing commit alters the current overfitting issues eliminates the need for a validation split,
version of the method, the version in the initial commit which is primarily used for regularization and avoiding the
cannot reliably be labeled as “fixed” since it required further models to be overfitted to the training data [26]. Instead, by
adjustment. In such cases, the version of the method after the limiting the dataset to training and test splits, we increase the
later commit is designated as the “fixed” code. In contrast, number of training samples, potentially allowing the model to
the version before the initial commit is treated as the “buggy” learn more effectively from the data. However, as shown in
code, preserving the validity of the bug-fix pairing. Section IV, our experiments reveal negligible improvements
Two-Year Stability Period: Inspired by Chowdhury et under this setting.
al. [17], a method labeled as “fixed” is required to remain Given that bug detection and APR are time-sensitive tasks
unmodified for at least two years after the bug fix to ensure [17], we ensure that the models are trained on older data than
the fix’s stability. The commit is excluded from the dataset if what is included in the test set.
the period between subsequent changes is less than two years. III. L ANGUAGE M ODELS AND E VALUATION M ETRICS
Excluding Methods in Test Files: Since modifications in This section covers different techniques incorporated in the
test files typically mirror changes in the business logic, all study to tune or prompt the models in the chosen downstream
methods in files identified as test files are excluded. Test tasks of bug detection and APR for the Ruby programming
files are recognized using a regular expression that detects language.
variations of the term “test” in file names.
The methodology described yielded a collection of 10, 702 A. The Choice of Language Models
real-world bug-fix pairs for Ruby sourced from 1, 289 reposi- CodeT5 is a pre-trained Transformer model designed for
tories. Notably, while the initial repository list included 2, 114 code understanding and generation tasks. It employs a unified

3
framework that utilizes the coding semantics conveyed through TABLE II: The initial value of hyper-parameters used to fine-
the identifiers assigned by developers, allowing for a more tune the PLM for both bug detection and APR tasks.
seamless and accurate understanding of code. Moreover, it will Hyper-parameter Value
enable multi-task learning, making it a versatile and powerful
Learning rate 1e-5
tool for developers to use in their projects [27]. It also performs Optimizer AdamW
well in vulnerability detection compared to the novel deep Warm up steps 10%
learning method at a function level scope [28] that fits the Adam-epsilon 1e-8
Training batch size 16
tasks and problems we aim to investigate in this study. Validation batch size 16
Code-Llama-7b-Instruct This model is one of the latest
state-of-the-art open-source Large Language Models and part
of the CodeLlama model series released from Meta [29]. sizes expand [6]. Building upon prior research [9, 32], we
CodeLlama is a highly specialized version of Llama2, de- undertake fine-tuning of a PLM, i.e., CodeT5 (base) [27],
signed to cater specifically to the needs of developers and aimed to analyze the Ruby dataset for the bug detection and
programmers. It results from further fine-tuning of the Llama-2 APR tasks. Following the prior research [33], we set the initial
model on programming language corpus [29]. This has enabled value of hyper-parameters for our experiments as shown in
CodeLlama to develop a deep understanding of the intricacies Table II.
of coding languages and their nuances, making it a powerful We set the models’ input token size for both tasks to 256
tool for developers to streamline their coding process. Given tokens. The model’s output size for the APR task is also set to
our limited resources, we use the 7B version of this model 256 tokens. We train the models on all dataset variants, both
to enable experimentation on a single GPU. Additionally, for RQ1 and RQ2, for three epochs and report the results on
since we apply zero-shot prompting without additional fine- the validation set after the final epoch. All experiments are
tuning, we rely on the instruction-tuned version to generate conducted on a single NVIDIA Tesla V100 32GB GPU.
task-specific responses. Without instruction tuning, pre-trained 2) Zero-shot learning: Zero-shot learning or prompting is
versions of large language models (LLMs) tend to provide one of the primary techniques in in-context learning, where
open-ended answers that do not meet our requirements. In the model is prompted with the task to be performed without
Section IV, we elaborate on the prompt design used to obtain being given any examples of the task [34]. In most use cases,
the desired output from the model. developers apply zero-shot prompting to LLMs to solve their
GPT-4o This model is one of the most popular and latest problems in the software development process. In this study,
commercial language models developed by OpenAI, capable we use zero-shot prompting of LLMs, i.e., Code-Llama-7b-
of handling various tasks and equipped with advanced rea- Instruct and GPT-4o, as a widely used and primary approach
soning capabilities [30]. Introduced in May 2024, this model to evaluate bug detection and APR tasks using such models.
can respond to text, image, and audio queries. It performs For the bug detection task, we use prompts to direct the
better than its competitors on the HumanEval [20] benchmark, models to classify the provided code as either ‘buggy’ or ‘non-
one of the most widely recognized benchmarks for code buggy’. Initially, these models are adjusted to generate more
generation and intelligence. This model was selected because information about the program snippets. To ensure we can
it is currently the most widely used for solving various tasks, identify the code label in the model output, we prompt them
including programming. to provide only the label of the given code without additional
Table I presents an overview of the PLMs and LLMs used information. The following is the prompt for bug detection by
in this study, including their sizes and corresponding data used LLMs: “Classify the following Ruby code as ‘buggy’ or ‘non-
during their training phase. buggy.’ Only answer with ‘buggy’ or ‘non-buggy’: {CODE}”,
where {CODE} represents the input code, which may be either
TABLE I: PLMs and LLMs used in our experiments. buggy or non-buggy.
Model Training Data Parameter Sizes We ask the models to generate a fixed code version for the
APR task without analyzing the existing bugs. This allows us
CodeT5 Ruby, JavaScript, Go, Small (60M) and Base (220M)
Python, Java, PHP, C, C# to retrieve only the fixed code snippet and compare it with
CodeLlama C++, Java, PHP, Type- 7B the ground truth in the dataset. The prompt for this task is as
Script, C#, Bash follows: “Fix the bug in the given code and output the fixed
GPT-4o NA NA
version of the code; only respond with the fixed code, without
any extra explanations: {CODE}”, where {CODE} represents
the given buggy code provided to the model.
B. Learning methods
1) Fine Tuning: Fine-tuning PLMs represents an initial ap- C. Machine Learning Methods
proach in transfer learning aimed at adapting language models Multiple studies have demonstrated that machine learning
to specific downstream tasks [31]. This technique updates all methods are highly effective in identifying defects [35, 36].
model parameters during the adaptation phase, necessitating As such, we have carefully selected two models that have
significant time and memory resources, particularly as model shown exemplary performance in classification tasks.

4
TABLE III: Accuracy, precision, recall, and F1-score of different models on bug detection. The results for Code-Llama-7b-
Instruct and GPT-4o models are reported in over 100 samples.
Model Accuracy Precision Recall F-Score
Random Forest 56.97% 56.99% 56.97% 56.96%
XGBoost 58.78% 58.79% 58.78% 58.74%
CodeT5 50.95% 51.05% 50.95% 50.54%
Code-Llama-7b-Instruct 44.00% 44.19% 44.00% 37.87%
GPT-4o 49.00% 49.08% 49.00% 49.04%

TABLE IV: BLEU score of language models on APR task. total number of predictions. Precision evaluates the quality
The results of Code-Llama-7b-Instruct and GPT-4o models are of positive predictions by calculating the ratio of correctly
reported on 100 dataset samples. predicted positive samples (e.g., buggy code) to the total
Model BLEU
number of samples predicted as positive. Recall is the ratio
of correctly predicted positive samples to the total number of
CodeT5 90.12%
Code-Llama-7b-Instruct 61.57% positives. Finally, the F1-score provides a harmonic mean of
GPT-4o 78.68% precision and recall, balancing their trade-offs. These metrics
collectively offer a comprehensive assessment of the model’s
performance in the bug detection task.
1) XGBoost: XGBoost [37] is an open-source machine- For the APR task, the generated code from the models is
learning method that utilizes an advanced algorithm called expected to be assessed about the ground truth code in the
gradient boosting to power scalable and distributed decision dataset. To measure the similarity of these code snippets,
trees. It is designed to work seamlessly with large datasets we use the widely used BLEU score [39] similar to prior
and can be used for various applications, including regression, studies [27, 33, 40]. This metric measures the similarity of
classification, and ranking problems. The parallel tree boosting two sequences of tokens by incorporating the overlapping n-
functionality ensures faster and more accurate model training, grams in the sequences. The exact definition of this metric is
making XGBoost the leading choice for data scientists and formulated as follows:
machine learning enthusiasts worldwide.
2) Random Forest: Ensemble classifiers are a group of N
X
machine learning algorithms that combine the predictions BLEU = BP × exp( wn log(pn )) (1)
of multiple models to improve overall performance. One n=1
commonly used ensemble classifier is Random Forest [38]. Where BP is the brevity penalty, calculated based on the
This algorithm is known for its ease of use and flexibility, generated output’s length and the dataset’s ground truth code.
as it can handle both classification and regression problems. Additionally, pn represents the modified precision for n-grams,
When constructing decision trees in a Random Forest, a and wn denotes the weight of each n-gram. We report the
random selection of attributes is chosen, and individual trees BLEU score with n = 4 in all the experiments similar to
are created using a simple algorithm. Unlike other decision prior studies [33].
tree algorithms, pruning is not performed at each node, and
attributes are sampled randomly. The unlabelled example is IV. R ESULTS
classified based on the majority of voting. One significant
This section presents the results of applied methods to
advantage of Random Forest is its speed and ability to handle
evaluate PLMs, LLMs, and ML algorithms for both bug
many input attributes [36]. This algorithm is also less prone
detection and APR tasks.
to overfitting, a common problem in decision tree algorithms.
To identify optimal hyperparameter configurations for train- A. PLMs and LLMs Performance for Bug Detection and APR
ing machine learning algorithms, we conducted experiments
using different values of maximum depth of the trees in To answer this question, we examine the bug detection
{16, 64, 256, 512} and number of estimators in {2, 10, 100}. capability of classical machine learning methods on the Ruby
Based on these experiments and their results, we set the dataset and compare it with the results of the language model’s
maximum tree depth and the number of tree estimators to fine-tuning and zero-shot prompting. The results are repre-
256 and 10, respectively. We opt for these values across all sented in Table III and Table IV for bug detection and APR
experiments involving machine learning algorithms. tasks, respectively. The Ruby dataset used for this research
question contains more than 10K samples, whereas for the
D. Evaluation Metrics larger models, i.e., Code-Llama-7b-Instruct and GPT4-o, we
This study reports accuracy, precision, recall, and F1-score report the results on 100 randomly selected records. This
as commonly used metrics for evaluating classification tasks, choice is made for LLMs, as we use the prompting strategy
such as bug detection [17]. Accuracy measures the propor- rather than training the models. As observed for bug detection
tion of correct predictions made by the model out of the tasks, when comparing the classical machine learning methods

5
TABLE V: Accuracy, precision, recall, and f1-score of CodeT5 model vs. machine learning algorithms, i.e., Random Forest
and XGBoost, on bug detection in the cross-project setting.
Dataset Splits Model Accuracy Precision Recall F1-Score
CodeT5 47.07% 46.24% 47.07% 43.28%
Train(3156) / Test(890) Random Forest 59.48% 59.51% 59.48% 59.47%
XGBoost 55.50% 55.57% 55.50% 55.44%
CodeT5 46.43% 46.60% 46.43% 46.39%
Train(3392) / Test (654) Random Forest 53.88% 54.32% 53.88% 53.66%
XGBoost 53.57% 53.58% 53.57% 53.57%
CodeT5 54.83% 55.00% 54.83% 54.87%
Train(3451) / Test (595) Random Forest 63.19% 63.76% 63.19% 63.16%
XGBoost 56.66% 57.09% 56.66% 56.65%
CodeT5 51.43% 51.57% 51.43% 51.37%
Train(3187) / Test (859) Random Forest 50.96% 50.80% 50.96% 50.62%
XGBoost 53.71% 53.69% 53.71% 53.69%
CodeT5 49.84% 50.13% 49.84% 49.80%
Train(3302) / Test (744) Random Forest 56.37% 57.28% 56.37% 55.95%
XGBoost 59.32% 59.55% 59.32% 55.96%
CodeT5 50.53% 50.67% 50.53% 50.55%
Train(2919) / Test (1127) Random Forest 47.86% 48.17% 47.86% 47.69%
XGBoost 51.79% 51.80% 51.79% 51.79%
CodeT5 49.09% 49.10% 49.09% 49.09%
Train(3556) / Test (490) Random Forest 52.55% 52.54% 52.55% 52.33%
XGBoost 53.04% 53.03% 53.04% 52.94%
CodeT5 51.30% 51.67% 51.30% 50.61%
Train(3560) / Test (486) Random Forest 53.57% 53.51% 53.57% 53.39%
XGBoost 51.90% 52.02% 51.90% 51.84%
CodeT5 52.15% 53.06% 52.15% 50.16%
Train(2924) / Test (1122) Random Forest 56.11% 56.08% 56.11% 56.06%
XGBoost 53.18% 53.27% 53.18% 53.15%
CodeT5 50.63% 50.29% 50.63% 50.14%
Train(3019) / Test (1027) Random Forest 52.64% 52.66% 52.64% 52.65%
XGBoost 51.37% 51.61% 51.37% 51.36%
CodeT5 50.33% 50.43% 50.33% 49.63%
Average Random Forest 54.66% 54.86% 54.66% 54.50%
XGBoost 54.00% 54.12% 54.00% 53.64%

TABLE VI: The BLEU Score of CodeT5 model on 10 cross- suggests that the learning capability of the models on the low-
project data splits for automated program repair task. The resource language Ruby for this task is limited. In contrast,
Naive Copy column shows the score when no changes is made classical machine learning approaches still yield better perfor-
in the buggy code. mance for this task.
Dataset Split CodeT5 Naive Copy More interestingly, the performance of zero-shot prompting
with Code-Llama-7b-Instruct and GPT-4o models is even be-
Train(3156) / Test (890) 88.33% 84.45%
Train(3392) / Test (654) 90.78% 80.43% low 50%, indicating that even larger models fail to detect bugs
Train(3451) / Test (595) 85.46% 86.56% effectively. In scenarios with only the buggy code, learning-
Train(3187) / Test (859) 90.09% 80.82% based methods, such as traditional machine learning algo-
Train(3302) / Test (744) 86.16% 86.21%
Train(2919) / Test (1127) 90.97% 87.22% rithms and fine-tuning pre-trained language models, continue
Train(3556) / Test (490) 91.46% 85.20% to show superior performance.
Train(3560) / Test (486) 90.83% 84.60%
Train(2924) / Test (1122) 89.38% 84.65%
Despite the poor performance of the pre-trained language
Train(3019) / Test (1027) 90.71% 84.50% model on the bug detection task, its results on the automated
Average 89.42% 84.46% program repair task is relatively high. Table IV reports the
results for APR. Machine learning methods are only applied
to the bug detection task, as these methods are compatible with
classification tasks and not the generative task of APR. The
with language models, the Random Forest algorithm achieves BLEU score of the CodeT5 model is 90.12%, which is higher
better results with an accuracy of 58.78% than the naive copy score in our dataset. The naive copy score
The results of all language models are close to 50%, similar for Ruby on APR is 84.36%. This score is obtained when the
to randomly guessing the labels of bugs in the dataset. This initial buggy code is considered as the fixed code and is thus

6
compared with the ground truth fixed code. In other words, Finding 2: In analyzing 10 Ruby datasets with
this suggests that the model does not learn to introduce any different domains for their train/test splits, fine-tuned
changes to the initial code. We consider the naive copy of the CodeT5 does not outperform traditional machine learn-
input code as the baseline similar to previous works [40]. ing methods. The Random Forest algorithm performs
The results of zero-shot prompting for LLMs are lower than better in most cases for detecting bugs than other
the baseline score, with scores of 61.57% and 78.68% for methods in this cross-project evaluation.
the Code-Llama-7b-Instruct and GPT-4o models, respectively. The enhanced performance of the PLM in the APR
This indicates that these models tend to produce more cor- task is still observable across different domains com-
rupting changes to the initial code rather than fixed patches. pared to naive copy. However, the average performance
In Section V, we further investigate this issue by providing an of the model on all cross-project datasets drops when
example of for each of the models. compared with the intra-project setting (RQ1) indicat-
ing that the having samples from same projects and
Finding 1: Results of the first RQ show that Random domains in both train and test data splits helps the
Forest outperforms PLMs and LLMs in bug detection performance improvement of the model.
tasks for the Ruby language. All language models
achieve near 50% accuracy, equivalent to random V. D ISCUSSION
guessing, indicating that such models fail to address
A. Result Analysis
the bug detection task for Ruby effectively. In contrast,
fine-tuning PLMs on the Ruby dataset shows improved In Section IV, the performance of the selected PLM for
results compared to the baseline score on the APR the bug detection task was discussed. However, the results
task. Such improvement is not observed when using were unsatisfactory, with the maximum accuracy reaching
LLMs with zero-shot prompting. This suggests that only 50.95%, comparable to guessing randomly. Additionally,
fine-tuning PLMs has an advantage in APR tasks for the models were fine-tuned and evaluated in a cross-project
low-resource languages over zero-shot prompting of setting, where the distributions of the training and testing splits
larger models, indicating the need for other techniques differed, each containing distinct projects. Similarly, signifi-
to use LLMs for APR in Ruby. cant changes in the results were not observed, and the models’
performance remained poor. In both cross-project and intra-
project settings, the machine learning algorithms outperformed
B. Models’ Performance on Bug Detection and APR in Cross- the PLM and the LLMs for the first RQ, highlighting the
Project Setting effectiveness of such methods for low-resource programming
languages like Ruby.
Table V shows the results of the fine-tuned PLM and ML Other studies [15, 40] have demonstrated higher perfor-
algorithms on different dataset variants for the bug detection mance of fine-tuning PLMs for the bug detection task in the
task, along with the train and test split sizes. Similar to C programming language (62.08%). Considering the lower
the previous RQ, we observe that the performance of ML readability and, consequently, reduced understandability of C
algorithms significantly outperforms that of the fine-tuned code snippets by language models compared to Ruby, the
PLM. The performance decreases compared to RQ1, indicat- higher performance of PLMs on C is noteworthy. These results
ing the models’ performance drops when tested on new data, highlight the significant impact of data quality and size when
i.e., projects. Among these algorithms, Random Forest scores training these models for the bug detection task. Notably, the
higher than XGBoost in 6 of the ten experiments. dataset used in the other study contains nearly 20K samples,
More specifically, the results of the fine-tuned PLM are near which is double the size of our dataset. This could be a
50% for all dataset variants, with several cases falling below potential reason for the higher performance of the models on
50% for all metrics. Such poor performance demonstrates C, in addition to the data quality and the types of bugs present
the inefficiency of fine-tuning PLMs when the domain of in the dataset.
bugs differs between the train and test splits. However, ML
algorithms maintain their performance even when the domain B. Example Cases
of the training data differs from the test data. These algorithms Despite of the high score of the PLM and good scores of
are more robust in low-resource scenarios where the available LLMs for the APR task, the real capability of such models in
data for training is limited. fixing buggy codes is limited. Figure 1 illustrate an example
Table VI shows the results for the same data distributions on of the APR task in which the buggy code, ground truth fixed
the APR task with the fine-tuned CodeT5 model. As observed, code, and generated codes of the models are represented. As it
the results are higher than the naive copy score in 8 out of 10 is observed, the generated codes by all models do not correctly
experiments, with 91.54% as the highest and 85.46% as the address the initial bug in the input code. Particularly, among
lowest BLEU scores. Such scores indicate the effectiveness those three models, Code-Llama-7b-Instruct fails to capture
of fine-tuning models for APR, even when the train and test the buggy line of the code and makes unnecessary changes in
splits distributions differ. wrong parts. CodeT5 and GPT-4o models correctly capture the

7
 (a) Input buggy code. (b) Fixed code as ground truth.

(c) Generated code by CodeT5 model. (d) Generated code by Code-Llama-7b-Instruct model.

(e) Generated code by GPT-4o model.

Fig. 1: An example of buggy code and its fixed version and generated codes by different models. Highlighted lines represent
the buggy part of the input code (a) and the changes in others (b, c, d, and e).

buggy line of the code yet both of them fail to fix it correctly. the projects’ specific business requirements. Some buggy
Additionally, the generated codes by larger models, i.e., Code- methods are heavily dependent on the context of the project.
Llama-7b-Instruct and GPT-4o, are syntactically correct while In other words, a specific method might be considered buggy
the generated code by CodeT5 is corrupted. in one project while functioning correctly in another due to
A manual analysis of the results of the initial experiments differences in their business requirements. To this end, since
(fine-tuned CodeT5) was conducted to investigate the reasons we are not providing further information about the existing
for the poor performance further. A random dataset of 200 bugs in the code, it is possible to achieve poor performance
predicted bug-fix pairs was curated, including 50 samples on these tasks.
each of True Positive, True Negative, False Positive, and Another issue we found in our manual analysis is related
False Negative. These samples were independently analyzed to the scope of the collected bugs. In our data collection
by two authors to identify any meaningful patterns between phase, we consider the changed methods to be the scope of
the correctly predicted samples and the failures. However, this the bug and only feed these methods to the model. However,
analysis did not reveal any specific pattern that the model based on the manual analysis, we observed many methods
followed to produce better results for certain types of bugs. that produced bugs in other methods or files. In other words,
In our manual analysis, it was also observed that detecting although the technique caused a bug in the system, it could
whether a code snippet contains a bug was challenging for not be discovered by considering the changed method. In fact,
the authors, primarily because they were not familiar with by considering only the changed method, the method would

8
be regarded as a non-buggy code. All these factors likely A. Bug detection
contributed to our study’s low-accuracy of the models. The field of software engineering has been grappling with
identifying bugs in code using various methods such as static
C. Actionable Insights analysis and machine learning. Different representations of the
Based on our analysis, detecting bugs and fixing them for bugs, such as abstract syntax trees or code snippets, have been
Ruby language is not easily doable by the current models, used before feeding them into the models [36], and empirical
nor can they classify the codes as buggy or non-buggy. Even evidence has shown no significance in a particular form of
though machine learning models achieved higher scores, they input given to the model [44]. A study introduced a new model
still have scores below 60% for bug detection. Similarly, as called DexBERT [45], based on the traditional BERT [46]
discussed above, though models achieved high BLEU scores, transformer representation that does bug detection and outper-
this is not a reliable metric to ensure that the generated code is forms the conventional deep learning method called smali2vec.
executable. These issues mainly arise from the lack of bench- Despite promising results, there has been a noticeable lack of
marks and the need for extended datasets. We recommend the research on solving this downstream task using PLMs, unlike
following directions for the researcher. many other downstream software engineering tasks [4]. Our
Insight 1: Benchmark datasets must ensure the generated research aims to address this gap in the literature and provide
fixed codes are executable. These benchmarks should have test insights into this unexplored yet longstanding area of interest.
cases, following a similar approach as HumanEval or other
B. APR
code generation benchmarks. As Ruby is not a high resource,
this task might require code translation from existing works A lot of traditional research has been done on APR over
to Ruby. time. The current papers on APR on PLMs [9] use datasets
Insight 2: The APR task requires a clear understanding from state-of-the-art deep learning/machine learning papers,
of the project context and cannot be effectively learned by such as CoCoNut [47] CodeRep [48] and MegaDiff [49].
simply providing buggy code alongside its corresponding fix. There has been work evaluating and fine-tuning Pre-Trained
Future studies could explore alternative methods for collecting Language Models for APR [32]. Across four APR bench-
datasets. For instance, researchers could link code commits to marks, the study finds that the best-performing PLM, when
their related bug reports (such as GitHub issues) and utilize unaltered, outperforms state-of-the-art deep-learning (DL)–
labels on these issues to identify bug-fixing commits. Addi- based APR techniques by 72% in bug fixing. Moreover, the
tionally, leveraging metadata from datasets, as seen in CodeNet study addresses the efficiency of different PLMs in terms
[41] could help recognize the project contexts associated with of size, time, and memory usage, offering promising direc-
buggy and corrected code. tions for improving PLMs and advocating for transparent
Insight 3: In addition to datasets, new methods are required reporting practices to mitigate data leakage concerns in future
for bug detection and APR tasks for the Ruby language. evaluations. Different sizes were considered to feed into the
These techniques should be tailored specifically to the Ruby model; the most noteworthy was using Simple Stupid Bugs to
language to improve the performance and could vary. For train CodeBert to solve APR task [9]. Previous research has
example, techniques can be developed to detect noise in the demonstrated that APR can be effectively addressed by PLMs
dataset or integrate the business logic/project concept in the with satisfactory performance. By employing fully fine-tuning
models by incorporating retrieval-augmented generated (RAG) and Parameter-Efficient Fine-Tuning (PEFT) [33] approaches,
[42] approaches or prompt engineering or prompt tuning with we can discover unexplored opportunities in low-resource pro-
LLMs that direct the model to include a Ruby-specific context. gramming languages. This paves the way for us to experiment
Insight 4: Bug detection using machine learning algorithms with innovative techniques and push the boundaries of what
demonstrated better performance, and their score did not can be achieved in this field.
decrease much in the cross-project setting. This might suggest VII. T HREATS TO VALIDITY
the robustness of these algorithms for this task. Researchers
We discuss the potential threats to provide more guidance
can rely on computationally cheaper models, i.e., machine
on the interpretation, limitation, and other alternatives of the
learning models when developing techniques for bug detec-
empirical experiment.
tion in Ruby. Similar results were seen in text classification
where XGBoost outperforms LLMs, such as GPT-4, in text A. Construct Validity
classification task [43].
This threat relates to the bug-fixing commit identification.
The reliance on RegEx to classify bug-fixing commits could
VI. R ELATED W ORK
lead to misclassification. While manual validation estimated
This section provides a summary of the research conducted a 92% precision, false positives might remain, affecting the
so far on the downstream tasks being considered. It covers a quality of the dataset. We however, followed the practices from
wide range of relevant studies and explores the intricacies of previous studies to ensure a reliable data collection.
the research, providing valuable insights into the current state Another threat to validity is the assumption that code
of the field. becomes bug-free after a bug-fixing commit. In this study, we

9
relied on this assumption to construct a dataset of bug-fix pairs, gramming languages, focusing on Ruby. Leveraging pre-
treating the code post-bug-fixing commit as a ”fixed” version. trained language models and large language models, we con-
However, this may not always be accurate, as undetected bugs ducted a comprehensive analysis of these tasks using our
might persist in the fixed code. To address this limitation, we newly introduced dataset, Defects4Ruby. Our contributions to
applied stringent filtering criteria. For instance, we excluded this study are 1) the Defects4Ruby dataset, mined from GitHub
methods associated with future bug-fixing commits and en- using established APR methodologies, which enables the study
sured the code remained unchanged for at least two years of software bug resolution in Ruby, and 2) the insights in
after the bug-fixing commit. These measures were intended evaluating PLMs and LLMs for this dataset in Ruby. Our
to increase confidence that the code was truly fixed. Despite findings reveal that while PLMs and LLMs show potential
these efforts, the possibility of residual bugs in the fixed for APR, their performance in bug detection remains subop-
code cannot be entirely eliminated. Such undetected bugs timal, with traditional machine learning models performing
could introduce noise into the dataset and affect the reliability competitively on certain metrics. This highlights a critical
of model evaluation. Future work could focus on improving need for new techniques tailored to low-resource languages
the methodology for verifying the correctness of fixed code, like Ruby. Specifically, PLMs such as CodeT5 demonstrate
refining filtering criteria, and aligning the dataset more closely promising capabilities for APR, achieving significantly higher
with the needs of bug detection tasks. performance than LLMs. However, bug detection scores falling
below 60% indicate substantial room for improvement.
B. External Validity To summarize, our study underscores the limitations of
This relates to the representativeness of datasets. In this existing approaches and opens avenues for future research,
work, the dataset focuses exclusively on Ruby projects from including the development of domain-specific models, im-
GitHub, which may not represent bugs and fixes from other proved data augmentation techniques, and a deeper explo-
closed-source projects, limiting the generalizability of the find- ration of fine-tuning and prompting strategies for bug-related
ings to other programming languages. Therefore, we cannot tasks. The study contributes to the existing research gap by
make a sound claim on the effectiveness of projects in another providing insights into two complex downstream tasks in
programming language, given we have not tested on other software bug resolution for Ruby. Our findings and the open-
scenarios. sourced Defects4Ruby dataset will serve as valuable resources
for researchers and practitioners working to address software
C. Internal Validity engineering challenges in low-resource languages, specially
Internal threats refer to factors arising from internal consid- Ruby.
erations. In this study, we set the hyperparameter values for
fine-tuning the PLM based on prior research and experimented IX. DATA AVAILABILITY
with various configurations to identify the optimal settings for
The data and scripts to reproduce our results are available
the machine learning algorithms. While this approach is widely
in the replication package.2
accepted, there remains a possibility that the selected values
may not represent the optimal configurations for the models R EFERENCES
and algorithms. However, based on the low obtained scores, we [1] K. Huang, B. Chen, C. Xu, Y. Wang, B. Shi, X. Peng, Y. Wu, and Y. Liu,
believe changing the parameters would not have a significant “Characterizing usages, updates and risks of third-party libraries in java
boost in the performance. projects,” Empirical Software Engineering, vol. 27, no. 4, p. 90, 2022.
[2] Y. Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha,
and N. Ubayashi, “A large-scale empirical study of just-in-time quality
D. Conclusion Validity assurance,” IEEE Transactions on Software Engineering, vol. 39, no. 6,
This relates to the potential risk that the manual validation pp. 757–773, 2012.
[3] C. Jones and O. Bonsignour, The Economics of Software Quality, 1st ed.
uses 100 commits. While useful, 100 commits may not be Addison-Wesley Professional, 2011.
representative of the entire dataset, leading to potential over- [4] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo,
estimation or underestimation of dataset quality. For all tests, J. Grundy, and H. Wang, “Large language models for software engi-
neering: A systematic literature review,” 2023.
we used widely used parameters for the bug detection and APR [5] D. Yan, Z. Gao, and Z. Liu, “A closer look at different difficulty
tasks. For APR, we were unable to apply Pass@K to ensure levels code generation abilities of chatgpt,” in 2023 38th IEEE/ACM
the fixed code is executable, due to the lack of availability International Conference on Automated Software Engineering (ASE).
IEEE, 2023, pp. 1887–1898.
of such dataset that also includes tests cases. We emphasize [6] M. Weyssow, X. Zhou, K. Kim, D. Lo, and H. Sahraoui, “Exploring
however, as discussed in the previous section, that high BLEU parameter-efficient fine-tuning techniques for code generation with large
scores do not mean that models are able to fix code reliably, language models,” 2024.
[7] T. Ahmed, K. S. Pai, P. Devanbu, and E. T. Barr, “Improving few-
and other techniques are required to improve the performance shot prompts with relevant static analysis products,” arXiv preprint
of APR for Ruby. arXiv:2304.06815, 2023.
[8] Y. Wu, Z. Li, J. M. Zhang, M. Papadakis, M. Harman, and Y. Liu, “Large
VIII. C ONCLUSION language models in fault localisation,” arXiv preprint arXiv:2308.15276,
2023.
This study addresses the underexplored challenges of bug
detection and automated program repair in low-resource pro- 2 [Link] only=dbd817dd5d564bf0b1c4b9efb72c9095

10
 [9] E. Mashhadi and H. Hemmati, “Applying codebert for automated pro- //[Link]/[Link]-main.685
gram repair of java simple bugs,” in 2021 IEEE/ACM 18th International [28] W. Tang, M. Tang, M. Ban, Z. Zhao, and M. Feng, “Csgvd: A
Conference on Mining Software Repositories (MSR), 2021, pp. 505–509. deep learning approach combining sequence and graph embedding
[10] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, for source code vulnerability detection,” Journal of Systems and
T. Liu, D. Jiang, and M. Zhou, “Codebert: A pre-trained model for Software, vol. 199, p. 111623, 2023. [Online]. Available: https:
programming and natural languages,” 2020. //[Link]/science/article/pii/S0164121223000183
[11] H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, [29] B. Rozière et al., “Code llama: Open foundation models for code,” 2023,
“Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv:2308.12950 [[Link]].
2020. [30] OpenAI, “gpt-4o,” [Link] 2024.
[12] S. Chandran and K. Abraham, “A correlative scrutiny on two program- [31] A. Radford and K. Narasimhan, “Improving language understanding
ming dialects: Ruby vs python,” International Journal of Engineering by generative pre-training,” 2018. [Online]. Available: [Link]
and Advanced Technology, vol. 9, pp. 4395–4404, 02 2020. [Link]/CorpusID:49313245
[13] S. Kaleba, O. Larose, R. Jones, and S. Marr, “Who you gonna call: [32] N. Jiang, K. Liu, T. Lutellier, and L. Tan, “Impact of code language mod-
analyzing the run-time call-site behavior of ruby applications,” in els on automated program repair,” in 2023 IEEE/ACM 45th International
Proceedings of the 18th ACM SIGPLAN International Symposium on Conference on Software Engineering (ICSE), 2023, pp. 1430–1442.
Dynamic Languages, 2022, pp. 15–28. [33] C. Wang, Y. Yang, C. Gao, Y. Peng, H. Zhang, and M. R. Lyu, “No
[14] GitHub Blog, “The state of open source and ai,” [Link] more fine-tuning? an experimental evaluation of prompt tuning in code
2023-11-08-the-state-of-open-source-and-ai/, November 2023. intelligence,” in Proceedings of the 30th ACM Joint European Software
[15] Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu, “Devign: Effective vulner- Engineering Conference and Symposium on the Foundations of Software
ability identification by learning comprehensive program semantics via Engineering, 2022, pp. 382–394.
graph neural networks,” 2019. [34] T. B. Brown, “Language models are few-shot learners,” arXiv preprint
[16] J. P. Near and D. Jackson, “Finding security bugs in web applications arXiv:2005.14165, 2020.
using a catalog of access control patterns,” in Proceedings of the [35] E. Ceylan, F. O. Kutlubay, and A. B. Bener, “Software defect iden-
38th International Conference on Software Engineering, ser. ICSE ’16. tification using machine learning techniques,” in 32nd EUROMICRO
New York, NY, USA: Association for Computing Machinery, 2016, p. Conference on Software Engineering and Advanced Applications (EU-
947–958. [Online]. Available: [Link] ROMICRO’06), 2006, pp. 240–247.
[17] S. Chowdhury, G. Uddin, H. Hemmati, and R. Holmes, “Method-level [36] S. Aleem, L. F. Capretz, and F. Ahmed, “Benchmarking machine
bug prediction: Problems and promises,” ACM Transactions on Software learning technologies for software defect detection,” 2015.
Engineering and Methodology, vol. 33, no. 4, pp. 1–31, 2024. [37] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,”
[18] M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and in Proceedings of the 22nd ACM SIGKDD International Conference on
D. Poshyvanyk, “An empirical study on learning bug-fixing patches in Knowledge Discovery and Data Mining, ser. KDD ’16. ACM, Aug.
the wild via neural machine translation,” ACM Transactions on Software 2016. [Online]. Available: [Link]
Engineering and Methodology (TOSEM), vol. 28, no. 4, pp. 1–29, 2019. [38] L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1,
[19] R. Dandamudi and G. Rodriguez-Perez, “A preliminary study of pp. 5–32, 10 2001. [Online]. Available: [Link]
multilingual code language models for code generation task using 1010933404324
translated benchmarks,” in Proceedings of the 39th IEEE/ACM [39] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for
International Conference on Automated Software Engineering automatic evaluation of machine translation,” in Proceedings of the 40th
Workshops, ser. ASEW ’24. New York, NY, USA: Association annual meeting of the Association for Computational Linguistics, 2002,
for Computing Machinery, 2024, p. 94–99. [Online]. Available: pp. 311–318.
[Link] [40] S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco,
[20] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou,
H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K.
G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, Deng, S. Fu, and S. Liu, “Codexglue: A machine learning benchmark
S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, dataset for code understanding and generation,” 2021.
C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, [41] R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V. Zolotov,
E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Dolby, J. Chen, M. Choudhury, L. Decker, V. Thost, L. Buratti,
J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, S. Pujar, S. Ramji, U. Finkler, S. Malaika, and F. Reiss, “Codenet: A
A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, large-scale ai for code dataset for learning a diversity of coding tasks,”
M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, 2021.
D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating [42] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal,
large language models trained on code,” 2021. H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, and
[21] O. Dabic, E. Aghajani, and G. Bavota, “Sampling projects in github for D. Kiela, “Retrieval-augmented generation for knowledge-intensive nlp
MSR studies,” in 18th IEEE/ACM International Conference on Mining tasks,” 2021. [Online]. Available: [Link]
Software Repositories, MSR 2021. IEEE, 2021, pp. 560–564. [43] M. Bohacek and M. Bravansky, “When XGBoost outperforms GPT-4
[22] GitHub. (2024) Fork a repository. Github. on text classification: A case study,” in Proceedings of the 4th
[Online]. Available: [Link] Workshop on Trustworthy Natural Language Processing (TrustNLP
collaborating-with-pull-requests/working-with-forks/fork-a-repo 2024), A. Ovalle, K.-W. Chang, Y. T. Cao, N. Mehrabi, J. Zhao,
[23] D. Spadini, M. Aniche, and A. Bacchelli, PyDriller: Python Framework A. Galstyan, J. Dhamala, A. Kumar, and R. Gupta, Eds. Mexico City,
for Mining Software Repositories, 2018. Mexico: Association for Computational Linguistics, Jun. 2024, pp.
[24] GitHub. (2024) About git cherry-pick. Github. [On- 51–60. [Online]. Available: [Link]
line]. Available: [Link] [44] C. Niu, C. Li, V. Ng, D. Chen, J. Ge, and B. Luo, “An empirical
cherry-picking-a-commit-in-github-desktop comparison of pre-trained models of source code,” 2023.
[25] F. Madeiral, T. Durieux, V. Sobreira, and M. Maia, “Towards an [45] T. Sun, K. Allix, K. Kim, X. Zhou, D. Kim, D. Lo, T. F. Bissyandé,
automated approach for bug fix pattern detection,” arXiv preprint and J. Klein, “Dexbert: Effective, task-agnostic and fine-grained repre-
arXiv:1807.11286, 2018. sentation learning of android bytecode,” IEEE Transactions on Software
[26] H. Li, G. K. Rajbahadur, D. Lin, C.-P. Bezemer, and Z. M. Jiang, Engineering, vol. 49, no. 10, pp. 4691–4706, 2023.
“Keeping deep learning models in check: A history-based approach to [46] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training
mitigate overfitting,” IEEE Access, 2024. of deep bidirectional transformers for language understanding,” 2019.
[27] Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “CodeT5: Identifier-aware [47] T. Lutellier, H. V. Pham, L. Pang, Y. Li, M. Wei, and L. Tan, “Coconut:
unified pre-trained encoder-decoder models for code understanding combining context-aware neural translation models using ensemble for
and generation,” in Proceedings of the 2021 Conference on program repair,” in Proceedings of the 29th ACM SIGSOFT International
Empirical Methods in Natural Language Processing, M.-F. Moens, Symposium on Software Testing and Analysis, ser. ISSTA 2020. New
X. Huang, L. Specia, and S. W.-t. Yih, Eds. Online and York, NY, USA: Association for Computing Machinery, 2020, p.
Punta Cana, Dominican Republic: Association for Computational 101–114. [Online]. Available: [Link]
Linguistics, Nov. 2021, pp. 8696–8708. [Online]. Available: https:

11
[48] Z. Chen and M. Monperrus, “The codrep machine learning on source “Megadiff: A dataset of 600k java source code changes categorized by
code competition,” 2018. diff size,” 2021.
[49] M. Monperrus, M. Martinez, H. Ye, F. Madeiral, T. Durieux, and Z. Yu,

12