LLM-Enhanced Software Patch Localization

Jinhong Yu1 , Yi Chen2 , Di Tang2 , Xiaozhong Liu1 , XiaoFeng Wang2 , Chen Wu3 , and Haixu Tang2
1 Worcester Polytechnic Institute
2 Indiana University Bloomington
3 Microsoft
arXiv:2409.06816v1 [[Link]] 10 Sep 2024

CVE-2019-10131
Abstract An off-by-one read vulnerability was discovered in ImageMagick before version
Open source software (OSS) is integral to modern product 7.0.7-28 in the formatIPTCfromBuffer function in coders/meta.c. A local attacker
may use this aw to read beyond the end of the buffer or to crash the program.
development, and any vulnerability within it potentially com- ImageMagick: cb1214c
promises numerous products. While developers strive to apply Commit
… only ellipsis
security patches, pinpointing these patches among extensive
OSS updates remains a challenge. Security patch localiza-
Figure 1: Unclear Association: CVE and Commit Example
tion (SPL) recommendation methods are leading approaches
to address this. However, existing SPL models often falter with implementing security patches. However, they typically
when a commit lacks a clear association with its correspond- learn about OSS vulnerabilities from public sources such as
ing CVE, and do not consider a scenario that a vulnerability the NVD/CVE vulnerability databases. These sources usually
has multiple patches proposed over time before it has been do not specify the exact location of the security patches, as
fully resolved. To address these challenges, we introduce these patches are often buried within a bundle of OSS updates
LLM-SPL, a recommendation-based SPL approach that lever- (e.g., commits on GitHub). As a result, when developers cus-
ages the capabilities of the Large Language Model (LLM) tomize patches for their projects or craft hot-fixes based on
to locate the security patch commit for a given CVE. More fl

the original patches, they might first have to navigate through

specifically, we propose a joint learning framework, in which numerous OSS modifications to identify the relevant security
the outputs of LLM serves as additional features to aid our fixes, which is a labor-intensive and error-prone process.
recommendation model in prioritizing security patches. Our Challenges in SPL. To address this dilemma, automated tech-
evaluation on a dataset of 1,915 CVEs associated with 2,461 niques have been proposed to locate the security patch for a
patches demonstrates that LLM-SPL excels in ranking patch given CVE, such as using auxiliary information (e.g. CVE-
commits, surpassing the state-of-the-art method in terms of ID) in vulnerability database [19, 30], leveraging the external
Recall, while significantly reducing manual effort. Notably, reference links in the CVE/NVD page [22, 30, 44, 45, 47], and
for vulnerabilities requiring multiple patches, LLM-SPL sig- training a recommendation model based on CVEs and com-
nificantly improves Recall by 22.83%, NDCG by 19.41%, and mits [36, 43]. Among them, the security patch localization
reduces manual effort by over 25% when checking up to the (SPL) recommendation methods notably stand out. However,
top 10 rankings. The dataset and source code are available at since ❶ the content of CVEs and commits is often complex
[Link] and intricate, existing SPL recommendation models cannot
1 Introduction handle cases if the commit has no clear association with the
Open source software (OSS) is now ubiquitous in product corresponding CVE. Figure 1 illustrates an example. The
development. A recent report by Synopsys reveals that 96% commit cb1214c in OSS ImageMagick addresses the “off-by-
of the 1,700 commercial codebases examined across 17 indus- one read” vulnerability, referenced in CVE-2019-10131, by
tries incorporate open source components [35]. Consequently, allocating an additional byte to memory (see code diffs in
a single vulnerability in OSS can compromise the security of Figure 1). Existing SPL methods fail to identify this commit
hundreds or even thousands of subsequent products. A promi- as it offers no descriptive content beyond the ellipsis, leaving
nent example is the Log4J vulnerability in Apache [5], which no shared terms or similar semantics between the commit and
sent shockwaves across the global tech community, prompting CVE to be leveraged. To recognize this commit as a patch
numerous vendors using Apache to urgently deploy security for the vulnerability, one is expected to have an in-depth un-
patches. To address such vulnerabilities from OSS, develop- derstanding of the CVE and commit, and possess security
ers who integrate OSS into their products are often tasked knowledge that associates adjusting memory allocation with

1
 CVE-2014-8275
OpenSSL before 0.9.8zd, 1.0.0 before 1.0.0p, and 1.0.1 before 1.0.1k does not enforce certain constraints on certi cate data, which allows remote attackers to defeat a ngerprint-based certi cate-blacklist
protection mechanism by including crafted data within a certi cate's unsigned portion, related to crypto/asn1/a_verify.c, crypto/dsa/dsa_asn1.c, crypto/ecdsa/ecs_vrf.c, and crypto/x509/x_all.c.

OpenSSL: 684400c OpenSSL: cb62ab4

Fix various certi cate ngerprint issues. use correct function name
By using non-DER or invalid encodings outside the signed portion of a certi cate the Reviewed-by: Rich Salz <rsalz@[Link]>
ngerprint can be changed without breaking the signature…. Reviewed-by: Matt Caswell <matt@[Link]>
CVE-2014-8275
Reviewed-by: Emilia Käsper <emilia@[Link]>

Figure 2: Example of a vulnerability fixed by multiple patches collaboratively.

a common remedy for the "off-by-one read" issue. Addition- ranked the patches for 92.74% CVEs within the top 10 po-
ally, oftentimes, a vulnerability has multiple distinct patches sitions, simultaneously delivering high-quality rankings as
proposed over time before it has been fully resolved, a sce- evidenced by the NDCG metric, which reached a high value
nario not considered by existing SPL solutions. As shown of 87.33%. This performance enables low manual effort, re-
in Figure 2, the commit cb62ab4 is a follow-up to the com- quiring the checking of only an average of 2.34 commits per
mit 684400c, which patch the OSS OpenSSL’s vulnerability CVE. When compared to the state-of-the-art SPL method,
CVE-2014-8275, as can be easily identified from its commit VCMatch, LLM-SPL consistently outperformed in all metrics
description. However, commit cb62ab4 has almost no direct – Recall, NDCG, and Manual Effort. Particularly for CVEs re-
association with the CVE in its content, thereby it would not quiring multiple collaborated patches, LLM-SPL significantly
be identified by existing SPL methods. We believe this is- improved Recall from 60.30% to 83.13% (a 22.83% increase),
sue can be addressed by incorporating the relations between enhanced NDCG rom 60.99% to 80.40% (a 19.41% increase),
the two commits into the model training. But, determining and reduced manual effort by over 25% when checking up
fi
fi
fi
fi
fi
fi
fi
fi
the inter-commit relationships poses another significant chal- to the top 10 rankings in the practical application of LLM-
lenge (❷). SPL. These results clearly demonstrate the effectiveness of
Our method. Addressing these challenges in SPL requires our LLM-SPL approach.
extensive knowledge of software and security, especially Contributions. The contribution of this paper is a twofold:
vulnerabilities, as well as adept handling of semantic infor- • New Technique. We proposed an innovative SPL solution,
mation from both natural language descriptions and pro- LLM-SPL, which leverages the intelligence of large language
gramming code. In some ways, Large Language Models models (LLMs) in comprehending CVEs, commits, and spe-
(LLMs) have demonstrated owing the relevant capabili- cialized knowledge in the field of software security. This
ties [2, 4, 7, 10, 12, 28, 29, 37]. Our experiments, in particular, solution efficiently and economically integrates insights from
revealed that LLM excels in comprehending CVEs and com- LLMs through a joint learning framework, utilizing them as
mits, notably by the high recall in recognizing their relation- additional features and feedback to refine the outcomes of a
ships. However, directly applying LLM to solve SPL receives recommendation model for SPL.
skepticism, evidenced by its high false positive rate (>85%), • Performance Enhancement. The evaluation results validate
which we will delve deeper into in Section 4.3. Therefore, LLM-SPL’s superior performance, consistently outperform-
to adequately leverage LLM’s capabilities, we proposed a ing the state-of-the-art in all metrics—Recall, NDCG, and
joint learning approach, in which the outputs of LLM serve Manual Effort. Notably, for vulnerabilities requiring multiple
as additional features to aid the recommendation model in pri- patches, LLM-SPL significantly improves Recall by 22.83%,
oritizing security patches. Specifically, our recommendation NDCG by 19.41%, and reduces manual effort by over 25%
model leverages two LLM-based features: LLM’s prediction when checking up to top 10 rankings.
of the relationship between CVEs and commits, and the LLM-
2 Background
endorsed inter-commit relationship graph of commits. In the
meanwhile, to make the cost of using LLM practical, we re- 2.1 Vulnerability and Patch
fined our recommendation algorithm by the LLM-feedback Common Vulnerabilities and Exposures. CVE is a globally
technique. This technique necessitates seeking LLM feedback acknowledged catalog detailing known cybersecurity vulnera-
solely for commits positioned within the top-k ranks, and suc- bilities. Every entry in the CVE comprises details of a specific
cessfully helped us drastically reduce costs from an initial vulnerability, encompassing a unique identifier (CVE-ID), an
estimate of 620,000 USD and a century-long processing time in-depth description of the vulnerability, essential references
using the GPT-3.5 model, down to a mere 880 USD with a pro- that often cover bug reports and information about patches
cessing duration of only 3 days. We name this LLM-enhanced and so forth. Figure 10 in Appendix illustrates an example.
SPL approach as LLM-SPL. For simplicity, we will use “CVE” to denote a CVE entry in
Results. We implemented LLM-SPL and evaluated its per- the following paper.
formance across a dataset of 1,915 CVEs with 2,461 asso- Open source software patch. Vulnerabilities in an OSS are
ciated patches. The results show that LLM-SPL effectively addressed with patches, which involve changes to certain code

2
segments. Typically, on GitHub [3], these changes are made Recommendation Recommendation
Feature Model θ1 Ranking Model θ2 Ranking′
into commits that are subsequently pushed into the OSS’s
repository. In a patch-related commit, there are code diffs that Feedback
Figure 3: One-iteration feedback recommendation process.
highlight changes made on code and indicate the specific file
being modified, together with the metadata including the au- introduce noise.
thor of this commit, the timestamp of this commit’s creation, Figure 3 illustrates one-iteration of feedback recommen-
tags representing the OSS’s version, and the commit-id of the dation process, where the feedback is from users in RF or
latest prior commit. Additionally, a commit contains a “mes- assumption on top k results in RPF. Given the scarcity of
sage” section, referred as the commit description, which might domain experts and the high costs associated with their en-
elucidate the specific vulnerability it seeks to resolve. We will 
deavor, employing RF to tackle the SPL challenge is nearly
use “patch” to denote a patch-related commit in following impractical. Meanwhile, relying on judgements from PRF
paper. An example is shown on Figure 11 in Appendix. poses a significant risk due to the prevalence of false posi-
2.2 Security Patch Localization and VCMatch tives in the top rankings produced by SOTA methods. In our
Security patch localization (SPL) is the task of pinpointing research, we proposed an approach, utilizing feedback from
commits within an OSS’s repository that serve as patches LLM, offers an invaluable, affordable and reliable solution
for addressing the vulnerability specified in a particular CVE. for this precarious situation (see Section 5.1).
Currently, the state-of-the-art (SOTA) SPL approach is VC- 3 Challenges in Effective SPL
Match [43], as supported by our experimental results in Sec- Recent advances in SPL have utilized recommendation
tion 6. It forms the SPL as a recommendation problem, thus models, drawing on the content of CVEs and commits for
prioritizing patches with a higher ranking over non-patch com- model training, with the aim of identifying the commits that
mits. The recommendation algorithm employed by VCMatch most likely address the vulnerabilities specified in given
utilizes five kinds of features: code behavior features, com- CVEs [36, 43]. The efficacy of these models is inherently
mit message identifiers, textual similarity features, security tied to their ability to comprehend such content. However, the
relevance features and temporal features. A comprehensive complexity and intricacy of CVE and commit content presents
breakdown of these features can be found in Table 4 at Ap- a significant challenge. Furthermore, certain vulnerabilities
pendix. necessitate multiple distinct patches over time to achieve full
2.3 Machine Learning resolution, a scenario inadequately addressed by current SPL
solutions. We contend that incorporating the relationships
Large language model. Large Language Models (LLMs) are
among commits into the model training can effectively re-
advanced artificial intelligence systems capable of understand-
solve this issue. Nevertheless, discerning these relationships is
ing and generating human-like text. These models not only
far from straightforward, presenting another challenge. Below,
exhibit exceptional proficiency in processing and producing
we will detail the two key challenges.
both natural language and code [2, 8, 11, 20, 31, 42, 46, 48], but
also possess extensive domain knowledge, including expertise 3.1 Challenge 1: Complexity of Content
in software and security [12, 18, 28]. Understanding or representing the content of CVEs and com-
Such capabilities suggest that LLMs have the potential mits is crucial for success of SPL, as it not only facilitates the
to comprehend the content of CVEs and commits, as well precise mapping of commits to their corresponding CVEs but
as discern their relationships, making them invaluable for is also essential for the effective feature extraction that is vital
supporting the SPL task. However, LLMs are not without for training models. However, the content of both CVEs and
their limitations; they occasionally produce inaccuracies and commits is inherently complicated, demanding specialized
hallucinations, which can compromise their reliability [15]. In knowledge for accurate comprehension.
Section 4, we will discuss in detail both the potential and the CVE. As introduced in Section 2.1, a CVE is purely textual,
limitations of LLMs within the SPL context. Subsequently, in detailing vulnerability information. To find what specific in-
Section 5, we will introduce our methodology for leveraging formation CVEs actually provided, we carefully reviewed
LLMs to enhance SPL. 100 randomly selected CVEs and summarized the types of in-
Relevance/Pseudo-relevance feedback for recommenda- formation encountered. These findings, along with examples
tion. In information retrieval, feedback mechanisms are cru- of how they are presented by CVEs, are shown in Table 5 in
cial for refining search precision. Relevance feedback (RF) the Appendix. The content within CVEs, as illustrated in the
[32, 33] involves users in the retrieval process, enhancing table, offers a wealth of information related to vulnerabilities,
results based on their input about relevant documents. This which can be categorized into four main areas: software in-
iterative process improves search results but at a high cost. formation, vulnerability information, attack information, and
Pseudo-relevance feedback (PRF) [21, 34] automates this pro- patch information. Each category is further divided into sub-
cess by assuming the top k results from an initial search are categories. For example, the vulnerability information cate-
relevant. While this can enhance ranking performance, it may gory may detail the type of vulnerability, the software’s faulty

3
functionality, and the names of files affected by the vulnerabil- understanding demands not only programming knowledge
ity, among others. Similarly, the attack information category and familiarity with the affected software’s architecture and
might cover the payload of an attack packet, the method of design but also a deep understanding of the context and ratio-
attack, and the attack’s impact. nale behind the changes. More specifically, it is essential to
However, understanding the information contained in a understand how a specific modification interacts with other
CVE is challenging. It demands not only knowledge of soft- code segments and addresses particular issues. Acquiring
ware security but also, in many cases, familiarity with the such comprehensive knowledge is challenging, necessitating
specific software affected by the vulnerability. For instance, advanced programming skills and specialized training in soft-
without a background in security, one might not understand ware security.
that integer overflow, as documented in CVE-2012-2386,
refers to a vulnerability type. Similarly, without knowledge
3.2 Challenge 2: Inter-Commit Relations
An analysis of all current CVEs reveals that 21.04% of them
of the Linux kernel, which carries a vulnerability documented
require multiple patches for complete resolution. This obser-
in CVE-2020-11608, it becomes difficult to recognize that
vation is understandable, as fully addressing an identified vul-
ov511_mode_init_regs and ov518_mode_init_regs are the
nerability sometimes necessitates several distinct patches over
names of the files affected by the vulnerability. Classical text
time. For instance, additional or subsequent patches may be
learning methods, without dual knowledge in both security
required in situations where an initial patch fails to completely
concepts and the affected software, can hardly generate high
address the vulnerability or introduces regression errors, or
quality CVE/commit representations for effective recommen-
when more efficient or effective solutions are later developed.
dations.
Moreover, in cases where a vulnerability impacts various
Commit. Compared to CVEs, commits feature a more com-
components across a system, every affected component might
plex content structure, including not only a textual description
require its own patch to ensure full protection.
that notes the changes but also code diffs, which are the actual
modifications made to the source code, as we introduced in Unfortunately, existing SPL approaches do not work well
Section 2.1. To find the specific information offered by com- in these situations, where one vulnerability is associated
mit content – similarly to how we approached CVE content – with multiple patches, which we refer to as “1-N” in this
we carefully reviewed 100 commits, randomly selected from paper. In our test set, nearly 40% of 1-N CVEs1 are not
50 GitHub repositories. This review focused on both textual fully matched with their respective patches using the cur-
descriptions and code diffs. The diverse types of information rent SOTA SPL method [43]. We attribute this issue to the
encountered, along with examples, are detailed in Tables 6 fact that existing SPL studies establish the association be-
and 7 in the Appendix. tween a commit and a CVE based solely on the content of
each commit and CVE, relying on a straightforward one-to-
From Table 6, we can see that the textual descriptions
one relationship for determination. Consequently, patches
within commits closely align with the types of information
that do not exhibit a clear relationship a CVE are often over-
found in CVE content, including but not limited to the vulner-
looked by these studies. However, this phenomenon is par-
ability type, attack impact, and affected files. Moreover, we
ticularly prevalent in 1-N situations, where we observed that
observed that commit descriptions often provide information
many patches lack a clear relationship with their correspond-
about fixes, such as fixing strategies or methods, as exempli-
ing CVEs. Take, for example, the patch with commit ID
fied in Table 6. This is logical, as their purpose is to describe
cb62ab4 for OpenSSL, one of several patches collectively
or explain the modifications made. Understanding commit
addressing the vulnerability CVE-2014-8275 (refer to Fig-
descriptions, akin to understanding CVE content, requires not
ure 2). This patch contains a brief description: “use correct
only knowledge of security concepts but also a deep familiar-
function name ...” and involves a one-line code modifica-
ity with the affected software’s source code, especially details
tion, replacing the parameter “ASN1_F_ASN1_VERIFY” with
related to fix information. This may present a challenge that,
“ASN1_F_ASN1_ITEM_VERIFY”. Clearly, the content of this
in some aspects, is even more substantial than understanding
patch seems unrelated to the content within the CVE, making
CVEs.
it challenging for even human experts to identify this com-
As illustrated in Table 7, code diffs also convey diverse and mit as a patch for the CVE based solely on their individual
rich information relevant to vulnerabilities. While some code content.
comments, written in natural language, clarify the purpose
Upon revisiting commit cb62ab4, we discovered that to
of modifications and functionalities of the modified code, the
fix the vulnerability CVE-2014-8275, this commit works in
majority of code diffs comprise source code changes. These
conjunction with another commit, 684400c. The latter con-
changes not only pinpoint the exact location of modifica-
tains numerous clues indicating its role as a patch for CVE-
tions but also provide clear insights into the functionalities
2014-8275, even with the term “CVE-2014-8275” directly
of specific code segments. Indeed, comprehending source
code is widely recognized as challenging [13]; however, un- 1 For clarity, a 1-N CVE refers to a CVE associated with multiple patches
derstanding code diffs presents even greater difficulties. This in the 1-N situation.

4
mentioned in its description, as shown in Figure 2. Therefore, 4.1 LLM Potential for Comprehension
if relationships like the one between 684400c and cb62ab4 are
identified (as highlighted in blue in Figure 2), and such con- The advancements in LLMs have demonstrated their near-
nections are considered during the model’s learning process, human performance across a broad range of natural language
it is likely that the model would not only recognize commit processing (NLP) tasks, such as entity recognition [20, 48],
684400c as a potential patch for CVE-2014-8275 based on information extraction [42, 46], and semantic understand-
their direct content association (as shown in green in Fig- ing [11, 31]. These capabilities are precisely what are needed
ure 2) but also deduce a high probability of commit cb62ab4 to comprehend the textual descriptions within CVEs and
being another patch for this CVE, taking into account the commits, as shown in Table 5 and 6. For example, entity
inter-commit relationships (as indicated in red in Figure 2). recognition enables the identification of key elements, such as
Based on this insight, we believe that identifying the inter- software names, vulnerability types, and erroneous function
commit relationships and incorporating them into the model’s names. Information extraction facilitates the gathering of de-
learning process would be a viable solution for addressing tailed descriptions about faulty functionalities, attack methods,
1-N situations. and fix strategies, among others. And semantic understanding
In our examination of the gathered 1-N data, we witnessed allows for the comprehension of the inherent implications
a strong linkage between the content of these commits, as behind these texts. When it comes to code, LLMs have also
evidenced by both the code diffs and the commit descrip- demonstrated remarkable proficiency in code-related tasks,
tions. Specifically, our investigation into the code revealed such as code comprehension (understanding the logic and
that these 1-N commits often modify the same sections of the functionality of code segments), code summarization (cre-
codebase, focus on identical or closely related functionalities, ating concise descriptions of code functionality and its pri-
or utilize similar modification patterns in various locations. mary purpose), and code-to-comment alignment (associating
In the commit descriptions, we observed not only similar- code parts with their corresponding comments or descrip-
ities in information related to vulnerabilities, attacks, and tions) [2,8]. Such capabilities are crucial for a thorough under-
fixes, as categorized in Table 6, but also additional details standing of code diffs, enabling comprehension of modified
indicating connections among commits. For example, some code functionalities and inference of the intentions behind
commits share the same author or address the same issue ID, these changes. Particularly, as a generalized model, LLM pos-
or one commit directly references another as a supplement sesses extensive domain knowledge including security and
or complement. However, understanding and identifying this software. This is showcased by LLMs’ success in recogniz-
information to build the inter-commit relationships demands ing various types of vulnerabilities [28], identify malicious
extensive security knowledge and deep familiarity with the or suspicious behaviors within code [12], and comprehend
software, which, as previously discussed in Section 3.1, is textual bug reports [18]. Given these capabilities, we are con-
significantly challenging. vinced that LLMs hold significant potential for enhancing the
comprehension of the content within CVEs and commits.

4 A Potential Solution: Large Language Model In addition, we conducted experiments using a real-world
LLM model, specifically GPT-3.5, to empirically evaluate the
As discussed in Section 3, improving SPL recommendation LLM’s capability in understanding CVEs and commits. We
methods fundamentally hinges on two challenges: understand- presented GPT-3.5 with 25 CVE and 25 commit samples, ask-
ing the content within CVEs and commits and recognizing the ing it to elucidate their contents. The results were compelling.
relationships among pieces of information in these contents. The LLM not only demonstrated a deep and accurate under-
Our analysis, detailed in Section 3, reveals that overcoming standing of the technical details within each sample but also
these challenges necessitates a set of closely related compre- successfully grasped their wider implications and context. For
hensive abilities: processing natural language, interpreting instance, when we presented the CVE and commit illustrated
code, and possessing in-depth dual knowledge in the domains in Figure 1 to the LLM, it accurately interpreted the term
of security and software. Emerging Large Language Model “off_by_one read” in CVE-2019-10131 as a vulnerability type,
(LLM) technologies appear particularly well-suited to meet explaining it as related to a buffer issue: “In an off_by_one
these demands. LLMs exhibit exceptional proficiency and read vulnerability, the program incorrectly reads data from
accuracy in comprehending both natural language and code, a buffer, by either accessing data beyond the buffer’s bounds
as well as possessing extensive domain knowledge, includ- or ...”. Regarding the commit cb1214c, the LLM effectively
ing in security and software. Consequently, we believe that inferred the rationale behind the code modification to adjust
LLMs hold significant potential to address these challenges memory allocation, suggesting that “Such a change likely
for enhancing SPL methods. In following sections, we will indicates that there were some memory boundary issues in
discuss both the potential and the limitations of LLMs in the the original implementation.” These empirical results further
SPL context. strengthen our belief in the LLM’s potential to comprehend
the contents of CVEs and commits, thereby enhancing SPL.
5
4.2 LLM Potential for Relation Recognition exceeding 85%.
Furthermore, in recognizing relationships among pieces of This high false positive rate of 85% indicates that for every
information within CVEs and commits, we believe that LLMs 10 patches identified by the LLM, more than 8 are incorrect.
have already mastered these connections, thanks to their inher- In extreme cases, such as for CVE-2014-8545, the LLM iden-
ent strengths in pattern recognition and their diverse training tified as many as 38 commits as its patches, with 37 of them
data. Pattern recognition refers to identification of contex- being incorrect, pushing the false positive rate above 97%.
tual connections within data, which is achieved by attention This significant high false positive rate necessitates substan-
mechanism [39] in LLMs. This mechanism enables LLMs tial manual effort from experts to sift through and identify the
not only to recognize recurring structures but also uncover actual patches. To understand why the LLM errs in identify-
connections between seemingly unrelated pieces of informa- ing patches for CVEs so frequently, we analyzed 50 samples
tion. Moreover, the diversity of LLMs’ training data, which randomly selected from our dataset. Our findings suggest that
includes both textual content and code [37], empowers LLMs these false positives may stem from the LLM’s excessive
to forge relationships between text and code. Thus, given the reliance on, or sensitivity to, specific patterns between a vul-
extensive training set of LLMs – a well-established fact – it is nerability and its patch. For instance, commit d4fdceb was
likely that LLMs have encountered and learned from patterns mistakenly identified as a patch for CVE-2016-2555, a SQL
akin to those in CVEs and commits, perhaps not directly from injection vulnerability in Atutor software. This error likely
these sources but in similar forms, multiple times during their occurred because the commit includes code for sanitizing
training. Consequently, we believe that LLMs have already a POST string, resembling an action to sanitize SQL input.
acquired the ability to discern patterns and establish these However, this change is unrelated to the vulnerability.
relationships among CVEs and commits. Additionally, the LLM missed 320 commits (false nega-
To further validate our hypothesis, we conducted tests on tives). To investigate the underlying reasons, we analyzed 50
LLM (specifically GPT-3.5) to assess its ability in recogniz- random samples from these overlooked commits. What we
ing the relationships between CVEs and commits, as well as observed was that the content of these commits lacks a clear
among commits themselves. Specifically, we presented LLM association with the CVEs they actually patch, as demon-
with the content of 25 CVEs and their associated patches, strated by the relationship between CVE-2014-8275 and its
asking whether the commits serve as patches for these CVEs; patch commit cdb2ab4, illustrated in Figure 2. However, we
and we also provided LLM with 25 pairs of commits, inquir- discovered that 26 of these commits were part of collaborative
ing if they collaboratively addressed the same vulnerability. efforts with other commits to patch their associated CVEs.
The results were encouraging. LLM not only made correct Notably, these collaborating commits had already been suc-
judgments in the majority of cases 22 out of 25 for the first set, cessfully identified by the LLM as patches for the CVEs. In
19 out of 25 for the second set) but also provided reasonable the meanwhile, it was possible to establish the inter-commit
and logical explanations for its decisions. Such experimental relationships between these 26 commits and their collaborat-
outcomes reinforce our belief that LLM has indeed learned ing commits through their content. This observation aligns
these correlations, showcasing its potential as a valuable tool with the 1-N scenario we discussed in Section 3.2, reinforcing
for establishing inter-commit relationships to enhance SPL our idea that inter-commit relationships could help rectify
recommendations. these missed instances. Given these insights, one may won-
der why we cannot simply adjust the prompt to include all
4.3 LLM Alone is Not Enough commits of the affected software along with the CVE, aiming
Given the LLM’s potential to comprehend CVEs and com- to fully leverage LLMs for this purpose. However, due to the
mits content and its capability to identify relationships among input constraints of LLMs, such an approach is impractical.
these pieces of information, one might wonder why not di- Each interaction with an LLM is limited by input length and is
rectly apply LLMs to SPL by testing each commit of the independent of others [7]. For example, the GPT-3.5 we used
affected software to see if it serves as a patch for a CVE. restricts a prompt to no more than 4,097 tokens [29]. Con-
To assess the effectiveness of this straightforward approach, sidering the substantial size of commits, which often include
we conducted experiments with GPT-3.5, using our collected extensive code diffs, inputting multiple commits into the LLM
dataset (for details of data collection, see Section 6.1). For the simultaneously becomes challenging, if not impossible.
1,915 CVEs in our dataset, we selected 50 commits for each, Our experiments and analysis indicate that directly using
including the actual patches, from the affected software. We LLMs for SPL is inadequate, because the LLM has an exces-
then formed a distinct CVE-commit pair for every individual sively high false positive rate exceeding 85%, along with the
commit, resulting in a total of 95,750 (1,915*50) pairs, which omission of certain commits that utilizing inter-commit rela-
we used as input for the LLM. The experimental results, as tionships could potentially rectify. Nevertheless, the LLM’s
depicted in Table 8 in the Appendix, reveal that directly ap- recall rate of 87% highlights its potential, specifically in iden-
plying LLMs to SPL is impractical: although the recall rate is tifying and establishing relationships between CVEs and com-
reasonable at 87%, the false positive rate is alarmingly high, mits.

6
 Ranking

F0
Numerical Encoder embedding
F1 Fusion Gate Ranking
F2 Graph Encoder embedding

F0 numerical
the number of these queries is on the order of O(nm), total-
Numerical Encoder
F1 embedding
Fusion Gate Ranking
graph ing 3 million in our dataset. Assessing whether two commits
F2 Graph Encoder
embedding collaboratively address a vulnerability by GPT-3.5 requires
Figure 4: Design of joint learning framework. a slightly higher cost of 0.003 USD per query, due to more
tokens typically in commits than CVEs. These queries scale
5 LLM-SPL: Design and Implementation with O(km2 ), totaling 206 million in our dataset. Cumula-
5.1 Design tively, the estimated total expense for our dataset exceeds
As discussed in Section 4, relying solely on LLMs to de- 620,000 USD. Regarding computational cost, each query,
termine CVE-commit relationships proves inadequate and on average, takes 15 seconds to process in GPT-3.5, lead-
impractical. However, LLMs’ impressive capabilities in com- ing to time costs of O(nm) + O(km2 ). For our dataset, a lone
prehending CVE and commit content, along with identifying user would take about “a century” to complete all queries
their complex relationships, align perfectly with the needs for if working without any interruption2 . Notably, our dataset
addressing the challenges SPL currently faces. Consequently, is relatively modest, only related to 200 software including
to leverage such capabilities for SPL, we proposed an LLM- 1,915 CVEs and a maximum of 1,500 commits for each soft-
enhanced approach, which, within a joint learning framework, ware (as detailed in Section 6.1). Yet, the estimated costs have
augments the SPL recommendation method by considering already reached a prohibitive level. Any further expansion
LLM’s judgement on CVE-commit relationships and LLM- of the dataset for enhanced performance would quadratically
endorsed inter-commit relationships. Specifically, in training increase associated costs, making the application of LLMs
our model, besides using all the features from the SOTA SPL untenable.
method (denoted as F0 ) to preserve existing performance lev- Solution. To address this cost challenge of using LLMs, we re-
els, we incorporate two additional sets of features endorsed ferred to pseudo-relevance feedback recommendation, where
by the LLM: ❶ the LLM’s determination of whether a com- the main concept is to annotate the top k rankings from an
mit patches a CVE (denoted as F1 ), and ❷ the inter-commit initial model and then use these annotations as feedback to
relationship graph established with the assistant of the LLM refine the model (see Section 2.3). Inspired by this, we uti-
(denoted as F2 ). lized F1 and F2 as LLM-feedback to refine the SPL recom-
Joint learning. Considering the diversity of features F0 , F1 , mendation model, but we limited LLM queries for F1 and
and F2 – with F0 and F1 presented in structured numerical F2 to only the top k ranked commits from the initial SPL
formats and F2 in an unstructured graph-based format – and method, so that we can avoid a vast number of LLM queries
stemming from different data modalities (notably, F0 cap- for those ranked lower. This strategy not only significantly
tures the semantic meaning of texts, while F1 and F2 describe reduces costs but also effectively leverages LLM insights to
the relationships between objects) – our design utilized joint optimize the model’s ranking. Notably, regarding F2 , an inter-
learning. This method is well-suited for integration of varied commit relationship graph, as discussed in Section 3.2, it is
feature representation and multi-modality features, ensuring designed to elevate commits that might rank low individu-
our model effectively learns from these heterogeneous sets ally but have strong ties to higher-ranked ones. This means,
of features. The primary idea behind joint learning for han- for an inter-commit relationship to be impactful, at least one
dling heterogeneous features is individually encoding each commit within it needs to be highly ranked. Thus, building
feature type and then strategically fusing them into a cohesive a graph consisting solely of the top k ranked commits is not
representation. Based on this, we designed our joint learn- just a cost-saving measure but also a practical solution which
ing framework, as illustrated in Figure 4. This framework eliminates the effort of forming unnecessary connections in
includes two specialized encoders tailored to their respective the graph. In our experiments, instead of an estimated cost of
feature types: a Numerical Encoder for F0 and F1 , and a 620,000 USD and a 100-year time span on our dataset, our
Graph Encoder for F2 . The outputted embeddings are then method spent only 880 USD and just 3 days for a single user
blended by a Fusion Gate component, fusing the features to complete the task.
into a singular score that represents the likelihood of a commit Architecture. Figure 5 depicts the architecture of our
being a patch for a CVE. This score subsequently informs the method, named LLM-SPL (LLM-Enhanced Security Patch
ranking of commits. Further details on the two encoders and Localization). The process is divided into three steps: 1) ob-
the fusion gate are elaborated in Section 5.3. taining F1 (indicated in blue); 2) establishing F2 (marked in
Challenges. However, during the model training with our red); and 3) training the joint learning model (highlighted
dataset, we encountered prohibitive costs in both financial in green). Specifically, in the first step, we use the LLM to
and temporal aspects when obtaining features F1 and F2 from label each of the top-ranked commits from an initial ranking
the LLM. Financially, querying the LLM, GPT-3.5, that we (Ranking0 ), which is generated by the base SPL recommenda-
used to determine whether a commit patches a CVE incurs tion model, as either patching a specific CVE or not, thereby
an average cost of approximately 0.002 USD. For a dataset 2 Other open-source LLM solutions, e.g., Llama, also have these compu-
of k software including n CVEs and m commits per software, tational cost challenges.

7
 ⊕
topk topk
F0 base SPL θ1 Ranking0 base SPL θ2 Ranking1 joint learning
LLM LLM model ϕ
Ranking2

F1 F2

Note: The detailed design of the joint learning model is illustrated in Figure 4.
Figure 5: Architecture of LLM-SPL.
obtaining the LLM-feedback F1 . Notably, since a better ini- engineering practices, we inform the LLM of its role as a
tial ranking enhances the final outcome of the model in the software security analyst, highlighted in olive in the figures,
pseudo-relevance feedback, in our implementation, we uti- to leverage the LLM’s expertise in software security. Then,
lized the SOTA SPL model, VCMatch [43], as our base SPL. we provide the LLM with a CVE and a commit (for F1 ) or
The F0 in Figure 5 represents all features used by VCMatch two commits (for F2 ), highlighted in green in figures. The
(introduced in Section 2.1). In the second step, for reasons task description, highlighted in red, instructs the LLM to an-
similar to why we used the SOTA as the base SPL in Step 1, alyze content and determine relationships. For F1 , the LLM
we hope the initial ranking used for establishing the graph F2 accesses if the commit serves as a patch to the CVE. For
is also as optimal as possible. Considering that we already F2 , the LLM judges if two commits collaboratively address
have feedback F1 , which can potentially enhance a model’s the same vulnerability. To ensure clear comprehension of
ranking capabilities, thus, we take this feedback F1 as the in- the LLM’s decision, our prompts (highlighted in blue) direct
put with F0 through concatenation operation to train the base the LLM to conclude with a single line stating YES, NO or
SPL (VCMatch, again) to produce the initial ranking for this Unknown. This structure allows us to efficiently extract the
phase (denoted as Ranking1 ). After that, we establish inter- LLM’s determination for F1 and F2 through simple string in-
commit relationships specific to the top-ranked commits from spection. Notably, when the LLM returns an UNKNOWN
Ranking1 , to finally construct the graph F2 . Finally, in the decision, considering its high recall rate and associated high
third step, incorporating three features F0 , F1 and F2 , we train false positive rate as discussed in Section 4.3, we treat such
the joint learning model, as illustrated in Figure 4, to deliver determinations as equivalent to NO.
the ultimate commit ranking (Ranking2 ) as our LLM-SPL ’s Generation of F1 . As introduced in Section 5.1, the feature
output. F1 , serving as LLM-feedback, represents the LLM’s deter-
Notably, in our architecture, why we first obtain F1 before mination of whether each of the top k commits in Ranking0
F2 is, once again, for saving costs. This design is based on two patches a particular CVE. In our model, we use a k × 1 vector
primary considerations: 1) As just analyzed, procuring F2 en- to represent it, where k specifies the number of top-ranked
tails considerably greater costs than obtaining F1 ; 2) Given the commits considered. Each entry in the vector is set to 0 if the
principle behind the effectiveness of F2 , as discussed in Sec- LLM determines the commit does not patch the CVE, or 1
tion 3.2, optimizing the quality of the initial ranking (refer to if it does. To construct the F1 vector, for every CVE in our
Ranking1 in Figure 5) makes it possible to create a practically dataset, we query the LLM with k prompts (see Figure 12 in
useful inter-commit relationship graph with fewer top-ranked Appendix) regarding the top k commits from Ranking0 , ask-
commits that is just as effective as one using broader set of ing for the LLM’s determination of their relevance to patching
commits. Therefore, our strategy is to first get F1 and use it the CVE. The responses from the LLM are used to populate
as a feedback to optimize the model, aiming to generate an the corresponding entries in the F1 vector.
improved ranking as the foundation for constructing F2 . Then, In our experiment, we set k = 100. To find the optimal
when LLM-SPL constructs the F2 graph based on this rank- value for k, we tested various numbers to see how often at
ing’s top k commits, the k can be chosen as a relatively small least one patch for CVEs being ranked within top k by the
number. This strategy, while preserving the effectiveness of base SPL model. Here, we considered only one patch because
F2 , offers significant cost savings for building the graph. we believe F2 would rectify any other commits that jointly
address the CVE. The experimental result is illustrated in
5.2 Feature Generation based on LLM
Figure 6. As it demonstrates, we observed an inflection point
As defined in Section 5.1, features F1 and F2 refer to the at k = 100, where the recall of CVEs reached 98.12%. In-
LLM’s determination of whether a commit patches a CVE and creasing k beyond this results in negligible improvements
whether two commits jointly address the same vulnerability. in detection. Thus, we chose k = 100 in our experiment to
In the following, we detail the prompts we used to generate balance identification with overall costs.
F1 and F2 in LLM-SPL , and elaborate on their generation Generation of F2 . As introduced in Section 5.1, the feature
during model learning. F2 serves as another LLM-feedback, representing the inter-
Prompt templates. commit relationship graph endorsed by the LLM, which per-
Figure 12 and 13 in Appendix illustrate the prompt tem- tains to the top k commits in Ranking1 . In our study, F2 is
plates we used to query the LLM for F1 and F2 . Each prompt formally defined as a graph G : {V, E}, where V represents
consists of four parts. Initially, Following established prompt a set of nodes, each node representing a commit, and E rep-

8
 ×106
95% 100%
91.32% 95.04%
100% 95%

Query Number
98.12% 85% 1.0 90%
98%

Recall
σ

Recall
85%
536,413
Recall

96% 75% 0.5 80% DNN

75%
94%
65% 0.0 70%
92% 0 4056 80 120 160 200 0 5 8 10 15 20 25
Time Interval Top k
0 50 100 150 200
Top k sparse features dense features

Figure 6: The recall of Figure 7: The recall of 1- Figure 8: The recall of

CVE with at least one patch N CVEs and the number of CVE with at least one Figure 9: Numerical encoder.
ranked in top k on Ranking0 . LLM queries over the time in- patch ranked in top k on
terval. Ranking1 .
resents a set of undirected edges, each edge representing a commit u, s(Vu ) is the size of the set Vu . Upon querying, if the
relationship between a pair of commits. For any given edge response from the LLM is YES, indicating the two commits
e = {u, v} which connects nodes u and v, the edge weight is u and v jointly address a vulnerability, we assign a weight of
assigned a value of 1 if the LLM determines that commits 1 to the edge e = {u, v}; otherwise, it is set to 0.
u and v collaboratively address the same vulnerability, or 0 Furthermore, during the graph building process for F2 , we
otherwise. set k = 8. This choice comes from the similar analysis as
When constructing the graph G, a direct approach is to pair we did to find the optimal top k for obtaining F1 (as referred
each top k commit from Ranking1 with every other commit in to Figure 6). Similarly, we tested with various values of k,
the software and query the LLM to determine if they collabo- assessing how many CVEs had at least one of their patches
ratively address the same vulnerability. This method, for our ranked within top k in Ranking1 . The result is illustrated
dataset, would require approximately 9 million LLM queries, in Figure 8. This figure reveals an inflection point at k =
still leading to substantial costs (refer to Section 5.1). To miti- 8, achieving a recall of 95.04% for the CVEs. Any further
gate this, we proposed a strategy to further reduce the number increase beyond k = 8 results in only marginal improvements
of queries to the LLM. Specifically, we only ask the LLM in coverage. Thus, to balance the identification efficacy with
when a pair of commits satisfies the following conditions: 1) overall costs, we selected k = 8 in our experiment.
sharing the same author or committer; and 2) being submit- 5.3 Joint Learning Model
ted within a specific time frame. This strategy comes from Figure 4 illustrates our design of the joint learning framework.
our observations about the common practices of collaborated We use a numerical encoder for F0 and F1 , a graph encoder for
patches. In our study, we observed that they are typically han- F2 , and then a fusion gate to combine embeddings from these
dled by the same individual, either the author, who designs encoders. The resulting score represents the probability of a
the patch and crafts the commit, or the committer, who inte- commit being a patch for a CVE, which informs LLM-SPL’s
grates the commit into the software. Additionally, they are recommendation ranking.
often consecutively applied in the software. Thus, using this Numerical Encoder. We utilize this encoder to merge shallow
strategy, we query the LLM for the commit pairs that are more numerical features: F0 and F1 , creating a consolidated embed-
likely to address the same vulnerability, thereby achieving ding to enhance subsequent ranking outcomes. Specifically,
cost savings. Notably, the information needed to determine F1 is a sparse feature with values of either 0 or 1 (commit-
if two commits meet these two conditions is presented in a relevance feedback from LLM), and F0 encompasses 32 di-
standardized format within commit metadata (see Figure 11), verse features that could be dense or sparse. All features
making it both easily accessible and straightforward for as- within F0 and their respective types are detailed in Table 4.
sessment. Moreover, for the second condition, we set the time For sparse features, cross-product transformations efficiently
frame at 56 days based on our analysis. As Figure 7 illus- capture interpretable interactions. However, this requires sig-
trates, in our dataset, as the time frame is extended beyond nificant feature engineering for generalization. Deep neural
56 days, there is no significant increase in the coverage of networks (DNN) exhibit strong generalizations on dense fea-
collaborated commits, yet the number of queries continues to tures. Yet, DNNs tend to over-generalize and perform sub-
increase linearly. optimally with sparse features. Therefore, we leverage the
More specifically, to establish the graph G of F2, for each Wide & Deep framework [9], adept at balancing memoriza-
CVE, we enumerate the top k commits in Ranking1 , querying tion and generalization, to efficiently manage both kinds of
the LLM using the number of ∑ki=1 s(Vui ) prompts (refer to features concurrently.
Figure 13), seeking its determination of whether commit u To enrich the original F0 and F1 , we incorporate dense
and commit v from Vu collaboratively address the same vul- feature categorization. Specifically, we derive an additional
nerability. In this context, u ∈ {u1 , u2 , ..., uk }, denoting the sparse feature by segmenting a dense feature into bins, which
top k commits in Ranking1 , Vu is a set of commits which sat- are determined based on its variance within our dataset. For-
isfy the previously mentioned two conditions in relation to mally, for a dense feature x, we derive a sparse one x′ by:

9
 x′ = x−x
xintv , xintv =
min xmax −xmin
nbins , Loss Function. By integrating a linear classifier after the fu-
where xmax and xmin represent the maximum and minimum sion gate, we perform binary classification to determine if
values of x in our dataset, and nbins denotes the chosen number a commit acts as a patch for a designated CVE. Given the
of bins. skewness in our dataset, where patch commits are vastly out-
The overall structure of our numerical encoder is shown in numbered by non-patch commits, we use the Focal loss [24],
Figure 9. Formally, it produces embeddings according to: as employed in SOTA SPL [43], to mitigate this imbalance.
σ(wTwide xsparse + wTdeep fdeep (xdense ) + b). Essentially, the Focal loss diminishes the relative loss for cor-
Here, wwide and wdeep represent the weights for the wide rectly classified instances and prioritizes challenging, misclas-
and deep components, respectively. The term xsparse denotes sified cases. Furthermore, the Focal loss introduces a parame-
all sparse features including those from dense ones, and ter to balance the weights of positive and negative samples,
1
fdeep (xdense ) refers to the embeddings from a DNN model which, according to our dataset, is set to be 1500 .
fdeep specific to the dense features xdense . Additionally, b is After adequately training our joint learning model, we ob-
the bias term, and σ(·) is a element-wide sigmoid function. tain a score from the final linear classifier indicating the likeli-
Graph Encoder. We employ the graph encoder to generate a hood of a commit being a patch for a specific CVE. By sorting
semantically-rich embedding for every node in the graph, these scores for each commit, we obtain the final ranking of
elevating previously lower-ranked but actual patches of a them, Ranking2 .
CVE. While our graph, constructed based on LLM’s feed- 6 Evaluation
back, includes edges that link high-ranked patches with their
lower-ranked counterparts for positive impacts, it also has 6.1 Experimental Setup
edges connecting non-patch commits that may introduce side Data. Our data set is collected from the National Vulnerabil-
effects. To ensure our model emphasizes the beneficial con- ity Database (NVD) [27] and Open Source Software (OSS)
nections (edges) and minimizes the influence from the less repositories on GitHub [3]. We gathered all the commits from
desirable ones, we’ve integrated an attention mechanism into 200 OSS repositories on GitHub, along with their associated
our model. Specifically, we utilized a Graph ATtention net- CVEs that are listed in the NVD. We selected these 200 OSS
work (GAT) [40] to generate each node’s embedding. repositories based on the most recent CVEs reported to the
The embedding generation process of GAT is demonstrated NVD. For a CVE, we identify a commit as its patch only if
as follows. Initially, each node’s embedding is initialized by the commit is explicitly mentioned in the CVE’s report (as
using a representation of its corresponding commit, gener- illustrated in Figure 10). Any other commits related to the
ated via a pre-trained language model (BERT). Subsequently, associated OSS are not considered as patches for this CVE.
the embedding of node-i undergoes iterative refinement by It’s important to note that the information of a CVE we use
computing a weighted average of the feature vectors from its for labeling commits is not included in our to-be-analyzed
neighboring nodes: hi = σ(∑k∈Ni αik WT hk ), where αik signi- content of CVE. In total, from 200 OSS repositories, we col-
fies the attention coefficient, hk denotes node-k’s embedding, lected 1,915 CVEs and 2,461 associated patches across all
and the feature vector of node-k is WT hk , the matrix-vector their commits.
product of the shared weight matrix W with its node embed- When constructing our dataset, we define a positive sample
ding hk . as a CVE paired with its patch, and a negative sample as a
The attention coefficient, αi j , between nodes-i and - j is CVE paired with a commit that is not its patch. To ensure that
derived using the softmax function on the raw attention scores, positive samples are not overshadowed by negative ones, we
ei j , over all neighbors of node-i, represented by Ni : follow previous SPL works [36, 43] by restricting the number
exp(ei j ) T T T of non-patch commits used to create negative samples for
αi j = Σ exp(e
k∈Ni ik ) , ei j = a(we [W hi , W h j ]).
each CVE to an experimental maximum of 1,500. In total,
The raw attention score, ei j , is obtained by applying the
our dataset contains 2,461 positive samples and 3,120,898
LeakyReLU activation function, a(·), to the dot product of
negative samples.
the weight vector, we , with the concatenated feature vectors
To comprehensively evaluate LLM-SPL, we established
of nodes, [WT hi , WT h j ]. Notably, the weights W and we are
three datasets: Full, 1-1, and 1-N. The Full dataset includes
consistent and used for all nodes in the graph.
every sample we labeled. The 1-1 dataset consists of samples
Fusion Gate. We use it to combine two embeddings produced
corresponding to 1,512 CVEs that can be fully resolved with
by encoders. Specifically, for a commit, it has a numerical
just one patch. The 1-N dataset includes samples related to
embedding EN created by the numerical encoder and a graph
403 CVEs each requiring multiple patches to fix jointly.
embedding EG optimized by the graph encoder. The fusion
Implementation. In the first two steps of LLM-SPL, we uti-
gate combines them by:
exp(wT E ) exp(wT E ) lized the SOTA VCMatch framework as the base SPL and
E N G G
EN exp(wT E )+exp(w T E ) + EG exp(wT E )+exp(wT E ) , followed its configuration. Step 1 used feature set F0 (model
E N G G E N G G
where wE and wG are two weight vectors indicating the em- parameter θ1 , while step 2 concatenated F0 and F1 (parameter
phasis placed on each type of embedding. θ2 ). Both models were trained on the same labeled dataset.

10
The third step’s joint learning model used Adam optimizer by LLM-SPL and users choose to check the top k commits, the
(learning rate 0.001, L2 regularization 1e-5, dropout 0.4), manual efforts needed for this CVE is calculated as min(r, k).
trained over 100 epochs with batch size 10240. Numerical For a dataset of n CVEs, the average manual effort, M@k, is
Encoder using 10 bins for discretization, followed by a three- ∑n min(r ,k)
calculated using this formula: M@k = i=1 n i , where ri
layer MLP(256 dimensions per layer.) The Graph Encoder is the ranking position of the lowest patch for the i-th CVE
employed a two-layer GAT with 4 attention heads and 256 within the top k results.
hidden sizes per layer. Overall results. Tables 1, 2, and 3 showcase the perfor-
Experimental environment. We used an Ubuntu 20.04.6 LTS mance results of LLM-SPL across three datasets: 1-1, 1-N,
64-bit machine (with 503 GB memory, an AMD EPYC 7543 and Full, as detailed in Section 6.1. For the 1-1 dataset, LLM-
32-Core Processor and 1 NVIDIA A100 GPU) for model SPL ranked patches in the first position for 82.14% of CVEs
training. and within top 10 for 95.30%. High ranking quality (N@10
Evaluation method. To evaluate LLM-SPL’s performance, = 89.18%) allow locating all patches for 95.30% by check-
we adopt a five-fold cross-validation, consistent with the ing only 1.72 commits on average (M@10 = 1.72) In the
SOTA VCMatch, against which we compared in our eval- 1-N dataset, LLM-SPL successfully identified and ranked all
uation. The results presented are the average values of each the collaborated patches within top 10 for 83.13% of 1-N
metric obtained over the five iterations. CVEs. Moreover, the high quality of these rankings, as evi-
6.2 Effectiveness denced by the NDCG value of N@10=80.40%, enables the
Research questions. To evaluate the effectiveness of LLM- examination of only 4.67 commits (M@10=4.67) on average.
SPL, we conducted experiments focusing on the following In the Full dataset, LLM-SPL achieved high recall values
two questions: ❶ How thoroughly can LLM-SPL retrieve all (over 90% when k >= 7) and consistently high NDCG values
patches for a CVE? This assessment is crucial as it directly (above 80% for every k from 1 to 10). These results clearly
correlates with LLM-SPL’s practical utility. Specifically, the demonstrate LLM-SPL’s effectiveness in locating patches for
ability to uncover all patches in 1-N scenarios is vital to en- vulnerabilities with minimal manual effort.
sure comprehensive vulnerability remediation. ❷ How effec- To better understand the effectiveness of LLM-SPL, we
tively can LLM-SPL prioritize patches by ranking them above compared its performance against prior SPL works, including
non-patch commits? This question assesses the quality of the VCMatch [43], PatchScout [36], and FixFinder [14]. The com-
ranking produced by LLM-SPL, which is crucial for enabling parative results, as illustrated in Tables 1, 2, 3, and Figure 14
users to quickly identify a CVE’s patch(es). in Appendix, indicate that LLM-SPL significantly outper-
Metrics. To answer these questions, our metrics are: forms all prior works, across all datasets (Full, 1-1, and 1-N)
• Recall. To answer the first questoin, we utilize the metric for all k values (from 1 to 10) and across all metrics (Recall,
Recall at top k (R@k). This metric measures the proportion NDCG, and Manual Effort). Notably, our experimental results
of CVEs for which LLM-SPL successfully locates all cor- indicate that VCMatch is the state-of-the-art (SOTA) SPL ap-
responding patches within the top k results, with k being a proach. In the following, we detail the specific improvements
parameter configurable by users. Specifically, R@k is the ratio our method offers over VCMatch.
of the number of CVEs whose patches are all located within For Recall, while VCMatch achieves high recall on the
the top k rankings to the total number of CVEs analyzed. 1-1 dataset (e.g., R@4 = 90.67%, R@10 = 94.25%), LLM-
• NDCG. For the second question, we utilize the metric Nor- SPL increases average recall by 1.77% across all k values (1
malized Discounted Cumulative Gain (NDCG) [1] at top k to 10). Notably, it improves R@1 by 3.44%, demonstrating
(denoted as N@k). N@k is a widely recognized metric for the enhanced capability to rank patches in the topmost posi-
evaluating ranking quality; it not only considers the position tion compared to VCMatch. In 1-N scenarios, improvements
of relevant items within the top k rankings but also assigns are substantial: R@5 increases from 47.89% to 69.48% and
greater weight to those items positioned higher in the list. R@10 from 60.30% to 83.13%, gains of 21.59% and 22.83%,
An N@k score, ranging from 0 to 1, indicates that a higher respectively. In our 403-CVE 1-N dataset, this allows the
score reflects a more favorable ranking of patches over non- complete remediation of 86 additional CVEs in the top 5 and
patch commits within the top k results, thus reflecting a more 92 in the top 10 rankings. Regarding NDCG ( 2 and Fig-
effective recommendation system. ure 14(b)), LLM-SPL shows consistent improvements, with
• Manual Effort. Besides using the metric NDCG to measure average increases of 6.01% (Full dataset), 2.46% (1-1) and
the quality of rankings, we calculate the Manual Effort at top 19.34% (1-N). This improved ranking quality significantly
k (denoted as M@k), following precedents established in pre- reduces manual effort( 3 and Figure 14(c)). Compared to VC-
vious SPL recommendation studies [36, 43]. M@k quantifies Match, Users experience an average 10% effort reduction in
the average number of commits that users need to manually the Full dataset, with the 1-N dataset showing about 20% re-
review in the list of top k items from top to bottom before duction at M@7 and over 25% at M@10. Overall, LLM-SPL
locating all the patches for a CVE. Specifically, for a given not only identifies all necessary patches for CVEs but also
CVE, if the lowest ranked patch is placed at the r-th position ranks them higher, substantially reducing practical manual

11
 Table 1: Recall (R@k) of PatchScout, FixFinder, VCMatch and LLM-SPL.
Full 1-1 1-N
k
PatchScout FixFinder VCMatch LLM-SPL PatchScout FixFinder VCMatch LLM-SPL PatchScout FixFinder VCMatch LLM-SPL
1 45.69% 44.28% 62.14% 64.86% ↑ 57.87% 56.08% 78.70% 82.14% ↑ 0.00% 0.00% 0.00% 0.00% =
2 57.75% 55.77% 74.99% 79.22% ↑ 69.11% 67.59% 87.57% 89.22% ↑ 15.14% 11.41% 27.79% 41.69% ↑
3 61.67% 61.57% 78.49% 84.18% ↑ 73.21% 73.74% 89.55% 91.60% ↑ 18.36% 15.88% 36.97% 56.33% ↑
4 64.02% 64.54% 80.68% 86.58% ↑ 75.66% 76.85% 90.67% 92.72% ↑ 20.35% 18.36% 43.18% 63.52% ↑
5 65.33% 66.21% 82.45% 88.56% ↑ 77.05% 78.57% 91.67% 93.65% ↑ 21.34% 19.85% 47.89% 69.48% ↑
6 66.37% 67.57% 83.66% 89.45% ↑ 78.17% 80.03% 92.39% 94.18% ↑ 22.08% 20.84% 50.87% 71.71% ↑
7 67.10% 68.41% 84.75% 90.34% ↑ 79.03% 80.82% 93.12% 94.71% ↑ 22.33% 21.84% 53.35% 73.95% ↑
8 67.68% 69.56% 85.43% 90.97% ↑ 79.63% 81.81% 93.58% 94.84% ↑ 22.83% 23.57% 54.84% 76.43% ↑
9 67.99% 70.29% 86.27% 91.64% ↑ 79.96% 82.61% 93.98% 94.84% ↑ 23.08% 24.07% 57.32% 79.65% ↑
10 68.56% 71.17% 87.10% 92.74% ↑ 80.22% 83.60% 94.25% 95.30% ↑ 24.81% 24.57% 60.30% 83.13% ↑

Table 2: NDCG (N@k) of PatchScout, FixFinder, VCMatch, and LLM-SPL.

Full 1-1 1-N
k
PatchScout FixFinder VCMatch LLM-SPL PatchScout FixFinder VCMatch LLM-SPL PatchScout FixFinder VCMatch LLM-SPL
1 51.36% 50.39% 73.63% 80.10% ↑ 57.90% 56.08% 78.70% 82.14% ↑ 26.80% 29.03% 54.59% 72.46% ↑
2 56.48% 55.38% 77.20% 82.92% ↑ 64.98% 63.35% 84.30% 86.61% ↑ 24.59% 25.48% 50.56% 69.10% ↑
3 58.38% 58.33% 78.58% 84.62% ↑ 67.03% 66.42% 85.29% 87.80% ↑ 25.95% 27.97% 53.43% 72.71% ↑
4 59.39% 59.65% 79.35% 85.48% ↑ 68.07% 67.76% 85.77% 88.28% ↑ 26.84% 29.24% 55.27% 74.95% ↑
5 59.95% 60.32% 79.94% 86.10% ↑ 68.60% 68.42% 86.16% 88.64% ↑ 27.47% 29.92% 56.62% 76.58% ↑
6 60.37% 60.85% 80.36% 86.44% ↑ 69.03% 68.94% 86.41% 88.83% ↑ 27.89% 30.50% 57.62% 77.47% ↑
7 60.66% 61.19% 80.74% 86.73% ↑ 69.30% 69.21% 86.66% 89.01% ↑ 28.23% 31.11% 58.53% 78.17% ↑
8 60.86% 61.55% 81.01% 86.93% ↑ 69.49% 69.52% 86.80% 89.05% ↑ 28.49% 31.64% 59.29% 78.99% ↑
9 60.97% 61.82% 81.31% 87.09% ↑ 69.59% 69.76% 86.92% 89.05% ↑ 28.64% 32.06% 60.25% 79.74% ↑
10 61.15% 62.12% 81.52% 87.33% ↑ 69.67% 70.05% 87.00% 89.18% ↑ 29.21% 32.38% 60.99% 80.40% ↑

effort compared to VCMatch. a one-line code change to update release information, offering
minimal insight into its relevance to the CVE or its connec-
Declined rankings in LLM-SPL against VCMatch. In our
tions with other patches. Such sparse information results in
analysis, we observed that among 2,461 security patches in
LLM’s failures to identify a commit that actually patches a
our dataset, 97 (3.94%) were ranked lower by our LLM-SPL
CVE (resulting in a FN for F1 ) and to recognize jointly work-
compared to VCMatch. Upon a detailed manual analysis of
ing patches (leading to a FN for F2 ). Another factor contribut-
these 97 commits, we discovered that the primary reason for
ing to false negatives is the extensive length of some commit
their lower rankings was due to false determinations made by
diffs. For instance, the security patch commit 2430929 for
the LLM (GPT-3.5 used in our research), which included both
CVE-2021-23365 contains 50,434 tokens, far exceeding the
false positives (FPs) and false negatives (FNs) in generating
maximum token limit of a single GPT-3.5 prompt, which is
features F1 and F2 . Specifically, with the feature F1 , when the
4,097. In such cases, we truncated the extensive code diffs
LLM failed to recognize an actual CVE patch (FN) and simul-
during prompt construction. Unfortunately, this truncation
taneously misidentified a non-patch commit as the patch for
often resulted in the loss of critical information, consequently
this CVE (FP), our model tended to assign a higher ranking
impeding the LLM’s ability to recognize the relationships
to the FP commit, which consequently resulted in the genuine
necessary for F1 and F2 , leading to false negatives.
patch being ranked lower. For the feature F2 , particularly in
However, such a decrease was minimal in our results.
1-N scenario, if the LLM did not detect a collaborated rela-
Among these 97 commits, the average drop was 4.77 po-
tionship between actual patches (FN) and incorrectly linked
sitions, with 36.08% declining by only one position, 57.73%
a non-patch with a high-ranked patch (FP), our model likely
falling within three positions, and 75.26% dropping within
elevated the ranking of the FP non-patch, thereby causing the
five positions.
actual patches to be ranked lower. Among the 97 commits
analyzed, 41 were affected by issues related to F1 , 30 by F2 , 6.3 Ablation Study
and the remaining 26 were influenced by a combination of As introduced in Section 5, our LLM-SPL incorporates two
errors in both features. key features, F1 and F2 , endorsed by the LLM to enhance the
As discussed in Section 4, the LLM indeed exhibits a high SPL recommendations. The substantial enhancements result-
false positive rate. However, it also maintains a high recall rate, ing from the entire design have been demonstrated in Sec-
indicating that false negatives are comparatively infrequent. tion 6.2. This section investigates the individual contributions
Thus, we further analyzed the causes of false negatives in fea- of F1 and F2 to the model’s performance, respectively. To this
tures F1 and F2 , ultimately identifying two primary reasons end, we conducted ablation studies using models with differ-
for these occurrences. Firstly, insufficient information within ent feature combinations: 1) F0 alone, which corresponds to
CVEs and commits significantly hampers the LLM’s perfor- VCMatch, 2) F0 and F1 only, excluding F2 , and 3) all features
mance. For example, CVE-2020-7772 is briefly described (F0 , F1 , and F2 ), which constitute our LLM-SPL. The results
as “This affects the package doc-path before 2.1.2.”, severely of these experiments are illustrated in Tables 9, 10, and 11 in
lacking details about the vulnerability (refer to Table 5 for Appendix.
typical details included in CVE descriptions). Similarly, the To evaluate the impact of feature F1 , we compared models
security patch for CVE-2020-12607, commit 7b64e3e, con- using only F0 against both F0 and F1 . Results (as detailed
tains merely the message “Update docs to v2.1.2” along with in Tables 9, 10, and 11 and illustrated in Figures 15, 16),

12
 Table 3: Manual Effort (M@k) of PatchScout, FixFinder, VCMatch, and LLM-SPL.
Full 1-1 1-N
k
PatchScout FixFinder VCMatch LLM-SPL PatchScout FixFinder VCMatch LLM-SPL PatchScout FixFinder VCMatch LLM-SPL
1 1.00 1.00 1.00 1.00 = 1.00 1.00 1.00 1.00 = 1.00 1.00 1.00 1.00 =
2 1.54 1.56 1.38 1.35 ↓ 1.42 1.44 1.21 1.18 ↓ 2.00 2.00 2.00 2.00 =
3 1.97 2.00 1.63 1.56 ↓ 1.73 1.76 1.34 1.29 ↓ 2.85 2.89 2.72 2.58 ↓
4 2.35 2.38 1.84 1.72 ↓ 2.00 2.03 1.44 1.37 ↓ 3.67 3.73 3.35 3.02 ↓
5 2.71 2.74 2.04 1.85 ↓ 2.24 2.26 1.54 1.44 ↓ 4.46 4.54 3.92 3.38 ↓
6 3.06 3.08 2.21 1.97 ↓ 2.47 2.47 1.62 1.51 ↓ 5.25 5.34 4.44 3.69 ↓
7 3.39 3.40 2.38 2.07 ↓ 2.69 2.67 1.69 1.56 ↓ 6.03 6.14 4.93 3.97 ↓
8 3.72 3.72 2.53 2.17 ↓ 2.90 2.86 1.76 1.62 ↓ 6.80 6.92 5.40 4.23 ↓
9 4.04 4.02 2.67 2.26 ↓ 3.10 3.04 1.83 1.67 ↓ 7.58 7.68 5.85 4.47 ↓
10 4.36 4.32 2.81 2.34 ↓ 3.30 3.22 1.89 1.72 ↓ 8.34 8.44 6.28 4.67 ↓

show consistent improvement across all metrics (R@k, N@k, and commits, enhancing the semantic understanding of vul-
and M@k for k = 1 to 10) across all dataset(Full, 1-1 and 1- nerability and patches. Our work, LLM-SPL, not only adopts
N). At k = 5, including F1 improved Recall (R@5) by 5.01%, these features but also leverages LLMs to comprehend CVEs,
1.59%, and 17.87% for Full, 1-1, and 1-N datasets respectively. commits, and specialized software security knowledge, signif-
NDCG (N@5) increased by 5.32%, 2.05%, and 17.57%, while icantly improving SPL recommendations. Moreover, unlike
Manual Effort (M@5) reduced by 8.05%, 5.47%, and 11.84%. previous works focused on 1-1 scenarios (single patch per
These improvements account for approximately 85% of LLM- vulnerability), our approach effectively addresses complex
SPL’s overall enhancement (Figures 15-17), confirming F1 ’s 1-N scenarios where multiple patches collaboratively resolve
effectiveness in leveraging LLM capabilities for CVE patch a vulnerability.
identification and prioritization. Recommendation algorithm. Our approach incorporates
To assess the contribution of F2 , we compared the perfor- LLM feedback into recommendation algorithms, combin-
mance of LLM-SPL, which incorporates all features (F0 , F1 , ing the advantages of Relevance Feedback (RF) and Pseudo-
and F2 ), against a model that uses only F0 and F1 . Given that Relevance Feedback (PRF). It mitigates RF’s high user in-
F2 is specifically designed for the 1-N scenario, our analysis volvement cost [32, 33] and PRF’s potential biases from
focused on the 1-N dataset. As illustrated in Tables 9, 10, relying solely on top k results [21, 34]. Additionally, our rec-
and 11 and Figures 15(c), 16(c), and 17(c) in Appendix, F2 ommendation algorithm leverages information from graphs.
provided comprehensive improvements across all metrics (Re- Graph-based recommendations have proven effective, espe-
call, NDCG and Manual Effort) in the 1-N dataset. For in- cially for candidates with inherent relationships: pharmaceu-
stance, at k = 10, F2 increased Recall(R@10) from 76.18% tical recommendations use medical knowledge graphs [6],
to 83.13%, a boost of 6.95%, and enhanced NDCG(N@10) travel recommendations employ map graphs [38], social me-
by 3.10%, which in turn led to a reduction in manual effort dia leverages social networks [16,23,41] and citation systems
by 4.23% (M@10). This means that in our 1-N dataset of 403 utilize scholarly graphs [17, 25, 26]. Unlike these systems us-
CVEs, on top of the reduction from manually checking 540 ing pre-existing graphs built on accumulated knowledge, our
commits achieved by F1 , F2 further reduced the need to check SPL research requires constructing a domain-specific security
an additional 100+ commits. Overall, F2 contributed approx- graph from scratch. We utilize LLMs to assist in building
imately 15% to the improvement in each performance met- this graph, demonstrating the potential of LLM-based graph
ric for LLM-SPL, as shown in Figures Figures 15(c), 16(c), generation for enhancing SPL.
and 17(c). These results demonstrate the efficacy of F2 , indi- 8 Conclusion
cating its ability to effectively address 1-N scenarios. Further- In this study, we proposed LLM-SPL, a novel SPL (Secu-
more, in the 1-1 dataset, F2 had minimal impact, which aligns rity Patch Localization) recommendation model leveraging
with our expectations. Large Language Models (LLMs) to address the challenges
inherent in SPL. Our approach extends beyond mere utiliza-
7 Related Work tion of LLM outputs; through a joint learning framework,
Security patch localization. Initial approaches [19, 22, 30, we incorporate the determinations of LLMs on CVE-commit
44, 45, 47] relied heavily on direct references between CVE and commit-commit relationships as additional features/feed-
descriptions and patches. However, the effectiveness of these back to refine the recommendation model to improve its per-
methods was limited by the availability of explicit infor- formance. Our evaluation demonstrates the effectiveness of
mation. To address this limitation, subsequent studies like LLM-SPL. Compared to the state-of-the-art work, VCMatch,
FixFinder [14], PatchScout [36] modeled SPL as a recom- our LLM-SPL consistently outperformed it in terms of Recall
mendation problem. They extracted rule-based features to while also achieving a notable reduction in manual effort. No-
describe CVEs-commit relationships, applying predefined fea- tably, for scenarios where one vulnerability requires multiple
ture weights or various machine learning models to rank the jointly working patches, LLM-SPL demonstrates significant
commits most likely associated with a given CVE. Building improvements in all metrics. These results underscore the po-
on these efforts, VCMatch [43], the current SOTA, extended tential of LLM-SPL as a valuable SPL approach in real-world
these features by introducing embeddings to represent CVEs applications.

13
References [12] Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin
Liu, Ruoyu Zhang, Ruijie Fang, Asmita Asmita, Ryan
[1] Discounted Cumulative Gain. [Link] Tsang, Najmeh Nazari, Han Wang, et al. Large language
[Link]/wiki/Discounted_cumulative_gain. models for code analysis: Do llms really do their job?
arXiv preprint arXiv:2310.12357, 2023.
[2] Evaluating large language models trained on code, 2021.
[13] Michael Hansen, Robert L. Goldstone, and Andrew
[3] GitHub. [Link] None. Lumsdaine. What makes code hard to understand?,
2013.
[4] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al-
shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mer- [14] Daan Hommersom, Antonino Sabetta, Bonaventura Cop-
ouane Debbah, Etienne Goffinet, Daniel Heslow, Julien pola, Dario Di Nucci, and Damian A Tamburri. Auto-
Launay, Quentin Malartic, Badreddine Noune, Baptiste mated mapping of vulnerability advisories onto their fix
Pannier, and Guilherme Penedo. Falcon-40B: an open commits in open source repositories. ACM Transactions
large language model with state-of-the-art performance. on Software Engineering and Methodology, 2021.
2023.
[15] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong,
[5] Apache. Apache Log4j Security Vulnerabili- Zhangyin Feng, Haotian Wang, Qianglong Chen, Wei-
ties. [Link] hua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on
2021. hallucination in large language models: Principles, tax-
onomy, challenges, and open questions. arXiv preprint
[6] Suman Bhoi, Mong Li Lee, Wynne Hsu, Hao Sen An-
arXiv:2311.05232, 2023.
drew Fang, and Ngiap Chuan Tan. Personalizing med-
ication recommendation with a graph-based approach. [16] Mohsen Jamali and Martin Ester. A matrix factorization
ACM Transactions on Information Systems (TOIS), technique with trust propagation for recommendation
40(3):1–23, 2021. in social networks. In Proceedings of the fourth ACM
conference on Recommender systems, pages 135–142,
[7] Tom Brown, Benjamin Mann, Ryder, et al. Language
2010.
models are few-shot learners. In H. Larochelle, M. Ran-
zato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- [17] Chanwoo Jeong, Sion Jang, Eunjeong Park, and
vances in Neural Information Processing Systems, vol- Sungchul Choi. A context-aware citation recommenda-
ume 33, pages 1877–1901. Curran Associates, Inc., tion model with bert and graph convolutional networks.
2020. Scientometrics, 124:1907–1922, 2020.
[8] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, [18] Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large
Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, language models are few-shot testers: Exploring llm-
Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks based general bug reproduction. In 2023 IEEE/ACM
of artificial general intelligence: Early experiments with 45th International Conference on Software Engineering
gpt-4. arXiv preprint arXiv:2303.12712, 2023. (ICSE), pages 2312–2323. IEEE, 2023.
[9] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal [19] Seulbae Kim, Seunghoon Woo, Heejo Lee, and Hakjoo
Shaked, Tushar Chandra, Hrishi Aradhye, Glen Ander- Oh. Vuddy: A scalable approach for vulnerable code
son, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, clone discovery. In 2017 IEEE Symposium on Security
Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Privacy (SP), pages 595–614. IEEE, 2017.
and Hemal Shah. Wide & deep learning for recom-
mender systems, 2016. [20] Guillaume Lample, Miguel Ballesteros, et al. Neural ar-
chitectures for named entity recognition. In Proceedings
[10] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, of the 2016 Conference of the North American Chapter
Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan of the Association for Computational Linguistics: Hu-
Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Sto- man Language Technologies, pages 260–270, San Diego,
ica, and Eric P. Xing. Vicuna: An open-source chat- California, June 2016. Association for Computational
bot impressing gpt-4 with 90%* chatgpt quality, March Linguistics.
2023.
[21] Canjia Li, Yingfei Sun, Ben He, Le Wang, Kai Hui, An-
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and drew Yates, Le Sun, and Jungang Xu. Nprf: A neural
Kristina Toutanova. Bert: Pre-training of deep bidirec- pseudo relevance feedback framework for ad-hoc infor-
tional transformers for language understanding, 2019. mation retrieval, 2018.

14
[22] Frank Li and Vern Paxson. A large-scale empirical study [33] S. Rajaram, A. Garg, X.S. Zhou, and T.S. Huang. Classi-
of security patches. In Proceedings of the 2017 ACM fication approach towards ranking and sorting problems.
SIGSAC Conference on Computer and Communications In N. Lavrač, D. Gamberger, H. Blockeel, and L. Todor-
Security, pages 2201–2215, 2017. ovski, editors, Machine Learning: ECML 2003, volume
2837 of Lecture Notes in Computer Science, Berlin, Hei-
[23] Zhepeng Li, Xiao Fang, and Olivia R Liu Sheng. A sur- delberg, 2003. Springer.
vey of link recommendation for social networks: Meth-
ods, theoretical foundations, and future research direc- [34] Joseph Rocchio. Relevance feedback in information
tions. ACM Transactions on Management Information retrieval, pages 313–323. The Smart Retrieval System-
Systems (TMIS), 9(1):1–26, 2017. experiments in Automatic Document Processing, 1971.

[24] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, [35] SYNOPSYS. 2023 Open Source Security and Risk
and Piotr Dollár. Focal loss for dense object detection. Analysis Report. [Link]
In Proceedings of the IEEE international conference on tent/dam/synopsys/sig-assets/reports/rep-o
computer vision, pages 2980–2988, 2017. [Link], 2023.
[25] Xiaozhong Liu, Yingying Yu, Chun Guo, and Yizhou [36] Xin Tan, Yuan Zhang, Chenyuan Mi, Jiajun Cao, Kun
Sun. Meta-path-based ranking with pseudo relevance Sun, Yifan Lin, and Min Yang. Locating the se-
feedback on heterogeneous graph for citation recommen- curity patches for disclosed oss vulnerabilities with
dation. In Proceedings of the 23rd acm international vulnerability-commit correlation ranking. In Proceed-
conference on conference on information and knowledge ings of the 2021 ACM SIGSAC Conference on Computer
management, pages 121–130, 2014. and Communications Security, pages 3282–3299, 2021.
[26] Shutian Ma, Chengzhi Zhang, and Xiaozhong Liu. A [37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
review of citation recommendation: from textual content Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap-
to enriched context. Scientometrics, 122:1445–1472, tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar,
2020. et al. Llama: Open and efficient foundation language
models. arXiv preprint arXiv:2302.13971, 2023.
[27] NIST. National Vulnerability Database.
[Link]
[38] Francesco Maria Turno and Irina Yatskiv Jackiva.
[28] David Noever and Kevin Williams. Chatbots as fluent Graph-based approach for personalized travel recom-
polyglots: Revisiting breakthrough code snippets. arXiv mendations. Transport and Telecommunication Journal,
preprint arXiv:2301.03373, 2023. 24(4):423–433, 2023.

[29] OpenAI. Gpt 3.5 model documentation. [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
[Link] Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser,
and Illia Polosukhin. Attention is all you need, 2023.
[30] Henning Perl, Sergej Dechand, et al. Vccfinder: Finding
potential vulnerabilities in open-source projects to assist [40] Petar Veličković, Guillem Cucurull, Arantxa Casanova,
code audits. In Proceedings of the 22nd ACM SIGSAC Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph
Conference on Computer and Communications Security, attention networks. arXiv preprint arXiv:1710.10903,
CCS ’15, page 426–437, New York, NY, USA, 2015. 2017.
Association for Computing Machinery.
[41] Frank Edward Walter, Stefano Battiston, and Frank
[31] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Schweitzer. A model of a trust-based recommendation
Gardner, Christopher Clark, Kenton Lee, and Luke system on a social network. Autonomous Agents and
Zettlemoyer. Deep contextualized word representations, Multi-Agent Systems, 16:57–74, 2008.
2018.
[42] Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong,
[32] Filip Radlinski and Thorsten Joachims. Query chains: Jie Tang, and Dawn Song. Zero-shot information extrac-
Learning to rank from implicit feedback. In Proceedings tion as a unified text-to-triple translation. In Proceedings
of the Eleventh ACM SIGKDD International Conference of the 2021 Conference on Empirical Methods in Nat-
on Knowledge Discovery in Data Mining, KDD ’05, ural Language Processing, pages 1225–1238, Online
page 239–248, New York, NY, USA, 2005. Association and Punta Cana, Dominican Republic, November 2021.
for Computing Machinery. Association for Computational Linguistics.

15
 Product Solutions Open Source Enterprise Pricing Product Solutions Open Source Enterprise
Search or jumpPricing
to... Sign in Sign up Search

torvalds / linux Public Commit

torvalds / linux Public Notifications Fork 52.2k Star 173k

Code Pull requests 354 Actions Projects Security Insights

Code Pull requests 354 Actions Projects Security Insights
loopback: off by one in tcm_loop_make_naa_tpg()
This is an off by one 'tgpt' check in tcm_loop_make_naa_tpg() that could result
Commit Commit
in memory corruption.

Signed-off-by: Dan Carpenter <error27@[Link]>

loopback: off by one in tcm_loop_make_naa_tpg() loopback: off by one in tcm_loop_make_naa_tpg() Browse files
[43] Shichao Wang, Yun Zhang, Liagfeng Bao, Xin Xia, and
This is an off by one 'tgpt' check in tcm_loop_make_naa_tpg() that could result
Linux: 12f09cc
Signed-off-by: Nicholas A. Bellinger <nab@[Link]>
This is an off by one 'tgpt' check in tcm_loop_make_naa_tpg() that could result
in memory corruption. in memory corruption.
loopback: o by one in tcm_loop_make_naa_tp()
Minghui Wu. VCMatch: A Ranking-based Approach
Signed-off-by: Dan Carpenter <error27@[Link]>
Thismaster
is an off by one 'tgpt' check in tcm_loop_make_naa_tpg() that could result
in memory corruption.
v6.10-rc2 …Danv3.1-rc1
Signed-off-by: Carpenter <error27@[Link]>
Commit Description

for Automatic Security Patches Localization for OSS

Signed-off-by: Nicholas A. Bellinger <nab@[Link]> Signed-off-by: Dan
Signed-off-by: Carpenter
Nicholas <error27@[Link]>
A. Bellinger <nab@[Link]>
Signed-off-by: Nicholas A. Bellinger <nab@[Link]>
error27 authored and nablio3000 committed on Jul 22, 2011
Metadata

Vulnerabilities. In 2022 IEEE International Confer-

master
v6.10-rc2 … v3.1-rc1
master
v6.10-rc2 … v3.1-rc1

ence on Software Analysis, Evolution and Reengineering

error27 authored and nablio3000 committed on Jul 22, 2011
Showing 1 changed file with 1 addition and 1 deletion.
error27 authored and nablio3000 committed on Jul 22, 2011 1 parent 21bca31 commit 12f09cc

(SANER), pages 589–600. IEEE, 2022. Showing 1 changed file with 1 addition and 1 deletion. 2
Showing 1 changed drivers/target/loopback/tcm_loop.c
file with 1 addition and 1 deletion. Whitespace Ignore whitespace
Code Diffs
Split Unified

@@ -1205,7 +1205,7 @@ struct se_portal_group *tcm_loop_make_naa_tpg(

2 drivers/target/loopback/tcm_loop.c 2 drivers/target/loopback/tcm_loop.c
1205 1205 tpgt_str += 5; /* Skip ahead of "tpgt_" */

[44] Xinda Wang, Kun Sun, Archer Batcheller, and Sushil

1205 1205
@@ -1205,7 +1205,7 @@ struct se_portal_group *tcm_loop_make_naa_tpg(

tpgt_str += 5; /* Skip ahead of "tpgt_" */ 1205

1206
1207
1205
1206
1207
@@ -1205,7 +1205,7 @@ struct se_portal_group *tcm_loop_make_naa_tpg(
tpgt = (unsigned short int) simple_strtoul(tpgt_str, &end_ptr, 0);
tpgt_str += 5; /* Skip ahead of "tpgt_" */

Jajodia. Detecting" 0-day" vulnerability: An empir-

1206
1207
1206
1207
tpgt = (unsigned short int) simple_strtoul(tpgt_str, &end_ptr,
12060); 1206
1207
1208
1207
-
tpgt = (unsigned short int) simple_strtoul(tpgt_str, &end_ptr, 0);
if (tpgt > TL_TPGS_PER_HBA) {
1208- + if (tpgt >= TL_TPGS_PER_HBA) {
ical study of secret security patch in oss. In 2019
1208
1208
-
+
if (tpgt > TL_TPGS_PER_HBA) {
if (tpgt >= TL_TPGS_PER_HBA) {
1208
1209 1208
1209+
if (tpgt > TL_TPGS_PER_HBA) {
printk(KERN_ERR
if (tpgt >= TL_TPGS_PER_HBA) { "Passed tpgt: %hu exceeds TL_TPGS_PER_HBA:"

49th Annual IEEE/IFIP International Conference on

1209 1209 printk(KERN_ERR "Passed tpgt: %hu exceeds TL_TPGS_PER_HBA:"
1209
1210 1209
1210 printk(KERN_ERR "Passed tpgt: %hu exceeds
" %u\n", tpgt,TL_TPGS_PER_HBA:"
TL_TPGS_PER_HBA);
1210 1210 " %u\n", tpgt, TL_TPGS_PER_HBA); 1210
1211 1210
1211 " %u\n", tpgt, TL_TPGS_PER_HBA);
return ERR_PTR(-EINVAL);
1211 1211 return ERR_PTR(-EINVAL); 1211 1211 return ERR_PTR(-EINVAL);
Dependable Systems and Networks (DSN), pages 485–
492. IEEE, 2019. 0 comments on commit 12f09cc 00comments
commentson on
Figure 11: Example of commit information.
commit
commit 12f09cc
12f09cc CVE-Commit
Please sign in to comment. Please sign in to comment.
Please sign in to comment.

[45] Xinda Wang, Shu Wang, Pengbin Feng, Kun Sun, and © 2024 GitHub, Inc. Terms Privacy
Role: You are a software security analyst with expertise in analyzing, fixing,
and
Security patching
Status vulnerabilities.
Docs Contact Manage cookies Do not share my personal
© 2024 GitHub, information
Inc. Terms Privacy Security Status Docs Contact Manage cookies Do not share my persona

Sushil Jajodia. Patchdb: A large-scale security patch © 2024 GitHub, Inc. Terms Privacy Security Status Docs Contact Manage cook
Considering the following provided specific vulnerability information from
dataset. In 2021 51st Annual IEEE/IFIP Interna- CVE (including CVE ID and CVE description) and the corresponding commit
data from its related repository (including commit ID, commit description, and
tional Conference on Dependable Systems and Networks ff code diff). Your task is to analyze the vulnerability and commit information to
determine whether this commit is a patch for the vulnerability. Provide a
(DSN), pages 149–160. IEEE, 2021. conclusion and reasoning to support your answer.
Additionally, at the end of your response, please give the answer again to the
question whether this commit is a patch for the vulnerability using a single
[46] Fei Wu and Daniel S. Weld. Open information ex- line. The answer can only be either YES, NO, or UNKNOWN.

traction using Wikipedia. In Proceedings of the 48th CVE ID:

CVE description:
Annual Meeting of the Association for Computational
Commit ID:
Linguistics, pages 118–127, Uppsala, Sweden, July 2010. Commit description:
Association for Computational Linguistics. Commit code diffs:

Figure 12: Template of prompt for F1 .

[47] Congying Xu, Bihuan Chen, Chenhao Lu, Kaifeng
Huang, Xin Peng, and Yang Liu. Tracking patches for
Role: You are a software security analyst with expertise in analyzing, fixing, and
open source software vulnerabilities. In Proceedings patching vulnerabilities.
of the 30th ACM Joint European Software Engineer- Consider the following provided two commits, each containing the commit ID,
ing Conference and Symposium on the Foundations of commit description, and code diff. Your task is to analyze the commit
information to determine whether these two commits jointly address a same
Software Engineering, pages 860–871, 2022. problem, bug, issue, or vulnerability. Provide a conclusion and reasoning to
support your answer.
Additionally, at the end of your response, please give the answer again to the
question whether these two commits jointly address a same problem, bug, issue,
[48] Andrej Žukov-Gregorič, Yoram Bachrach, and Sam or vulnerability using a single line. The answer can only be either YES, NO or
UNKNOWN.
Coope. Named entity recognition with parallel recurrent Commit 1:
neural networks. In Proceedings of the 56th Annual Commit ID:
Commit description:
Meeting of the Association for Computational Linguis- Commit code diffs:
tics (Volume 2: Short Papers), pages 69–74, Melbourne, Commit 2:
Australia, July 2018. Association for Computational Lin- Commit ID:
Commit description:
guistics. Commit code diffs:

Figure 13: Template of prompt for F2 .

CVE-ID

lib/thin/[Link] in Thin web server before 1.2.4 Description

relies on the X-Forwarded-For header to determine
the IP address of the client, which allows remote
attackers to spoof the IP address and hide activities Published Time
via a modi ed X-Forwarded-For header. References

Patch

Common Weakness Enumeration (CWE) type

Figure 10: Example of CVE information.

16
 VCMatch vs LLM-SPL

Full dataset 1-1 dataset 1-N dataset

Increase

Increase

Saving
Recall
Top k Top k Top k
(a) Recall (R@k) (b) NDCG (N@k) (c) Manual Effort (M@k)

Figure 14: Increase in Recall, NDCG and Manual Effort for LLM-SPL over VCMatch.

Increase by F1 Increase by F2
Increase

Increase

Increase
NDCG
Top k Top k Top k
(a) Full dataset (b) 1-1 dataset (c) 1-N dataset

Figure 15: Ablation Study: Impact on F1 and F2 in Recall

Increase by F1 Increase by F2
Increase

Increase

Increase

Labor
Top k Top k Top k
(a) Full dataset (b) 1-1 dataset (c) 1-N dataset

Figure 16: Ablation Study: Impact on F1 and F2 in NDCG.

Saving by F1 Saving by F2
Saving

Saving

Saving

Top k Top k Top k

(a) Full dataset (b) 1-1 dataset (c) 1-N dataset
Figure 17: Ablation Study: Impact on F1 and F2 in Manual Effort.

17
 Table 4: VCMatch Feature List (F0 )
Note: Type S denotes sparse feature, D denotes dense feature.
Feature Category Feature Description Type
Code Added Num # of lines of code added in the commit S
Code Deleted Num # of lines of code deleted in the commit S
Code Modified Num # of lines of code modify in the commit S
Same Filepath Num # of filepaths that exists in both commit and vul desciption S
Same Filepath Ratio # of same filepath num / # of filepath modified by commit, S
Code Behavior Unrelated Filepath Num # of filepaths existed in the commit but not mentioned in vuln description S
Features Same File Num # of files that exist in both commit and vuln description S
Same File Ratio # of same files / # of files modified by the commit S
Unrelated File Num # of files that appear in commit but not mentioned by the code diff S
Same Function Num # of functions that exist both in the commit diff and vul description. S
Same Function Ratio # of same function / # of functions modified by the code. S
Unrelated Function Num # of functions that exist in commit diff but not mentioned in the vul description. S
CVE Num # of CVE IDs in commit message. S
Bug Num # of bug IDs in commit message S
Commit Message Issue Num # of issue IDs in commit message S
Identifiers URL Num # of URLs in commit message. S
CVE Match Whether the commit mentions CVE-ID in the NVD Page. S
Bug Match Whether the code commit mentions the Bug-ID in the NVD Page. S
Vul-CWE-Msg Same Num # of the same tokens between commit message and CWE name. D
Vul-CWE-Msg Same Ratio Vul-CWE-Msg Same Num/#ofCWEname tokens. D
Vul-Commit TfIdf Similarity Cosine similarity of vulnerability tfidf and commit tfidf. D
Shared-Vul-Msg-Word Num # of shared words between vul description and commit message. D
Shared-Vul-Msg-Word Ratio # of Shared-Vul-Msg-Words / # of words in vul description. D
Textual Similarity
Shared-Vul-Code-Word Num # of shared words between vul description and code diff. D
Features
Shared-Vul-Code-Word Ratio # of Shared-Vul-Code-Words / # of words in vul description. D
Max/Sum/Average/Variance of
The max/sum/average/variance of the frequencies for all Shared-Vul-Msg-Words. D
Shared-Vul-Msg-Word Frequency
Max/Sum/Average/Variance of
The max/sum/average/variance of the frequencies for all Shared-Vul-Code-Words. D
Shared-Vul-Code-Word Frequency
Deep Textual CVE
A vector representation of CVE description encoded using a pre-trained language model. D
Description Representation
Deep Textual Commit
A vector representation of commit content encoded using a pre-trained language model. D
Content Representation
Security Relevance
Vulnerability Type Relevance The relevance of the vulnerability texts between NVD and commit D
Features
Temporal Features Time Interval Time interval between code commit time and CVE-ID assigned time D

18
 Table 5: Information categories within CVE content.

Category Examples
Software CVE-2022-39252: ... is an implementation of a Matrix client-server library in Rust, ...
Software name CVE-2022-21208: The package node-opcua before 2.74.0 are vulnerable to Denial of Service ...
information Software CVE-2014-5273: ... vulnerabilities in phpMyAdmin 4.0.x before [Link], 4.1.x before [Link], and 4.2.x before [Link] allow ...
version CVE-2018-18227: In Wireshark 2.6.0 to 2.6.3 and 2.4.0 to 2.4.9, ...
Vulnerability CVE-2012-2386: Integer overflow in the phar_parse_tarfile function in ...
type CVE-2010-3429: flicvideo.c in ..., related to an "arbitrary offset dereference vulnerability"
Faulty CVE-2022-23567: ... We are missing some validation on the shapes of the input tensors as well as ...
functionality CVE-2018-10853: ... It did not check current privilege (CPL) level while emulating unprivileged instructions ...
Vulnerability Erroneous CVE-2020-11608: ... in the Linux kernel before 5.6.1. drivers/media/usb/gspca/ov519.c allows ...
information file CVE-2014-9219: ... vulnerability in the redirection feature in [Link] in phpMyAdmin ...
Erroneous CVE-2012-0851: The ff_h264_decode_seq_parameter_set function in ...
function CVE-2020-11608: ... allows NULL pointer dereferences in ov511_mode_init_regs and ov518_mode_init_regs when ...
Related CVE-2015-2923: ... allows remote attackers to reconfigure a hop-limit setting via a small hop_limit value in ...
parameter CVE-2017-16939: ... allows local users to ... via a ... call in conjunction with XFRM_MSG_GETPOLICY Netlink messages.
Attack CVE-2011-3973: ... allows remote attackers to ... via an invalid bitstream in a Chinese AVS video (aka CAVS) file ...
payload CVE-2011-4031: ... allows remote attackers to ... via a crafted ASF packet.
Attack Attack CVE-2022-21208: ... An attacker can exploit this vulnerability by sending an unlimited number of huge chunks (e.g. 2GB each) ...
information method CVE-2018-10087: ... allow local users to cause ... by triggering an attempted use of the -INT_MIN value.
Attack CVE-2022-21678: ..., the bios of users who made their profiles private were still visible in the ‘<meta>‘ tags on their users’ pages ...
impact CVE-2018-12232: ... there is a race condition ..., leading to a NULL pointer dereference and system crash.
Patch CVE-2022-21682: ... This has been resolved in ... by changing the behaviour of ‘–nofilesystem=home‘ and ‘–nofilesystem=host‘.
Patch method CVE-2022-42725: ...The issue has been fixed with a parameter check on user input
information Patch CVE-2021-39207: ... See commit 507d066ef432ea27d3e201da08009872a2f37725 for details.
reference CVE-2017-3738: ... The fix is also avaliable in commit e502cc86d in the OpenSSL git repository.

Table 6: Information categories within commit description.

Category Example
Vulnerability owncloud:38271de: Added CSRF checks
Vulnerability type AuroCMS:790f66f: ... Update Vulnerability SQL Injection in [Link]
information Faulty djblets: 77ac646: ... The generated gravatar HTML wasn’t handling escaping of the display name of the user, allowing ...
functionality ffi:e0fe486: ... Symbols were sent directly to FFI::[Link] in the first attempt, resulting in ...
Attack djblets:77a68c0: ... This allows an attacker who can provide part of a JSON-serializable object to craft a string ...
Attack method fbthrift:3f15620: ... This allows malicious attacker to send few bytes message and cause server to allocate GBs of memory ...
information Attack djblets:77a68c0: ... This allows ... that can break out a <script>tag and create its own, injecting a custom script.
impact revive-adserver:a323fd6: ... such a vulnerability could be used by an attacker to steal the session ID of an authenticated user ...
File:71a8b6c: ... Limit regex search for BEGIN to the first 4K of the file.
Method
djblets:77a68c0: ... To fix this, we escape ’<’, ’>’, and ’&’ characters in the resulting string, preventing ...
Erroneous djblets: 77ac646: Fix a XSS vulnerability in the gravatar template tag ...
Fix file FFmpeg:a5d849b: avformat/avide.c: Limit formats in gab2 to srt ...
information Erroneous Tcpdump:a1eefe9: ... prevent a possible buffer overread in chdlc_print() ...
function openfortivpn:60660e0: ... CVE-2020-7041 incorrect use of X509_check_host (regarding return value) is fixed with ...
Related Linux:89d7ae3: ... This patch fixes this by first checking to ensure that the skb is non-NULL before using it to ...
parameter flatpak:6d1773d: ... if ..., they cannot be used to inject arbitrary code into a non-setuid bwrap via mechanisms like ...

Table 7: Information categories within commit code diff. Table 8: Confusion matrix for LLM’s determination.
LLM Predicted
Component Category
Positive Negative
Comment Program functionality Modification reason
Positive 2,141 320
Source code Program functionality Modification location Actual
Negative 12,437 80,852

19
 Table 9: Result of ablation study on Recall (R@k)

Data Feature 1 2 3 4 5 6 7 8 9 10
F0 62.14% 74.99% 78.49% 80.68% 82.45% 83.66% 84.75% 85.43% 86.27% 87.10%
Full F0 + F1 64.23% ↑ 78.80% ↑ 83.39% ↑ 86.27% ↑ 87.47% ↑ 88.41% ↑ 89.19% ↑ 89.82% ↑ 90.39% ↑ 91.07% ↑
F0 + F1 + F2 64.86% ↑ 79.22% ↑ 84.18% ↑ 86.58% ↑ 88.56% ↑ 89.45% ↑ 90.34% ↑ 90.97% ↑ 91.64% ↑ 92.74% ↑
F0 78.70% 87.57% 89.55% 90.67% 91.67% 92.39% 93.12% 93.58% 93.98% 94.25%
1-1 F0 + F1 81.35% ↑ 89.48% ↑ 91.27% ↑ 92.79% ↑ 93.25% ↑ 93.85% ↑ 94.11% ↑ 94.38% ↑ 94.58% ↑ 95.04% ↑
F0 + F1 + F2 82.14% ↑ 89.22% ↓ 91.60% ↑ 92.72% ↓ 93.65% ↑ 94.18% ↑ 94.71% ↑ 94.84% ↑ 94.84% ↑ 95.30% ↑
F0 0.00% 27.79% 36.97% 43.18% 47.89% 50.87% 53.35% 54.84% 57.32% 60.30%
1-N F0 + F1 0.00% = 38.71% ↑ 53.85% ↑ 61.79% ↑ 65.76% ↑ 67.99% ↑ 70.72% ↑ 72.70% ↑ 74.69% ↑ 76.18% ↑
F0 + F1 + F2 0.00% = 41.69% ↑ 56.33% ↑ 63.52% ↑ 69.48% ↑ 71.71% ↑ 73.95% ↑ 76.43% ↑ 79.65% ↑ 83.13% ↑

Table 10: Result of ablation study on NDCG (N@k)

Data Feature 1 2 3 4 5 6 7 8 9 10
F0 73.63% 77.20% 78.58% 79.35% 79.94% 80.36% 80.74% 81.01% 81.31% 81.52%
Full F0 + F1 78.85% ↑ 82.22% ↑ 83.77% ↑ 84.84% ↑ 85.26% ↑ 85.57% ↑ 85.85% ↑ 86.05% ↑ 86.21% ↑ 86.37% ↑
F0 + F1 + F2 80.10% ↑ 82.92% ↑ 84.62% ↑ 85.48% ↑ 86.10% ↑ 86.44% ↑ 86.73% ↑ 86.93% ↑ 87.09% ↑ 87.33% ↑
F0 78.70% 84.30% 85.29% 85.77% 86.16% 86.41% 86.66% 86.80% 86.92% 87.00%
1-1 F0 + F1 81.35% ↑ 86.48% ↑ 87.37% ↑ 88.03% ↑ 88.21% ↑ 88.42% ↑ 88.51% ↑ 88.59% ↑ 88.65% ↑ 88.79% ↑
F0 + F1 + F2 82.14% ↑ 86.61% ↑ 87.80% ↑ 88.28% ↑ 88.64% ↑ 88.83% ↑ 89.01% ↑ 89.05% ↑ 89.05% ↑ 89.18% ↑
F0 54.59% 50.56% 53.43% 55.27% 56.62% 57.62% 58.53% 59.29% 60.25% 60.99%
1-N F0 + F1 69.48% ↑ 66.22% ↑ 70.23% ↑ 72.85% ↑ 74.19% ↑ 74.86% ↑ 75.85% ↑ 76.52% ↑ 77.03% ↑ 77.30% ↑
F0 + F1 + F2 72.46% ↑ 69.10% ↑ 72.71% ↑ 74.95% ↑ 76.58% ↑ 77.47% ↑ 78.17% ↑ 78.99% ↑ 79.74% ↑ 80.40% ↑

Table 11: Result of ablation study on Manual Effort (M@k)

Data Feature 1 2 3 4 5 6 7 8 9 10
F0 1.00 1.38 1.63 1.84 2.04 2.21 2.38 2.53 2.67 2.81
Full F0 + F1 1.00 = 1.36 ↓ 1.57 ↓ 1.74 ↓ 1.87 ↓ 2.00 ↓ 2.11 ↓ 2.22 ↓ 2.32 ↓ 2.42 ↓
F0 + F1 + F2 1.00 = 1.35 ↓ 1.56 ↓ 1.72 ↓ 1.85 ↓ 1.97 ↓ 2.07 ↓ 2.17 ↓ 2.26 ↓ 2.34 ↓
F0 1.00 1.21 1.34 1.44 1.54 1.62 1.69 1.76 1.83 1.89
1-1 F0 + F1 1.00 = 1.19 ↓ 1.29 ↓ 1.38 ↓ 1.45 ↓ 1.52 ↓ 1.58 ↓ 1.64 ↓ 1.70 ↓ 1.75 ↓
F0 + F1 + F2 1.00 = 1.18 ↓ 1.29 = 1.37 ↓ 1.44 ↓ 1.51 ↓ 1.56 ↓ 1.62 ↓ 1.67 ↓ 1.72 ↓
F0 1.00 2.00 2.72 3.35 3.92 4.44 4.93 5.40 5.85 6.28
1-N F0 + F1 1.00 = 2.00 = 2.61 ↓ 3.07 ↓ 3.46 ↓ 3.80 ↓ 4.12 ↓ 4.41 ↓ 4.68 ↓ 4.94 ↓
F0 + F1 + F2 1.00 = 2.00 = 2.58 ↓ 3.02 ↓ 3.38 ↓ 3.69 ↓ 3.97 ↓ 4.23 ↓ 4.47 ↓ 4.67 ↓

20