Guiding AI to Fix Its Own Flaws: An Empirical Study on

LLM-Driven Secure Code Generation

Hao Yan Swapneel Suhas Vaidya
George Mason University George Mason University
Fairfax, USA Fairfax, USA
hyan5@[Link] svaidya4@[Link]

Xiaokuan Zhang Ziyu Yao

George Mason University George Mason University
Fairfax, USA Fairfax, USA
xiaokuan@[Link] ziyuyao@[Link]
arXiv:2506.23034v1 [[Link]] 28 Jun 2025

Abstract
Coding Question Coding Question LLM-generated
Large Language Models (LLMs) have become powerful tools for Vulnerable Code
automated code generation. However, these models often overlook
critical security practices, which can result in the generation of
insecure code that contains vulnerabilities—weaknesses or flaws in
Self-generated
the code that attackers can exploit to compromise a system. How- Vul. Hints
ever, there has been limited exploration of strategies to guide LLMs Explained
in generating secure code and a lack of in-depth analysis of the CodeQL
effectiveness of LLMs in repairing code containing vulnerabilities. Feedback

In this paper, we present a comprehensive evaluation of state-of-

the-art LLMs by examining their inherent tendencies to produce
insecure code, their capability to generate secure code when guided Assessment of Proactive Vulnerability Post-Hoc
LLMs on Code Prevention w. Self- Vulnerability Repair
by self-generated vulnerability hints, and their effectiveness in Vulnerability generated Hints w. Feedback
repairing vulnerabilities when provided with different levels of
feedback. Our study covers both proprietary and open-weight mod- Figure 1: We performed a comprehensive evaluation of LLMs
els across various scales and leverages established benchmarks to in secure code generation and repair across three dimensions.
assess a wide range of vulnerability types. Through quantitative
and qualitative analyses, we reveal that although LLMs are prone
to generating insecure code, advanced models can benefit from
GitHub Copilot generated vulnerable code in 40% of cases across
vulnerability hints and fine-grained feedback to avoid or fix vulner-
18 different types of vulnerabilities. These vulnerabilities are cate-
abilities. We also provide actionable suggestions to developers to
gorized under the Common Weakness Enumeration (CWE) [1], a
reduce vulnerabilities when using LLMs for code generation.
well-known framework that standardizes software weaknesses.
While existing studies highlight concerns regarding vulnera-
CCS Concepts
bilities in LLM-generated code, a comprehensive evaluation of
• Security and privacy → Software security engineering. LLMs across varying scales and series remains lacking. Most prior
works [4, 26, 38] have focused on a narrow set of LLMs targeting
Keywords large-scale and closed-source models (e.g., GPT-3.5/4 or GitHub
Large Language Models, Secure Code Generation, Vulnerability Copilot, etc). However, open-weight models are increasingly used in
Repair real-world applications for their flexibility, lower cost, and reduced
latency. Their efficiency makes them ideal for resource-constrained
1 Introduction environments, privacy-sensitive settings, and on-device code assis-
The adoption of large language models (LLMs) for code generation tants. Despite their growing use, comprehensive evaluations of their
has grown rapidly. These models, such as GitHub Copilot [20] and security risks and comparisons with proprietary models are still
OpenAI ChatGPT [37], have demonstrated remarkable capabilities lacking. In addition, although recent studies have shown promis-
in completing partial code snippets or generating code from natural ing results in employing LLMs especially smaller-scale LLMs [57]
language descriptions. Despite these advances, a critical concern (ranging from 220M to 20B) to ensure code functionalities, the
remains: the security of the code generated by these models. Mul- effectiveness of these models in addressing vulnerability-related
tiple studies have shown that LLM-generated code often contains problems remains largely unexplored.
vulnerabilities—defects that can be exploited to compromise soft- Addressing the security risks of LLM-based code generation
ware functionality, integrity, or confidentiality [4, 26, 38, 40, 41, 44]. requires developing effective detection [6, 14, 30, 46, 63] and mitiga-
For instance, Perry et al. [41] showed that using AI code assistants tion strategies [10, 13, 49, 56, 61]. Our work focuses on vulnerability
can produce more insecure code. Pearce et al. [38] reported that mitigation and uses CodeQL for vulnerability detection. Existing
Preprint, 2025, Yan et al.

efforts can be summarized in two paradigms, differing in their point feedback. We begin by using CodeQL [19]—an external vulnerabil-
of intervention during the code generation process. The first para- ity detection tool—to identify security issues within the generated
digm, proactive vulnerability prevention, is applied before the LLM code. The first type of feedback, termed direct feedback, is provided
to generate the code. In this paradigm, researchers design care- directly by CodeQL and includes only the raw vulnerability in-
fully crafted prompts, incorporating predetermined vulnerability formation with a short description and localization information.
information [50, 53] or demonstrations of safe coding practices [61] The second type, explained feedback, is obtained by prompting the
to guide the LLM away from known security pitfalls and steer it GPT-4o [37] to explain the CodeQL results, which provides both
toward generating secure code directly. However, these techniques the vulnerability information and detailed suggestions and actions
assume users to either possess the required expertise in construct- for resolving the issue. This design of feedback draws inspiration
ing the prompt or have access to a comprehensive code base, which from recent works on feedback-driven code debugging [12, 27, 33],
limits their broader applications. where an LLM is prompted to first enrich the raw feedback received
On the other hand, the second paradigm, post-hoc vulnerability from external evaluators and then utilize it for better debugging.
repair, is applied after vulnerable code has been generated. This However, as far as we know, none of the prior research has exam-
paradigm focuses on correcting vulnerabilities that were not pre- ined the application of this advanced approach for vulnerability
emptively addressed during code generation. While earlier works repair. By analyzing their responses to these two types of feedback,
trained or fine-tuned standalone neural models as code repair- we assess LLMs’ ability to repair vulnerable code under varying
ers [10, 13, 56], they heavily relied on existing vulnerability repair levels of guidance. This phase emulates the reviewing and refining
datasets and were unable to resolve unseen vulnerability types. In workflow that human developers adopt to enhance code security.
addition, these approaches require extensive computing resources By thoroughly examining these aspects, our study provides a
when an LLM (e.g., CodeLlama [49]) is tasked to be the code re- comprehensive understanding of modern LLMs’ strengths and
pairer. As a result, there has been an increasing interest in directly weaknesses in security-critical programming tasks. Our evaluation
prompting an LLM for vulnerable code repair [34, 35, 39], particu- shows that most LLMs generate vulnerable code at rates ranging
larly drawing on the success of reasoning prompts such as Chain from 9.8% to 42.1% across diverse vulnerabilities, with similar distri-
of Thought [55]. However, the exploration of prompting-based vul- butions observed among the most frequent types. Although vulner-
nerable code pair is only in its infancy, and many approaches are ability hints help reduce insecure code generation, self-generated
limited to leveraging only the superficial error messages returned hints often include irrelevant or incorrect information. Moreover,
by a vulnerable code detector for repair. post-hoc vulnerability repair proves effective for models with ad-
In this work, we aim to systematically examine the capabilities vanced instruction-following capabilities, and its performance is fur-
of modern LLMs in handling code vulnerabilities (Figure 1). Our ther enhanced by incorporating explained, contextualized feedback.
exploration is structured into three dimensions, each designed to ad- Contributions. To summarize, our contributions are:
dress specific gaps in understanding and improving the secure code
generation performance of LLMs, thereby addressing the following
research questions (RQs): • We systematically assess the tendency of a broad range of both
proprietary and open-weight LLMs in producing insecure code.
RQ1. Do LLMs generate vulnerable code? We examine a broader
• We investigate the effectiveness of self-generated vulnerability
range of LLMs—including both proprietary and open-weight models
hints for various LLMs in proactively detecting and preventing
spanning various scales and series to show the tendency of how
security vulnerabilities, and we offer practical guidelines for
often these LLMs generate vulnerable code. To do this, we use two
constructing effective hints.
benchmarks, SecurityEval[47] and SecCodePLT[58], which are de-
• We analyze the ability of LLMs to repair their vulnerabilities
signed to test secure code generation in Python. These benchmarks
under two levels of feedback, providing insights into their adapt-
cover a broad range of vulnerability types defined under CWE. By
ability and practical applicability in real-world development.
using these benchmarks, we provide a clear picture of the types
and frequencies of vulnerabilities each LLM is likely to produce.
RQ2. Does providing self-generated vulnerability hints help Code availability. Code can be accessed through the following
vulnerability prevention? Corresponding to proactive vulnera- anonymized repo: [Link]
bility prevention, we investigate how self-generated vulnerability Code_Generation-E89E.
hints can guide LLMs in producing secure code, thereby eliminating
the reliance on human expertise for prompt construction. The self-
generated vulnerability hints are suggestions generated by the LLM 2 Background and Related Work
itself to highlight potential security risks during code generation, 2.1 LLM-based Code Generation
guiding LLMs to prevent them. We assessed various LLMs to see
if they can effectively use these self-generated hints to minimize State-of-the-art LLMs were mostly tuned to follow natural language
code vulnerabilities. instructions [9, 43]. Therefore, the applications of LLMs are often
formulated in an instruction-following format. Specifically, given
RQ3. How does LLMs leverage different levels of feedback in
an input sequence X = (𝑥 1, 𝑥 2, ..., 𝑥 𝑁 ) comprising a natural lan-
post-hoc vulnerability repair? We test the ability of LLMs to re-
guage instruction and task-specific context (e.g., function name
pair their code vulnerabilities by incorporating two distinct types of
and arguments in code generation), the objective of an LLM is to
generate a response Y = (𝑦1, 𝑦2, ..., 𝑦𝑀 ) (e.g., code snippets in code
Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation Preprint, 2025,

generation) that aligns with the goals described in X: the model to review and debug its own code to identify and fix
𝑇 vulnerabilities, and (3) a CoT based method. The study found that
Ö
𝑃 (𝑌 | 𝑋 ) = (𝑦𝑡 | 𝑋, 𝑦1, 𝑦2, ..., 𝑦𝑡 −1 ). the RCI technique delivered the best performance, with the CWE-
𝑡 =1 specific template ranking second. Their work demonstrated the
potential of LLMs to avoid generating vulnerable codes without
The LLM generates the output sequence Y recurrently, one token
fine-tuning. Similarly, Wang et al. [53] explored enhancing secure
at a time 𝑡, by sampling based on the conditional probability distri-
code generation by incorporating vulnerability information into
bution at the current time step.
prompts. They manually refined task descriptions with detailed
definitions, security conditions, and required actions. While prior
2.2 Security Issues of LLM-generated Code
work shows the potential of using vulnerability information to
Despite advancements of LLMs in code generation, the evaluation guide LLMs in generating secure code, their approaches depend on
of LLM-generated code has predominantly focused on functional humans to define the vulnerability information in advance. This
correctness [5, 11, 22] rather than security. Recent studies have requirement limits their practicality. Our approach addresses this
highlighted a significant security concern in LLM-generated codes. by enabling LLMs to autonomously generate vulnerability hints,
Pearce et al. [38] manually designed 54 distinct scenarios spanning thereby reducing human intervention. Another line of research
18 different CWEs to evaluate GitHub Copilot and reported 40% of highlights the effectiveness of fine-tuning LLMs on security-focused
the generated code being vulnerable. Siddiq and Santos [47] con- datasets [21, 29].
structed the SecurityEval benchmark with 121 coding questions
covering 69 CWEs and revealed that approximately 68% of the code
produced by InCoder [16] and 74% generated by GitHub Copilot 2.4 Post-hoc Code Vulnerability Repair
contained vulnerabilities. Yang et al. [58] introduced SecCodePLT, While the ability of LLM-based automated program repair (APR)
a dataset containing 1,345 synthesized coding problems derived has been demonstrated effective in repairing functionality bugs [28,
from five manually designed seed questions for each covered CWE. 42, 57], their ability in fixing vulnerability lacks comprehensive
They demonstrated SecCodePLT’s effectiveness in assessing LLMs’ exploration. Earlier works [13, 17, 56] focused on training specified
secure coding capabilities across 4 LLMs and indicated 40% to 65% vulnerability repair models based on established datasets, but it
vulnerable codes were generated by those LLMs. The CyberSecEval usually overfits the vulnerabilities covered by the training data
series [7, 8, 52], proposed by Meta, provides a comprehensive bench- and the programming language. An alternative strategy to im-
mark for evaluating LLMs across various security aspects, including prove security is fine-tuning LLMs via reinforcement learning with
insecure code generation and facilitating cyber attacks. Their eval- security-focused rewards. For example, Islam et al. [25] introduced
uation of insecure code generation reveals significant failures in SecureCode, a reinforcement learning framework that fine-tunes
passing security tests using code generated by Llama2 and CodeL- LLMs with dual rewards of functionality and security. Their re-
lama. Furthermore, Khoury et al. [26] showed that GPT-3.5 [37] sults showed improvements in secure coding practices, but at the
produced 76% vulnerable codes. While these findings underscore cost of expensive training and data annotations. These challenges
the potential risks inherent in LLM-based automated code genera- have thus inspired the research of prompting-based code vulnera-
tion, a common limitation is that these explorations often focus on bility repair. A few studies have investigated the capability of LLMs
a narrow set of LLMs. This focus neglects the security performance to directly generate vulnerability patches [18, 35, 39]. For exam-
of many open-weight and smaller-scale LLMs, which leaves an ple, Pearce et al. [39] explored zero-shot vulnerability repair by
important gap in our understanding of how model size and archi- embedding vulnerability information into the prompt and experi-
tecture influence vulnerability generation. Our work tackles this menting with different prompt templates. They found that every
by evaluating a diverse range of LLMs across different sizes and test case could be successfully repaired by at least one combination
series, which offers a comprehensive analysis of how model design of templates. Nong et al. [35] carefully designed the workflow to
influences vulnerability generation and provides valuable insights explore the root cause in the vulnerable code, then retrieved ex-
into the secure coding capabilities of a broader spectrum of LLMs. amples with similar vulnerability issues, and incorporated these
examples with ground-truth annotations as demonstrations into
2.3 Proactive Code Vulnerability Prevention the prompt to guide the LLM in generating correct patches. Fu
Given the prevalence of LLM-based code generation, prompt-based et al. [18] systematically examined ChatGPT (3.5 and 4 [2]) on tasks
improvements has been largely explored in guiding the model to of code vulnerability detection, classification, and repair, and re-
generate both functional and secure code. Yin et al. [60] also ex- ported poor performance across all three aspects. These findings
plored a Chain-of-Thought [55] based method that requires the highlight the need for improved approaches to vulnerability repair.
model to explicitly articulate its reasoning steps. Their findings However, prior work on LLM-based vulnerability repair has largely
indicated significantly improved patch success rates, reducing false overlooked the impact of structured feedback. While some stud-
positives by 35% and increasing security-aware fixes to 62%. Tony ies have tested zero-shot repair capabilities, they lack an analysis
et al. [50] evaluated various prompting techniques on GPT-series of how different levels of vulnerability feedback influence repair
LLMs. Among the various prompting methods examined, three effectiveness. Our work analyzes how LLMs adapt to structured
best-performing approaches are: (1) a CWE-specific template that security feedback and whether they meaningfully apply security
embeds explicit vulnerability information into the prompt, (2) a Re- fixes based on different levels of guidance. By incorporating de-
cursively Criticizes and Improves method (RCI [27]) that prompts tailed explanations on how to fix the vulnerability, our work aims
Preprint, 2025, Yan et al.

Task Definition Prompt for Vulnerability Hints Generation

You are a professional Python developer, your task is: [Placeholder for Task Definition]
Description: Get file name from request, validate the List 5 potential vulnerabilities likely to be triggered during
request, and delete the image file. the code implementation process. Each vulnerability must
Function Definition: be in the format: ‘CWE-ID: short description’. Do not
import os include Python code, mitigations, suggestions, or any text
from flask import Flask, request other than the list of vulnerabilities in the specified format.
app = Flask(__name__)
@[Link]('/deleteImage') Example of Self-generated Vulnerability Hints
def deleteImage(): – CWE-22 Path Traversal: If the file name from the re-
(Optional) Additional function definitions defined by the quest is not properly validated, an attacker could ma-
dataset: Arguments, Return, Raise, etc. nipulate the file path to delete arbitrary files.
– ... (5 in total)
Vanilla Prompt for Code Generation
Prompt for Code Generation w. Vulnerability Hints
[Placeholder for Task Definition]
Please implement the function according to the descrip- [Placeholder for Task Definition]
tion. Return pure Python code without additional text. Here are 5 potential vulnerabilities that might be triggered
during the code implementation process:
Figure 2: Example prompt for vanilla code generation. [Placeholder for Self-generated Hints]
Please implement the function according to the de-
scription while avoiding the vulnerabilities. Return pure
Python code without additional text.
to determine whether more contextually rich feedback enhances
LLM repair performance. Figure 3: Up: Prompt for vulnerability hints generation and
an example response. Bottom: Prompt for code generation
3 Methodologies with self-generated vulnerability hints.
This section presents our methodologies for exploring and improv-
ing code security from three dimensions: 1) exploring the tendency
of LLMs to generate vulnerable code (Section 3.1), 2) improving
proactive vulnerability prevention through self-generated vulnera-
bility hints (Section 3.2), and 3) improving post-hoc vulnerable code
repair through by exploring two types of feedback (Section 3.3).
that LLMs not only introduce the target vulnerability but also gen-
3.1 Exploring Security Code Generation via erate other unintended ones. This underscores the need for a more
Vanilla Prompt robust approach to guide LLMs toward secure code generation.
We first assess how often LLMs generate insecure code using vanilla To address this, we propose to augment the input prompt with
programming prompts (see Figure 2). These prompts provide natu- self-generated vulnerability hints, which consists of two steps:
ral language instructions—including task descriptions and expected 1) Vulnerability Hints Prediction: Given the prompt (Figure 3,
functionalities-without explicit security requirements. This base- Top) defining the coding task, the LLM analyzes the described
line helps us understand the frequency and types of vulnerabilities functionality to identify potential security risks and generates a
in code generated by different LLMs. The task involves transform- list of the top-5 relevant vulnerabilities. For instance, when the
ing the vanilla programming prompt into a corresponding code task involves handling user inputs for database operations, the
snippet, which should ideally be both functional and secure. model might identify risks such as “SQL injection (CWE-89)” with
a concise description of the vulnerability. This prediction step is
essential to ensure that the vulnerability hints are relevant to the
3.2 Proactive Vulnerability Prevention via specific context of the task.
Self-generated Vulnerability Hints 2) Code Generation with Self-generated Vulnerability Hints:
To enhance code security, we explore proactive vulnerability pre- The vanilla prompt is augmented with the self-generated hints
vention—preventative identifying potential vulnerabilities and in- (shown in Figure 3, Bottom), Code Generation with Vulnerability
corporating this information into the prompt. Previous work [53] Hints. The LLM then is tasked to generate code using the aug-
showed that explicitly including the information of target vulner- mented prompt. The inclusion of such vulnerability hints provides
ability could improve code security. However, their approach re- the LLM with explicit guidance about the risks to consider while
quired manually identifying potential vulnerabilities, which limited generating the code. The expectation is that by explicitly informing
its general applicability and was impractical for handling diverse the model of potential vulnerabilities, the enhanced prompts will
real-world scenarios. In our preliminary experiments, we observed steer the LLM toward avoiding these security issues in its output.
Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation Preprint, 2025,

Prompt for Vulnerability Repair w. Direct Feedback Prompt for Explained Feedback Generation
User: [Placeholder for Vanilla Prompt] User: [Placeholder for Vanilla Prompt]
Assistant: ... (LLM-generated vulnerable code under Vanilla Assistant: ... (LLM-generated vulnerable code under Vanilla
Prompt; omitted for simplicity in presentation) Prompt; omitted for simplicity in presentation)
User: Following vulnerabilities are detected: User: Following vulnerabilities are detected:
– cwe-022: Accessing paths influenced by users can allow – cwe-022: Accessing paths influenced by users can allow
an attacker to access unexpected [Link] Line: an attacker to access unexpected [Link] Line:
17, Start Column: 27 17, Start Column: 27
Please fix all vulnerabilities. Return pure Python code For each detected vulnerability, can you provide an expla-
without additional text. nation for why the vulnerability is triggered and provide
suggestions on how to correct it. Please generate feedback
Figure 4: Example prompt for post-hoc vulnerable code repair only and do not write code.
with direct CodeQL feedback.
Example of Explained Feedback
– cwe-022: Accessing paths influenced by users can allow
3.3 Post-Hoc Code Vulnerability Repair via Two an attacker to access unexpected resources.
Levels of Feedback ■ **Explanation:** These vulnerabilities are triggered be-

Despite the promise of preventative code generation, vulnerabili- cause the file path provided by the user is directly used
ties may persist due to inherent limitations in the model’s under- without validation. This can lead to directory traversal
standing of secure practices or the complexity of the task. Post-hoc attacks where an attacker can access files outside the
vulnerability repair focuses on identifying and fixing vulnerabilities intended directory.
■ **Suggestion:** Validate and sanitize the file path to
in already generated code using a structured feedback loop. This
approach leverages external vulnerability detection tools to analyze ensure it does not contain any directory traversal char-
the code, provide feedback, and guide the LLM in generating secure acters (e.g., ‘../’). Use a whitelist of allowed file paths or
revisions. Our study uses CodeQL [19] as such a tool. CodeQL is restrict file operations to a specific directory.
a static analysis tool designed to identify vulnerabilities in source Please fix all vulnerabilities. Return pure Python code
code. The output of CodeQL includes a brief description of identi- without additional text.
fied vulnerabilities and their location within the code. Based on the
Prompt for Vulnerability Repair w. Explained Feedback
detection output of CodeQL, we explore two levels of feedback for
post-hoc vulnerable code repair. User: [Placeholder for Vanilla Prompt]
Vulnerability Repair with Direct Feedback: The results from Assistant: ... (LLM-generated vulnerable code under Vanilla
CodeQL are fed as follow-up user input in a conversational frame- Prompt; omitted for simplicity in presentation)
work, which is then used to guide the repair process (Figure 4). We User: Following vulnerabilities are detected:
denote this CodeQL feedback as “direct feedback”. [Placeholder for Explained Feedback]
Vulnerability Repair with Explained Feedback: Direct feed- Please fix all vulnerabilities. Return pure Python code
back often lacks the level of detail required to precisely guide the without additional text.
LLM in repairing vulnerabilities. Prior efforts [51, 54] have shown
that more fine-grained feedback enables models to better under- Figure 5: Up: Example prompt for explained feedback gener-
stand the instruction for refinement, which in turn leads to more ation and an example response. Bottom: Prompt for post-hoc
accurate and effective corrections. Inspired by these works, we vulnerable code repair with explained CodeQL feedback.
propose to perform post-hoc vulnerable code repair with explana-
tions of the direct CodeQL feedback, called the “explained feedback”
(Figure 5). Specifically, we feed the direct feedback from CodeQL, for evaluating secure code generation in Python using LLMs. Each
along with contextual information such as the model-generated coding task in these benchmarks is labeled with a corresponding
vulnerable code, to GPT-4o and then prompt it to explain the direct CWE-ID, called “target vulnerability”, that was used to construct
feedback and provide actionable suggestions for code repair. We the coding question.
posit that such question-grounded explained feedback can greatly SecCodePLT is predominantly a synthesized dataset covering
facilitate the vulnerability understanding the repair of an LLM. 27 CWE vulnerability types. Each coding question contained in
When prompting the LLM to repair its vulnerable code, we supply the benchmark is mutated from 5 expert-crafted seed problems by
the explained feedback as part of the instruction, as shown in the changing the description, function name, or argument names for
bottom box of Figure 5. each CWE. It contains up to 70 mutated questions per covered CWE,
resulting in a total of 1,345 coding questions. The synthesized nature
4 Experimental Setup of this dataset allows for a consistent and systematic exploration
Datasets. We conducted experiments using two benchmarks, Secu- of vulnerabilities across a wide range of security contexts. We
rityEval [48] and SecCodePLT [58], which are specifically designed excluded 6 CWEs, corresponding to 274 samples, because these are
Preprint, 2025, Yan et al.

Table 1: Summary of SecurityEval and SecCodePLT Datasets

Attribute SecCodePLT SecurityEval

Data Source Expert-annotated seeds + synthesized questions via mutation Real-world problems + expert-annotated questions
#Questions 1,071∗ 121
#CWEs Covered 21 69
Top CWEs (#Ques- CWE-22 (70), CWE-74 (60), CWE-77 (51), CWE-79 (51), CWE-20 (6), CWE-611 (6), CWE-601 (5), CWE-22 (4), CWE-327
tions per CWE) CWE-94 (51), CWE-95 (51), etc. (4), CWE-502 (4), CWE-79 (3), etc.
Overlapped CWEs CWE-22, CWE-78, CWE-79, CWE-94, CWE-95, CWE-200, CWE-295, CWE-327, CWE-347, CWE-367, CWE-400, CWE-502,
CWE-601, CWE-611, CWE-732, CWE-918 (totally, 16 overlapped CWE types)
∗ After excluding 274 samples not supported by CodeQL.

not covered by CodeQL, resulting in a final subset of 21 CWEs with Table 2: Covered LLMs’ parameter sizes, weight informa-
1,071 samples. tion, and benchmark performance (Accuracy in percentage).
SecurityEval consists of 121 coding problems spanning 69 CWE Model performance was sourced from the original model pub-
vulnerability types. As this is a relatively smaller dataset, each CWE lications unless denoted. (+ evaluated on MBPP-Plus [31]).
type is evaluated by much fewer coding questions (e.g., 6 questions
Model Size Weights HumanEval MBPP
for the most frequent type, CWE-20), compared to SecCodePLT.
7B Open 34.8% 44.4%
Each entry in the dataset was derived either from real-world coding CodeLlama-Instruct [45]
34B Open 41.5% 57.0%
practices or created manually by experts to reflect practical and
security-critical scenarios. This ensures that the dataset covers a Llama3.1-Instruct [15] 8B Open 72.6% 60.8%
broad range of vulnerabilities encountered in real-world applica- Llama3.2-Instruct [3] 3B Open 61.0% [59] 68.5% [59]
tions. The diverse nature of this dataset allows us to evaluate how StarCoder2- 15B Open 46.3% 66.2%
well models can handle a wide range of vulnerabilities. Instruct [32]
A comparison between the two datasets was shown in Table 1. DeepSeek-Coder-V2- 16B Open 81.1% 68.8%+
Lite-Instruct [64]
We note that 16 out of the 21 CWE types covered by SecCodePLT
are also evaluated on SecurityEval, and the broader coverage of GPT-3.5-turbo- N/A Closed 57.3% [23] 52.2% [23]
0125 [36]
SecurityEval assesses each LLM on another 53 CWEs.
GPT-4o-0513 [24] N/A Closed 90.2% 81.1%
LLM Coverage. To ensure a comprehensive evaluation of code
generation capabilities, we selected state-of-the-art LLMs (shown in
Table 2) through a systematic framework prioritizing three criteria: generated by each model using CodeQL. we introduce the Target
model diversity, scalability, and efficiency. Our goal is to choose Vulnerability Rate (TarV-R), which quantifies the percentage of
models that vary in design, cover different sizes, and reflect practi- vulnerable code snippets that include the target vulnerability:
cal constraints, while still grounding our choices in actual bench-
mark results. All models we considered were instruction-tuned # 𝑉𝑢𝑙𝑛𝑒𝑟𝑎𝑏𝑙𝑒 𝑐𝑜𝑑𝑒𝑠 𝑤𝑖𝑡ℎ 𝑡ℎ𝑒 𝑡𝑎𝑟𝑔𝑒𝑡 𝑣𝑢𝑙𝑛𝑒𝑟𝑎𝑏𝑖𝑙𝑖𝑡𝑦
versions so they can perform all tasks we introduced in Section 3, 𝑇 𝑎𝑟𝑉 -𝑅 =
# 𝑇𝑜𝑡𝑎𝑙 𝑠𝑎𝑚𝑝𝑙𝑒𝑠
and their coding capabilities are quantified using HumanEval [11]
and MBPP [5], the two standard code-generation benchmarks. Additionally, we define the All Vulnerability Rate (AllV-R) as
First, model diversity drives the inclusion of distinct architec- the percentage of code snippets containing at least one vulnerability,
tural families to capture differences in training paradigms and spe- regardless of whether it is the target vulnerability:
cialization. For instance, we compare the code-optimized CodeL-
lama series against the general-purpose Llama3 family and the # 𝑉𝑢𝑙𝑛𝑒𝑟𝑎𝑏𝑙𝑒 𝑐𝑜𝑑𝑒𝑠 𝑤𝑖𝑡ℎ 𝑎𝑛𝑦 𝑣𝑢𝑙𝑛𝑒𝑟𝑎𝑏𝑖𝑙𝑖𝑡𝑦
𝐴𝑙𝑙𝑉 -𝑅 =
performance-leading DeepSeek-Coder-V2 and GPT-4o. Second, scal- # 𝑇𝑜𝑡𝑎𝑙 𝑠𝑎𝑚𝑝𝑙𝑒𝑠
ability is addressed by selecting models across parameter counts For both metrics, a lower rate indicates that the LLM gener-
(3B to 34B), enabling direct comparisons of size-performance trade- ates fewer vulnerable code snippets, demonstrating better security
offs. For example, CodeLlama-34B (57.0% HumanEval) is paired performance in code generation (i.e., the lower the better).
with its smaller 7B variant (44.4%) to isolate scaling effects, while
StarCode2-15B and DeepSeek-Coder-V2-Lite bridges the mid-sized 5 Experimental Results
tier. Third, efficiency considerations justify lightweight models like
Llama3.2-3B, included despite lacking benchmark evaluations due
5.1 RQ1: Do LLMs Generate Vulnerable Code?
to its unique role as the smallest production-ready code model for Overall Evaluation Result. To begin with, we measure how of-
resource-constrained environments. This approach ensures each ten various LLMs generate vulnerable code using vanilla prompts
model serves a specific comparative role. (Section 3.1). The results, presented in Table 3, reveal substantial dif-
ferences among models across two datasets. On SecCodePLT, GPT-
Evaluations. To evaluate the performance of LLMs in secure code
4o exhibits the highest vulnerability rate (TarV-R: 15.0%, AllV-R:
generation, we measured the percentage of vulnerable code snippets
18.0%), while models such as StarCoder2-15B and DeepSeekV2-16B
Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation Preprint, 2025,

Table 3: The percentage of vulnerable codes detected in LLM- code. This finding is significant because it implies that different
based code generation using vanilla prompts. models tend to exhibit similar weaknesses. Such overlap may indi-
cate common limitations in the underlying model architectures or
SecCodePLT SecurityEval training datasets, suggesting that addressing these prevalent vul-
Model
TarV-R AllV-R TarV-R AllV-R nerability types could lead to improvements in the overall security
CodeLlama-7B 9.8% 14.7% 19.0% 28.9% of these models.
CodeLlama-34B 6.0% 11.8% 9.9% 16.5% Code-Optimized Model vs. Generalized Models. Models such
Llama3.1-8B 11.2% 15.3% 24.0% 40.5% as CodeLlama, StarCoder2, and DeepSeek-Coder-V2 are fine-tuned
Llama3.2-3B 9.8% 11.5% 23.1% 40.5% on extensive code repositories, which may equip them with better
StarCoder2-15B 4.0% 12.8% 24.0% 37.2% capability to generate code that is both syntactically and semanti-
DeepSeekV2-16B 4.2% 9.8% 27.3% 42.1% cally accurate. This specialized training on common coding patterns
GPT-3.5-turbo 7.7% 11.4% 12.4% 19.8% seems to help them steer clear of the vulnerabilities addressed by
GPT-4o 15.0% 18.0% 25.6% 41.3%
SecCodePLT, yet when it comes to managing a broader range of
vulnerabilities on SecurityEval, they do not demonstrate significant
strengths.
show significantly lower rates (TarV-R: 4.0% and 4.2%, respectively).
Comparing Models in the Same Family. In addition to the com-
In terms of the all-vulnerability rate (AllV-R), which indicates the
parison between individual LLMs, we found that within the same
percentage of all generated code containing any vulnerability, a
model families, scaling generally improves security, but exceptions
similar pattern emerges on SecCodePLT: DeepSeekV2-16B (AllV-R:
exist. CodeLlama-34B consistently outperforms its smaller coun-
9.8%) and GPT-3.5-turbo (AllV-R: 11.4%) outperform models like
terpart, CodeLlama-7B, with significantly lower vulnerability rates
Llama3.1-8B (AllV-R: 15.3%) and GPT-4o (AllV-R: 18.0%). However,
across both datasets. This trend aligns with previous findings that
when evaluated on SecurityEval, vulnerability rates increase dra-
larger models, with greater parameter capacity, better generalize
matically. DeepSeekV2-16B now reaches the highest rate (TarV-R:
secure coding practices [8]. However, an inverse pattern emerges
27.3%, AllV-R: 42.1%), and GPT-4o remains among the most vulner-
when comparing newer models to their predecessors. GPT-4o ex-
able (TarV-R: 25.6%, AllV-R: 41.3%).
hibits a higher vulnerability rate than GPT-3.5-turbo (SecCodePLT
The divergence in vulnerability rates across the two benchmarks
TarV-R: 15.0% vs. 7.7%, SecurityEval TarV-R: 25.6% vs. 12.4%). Simi-
can be attributed to the variations in the set of CWE types that
larly, Llama3 models show increased vulnerability rates compared
these datasets encompass (Table 1). In particular, while the two
to CodeLlama models, which were built on top of Llama2. This
datasets share a subset of CWE types, SecurityEval encompasses
raises critical questions about whether architectural modifications,
a broader variety of vulnerability types, typically with a few in-
broader training data coverage, or altered fine-tuning strategies
stances per type. Consequently, the higher vulnerability rate on
contribute to the increased vulnerability in newer models.
SecurityEval implies that the models face significant challenges
Takeaway. All models frequently generate vulnerable code across
when dealing with a diverse set of vulnerabilities. For models show-
diverse vulnerability types, with average rates on SecCodePLT at
ing dramatically different vulnerability rates on the two datasets
TarV-R: 8.5% and AllV-R: 13.2%, and on SecurityEval at TarV-R:
(e.g., DeepSeek-V2, with 4.2% TarV-R on SecCodePLT but 27.3%
20.7% and AllV-R: 33.4%. Moreover, all models tend to generate
TarV-R on SecurityEval), the results imply that they reveal different
vulnerabilities for both common (top-25) and less common types,
extents of vulnerability to various CWE types.
indicating similar weaknesses likely stemming from shared limi-
Finally, the comparison between TarV-R and AllV-R highlights
tations in their architectures or training data. This overlap high-
that LLMs can introduce additional, unintended vulnerabilities be-
lights the need for comprehensive training and robust vulnerability
yond those explicitly targeted. This observation underscores the
mitigation strategies, as focusing solely on a limited set of vulnera-
challenges these models face when handling the practical and di-
bilities is insufficient. Code-optimized models like CodeLlama and
verse coding problems present in real-world security scenarios.
StarCoder2 generally yield safer code, while newer models such as
Vulnerability Distribution. To gain deeper insight into the
GPT-4o and Llama3 exhibit higher vulnerability rates, suggesting
vulnerabilities triggered by each model, we plotted the most fre-
that increased complexity can lead to security gaps.
quently occurring vulnerabilities across models in Figure 6. Specifi-
cally, we present the top-10 vulnerabilities generated by each model Answering RQ1: Our evaluation shows that LLMs consistently
on SecCodePLT. We exclude SecurityEval as it includes only a few generate vulnerable code with diverse vulnerability types. Al-
questions per vulnerability type, which makes it unsuitable for a though the absolute vulnerability rates differ, the similarity in
meaningful plot. Notably, the top-10 vulnerabilities across the eight the distribution of top-10 vulnerability types across models indi-
models we experimented with largely overlap, resulting in only 11 cates similar weaknesses likely stemming from shared limitations
unique vulnerability types, which indicates that most models share in their architectures or training data.
similar frequent vulnerabilities. Out of these 11 vulnerabilities, 6
are not featured among the Top 25 vulnerabilities listed on the CWE 5.2 RQ2: Does Providing Self-generated
website.1 This suggests that the models are equally inclined to gen- Vulnerability Hints Help Vulnerability
erate vulnerable code to both common (ranked top-25) and less Prevention?
common (ranked out of top-25) vulnerabilities found in real-world
Overall Evaluation Result. We next evaluate whether self-generated
1 [Link] vulnerability hints can reduce code vulnerabilities. As shown in
Preprint, 2025, Yan et al.

Figure 6: The distribution of the top-10 vulnerabilities across all LLMs on SecCodePLT. In practice, models share most of the
frequent vulnerability types, which results in 11 unique vulnerabilities in total.

Table 4: Percentage of vulnerable code with self-generated Table 5: Breakdown of TarV-R categorized based on whether
vulnerability hints. Subscript numbers show changes vs. the hints include the target vulnerability. Numbers in paren-
vanilla prompts (Table 3). theses represent the count of all instances for each category.
The left part of each column shows the TarV-R under the
SecCodePLT SecurityEval same group of questions using the vanilla prompt.
Model
TarV-R AllV-R TarV-R AllV-R
CodeLlama-7B 9.8% (+0.0) 12.7% (−2.0) 22.3% (+3.3) 30.0% (+1.1) TarV-R TarV-R
Model
CodeLlama-34B 6.8% (+0.8) 8.8% (−3.0) 12.4% (+2.5) 24.0% (+7.5) (Hints w. TarV) (Hints w/o TarV)
Llama3.1-8B 6.6% (−4.6) 11.3% (−4.0) 15.7% (−8.3) 32.2% (−8.3) SecCodePLT
Llama3.2-3B 9.6% (−0.2) 9.9% (−1.6) 20.7% (−2.4) 44.6% (+4.1)
CodeLlama-7B 6.7% → 4.4% (270) 10.9% → 11.6% (801)
StarCoder2-15B 3.6% (−0.4) 7.7% (−5.1) 24.0% (+0.0) 38.8% (+1.6)
CodeLlama-34B 7.0% → 7.3% (412) 5.3% → 6.5% (659)
DeepSeekV2-16B 3.9% (−0.3) 8.1% (−1.7) 25.6% (−1.7) 45.4% (+3.3)
Llama3.1-8B 6.8% → 3.5% (515) 12.6% → 9.5% (556)
GPT-3.5-turbo 2.4% (−5.3) 7.1% (−4.3) 22.3% (+9.9) 37.2% (+17.4)
Llama3.2-3B 8.0% → 6.4% (188) 10.2% → 10.3% (883)
GPT-4o 2.6% (−12.4) 12.5% (−5.5) 13.2% (−12.4) 24.0% (−17.3)
StarCoder2-15B 2.8% → 1.5% (397) 4.7% → 4.9% (674)
DeepSeekV2-16B 3.5% → 2.0% (509) 4.8% → 5.7% (562)
GPT-3.5-turbo 7.2% → 1.0% (625) 8.5% → 4.5% (446)
Table 4, the outcomes vary across models. Notably, GPT-4o achieves GPT-4o 14.0% → 0.4% (765) 17.6% → 8.2% (306)

a 12.4% reduction in TarV-R on both datasets and 5.5% and 17.3% SecurityEval
reduction in AllV-R respectively, while Llama3.1-8B experiences CodeLlama-7B 11.8% → 17.6% (17) 20.2% → 23.1% (104)
declines of around 4.0% and 8.3% on in TarV-R and AllV-R across CodeLlama-34B 15.0% → 10.0% (40) 7.4% → 13.6% (81)
Llama3.1-8B 32.3% → 13.0% (31) 21.1% → 16.7% (90)
both datasets, respectively. These reductions suggest that newer Llama3.2-3B 45.5% → 27.0% (11) 20.9% → 20.0% (110)
models can effectively leverage security hints to improve code StarCoder2-15B 31.0% → 17.2% (29) 21.7% → 26.1% (92)
safety. In contrast, several other models show minimal change or DeepSeekV2-16B 27.5% → 15.0% (40) 27.2% → 30.9% (81)
GPT-3.5-turbo 23.5% → 21.6% (51) 4.3% → 22.9% (70)
even an unexpected increase in TarV-R and AllV-R when provided GPT-4o 29.0% → 8.1% (62) 22.0% → 18.6% (59)
with self-generated hints. These mixed observations have made
us wonder: are the opposite effects caused by potentially low qual-
ity of self-generated vulnerability hints, or are they caused by the
models’ inability in making use of the hints? To answer this ques-
tion, we start with analyzing the quality of models’ self-generated where they do not (Hints w/o TarV). Comparing their effectiveness
vulnerability hints. in reducing TarV-R to the vanilla prompt baseline (Table 5), we
Evaluating the Relevance and Preciseness of Self-Generated find that hints including the target vulnerability generally lead to
Vulnerability Hints. We evaluate these hints based on their rele- lower TarV-R (except a tie for CodeLlama-34B on SecCodePLT and
vance and preciseness. Relevance checks if the top-5 predicted hints an increase for CodeLlama-7B on SecurityEval), while hints that
include the target vulnerability, while preciseness ensures that the omit it tend to have a negative effect. In fact, when the hints do
descriptions match their official definitions. For the former, we di- not include the target vulnerability, all models except Llama-3.1-8B
vided the instances into two groups—those where self-generated and GPT-4o on both datasets show an increase in TarV-R, indicat-
hints include the target vulnerability (Hints w. TarV) and those ing that irrelevant hints may mislead the model and introduce an
Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation Preprint, 2025,

unintended increase in TarV-R. This underscores the importance of Table 6: Example of target CWE definition, self-generated
including only the most relevant vulnerability hints. hint by GPT-4o, and the contextualized target hint.
While relevance confirms that the hint contains the target vul-
Question Compute the n-th fibonacci number using loops.
nerability, preciseness ensures that its description is correct. We
assess preciseness by verifying that the predicted description of Target CWE CWE-835: Loop with Unreachable Exit Condition
the target vulnerability aligns with its formal definition, which (Definition) (’Infinite Loop’)
we obtained from the official website. This preciseness is essential, Self-generated CWE-835: Infinite Loop - If there is no condition
as any ambiguity and incorrect description can lead to misinter- Hint (GPT-4o) to break out of the loop, the function could run
pretation and diminish the hint’s effectiveness in guiding secure indefinitely
code generation. To evaluate preciseness, we followed the idea of Contextualized The vulnerability is likely to be introduced if the
“LLMs as judge” [62] and used GPT-4o to evaluate each model’s self- Hint (GPT-4o) loop’s exit condition is not properly defined or up-
dated within the loop. If the variables controlling
generated vulnerability hints. Specifically, we prompted GPT-4o
the loop’s termination (such as the loop counter or
with each model’s predicted target-vulnerability hint alongside the the Fibonacci sequence elements) are not correctly
CWE’s official definition, and then instructed it to provide a binary managed, the loop may never reach a condition
judgment on the preciseness of the hint description. We conducted where it can exit, leading to an infinite loop. This
human validation on 80 self-generated hint–definition pairs judged can occur due to errors in initializing, updating, or
by GPT-4o. Two evaluators assessed whether the hint accurately re- comparing these variables, making the loop’s exit
condition unattainable.
flected its definition. Disagreements (5 cases) were resolved through
discussion. GPT-4o’s judgments aligned with human evaluations at
95.65% precision and 91.67% recall.
From the results, GPT-4o was shown to generate the most precise target hints for all instances in SecurityEval and for 100 randomly
hint descriptions, with 98.6% of its self-generated hints being judged selected instances in SecCodePLT to ensure that each hint precisely
to be precise, which potentially contributed to its reduced TarV- addressed the target vulnerability in context. Compared to the self-
R when the target vulnerability hint was provided. In contrast, generated hints, the CWE Definition is less informative due to its
CodeLlama-7B showed a much worse performance, with only 46.5% lack of additional descriptive context, whereas the contextualized
of its self-generated hints judged to be precise, highlighting a likely hints offer a more grounded explanation under the coding scenario
reason for its limited or negative impact in lowering TarV-R. Other as shown in Table 6.
models achieved the following precision: CodeLlama-34B at 67.4%, We compared their effectiveness by measuring changes in TarV-
Llama-3.1-8B at 79.7%, Llama-3.2-3B at 72.5%, StarCoder2-15B at R (Table 7). Results indicate that contextualized hints generally lead
75.0%, DeepSeekCoderV2-16B at 82.6%, and GPT-3.5-turbo at 92.7%. to larger reductions in TarV-R compared to the CWE Definition es-
Exploring the Informativeness of Target Vulnerability Hints pecially on the SecurityEval dataset. This highlights their advantage
for Vulnerable Code Prevention. Our previous analysis empha- in clearly relating vulnerabilities to the provided coding context.
sizes the need for relevant and precise vulnerability hints in pre- Additionally, for models like GPT-4o and DeepSeek-Coder-V2, these
venting insecure code generation. However, even for GPT-4o, which hints also reduce AllV-R, suggesting that focused, context-aware
was shown to generate precise hints, it still produces target vulner- hints not only deepen understanding and provide clear guidance
ability on SecurityEval (8.1% TarV-R). We thus wonder: is there any of the target vulnerability, but also foster overall secure coding
other property that is necessary for an effective vulnerability hint? practices. These improvements underscore that while relevance is
In our further analysis, we examine the effect of contextualized essential, precise, context-aware vulnerability hints are critical for
vulnerability hints on preventing insecure code generation. Here, guiding LLMs toward secure code generation.
we define contextualized vulnerability hints as those that are not Model Instruction-following Capability. Despite the use of
only precise in their definition, but also provide explanations of precise and contextualized hints, some models (e.g., CodeLlama-7B
potential vulnerability based on the specific coding context (i.e., and 34B) still showed minimal improvement. This may indicate
the coding question and provided function signature, etc.). We hy- inherent limitations in these models’ ability to follow instructions
pothesize that such contextualized hints help LLMs better grasp and utilize provided information effectively. In contrast, advanced
security concerns and how to prevent them. models like GPT-4o, GPT-3.5-turbo, and DeepSeek-Coder-V2 consis-
Specifically, to isolate the confounding factor of vulnerability tently benefited from contextualized hints, emphasizing the role of
type relevance, we focused exclusively on the target vulnerability. both hint quality and the models’ instruction-following capabilities
That is, we only include descriptions about the target vulnerability in vulnerability prevention.
type in the hint, as opposed to listing all the top-5 potential vul- Takeaway. Our findings indicate that self-generated vulnera-
nerability types as in prior experiments. For each target CWE, we bility hints can enhance code security, but their impact relies on
introduced two types of hints. The first, termed CWE Definition, both their relevance to the correct vulnerability and the precise-
includes only the CWE-ID and the vulnerability name from the ness of their descriptions. When these hints are further contex-
official CWE website, which represents hints that are precise but tualized—explicitly linking the vulnerability to the specific sce-
not grounded onto the coding context. The second, Contextualized nario—they outperform basic definitions. Powerful models like
Target Hint, is generated using GPT-4o and expands on the CWE GPT-4o also show reduced AllV-R, implying that contextualized
definition by explicitly grounding the vulnerability onto the coding hints not only clarify the target vulnerability but also promote
context. We manually verified the correctness of the contextualized broader secure coding practices.
Preprint, 2025, Yan et al.

Table 7: The percentage of vulnerable codes with curated Table 8: The percentage of vulnerable codes after post-hoc
CWE Definition vs. Contextualized hints incorporating only repair under two types of feedback. Numbers in the subscript
the target vulnerability. Numbers in the subscript indicate indicate changes compared to using vanilla prompts (Table 3).
changes compared to using vanilla prompts (Table 3).
SecCodePLT SecurityEval
CWE Definition Contextualized Hints Model AllV-R AllV-R
Model
TarV-R AllV-R TarV-R AllV-R Direct Explained Direct Explained
SecCodePLT CodeLlama-7B 14.4% (−0.3) 13.9% (−0.8) 28.9% (+0.0) 28.9% (+0.0)
CodeLlama-7B 9.3% (−0.5) 14.7% (+0.0) 9.3% (−0.5) 14.7% (+0.0) CodeLlama-34B 10.2% (−1.6) 10.2% (−1.6) 16.5% (+0.0) 16.5% (+0.0)
CodeLlama-34B 5.9% (−0.1) 11.7% (−0.1) 5.4% (−0.6) 11.8% (+0.0) Llama3.1-8B 12.1% (−3.2) 10.1% (−5.2) 40.5% (+0.0) 40.5% (+0.0)
Llama3.1-8B 6.2% (−5.0) 14.0% (−1.3) 4.6% (−6.6) 13.1% (−2.2) Llama3.2-3B 10.0% (−1.5) 9.1% (−2.4) 40.5% (+0.0) 40.5% (+0.0)
Llama3.2-3B 7.7% (−2.1) 11.5% (+0.0) 5.7% (−4.1) 11.0% (−0.5) StarCoder2-15B 12.0% (−0.8) 11.7% (−1.1) 37.2% (+0.0) 37.2% (+0.0)
StarCoder2-15B 3.6% (−0.4) 10.9% (−1.9) 3.2% (−0.8) 10.0% (−2.8) DeepSeekV2-16B 5.4% (−4.4) 2.4% (−7.4) 28.9% (−13.2) 20.7% (−21.4)
DeepSeekV2-16B 2.8% (−1.4) 5.8% (−4.0) 2.3 % (−1.9) 5.8% (−4.0) GPT-3.5-turbo 4.3% (−7.1) 3.5% (−7.9) 10.7% (−9.1) 6.6% (−13.2)
GPT-3.5-turbo 1.1% (−6.6) 4.3% (−7.1) 0.9% (−6.8) 3.9% (−7.5) GPT-4o 6.8% (−11.2) 3.7% (−14.3) 13.2% (−28.1) 13.2% (−28.1)
GPT-4o 1.2% (−13.8) 7.0% (−10.2) 1.1% (−13.9) 5.2% (−12.8)
SecurityEval
CodeLlama-7B 19.0% (+0.0) 28.9% (+0.0) 19.0% (+0.0) 28.9% (+0.0) to infer the underlying reasoning. For models that show little or no
CodeLlama-34B 8.7% (−1.2) 16.5% (+0.0) 8.7% (−1.2) 18.2% (+1.7) improvement, this may be due to their limitations in instruction
Llama3.1-8B 12.0% (−12.0) 24.0% (−16.5) 10.8% (−13.2) 21.5% (−19.0)
Llama3.2-3B 19.8% (−3.3) 36.4% (−4.1) 16.5% (−6.6) 36.4% (−4.1)
following. These findings align with our observations in Section 5.2,
StarCoder2-15B 20.0% (−4.0) 33.0% (−4.2) 17.6% (−6.4) 33.0% (−4.2) where larger and newer models demonstrated a better ability to
DeepSeekV2-16B 20.9% (−6.4) 33.9% (−8.2) 18.8% (−8.5) 28.9% (−13.2) follow human instructions. In contrast, the limited impact of direct
GPT-3.5-turbo 12.4% (+0.0) 19.0% (−0.8) 9.1% (−3.3) 16.5% (−3.3) feedback suggests weaker reasoning capabilities for applying fixes
GPT-4o 9.9% (−15.7) 23.7% (−17.6) 8.3% (−17.3) 22.0% (−19.3)
without detailed guidance.
Takeaway. These findings highlight the effectiveness of feedback-
driven vulnerability repair. Direct feedback, based on raw CodeQL
Answering RQ2: Self-generated vulnerability hints can effectively results, requires models to reason through the issue and determine
reduce vulnerabilities, but it relies on whether the self-generated appropriate fixes on their own. In contrast, explained feedback,
hints can precisely include the target vulnerability and are contex- which explicitly details the vulnerability context and outlines the
tualized in the coding scenario. Imprecise or irrelevant hints, in necessary correction steps, significantly enhances repair effective-
contrast, may mislead the model and increase vulnerability rates. ness. On the other hand, models that struggle to utilize either type
of feedback effectively reveal limitations in their reasoning and
5.3 RQ3: How Does LLMs Leverage Different instruction-following abilities.
Levels of Feedback in Post-hoc Vulnerability Answering RQ3: More powerful LLMs are effective in post-hoc
Repair? vulnerability repair when provided with detailed, explained feed-
back rather than raw direct feedback. Conversely, less powerful
Overall Evaluation Result. Finally, we investigate how effectively
models struggle to fully utilize either type of feedback.
LLMs repair their self-generated vulnerable code when guided by
different levels of CodeQL feedback. For each LLM, we collected a
subset of its generated code where any vulnerability was detected,
6 Discussion
and then provided CodeQL feedback for self-repair. The explained Threats to validity. We acknowledge several threats to the validity
CodeQL feedback is generated using the most powerful GPT-4o of the results and findings of our study. First, our evaluation is based
for all subsets. We manually validated the explained feedback by on a static vulnerability detection tool, CodeQL, which effectively
randomly selecting 10 vulnerable code instances per model and identifies many common security issues through static analysis,
dataset, totaling 160 samples for assessment. The results indicate but fails to capture dynamic vulnerabilities or those that manifest
that over 98% of the explained feedback generated by GPT-4o is under specific runtime conditions. Second, our experiments focus
correct and offers contextualized, actionable suggestions for fixing exclusively on Python code. Although Python is widely used and
vulnerabilities. In the remaining 3 instances, although the actionable provides a practical testbed for secure code generation, its unique
suggestions were less clear, they still provided valuable guidance for features and common usage patterns may not represent the security
addressing the issues. We then apply these two types of feedback challenges of other programming languages such as C++ or Java.
for vulnerability repair. Consequently, the trend of unique vulnerabilities for specific lan-
As shown in Table 8, GPT-4o, DeepSeekCoderV2-16B, and GPT- guage needs to be noticed and explored separately. Third, though
3.5-turbo exhibit considerable improvements, particularly when we strive to cover a broad range of vulnerabilities, our study is
using explained feedback. This indicates that the explained feed- limited to two prominent datasets. These datasets might not cover
back helps guide effective vulnerability repairs. Explained feedback the entire range of potential vulnerabilities, potentially leaving out
goes beyond merely stating the issue; it provides detailed reason- some other issues. Moreover, our evaluation framework operates
ing that clarifies why a piece of code is vulnerable and outlines in a static, single-turn setting, which contrasts with real-world de-
how to fix it. This extra context simplifies vulnerability repair by velopment environments where developers engage in multi-turn
allowing the model to follow clear instructions rather than having interactions with LLMs to iteratively refine and debug code. This
Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation Preprint, 2025,

static approach may not fully capture the potential improvements References
achievable through dynamic, conversational exchanges. [1] [n. d.]. [Link]
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren-
Suggestions for developers. Our findings suggest several strate- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal
gies for developers aiming to integrate LLMs into secure coding Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
workflows. Firstly, adapting different LLMs at various stages could (2023).
[3] Meta AI. [n. d.]. Llama 3.2: Revolutionizing edge ai and vision with open, cus-
be promising: smaller code-optimized models can be used to gener- tomizable models. [Link]
ate initial code with fewer vulnerabilities, while larger models are edge-mobile-devices/
[4] Owura Asare, Meiyappan Nagappan, and N Asokan. 2023. Is github’s copilot
applied to vulnerability reasoning and repair. This mixed strategy as bad as humans at introducing vulnerabilities in code? Empirical Software
can offer a cost-effective path to safer code. Secondly, it is evident Engineering 28, 6 (2023), 129.
that incorporating fine-grained, explained feedback is essential for [5] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk
Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le,
effective vulnerability repair, as demonstrated by the significant et al. 2021. Program synthesis with large language models. arXiv preprint
improvements observed with detailed explanations and actionable arXiv:2108.07732 (2021).
suggestions. Developers should design their systems to provide [6] Pavel Avgustinov, Oege De Moor, Michael Peyton Jones, and Max Schäfer. 2016.
QL: Object-oriented queries on relational data. In 30th European Conference on
rich, actionable insights (e.g., incorporating a powerful reasoning Object-Oriented Programming (ECOOP 2016). Schloss Dagstuhl–Leibniz-Zentrum
model for vulnerability hints predictions and vulnerability repair für Informatik, 2–1.
[7] Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song,
suggestions) that guide the LLM in identifying and fixing security Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil,
flaws. Thirdly, expanding the set of vulnerabilities is crucial, as the et al. 2024. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for
lower performance of LLMs on the complex SecurityEval dataset in- large language models. arXiv preprint arXiv:2404.13161 (2024).
[8] Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evti-
dicates. By including a broader range of vulnerabilities, developers mov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo
can enhance model robustness and better address diverse security Fontana, et al. 2023. Purple llama cyberseceval: A secure coding benchmark for
risks in real-world scenarios. language models. arXiv preprint arXiv:2312.04724 (2023).
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
7 Conclusion Askell, et al. 2020. Language models are few-shot learners. Advances in neural
information processing systems 33 (2020), 1877–1901.
In this work, we conducted a comprehensive evaluation of LLMs [10] Quang-Cuong Bui, Riccardo Scandariato, and Nicolás E. Díaz Ferreyra. 2022.
in the domains of secure code generation and vulnerability repair. Vul4J: A Dataset of Reproducible Java Vulnerabilities Geared Towards the Study
of Program Repair Techniques. In 2022 IEEE/ACM 19th International Conference
Our study revealed that while LLMs are inherently prone to gener- on Mining Software Repositories (MSR). 464–468. doi:10.1145/3524842.3528482
ating insecure code, their security performance can be significantly [11] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde
improved through the incorporation of self-generated vulnerabil- De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph,
Greg Brockman, et al. 2021. Evaluating large language models trained on code.
ity hints and explained, contextualized feedback. Importantly, the arXiv preprint arXiv:2107.03374 (2021).
effectiveness of self-generated hints is contingent upon their rele- [12] Xinyun Chen, Maxwell Lin, Nathanael Schaerli, and Denny Zhou. 2023. Teach-
vance and preciseness. Contextualized feedback can result in lower ing Large Language Models to Self-Debug. In The 61st Annual Meeting Of The
Association For Computational Linguistics.
vulnerability rates in code repair tasks. Moreover, our comparative [13] Zimin Chen, Steve Kommrusch, and Martin Monperrus. 2022. Neural transfer
analysis across a diverse set of models highlighted that more ad- learning for repairing security vulnerabilities in c code. IEEE Transactions on
Software Engineering 49, 1 (2022), 147–165.
vanced models tend to benefit more from the provided hints and [14] Domenico Cotroneo, Roberta De Luca, and Pietro Liguori. 2025. DeVAIC: A
feedback. tool for security assessment of AI-generated code. Information and Software
Technology 177 (2025), 107572.
[15] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad
Acknowledgments Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan,
The project was sponsored by the Virginia Commonwealth Cyber et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024).
[16] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi,
Initiative (CCI). Hao Yan was also partially funded by the GRA Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A
Fellowship from the Center for Advancing Human-Machine Part- generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999
nership (CAHMP) at GMU. The project was supported by resources (2022).
[17] Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Van Nguyen, and Dinh Phung.
provided by the Office of Research Computing at George Mason 2022. VulRepair: a T5-based automated software vulnerability repair. In Pro-
University ([Link] and funded in part by grants from ceedings of the 30th ACM joint european software engineering conference and
symposium on the foundations of software engineering. 935–947.
the National Science Foundation (Award Number 2018631). [18] Michael Fu, Chakkrit Kla Tantithamthavorn, Van Nguyen, and Trung Le. 2023.
Chatgpt for vulnerability detection, classification, and repair: How far are we?. In
2023 30th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 632–636.
[19] GitHub. [n. d.]. [Link]
[20] GitHub. 2021. Copilot. [Link]
copilot-ai-pair-programmer/.
[21] Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Martin Vechev. 2024. In-
struction tuning for secure code generation. In Proceedings of the 41st International
Conference on Machine Learning. 18043–18062.
[22] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora,
Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob
Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. NeurIPS
(2021).
[23] Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming
Cui. 2023. Agentcoder: Multi-agent-based code generation with iterative testing
and optimisation. arXiv preprint arXiv:2312.13010 (2023).
Preprint, 2025, Yan et al.

[24] Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2025. [48] Mohammed Latif Siddiq and Joanna C. S. Santos. 2022. SecurityEval Dataset:
CODESIM: Multi-Agent Code Generation and Problem Solving through Mining Vulnerability Examples to Evaluate Machine Learning-Based Code Gen-
Simulation-Driven Planning and Debugging. arXiv preprint arXiv:2502.05664 eration Techniques. In Proceedings of the 1st International Workshop on Min-
(2025). ing Software Repositories Applications for Privacy and Security (MSR4P&S22).
[25] Nafis Tanveer Islam, Mohammad Bahrami Karkevandi, and Peyman Najafirad. doi:10.1145/3549035.3561184
2024. Code security vulnerability repair using reinforcement learning with large [49] André Silva, Sen Fang, and Martin Monperrus. 2023. Repairllama: Efficient
language models. arXiv preprint arXiv:2401.07031 (2024). representations and fine-tuned adapters for program repair. arXiv preprint
[26] Raphaël Khoury, Anderson R Avila, Jacob Brunelle, and Baba Mamadou Camara. arXiv:2312.15698 (2023).
2023. How secure is code generated by chatgpt?. In 2023 IEEE International [50] Catherine Tony, Nicolás E Díaz Ferreyra, Markus Mutas, Salem Dhiff, and Ric-
Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2445–2451. cardo Scandariato. 2024. Prompting techniques for secure code generation: A
[27] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. Language models can systematic investigation. arXiv preprint arXiv:2407.07064 (2024).
solve computer tasks. Advances in Neural Information Processing Systems 36 [51] Manya Wadhwa, Xinyu Zhao, Junyi Jessy Li, and Greg Durrett. 2024. Learn-
(2023), 39648–39677. ing to Refine with Fine-Grained Natural Language Feedback. In Findings of the
[28] Sophia D Kolak, Ruben Martins, Claire Le Goues, and Vincent Josua Hellendoorn. Association for Computational Linguistics: EMNLP 2024. 12281–12308.
2022. Patch generation with language models: Feasibility and scaling behavior. [52] Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich,
In Deep Learning for Code Workshop. Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman,
[29] Junjie Li, Aseem Sangalay, Cheng Cheng, Yuan Tian, and Jinqiu Yang. 2024. Fine Stephanie Ding, et al. 2024. Cyberseceval 3: Advancing the evaluation of cy-
tuning large language model for secure code generation. In Proceedings of the 2024 bersecurity risks and capabilities in large language models. arXiv preprint
IEEE/ACM First International Conference on AI Foundation Models and Software arXiv:2408.01605 (2024).
Engineering. 86–90. [53] Jiexin Wang, Xitong Luo, Liuwen Cao, Hongkui He, Hailin Huang, Jiayuan Xie,
[30] Ziyang Li, Saikat Dutta, and Mayur Naik. 2025. IRIS: LLM-assisted static analysis Adam Jatowt, and Yi Cai. 2024. Is Your AI-Generated Code Really Safe? Evaluating
for detecting security vulnerabilities. In The Thirteenth International Conference Large Language Models on Secure Code Generation with CodeSecEval. arXiv
on Learning Representations. preprint arXiv:2407.02395 (2024).
[31] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is [54] Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng,
Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large and Heng Ji. 2024. MINT: EVALUATING LLMS IN MULTI-TURN INTERACTION
Language Models for Code Generation. In Thirty-seventh Conference on Neural WITH TOOLS AND LANGUAGE FEEDBACK. In 12th International Conference
Information Processing Systems. [Link] on Learning Representations, ICLR 2024.
[32] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- [55] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi,
Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning
et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint in large language models. Advances in neural information processing systems 35
arXiv:2402.19173 (2024). (2022), 24824–24837.
[33] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah [56] Yi Wu, Nan Jiang, Hung Viet Pham, Thibaud Lutellier, Jordan Davis, Lin Tan, Petr
Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Babkin, and Sameena Shah. 2023. How effective are neural networks for fixing
2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural security vulnerabilities. In Proceedings of the 32nd ACM SIGSOFT International
Information Processing Systems 36 (2024). Symposium on Software Testing and Analysis. 1282–1294.
[34] Yu Nong, Mohammed Aldeen, Long Cheng, Hongxin Hu, Feng Chen, and Haipeng [57] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated
Cai. 2024. Chain-of-thought prompting of large language models for discovering program repair in the era of large pre-trained language models. In 2023 IEEE/ACM
and fixing software vulnerabilities. arXiv preprint arXiv:2402.17230 (2024). 45th International Conference on Software Engineering (ICSE). IEEE, 1482–1494.
[35] Yu Nong, Haoran Yang, Long Cheng, Hongxin Hu, and Haipeng Cai. 2024. Au- [58] Yu Yang, Yuzhou Nie, Zhun Wang, Yuheng Tang, Wenbo Guo, Bo Li, and Dawn
tomated software vulnerability patching using large language models. arXiv Song. 2024. SecCodePLT: A Unified Platform for Evaluating the Security of Code
preprint arXiv:2408.13597 (2024). GenAI. arXiv:2410.11096 [[Link]] [Link]
[36] OpenAI. [n. d.]. [Link] [59] Ziyi Yang, Fanqi Wan, Longguang Zhong, Tianyuan Shi, and Xiaojun Quan.
[37] OpenAI. 2023. ChatGPT. [Link] 2025. Weighted-Reward Preference Optimization for Implicit Model Fusion.
[38] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and In The Thirteenth International Conference on Learning Representations. https:
Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github //[Link]/forum?id=fq24pEb8SL
copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy [60] Xin Yin, Chao Ni, Shaohua Wang, Zhenhao Li, Limin Zeng, and Xiaohu Yang.
(SP). IEEE, 754–768. 2024. Thinkrepair: Self-directed automated program repair. In Proceedings of the
[39] Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis.
Dolan-Gavitt. 2023. Examining zero-shot vulnerability repair with large language 1274–1286.
models. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2339–2356. [61] Boyu Zhang, Tianyu Du, Junkai Tong, Xuhong Zhang, Kingsum Chow, Sheng
[40] Xutan Peng, Yipeng Zhang, Jingfeng Yang, and Mark Stevenson. 2022. On the Cheng, Xun Wang, and Jianwei Yin. 2024. SecCoder: Towards Generalizable
security vulnerabilities of text-to-sql models. arXiv preprint arXiv:2211.15363 and Robust Secure Code Generation. In Proceedings of the 2024 Conference on
(2022). Empirical Methods in Natural Language Processing. 14557–14571.
[41] Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2023. Do users [62] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu,
write more insecure code with AI assistants?. In Proceedings of the 2023 ACM Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging
SIGSAC Conference on Computer and Communications Security. 2785–2799. llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information
[42] Julian Aron Prenner, Hlib Babii, and Romain Robbes. 2022. Can OpenAI’s codex Processing Systems 36 (2023), 46595–46623.
fix bugs? an evaluation on QuixBugs. In Proceedings of the Third International [63] Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019.
Workshop on Automated Program Repair. 69–75. Devign: Effective vulnerability identification by learning comprehensive program
[43] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, semantics via graph neural networks. Advances in neural information processing
et al. 2019. Language models are unsupervised multitask learners. OpenAI blog systems 32 (2019).
1, 8 (2019), 9. [64] Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu,
[44] Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Wai Lam, and Lizhuang Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. DeepSeek-Coder-V2:
Ma. 2024. Codeattack: Revealing safety generalization challenges of large lan- Breaking the Barrier of Closed-Source Models in Code Intelligence. arXiv preprint
guage models via code completion. In Findings of the Association for Computa- arXiv:2406.11931 (2024).
tional Linguistics ACL 2024. 11437–11452.
[45] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao-
qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023.
Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950
(2023).
[46] Bernhard Scholz, Herbert Jordan, Pavle Subotić, and Till Westmann. 2016. On fast
large-scale program analysis in datalog. In Proceedings of the 25th International
Conference on Compiler Construction. 196–206.
[47] Mohammed Latif Siddiq and Joanna CS Santos. 2022. SecurityEval dataset: mining
vulnerability examples to evaluate machine learning-based code generation
techniques. In Proceedings of the 1st International Workshop on Mining Software
Repositories Applications for Privacy and Security. 29–33.