A Comprehensive Study of LLM Secure Code Generation

Shih-Chieh Dai Jun Xu Guanhong Tao

[Link]@[Link] junxzm@[Link] [Link]@[Link]
University of Utah University of Utah University of Utah
Salt Lake City, Utah, USA Salt Lake City, Utah, USA Salt Lake City, Utah, USA

Abstract generation methods. These techniques typically involve either ad-

Large language models (LLMs) are widely used in software de- justing LLM weight parameters or manipulating the input to or
velopment. However, the code generated by LLMs often contains output from the LLM. For example, SVEN [23] and SafeCoder [24]
vulnerabilities. Several secure code generation methods have been construct a training dataset containing code snippets with and
arXiv:2503.15554v1 [[Link]] 18 Mar 2025

proposed to address this issue, but their current evaluation schemes without vulnerabilities. They use such a dataset to fine-tune the
leave several concerns unaddressed. Specifically, most existing stud- model such that it can generate vulnerability-free code once trained.
ies evaluate security and functional correctness separately, using CodeGuard+ [21] instead controls the generation process of LLM
different datasets. That is, they assess vulnerabilities using security- inference. As LLMs use a decoding algorithm to determine the out-
related code datasets while validating functionality with general put, CodeGuard+ modifies this algorithm to favor outputs that lead
code datasets. In addition, prior research primarily relies on a single to secure code. Another state-of-the-art technique, PromSec [41],
static analyzer, CodeQL, to detect vulnerabilities in generated code, iteratively refines the task prompt based on the feedback from a
which limits the scope of security evaluation. vulnerability scanner on the generated code.
In this work, we conduct a comprehensive study to systemati- The evaluation for these secure code generation techniques typi-
cally assess the improvements introduced by four state-of-the-art cally considers whether the generated code contains vulnerabilities.
secure code generation techniques. Specifically, we apply both secu- A common practice is to use a vulnerability scanner, such as Cod-
rity inspection and functionality validation to the same generated eQL [7], to assess the security. These techniques also measure their
code and evaluate these two aspects together. We also employ three impact on normal model utility, that is, whether the enhanced LLM
popular static analyzers and two LLMs to identify potential vul- can still generate functional and usable code. A code benchmark
nerabilities in the generated code. Our study reveals that existing like HumanEval [15] is usually adopted for the evaluation.
techniques often compromise the functionality of generated code While the current evaluation schemes make sense, there are
to enhance security. Their overall performance remains limited a few problems. First, existing works mainly rely on the reported
when evaluating security and functionality together. In fact, many results by a single vulnerability detector, CodeQL, for assessing secu-
techniques even degrade the performance of the base LLM. Our rity, following the security evaluation practice proposed in prior
further inspection reveals that these techniques often either remove work [45]. This scanner is not the gold standard as it can miss or
vulnerable lines of code entirely or generate “garbage code” that misflag certain vulnerabilities. Our experiments in Section 4.1 show
is unrelated to the intended task. Moreover, the commonly used that CodeQL can miss more than 20% vulnerabilities in generated
static analyzer CodeQL fails to detect several vulnerabilities, further code. Second, the current evaluation scheme examines the security
obscuring the actual security improvements achieved by existing and functional correctness of generated code independently. In other
techniques. Our study serves as a guideline for a more rigorous and words, it uses a security-related code dataset to measure the se-
comprehensive evaluation of secure code generation performance curity (e.g., [53]) and use another general code dataset to assess
in future work. functionality (e.g., [44]). Such an evaluation scheme leaves a gap
of reported results between the two aspects. Are the generated code
from the functionality dataset secure? Are the generated code from
1 Introduction the security dataset functionally correct?
A straightforward idea is to use one of those datasets to evaluate
Software development is a time-consuming and repetitive task.
both security and functional correctness on the LLM-generated
The rapid evolution of large language models (LLMs) has greatly
code. However, there are a few issues with the datasets employed
benefited software developers. Given task requirements described in
in previous works. Most security-related code datasets do not have
natural language, LLMs can generate functional and easy-to-adopt
unit tests [13, 53, 56], meaning it is not feasible to rigorously eval-
code snippets, significantly accelerating the software development
uate the functionality of the generated code. The functionality
process. Many LLM-based code assistants are already integrated
benchmarks, such as HumanEval [15] and MBPP [12], on the other
into IDEs, such as Copilot [2] and Cursor [3].
hand, contain relatively simple tasks. They do not have the com-
However, just like human developers, LLMs can also make mis-
plexity of triggering security issues in the generated code, resulting
takes when producing code. One of the major concerns regarding
in nearly 100% security rate [53].
LLM-generated code is its security. As code snippets generated by
A few works craft new benchmarks for evaluating the security
LLMs are increasingly incorporated into industrial-level software
and functionality of LLM-generated code together, such as Code-
and systems, it is critical to ensure that LLM-generated code is free
Guard+ [21] and CWEval [47]. While promising, their sample size
of vulnerabilities that could be exploited by attackers.
is at a small scale, with CodeGuard+ and CWEval containing only
To address this, a number of techniques have been proposed
91 and 119 tasks, respectively, which limits a comprehensive and
to improve the security of LLM-generated code, called secure code
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Dai et al.

in-depth analysis. In addition, these works focus on either con- # Instruction: Runs the provided backup shell script and logs the start
time, end time, and exit status in a specified JSON log file
structing the benchmark for mainstream LLMs or assessing the
1 import subprocess
performance of a specific technique (e.g., SVEN [23]). They do 2 import os
3 import json
not aim to comprehensively evaluate many existing secure code 4 from datetime import datetime
5 def task_func(script_name, log_file):
generation techniques. 6 # Check if the script exists
7 > if not [Link](script_name):
In this work, we conduct a comprehensive study on the perfor- 13 # Execute the script
14 try:
mance of existing secure code generation techniques when con- 15 result = [Link](['bash', script_name])
16 exit_status = [Link]
sidering security and functionality simultaneously. Our goal is to 17 except [Link] as e:
18 exit_status = [Link]
understand not only their overall performance but also what con- 19 result = e
20
tributes to the improvement or reduction of the two aspects. To 21 # Get the end time
22 end_time = [Link]().strftime('%Y-%m-%d %H:%M:%S')
achieve this, we leverage two large datasets on code generation 23
24 # Prepare the log entry
tasks: BigCodeBench [70] and SecCodePLT [62], each containing 25
26 > log_entry =
more than 1,000 tasks. This allows for a more comprehensive as- 50 return log_entry Overlooked By CodeQL
sessment of existing techniques at a large scale. In addition, to
avoid relying on a single vulnerability scanner, we incorporate
three static analyzers: CodeQL [7], Bearer [17], and Bandit [48], for Figure 1: An example where CodeQL overlooks a vulner-
evaluating the security of LLM-generated code. We also consider ability in the generated code. Line 15 contains a CWE-78
two LLMs: Llama3.3-70B [22] and Qwen2.5-72B [61], for zero-shot vulnerability that CodeQL fails to identify.
vulnerability detection, as LLMs have shown promise in assessing
code security [32, 33, 55, 68]. Secure code generation needs to be evaluated from two aspects,
Our study reveals that while existing secure code generation functionality and security. Functionality measures whether the gen-
techniques improve the security of LLM-generated code to some erated code adheres to the task requirements, while security in-
extent, they often come at the cost of sacrificing functional correct- spects the existence of vulnerabilities in the generated code. Most
ness. A deeper analysis shows that these techniques may simply existing works evaluate the two aspects independently. That is, they
remove insecure lines of code or produce “garbage code” that is use one dataset to evaluate functionality (e.g., [44]) and another
irrelevant to the intended task. However, such problems cannot dataset to assess security (e.g., [53]). The results are then combined
be discovered with the current evaluation scheme, as the security to represent the technique’s performance. The functionality dataset
and functionality scores are reported based on separate evaluations. usually comes with unit tests, which can be applied directly for
Furthermore, our study demonstrates that vulnerability scanners evaluation. Yet, the security dataset offers no such measurements.
have different strengths in detecting certain types of vulnerabilities. Alternatively, the existing works use external vulnerability scanners
No single scanner can cover all potential security issues. Therefore, like CodeQL [7] to detect if the generated code has vulnerabilities.
it is critical to employ more scanners in security evaluations or use An intuitive question people often have is why the existing works
other advanced security assessment methods. The contributions of do not directly measure the security of the code generated for the
this work are summarized below: functionality dataset. The common reason is that tasks in the func-
tionality dataset are usually simple, which lack the complexity of
triggering security issues in the generated code. Thus, they are not
• We conduct a comprehensive study to systematically evalu-
suited for systematically assessing security.
ate state-of-the-art secure code generation techniques and
identify gaps in the current evaluation schemes.
2.2 Problems of Current Evaluation Schemes
• We investigate the performance disparity in functionality
and security of existing techniques and analyze their causes. Problem (I). The existing works commonly run CodeQL [7], a
• We highlight the limited performance improvement of exist- static vulnerability scanner, to assess the security of the generated
ing techniques under our evaluation framework and call for code. CodeQL employs rules referencing the CWE list [1] for de-
new approaches to secure code generation. tecting security vulnerabilities. However, the rules do not cover
the full CWE list. Further, the rules for a single CWE item can be
incomplete. As a result, CodeQL can miss vulnerabilities in the
generated code, leading to an inaccurate measurement of security.
2 Background and Motivation Figure 1 shows a piece of “secure” code generated by Qwen2.5-
2.1 Secure Code Generation and Evaluation Coder-7B-Instruct enhanced by CodeGuard+ [21], following the
Code generation aims to obtain a code snippet from an LLM through instruction at the top of the figure. CodeQL, using both the default
a prompt describing the desired task. Yet, the generated code can rules and GitHub-extended rules [8] for CWE-78 [39], detects no
carry vulnerabilities. To enhance security, several techniques have vulnerabilities in the code. However, line 15 in the code directly
been proposed to guide LLMs in generating code that not only executes the user-provided shell script without checking for ma-
accomplishes its intended tasks but also remains free of vulnera- licious commands, representing a CWE-78 issue. In this case, the
bilities [21, 23, 24, 41, 65]. These techniques commonly fine-tune security of CodeGuard+ is exaggerated.
the model, modify the input prompt, or manipulate the generation Problem (II). Independently evaluating functionality and security
process, for which more details are discussed in Section 3.2. can obscure the actual performance of secure code generation. For
A Comprehensive Study of LLM Secure Code Generation Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

# Instruction: Create n random txt files in a specific directory, write only a single digit random integer into each file, and
then reset the cursor to the beginning of each file.

1 import os 1 import os
2 import random 2 import random
3 def task_func(directory, n_files): 3 def task_func(directory, n_files):
4 # Create the directory if it doesn't exist 4 # Create the directory if it doesn't exist
5 if not [Link](directory): 5 - -
6 [Link](directory) 6 - -
7 7
8 # Generate and write to files 8 - -
9 for i in range(n_files): 9 - -
10 file_name = [Link](directory, f'file_{i+1}.txt') 10 - -
11 with open(file_name, 'w') as file: 11 - -
12 [Link](str([Link](0, 9))) 12 - -
13 13
14 # Reset the cursor to the beginning of each file 14 - -
15 for i in range(n_files): 15 - -
16 file_name = [Link](directory, f'file_{i+1}.txt') 16 - -
17 with open(file_name, 'r+') as file: 17 - -
18 [Link](0) 18 - -
19 19
20 return n_files DeepSeek-Coder 20 return n_files DeepSeek-Coder + CodeGuardPlus

Figure 2: Generated code by DeepSeek-Coder-V2-Lite before (left) and after (right) applying CodeGuard+. The code on the left
is functional but insecure. CWE-22 vulnerabilities exist at lines 6, 10, and 16. The code on the right is secure but not functional.

example, during security evaluation, the LLMs can be “encouraged” Table 1: Summary of existing secure code generation tech-
to generate simple yet task-irrelevant code, passing security tests niques. White-box denotes whether the technique requires
and leading to one-sided observations. access to the model weights, where , , and indicate full,
Figure 2 shows a case where DeepSeek-Coder-V2-Lite [5] is partial, and no access, respectively.
asked for code to create a list of files with a random number in
them and reset the cursor, using the prompt at the top of the figure. Method White-box Weight Update Prompt Mod. Decoding Ctrl External Tool
The generated code (on the left) satisfies the task requirements SVEN [23] ✓
SafeCoder [24] ✓
and passes all unit tests. However, without applying techniques CodeGuard+ [21] ✓ ✓
to improve the security, the code contains a vulnerability. Line PromSec [41] ✓ ✓
6 on the left-hand side uses external input to create a directory
without proper neutralization. This can be exploited to bypass the
limitation to restricted directories (CWE-22 [38]). Similar issues • (RQ2) How do secure code generation techniques perform
also exist at lines 10 and 16. To secure the code generation, we when security and functionality are evaluated together?
apply CodeGuard+ [21] on top of DeepSeek-Coder-V2-Lite to re-
• (RQ3) What leads to the disparity (if any) between function-
run the task. The newly generated code, presented on the right of
ality observed under combined measure and functionality
Figure 2, removed all code except for package import and function
assessed independently?
declaration. Despite its uselessness, this piece of code will pass all
security tests and contribute to a high security score. • (RQ4) What contributes to the disparity (if any) between
In fact, independently evaluating functionality and security leads security observed under combined measure and security
to many other problems like the example above. We categorize them assessed independently?
into five categories and present their details in Section 4.3.
3.2 Study Setup
3 Research Questions and Study Methodology Secure Code Generation Methods. We consider four state-of-the-
3.1 Scope and Research Questions art secure code generation techniques: SVEN [23], SafeCoder [24],
CodeGuard+ [21], and PromSec [41]. Table 1 summarizes these
Scope of Our Study. In this work, we focus on re-understanding techniques. SVEN and SafeCoder are fine-tuning-based methods.
the security and functionality of code generated by LLMs. Instead They construct a training dataset containing code snippets with
of considering end-to-end software development by LLMs, we focus and without vulnerabilities, which is used to fine-tune the model.
on function-level code generation, a commonly adopted scenario in Therefore, they require white-box access to the model’s weight
LLM-aided development. Our study does not aim at the performance parameters to update them. Once the model is updated, it functions
of the most advanced models but instead targets secure code gen- the same as the original model.
eration techniques (see Section 3.2) applied to mainstream LLMs. CodeGuard+ does not require fine-tuning but instead controls
the generation process during inference. Since LLMs produce an
Research Questions. We focus on the following research ques-
output sequence by generating tokens one by one, they rely on a
tions. We refer to the evaluation of both security and functionality
decoding algorithm to determine which token to choose at each step.
as the combined measure.
CodeGuard+ modifies the decoding algorithm to favor tokens that
• (RQ1) How do vulnerability scanners perform when assess- lead to a secure sequence. As a result, it requires gray-box access to
ing the security of LLM-generated code? the model, specifically to the decoding algorithm during inference.
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Dai et al.

Table 2: Summary of the employed datasets. CWE Label it difficult to construct unit tests. Additionally, since the data is
means if each task is accompanied with the annotation of extracted from example code on the MITRE CWE web page, there
potential CWE vulnerabilities. NL and CT stand for natural is a potential risk of data contamination. LLMs may have already
language instruction and code template, respectively. Sec- been trained on these examples, which limits the usefulness of this
CodePLT+ is an enhanced version SecCodePLT. The test cases dataset in evaluating their performance.
for SecCodePLT+ are self-prepared. Measurement. The goal of this study is to evaluate security and
functionality together, which has been overlooked by many existing
#Sample Language Prompt Unit Test Avg. Test Cases CWE Label works. One metric introduced by [21, 47] considers both aspects. It is
BigCodeBench 1,140 Python NL, CT Ë 5.6 é
SecCodePLT+ 1,201 Python NL, CT Ë 7.5∗ Ë called Secure-Pass@𝑘, which calculates the percentage of generated
code snippets that pass all unit tests and do not contain any security
vulnerabilities. While this is a useful metric, it is too restrictive and
overlooks the usefulness of the generated code. In many cases,
Additionally, CodeGuard+ modifies the original task prompt by
LLMs may not produce a fully functional code snippet that passes
incorporating security-related text, such as “use snprintf” to avoid
all unit tests. However, the generated code may still satisfy most
buffer overflow vulnerabilities.
parts of the task, missing only a few simple lines. Such a snippet
PromSec is a prompt engineering-based technique that itera-
can still be useful to developers with minimal effort.
tively refines the task prompt. Specifically, it first leverages external
Therefore, we propose a new metric, SAFE1 , which considers
tools such as Bandit [48] to identify vulnerabilities in the generated
both the security and functionality of LLM-generated code while
code. If any vulnerabilities are detected, PromSec utilizes a genera-
introducing a relaxation on functionality. Specifically, it is computed
tive adversarial network (GAN) model to enhance the security of
using the following formula:
the code. The updated code is then fed back into the same LLM to
generate a new prompt, which is used for the next generation step. 1 ∑︁ 𝑒 case-pass𝑖 − 1
SAFE@𝑘 := secure𝑖 · . (1)
This process repeats iteratively until no vulnerabilities are detected 𝑘 𝑖 𝑒 −1
in the generated code by the external tool.
Here, secure𝑖 denotes whether the 𝑖-th generated code snippet in
Code Datasets. There are several datasets available for evaluat- the top-𝑘 by an LLM contain vulnerabilities, where 1 indicates no
ing the functionality and security of code generation. However, as vulnerabilities and 0 otherwise. case-pass𝑖 represents the average
explained in Section 2, the tasks in the functionality dataset are usu- unit test passing rate for the 𝑖-th code in the top-𝑘. That is, we
ally simple, which cannot trigger security issues in the generated calculate the percentage of passed unit tests for this generated code.
code for assessing security. In contrast, security-related datasets Note that we leverage the exponential function to calibrate the unit
often do not include unit tests for assessing functionality. For our test passing score, encouraging LLMs to generate code with a higher
study, we use two public datasets: BigCodeBench [70] and SecCode- passing rate. This metric is more fine-grained than Secure-Pass@𝑘,
PLT [62], whose information is shown in Table 2. BigCodeBench as it takes the unit test passing rate into account.
includes unit tests, which makes it well-suited for evaluation. How- LLMs. We employ five popular open-source code LLMs for our
ever, SecCodePLT does not provide unit tests, limiting its ability to study: CodeLlama-7B [50], Qwen2.5-Coder-7B [26], DeepSeek-Coder-
assess functionality. V2-Lite [5], Mistral-7B [30], and StarCoder-1B [36]. Additionally, we
To ensure a comprehensive evaluation, we construct unit tests for include two commercial APIs: GPT-3.5-Turbo [43] and GPT-4o [27].
SecCodePLT. Since this dataset provides ground truth code for each We include only smaller-sized LLMs (e.g., Qwen2.5-7B instead of
generation task, we leverage an LLM, Qwen2.5-Coder-32B [26], to 32B) and older-version models (e.g., StarCoder instead of StarCoder
generate unit tests. Specifically, we prompt Qwen2.5-Coder-32B to 2) to be consistent with those used by existing works. We evaluate
create a set of test inputs based on the ground truth code. We then only PromSec on the commercial APIs since the other three tech-
execute the ground truth code with these inputs to obtain expected niques require white-box or gray-box access to the LLM, which is
outputs. Additionally, we include pre-existing inputs in the prompt not available for commercial models. Since PromSec is only applica-
to encourage Qwen2.5-Coder-32B to generate diverse test cases. The ble to instruction-following models, we exclude StarCoder from the
resulting input-output pairs serve as unit tests for evaluating code evaluation, as it is designed for code completion. In addition, when
generated by various methods. Using this approach, we generated we applied CodeGuard+ to CodeLlama and Mistral, we found that
an average of 7.5 unit test cases per task for SecCodePLT, even most of the generated code had indentation issues. To assess the
more than the 5.6 test cases per task provided by BigCodeBench. realistic functionality of the code, we correct these issues before
The enhanced SecCodePLT dataset, which we call SecCodePLT+, conducting unit tests.
can be found here [9].
Vulnerability Scanners. To evaluate the security of generated
There are other security-related datasets, such as CyberSecE-
code, a common practice is to use vulnerability scanners to de-
val [13] and SecurityEval [53]. CyberSecEval provides various task
tect potential security issues. We adopt three widely used static
prompts for secure code evaluation across multiple programming
analyzers: CodeQL [7], Bearer [17], and Bandit [48]. Additionally,
languages, along with ground truth code for each task. However,
we consider LLMs for vulnerability detection, as they have shown
the provided code consists only of function fragments rather than
promising results [32, 68]. In our study, we use two LLMs: Qwen2.5-
complete programs. As a result, it is challenging to generate unit
72B [61] and Llama3.3-70B [22], for security evaluation.
tests and conduct functional testing using this dataset. SecurityEval
contains only 130 samples and lacks ground truth code, making 1 Security and Functionality Evaluation.
A Comprehensive Study of LLM Secure Code Generation Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Base SVEN SafeCoder CodeGuardPlus PromSec

Model = Codellama Model = Qwen2.5-Coder Model = DeepSeek-Coder Model = Mistral Model = StarCoder
1.00
0.75
Score

0.50
0.25
0.00
CL BR BT Case Task CL BR BT Case Task CL BR BT Case Task CL BR BT Case Task CL BR BT Case Task
Score Type Score Type Score Type Score Type Score Type

Figure 3: The results of Secure@1 from each static analyzer and Pass@1 on BigCodeBench. The results are separated by a
dashed line. CL, BR, and BT represent CodeQL, Bearer, and Bandit, respectively. “Case” and “Task” indicate the Pass@1 scores
calculated at the test case level (considering percentage of passed unit tests) and the task level (passing all unit tests).

Base SVEN SafeCoder CodeGuardPlus PromSec

Model = Codellama Model = Qwen2.5-Coder Model = DeepSeek-Coder Model = Mistral Model = StarCoder
1.00
0.75
Score

0.50
0.25
0.00
CL BR BT Case Task CL BR BT Case Task CL BR BT Case Task CL BR BT Case Task CL BR BT Case Task
Score Type Score Type Score Type Score Type Score Type

Figure 4: The results of Secure@1 from each static analyzer and Pass@1 on SecCodePLT+. The results are separated by a
dashed line. CL, BR, and BT represent CodeQL, Bearer, and Bandit, respectively. “Case” and “Task” indicate the Pass@1 scores
calculated at the test case level (considering percentage of passed unit tests) and the task level (passing all unit tests).

4 Evaluation Results than those reported by CodeQL for most models. For example, on
Qwen2.5-Coder (in the 2nd chart), Bearer reports security scores
4.1 (RQ1) Vulnerability Scanner Performance
of around 65%, while Bandit reports around 75% for both the base
To evaluate the security of LLM-generated code, a common practice model and the three secure code generation methods. Although
is to use a vulnerability scanner to detect potential vulnerabilities. PromSec achieves higher scores, they are still lower than those
If no vulnerabilities are detected, the code is considered secure. reported by CodeQL.
Prior research [23, 24, 65] has primarily leveraged the static ana- Figure 4 reports the results for the SecCodePLT+ dataset. The ob-
lyzer CodeQL [7] to assess code security. However, as discussed servations are similar: CodeQL reports nearly 100% security scores
in Section 2, CodeQL can fail to detect certain vulnerabilities in for all models, while Bearer and Bandit show lower values. Addi-
generated code. Therefore, we also employ two other widely used tionally, we observe that Bearer reports a lower score for PromSec
static analyzers, Bearer [17] and Bandit [48], to measure security. compared to the base model for DeepSeek-Coder and Mistral (in
The first three groups in each chart in Figure 3 present the se- the 3rd and 4th charts). In contrast, Bandit shows the opposite:
curity results evaluated by the three vulnerability scanners on the PromSec improves security over the base model.
BigCodeBench dataset. CL, BR, and BT correspond to CodeQL,
Bearer, and Bandit, respectively. Each chart represents the results Finding 1: The security scores reported by different vulnerability
for an LLM, and each bar corresponds to a different secure code gen- scanners are inconsistent and can even be contradictory. Relying on
eration technique. The blue bar represents the base model. We did a single vulnerability scanner is insufficient for comprehensively
not report PromSec results for StarCoder, as PromSec requires the assessing the security of generated code.
base model to generate code solely based on instructions. However,
StarCoder only supports code completion tasks. Figure 5 presents the reported CWE vulnerabilities identified
As shown in Figure 3, CodeQL (first group) reports nearly 100% by the three static analyzers for Qwen2.5-Coder enhanced by Safe-
security scores for all techniques, including the base model. (Note Coder on the BigCodeBench dataset. The security scores reported
that we successfully reproduced CodeQL’s results on the dataset by the scanners are 0.9877, 0.6535, and 0.7482 for CodeQL, Bearer,
used by these techniques, as reported in the original papers.) How- and Bandit, respectively. From the figure, we observe that Cod-
ever, this does not necessarily mean the generated code is truly eQL identifies only a very small number of CWE vulnerabilities
secure. As we can see the results from Bearer and Bandit (second (i.e., 4). Bandit detects slightly more vulnerabilities than Bearer
and third groups), the security scores are approximately 25% lower but fails to identify certain vulnerabilities, such as CWE-328 and
 Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Dai et al.

200 Static Analyzer the generated code does not contain any logging functionality, yet
Number of Occurrences

CodeQL the LLM still reported the issue. While LLMs show promise in vul-
150 Bandit
Bearer nerability detection, their effectiveness remains an open research
question [18, 37].
100

50 Finding 3: LLMs are helpful in detecting vulnerabilities. However,

they also produce a significant number of false positives, which
0
20 22 78 89 94 95 259 319 327 328 330 377 400 502 703 916 hinders their application in security assessment for generated code.
CWE ID

Figure 5: CWE vulnerabilities identified by the three static

analyzers for Qwen2.5-Coder enhanced by SafeCoder on the
BigCodeBench dataset. 4.2 (RQ2) On the Combined Measure of Existing
Techniques
1.0 Existing secure code generation approaches assess the functionality
Base
0.8 SVEN and security of LLM-generated code independently, using different
SafeCoder Table 3: Confusion
0.6 CodeGuardPlus matrix of the man- datasets. Here, we evaluate both aspects on the same dataset. Fig-
Score

PromSec
ually validated secu- ure 3 presents the results on BigCodeBench. Existing techniques
0.4 can improve security scores for most models, although different
rity results for SVEN
0.2 with Mistral and Sec- vulnerability scanners may produce inconsistent relative rankings.
0.0 CodePLT+. For Mistral (in the 4th chart), Bearer reports a security reduction
CL BR BT Llama Qwen with CodeGuard+, while Bandit reports security reductions for
Score Type
Evaluator TP TN FP FN
three techniques: SVEN, SafeCoder, and CodeGuard+. PromSec
consistently enhances security across four models.
Figure 6: The security results on Mis- CodeQL 2 21 0 19
Bearer 9 15 6 12 However, when evaluating the functionality of generated code
tral with SecCodePLT+. CL, BR, and Bandit 3 19 2 18 using existing secure code generation techniques, we observe a
BT represent CodeQL, Bearer, and Qwen 20 3 18 1
Llama 21 0 21 0 notable disparity compared to the results reported in the original
Bandit, respectively.
papers. The last two groups in each chart in Figure 3 illustrate the
functional correctness of the generated code. The “Case” group
CWE-916, which are detected by either CodeQL or Bearer. The represents test case-level scores, where we calculate the percentage
most frequently detected vulnerability is CWE-327, identified by of passed unit tests for each task and then average the values across
Bearer with 200 occurrences, which relates to the use of a broken all tasks. This provides a fine-grained view of functional correct-
or risky cryptographic algorithm. ness. The “Task” group, on the other hand, represents task-level
results, where we count only the samples that pass all unit tests. We
Finding 2: Different vulnerability scanners have varying strengths observe that most techniques reduce the functionality of generated
in identifying different types of vulnerabilities. No single scanner code at both the test case and task levels. This suggests that these
can cover all potential security issues. techniques may not actually be fixing vulnerabilities in the gen-
erated code but instead sacrificing functionality to make the code
LLMs have shown promising results in detecting vulnerabilities “appear” more secure. We have previously discussed an example
in code. Therefore, we leverage two LLMs, Qwen2.5-72B [61] and case in Section 2 and will further explore this issue in the following
Llama3.3-70B [22], to evaluate the security of generated code. Fig- RQs. CodeGuard+ improves functionality scores on CodeLlama,
ure 6 presents the results reported by the LLMs and the three static Qwen2.5-Coder, and Mistral. However, its security improvements
analyzers for the Mistral model on SecCodePLT+. Surprisingly, the are limited or even decrease on Mistral, indicating that CodeGuard+
security scores reported by the LLMs are significantly lower than struggles to balance security and functionality in generated code.
those from the static analyzers. Additionally, the relative rankings The observations on the SecCodePLT+ dataset are similar, as
of different secure code generation techniques vary between the shown in Figure 4. All evaluated techniques reduce the functionality
two approaches. To understand this significant disparity, we con- of the generated code. SVEN and SafeCoder show a significant drop
duct a manual inspection. In particular, we use SVEN as an example (over 50%) in functional correctness for Codellama. This is likely
and randomly sample 44 cases. Table 3 summarizes our manual due to the challenging nature of SecCodePLT+, where even GPT-
analysis. For the three static analyzers, we observe a non-trivial 4o achieves only 8.74% task-level performance. Fine-tuning-based
number of false negatives (12–19 out of 44 samples), indicating methods like SVEN and SafeCoder may negatively impact code
that they fail to detect certain vulnerabilities. We have discussed generation quality, leading to lower functionality scores. Moreover,
such examples in Section 2. Conversely, the LLMs exhibit a high security improvements are also limited for most models. Notably,
false positive rate (nearly 50%), meaning they tend to over-report PromSec experiences nearly a 9% security degradation on Codel-
vulnerabilities that do not actually constitute security issues. For lama, as reported by Bearer.
instance, in one case, Qwen2.5-72B flags a CWE-532 vulnerability,
suggesting a sensitive information leak through logging. However,
A Comprehensive Study of LLM Secure Code Generation Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Base SVEN SafeCoder CodeGuardPlus PromSec

Codellama Qwen2.5-Coder DeepSeek-Coder Mistral StarCoder
0.4
0.3
Score

0.2
0.1
0.0
Sec-Pass SAFE Sec-Pass SAFE Sec-Pass SAFE Sec-Pass SAFE Sec-Pass SAFE
Score Type Score Type Score Type Score Type Score Type

Figure 7: The overall results of Secure-Pass@1 and SAFE@1 on BigCodeBench. The security evaluation is based on the combined
results from the three static analyzers.

Base SVEN SafeCoder CodeGuardPlus PromSec

Codellama Qwen2.5-Coder DeepSeek-Coder Mistral StarCoder
0.08
0.06
Score

0.04
0.02
0.00
Sec-Pass SAFE Sec-Pass SAFE Sec-Pass SAFE Sec-Pass SAFE Sec-Pass SAFE
Score Type Score Type Score Type Score Type Score Type

Figure 8: The overall results of Secure-Pass@1 and SAFE@1 on SecCodePLT+. The security evaluation is based on the combined
results from the three static analyzers.

Finding 4: Existing secure code generation techniques can enhance CodeGuard+ demonstrates notable improvements on CodeLlama
the security of generated code to some extent, but often at the and Mistral models but performs worse than the base models on
expense of functional correctness. DeepSeek-Coder and StarCoder. CodeGuard+ requires constraints
to be provided in the prompt, which were manually crafted in
As observed above, the trade-off between security and functional- the original paper. In our experiments, we select these constraints
ity makes direct head-to-head comparisons challenging. Therefore, based on the corresponding CWE labels reported by vulnerabil-
we need a metric that evaluates both aspects of generated code in a ity scanners on the code generated by the base model. This gives
unified manner. Secure-Pass@1, introduced by previous work [21], CodeGuard+ an advantage, as it preemptively knows what kinds of
measures the percentage of top-1 generated samples2 that pass all vulnerabilities should be avoided during generation. That is why it
unit tests and are free from detected vulnerabilities. Additionally, can improve performance on certain models.
we introduce a more fine-grained metric, SAFE@1, which considers Figure 8 presents the results on SecCodePLT+. Nearly all secure
the passing rate of unit tests for each task. This metric is further code generation techniques fail to improve Secure-Pass@1 and
explained in Section 3.2. SAFE@1 scores. This is largely due to the challenging nature of
Figure 7 presents the results using the two metrics on the Big- SecCodePLT+. Even GPT-4o, a state-of-the-art commercial LLM,
CodeBench dataset. Since we employ three static analyzers to evalu- struggles to achieve a high score as we will discuss later. Addi-
ate security in this paper, and each detects different types of vulner- tionally, on CodeLlama, we observe that CodeGuard+ outperforms
abilities, we aggregate their results by considering a code snippet PromSec in the Secure-Pass@1 metric. However, the results are re-
secure only if none of the three scanners detects a vulnerability. We versed for SAFE@1. This suggests that while PromSec may produce
do not use LLMs as scanners because they produce a large number fewer samples that pass all unit tests, the ones it does generate tend
of false positives, as discussed in RQ1. From the charts, we observe to be secure and closely aligned with the intended task. SAFE@1,
that most techniques fail to improve Secure-Pass@1 and SAFE@1 therefore, provides a more fine-grained interpretation of the results.
scores. For instance, all techniques except CodeGuard+ show a
significant reduction in both metrics compared to the base models Finding 5: Existing techniques show limited effectiveness in im-
(“Base” in the legend). PromSec exhibits more than a 50% drop on proving secure code generation when evaluating security and func-
Qwen2.5-Coder and DeepSeek-Coder. Despite being a state-of-the- tionality simultaneously.
art method published at CCS 2024 [41], PromSec’s performance is
less impressive when evaluating security and functionality together.
While Figure 7 and Figure 8 present results on open-source
2 An LLM can return a list of outputs ranked by their probabilities. models, we also evaluate two commercial APIs in our experiments.
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Dai et al.

Table 4: Results of PromSec using commercial APIs on BigCodeBench and SecCodePLT+.

BigCodeBench SecCodePLT+
Secure@1 Pass@1 Secure-Pass@1 SAFE@1 Secure@1 Pass@1 Secure-Pass@1 SAFE@1
Model Method CodeQL Bearer Bandit Test Task Overall Overall CodeQL Bearer Bandit Test Task Overall Overall
Base 0.9904 0.6447 0.7544 0.7658 0.5078 0.3219 0.4445 0.9886 0.7324 0.8132 0.2182 0.0574 0.0308 0.0979
GPT-3.5-Turbo
PromSec 0.9912 0.7325 0.8465 0.5809 0.3166 0.2447 0.3810 0.9862 0.7235 0.8043 0.2139 0.0599 0.0316 0.0996
Base 0.9904 0.6395 0.7482 0.8217 0.6043 0.3771 0.4794 0.9951 0.7510 0.8035 0.2391 0.0874 0.0399 0.1071
GPT-4o
PromSec 0.9904 0.6860 0.7454 0.7143 0.4622 0.3561 0.4765 0.9935 0.7776 0.8399 0.2158 0.0732 0.0349 0.1228

Table 5: Comparison of security and functionality of the generated code between the base model and secure code generation
techniques. represents security, where teal indicates secure code and red indicates insecure code. Ë and é represent the
functional correctness of the code.

BigCodeBench SecCodePLT+
Model Method Ëto é Ëto é Ëto Ë to Ëto é Ëto é Ëto Ë to
SVEN 18.81% 69.00% 0.00% 3.15% 43.58% 76.74% 0.00% 4.08%
SafeCoder 22.77% 73.00% 0.00% 3.09% 48.71% 62.79% 0.00% 5.37%
Codellama
CodeGuard+ 11.88% 33.55% 1.34% 4.07% 12.82% 39.53% 2.32% 6.42%
PromSec 12.87% 55.70% 0.67% 1.82% 5.10% 46.51% 11.62% 12.85%
SVEN 2.59% 33.87% 0.32% 1.74% 6.12% 23.68% 7.89% 6.32%
SafeCoder 1.03% 20.64% 0.32% 3.19% 12.24% 42.10% 5.26% 5.43%
Qwen2.5-Coder
CodeGuard+ 1.03% 16.77% 0.32% 1.88% 2.04% 28.94% 10.52% 7.07%
PromSec 35.75% 77.09% 0.96% 2.03% 8.16% 50.00% 5.26% 9.22%
SVEN 47.26% 59.45% 1.35% 1.43% 10.00% 36.58% 4.87% 5.53%
SafeCoder 13.69% 28.37% 1.35% 2.00% 4.00% 31.70% 4.87% 6.68%
DeepSeek-Coder
CodeGuard+ 21.23% 39.64% 0.45% 1.57% 2.00% 26.82% 2.43% 5.76%
PromSec 43.15% 69.81% 0.00% 0.42% 0.00% 34.14% 24.39% 14.99%
SVEN 8.33% 72.46% 0.00% 11.77% 42.30% 66.66% 9.09% 7.81%
SafeCoder 13.88% 84.05% 0.00% 11.77% 3.84% 27.27% 6.06% 7.44%
Mistral
CodeGuard+ 0.00% 8.69% 0.00% 12.76% 11.53% 45.45% 3.03% 6.71%
PromSec 30.55% 62.31% 0.00% 12.39% 15.38% 27.27% 21.21% 18.68%
SVEN 40.00% 68.42% 0.00% 5.15% 0.00% 21.33% 1.33% 0.98%
StarCoder SafeCoder 0.00% 52.63% 0.00% 0.00% 0.00% 21.33% 0.00% 3.67%
CodeGuard+ 80.00% 57.89% 0.00% 0.73% 0.00% 14.66% 0.00% 0.71%
GPT-3.5-Turbo PromSec 20.28% 31.33% 1.63% 1.88% 3.12% 24.32% 2.70% 7.89%
GPT-4o PromSec 8.10% 16.74% 0.46% 0.44% 3.50% 29.16% 2.08% 6.02%
Average 9.97% 42.04% 0.62% 4.11% 11.75% 35.04% 5.15% 6.87%

As previously discussed, all evaluated techniques except PromSec Finding 6: Commercial LLM APIs do not offer any additional
require white-box or gray-box access to the LLM, which is not advantages for existing secure code generation techniques.
available for commercial APIs. Therefore, this experiment primarily
focuses on PromSec. Table 4 reports results on the BigCodeBench
and SecCodePLT+ datasets. Columns Secure@1 and Pass@1 present
individual security scores from different vulnerability scanners 4.3 (RQ3) On the Performance Disparity of
along with unit test results, while the following columns show the Functionality
Secure-Pass@1 and SAFE@1 scores.
In RQ2, we observe the degradation in functionality of generated
PromSec shows a significant improvement in security for GPT-
code by existing techniques. We further analyze the results to bet-
3.5 on BigCodeBench. However, the Pass@1 scores decrease sub-
ter understand this phenomenon. Specifically, we collect all tasks
stantially. This is reflected in the Secure-Pass@1 and SAFE@1
where the generated code by the base models is functionally correct
scores, which drop by 23% (from 0.3219 to 0.2447) and 28% (from
and inspect the outcomes when applying the secure code genera-
0.4445 to 0.381), respectively. GPT-3.5 was the default model used
tion techniques. Table 5 presents the results on the two datasets.
for evaluation in the original PromSec paper [41], but it still can-
For each dataset, the first two columns show the results for the
not improve overall performance when security and functionality
aforementioned cases.
are measured together. Security improvements for GPT-4o and on
• The column “ Ëto é” denotes cases where insecure but func-
the SecCodePLT+ dataset are limited, with PromSec showing little
tional code by the base model becomes secure but non-functional
improvement in Secure-Pass@1 and SAFE@1 scores. Another ob-
after applying existing techniques.
servation is that even GPT-3.5 and GPT-4o exhibit very low Pass@1
• The column “ Ëto é” represents cases where secure and func-
scores on the SecCodePLT+ dataset. This highlights the challenging
tional code becomes secure but non-functional after applying
nature of the tasks in this dataset, which require advanced tech-
existing techniques.
niques to enhance the functional correctness of generated code.
We observe that around 10% to 30% of tasks fall under the “ Ëto
é” category. Notably, CodeGuard+ with StarCoder on BigCodeBench
A Comprehensive Study of LLM Secure Code Generation Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Table 6: Case study on the code generated by existing tech- 4.4 (RQ4) On the Performance Disparity of
niques using DeepSeek-Coder on BigCodeBench for the Security
“ Ëto é” category. The value in parentheses denotes the
number of cases for each technique in this category. “NFI” While existing secure code generation techniques were designed
represents cases where the generated code did not follow the to improve the security of LLM-generated code, they may inadver-
instruction. “FN” indicates cases where the code snippet con- tently degrade security for certain tasks. We have already observed
tains a vulnerability that was missed by the static analyzers. the performance reduction in Figure 3 and Figure 4. Here, we specif-
ically analyze the scenario where the code generated by the base
model was originally secure, but vulnerabilities are detected after
Removed Code Junk Code NFI FN Other
SVEN (69) 75.36% 8.7% 0% 2.89% 13.04%
applying existing techniques. We refer to this as the non-monotonic
SafeCoder (20) 60% 0% 0% 25% 15% security improvement of these methods.
CodeGuard+ (31) 67.74% 6.45% 12.9% 3.23% 9.68%
PromSec (63) 92.06% 0% 7.94% 0% 0% Table 5 presents the results for the scenario described above.
Specifically, we consider two cases:
• The column “ Ëto Ë” denotes cases where secure and func-
tional code by the base model becomes insecure but remains
functional after applying existing techniques.
leads to 80% of tasks becoming non-functional after improving secu-
• The column “ to ” represents cases where secure code be-
rity. The percentage is generally lower on the SecCodePLT+ dataset;
comes insecure after applying existing techniques, regardless of
however, for a few cases, such as SafeCoder with CodeLlama and
its functionality.
SVEN with Mistral, it remains above 40%. For the “ Ëto é” cat-
Observe that for the first case “ Ëto Ë”, although it rarely hap-
egory, the percentages are even higher, with most cases ranging
pens, around 1% of samples can still occur, such as CodeGuard+
from 20% to 60% on BigCodeBench and 20% to 40% on SecCode-
with Codellama and PromSec with GPT-3.5 on BigCodeBench. The
PLT+. This indicates that while existing techniques are effective in
percentage is much higher in SecCodePLT+. In fact, PromSec with
fixing vulnerabilities, they tend to compromise the functionality
DeepSeek-Coder exhibits 24.39% of samples becoming insecure
of generated code for samples that were previously correct. This
after applying the technique. When functionality is not considered,
explains the functionality degradation we observed earlier.
more instances of secure code becoming insecure are observed after
applying existing techniques, as shown in the column “ to ”. The
Finding 7: Existing secure code generation techniques negatively average percentages are 4.11% for BigCodeBench and 6.87% for
impact the base model, transforming originally functional code into SecCodePLT+, respectively.
non-functional code.
Finding 9: The security improvement provided by existing tech-
To better understand the issue, we manually inspect the cases in niques is not monotonic; they may introduce vulnerabilities into
the “ Ëto é” category. We use DeepSeek-Coder on BigCodeBench code that was previously secure.
as an example. Table 6 presents the results for the four existing
techniques. We classify the cases into five categories: Figure 9 shows an example where the task is to copy data from
• Removed Code: This category includes cases where techniques an Excel spreadsheet into a new Excel file and then read and return
simply remove vulnerability-related code to improve security, as the contents of the new file. The base model, Qwen2.5-Coder-7B-
shown in the example in Section 2. We observe a large number Instruct, can generate a secure code snippet as shown on the left.
of cases in this category, with PromSec removing insecure code However, after applying CodeGuard+[21], the generated code on
in 92.06% of the samples. the right introduces a vulnerability at line 9. It uses the system call
• Junk Code: The generated output is nonsensical, such as the [Link]() to execute the copy command, which is passed as
repeated word “task task task.” SVEN and CodeGuard+ show a string. If the original_location or new_location variables
around 6%-8% of cases in this category. contain malicious input, this could lead to command injection. This
• Not Following Instruction (NFI): In these cases, the LLM gen- vulnerability corresponds to CWE-78 [39].
erates code completely irrelevant to the task. CodeGuard+ and
PromSec have a non-trivial percentage of cases in this category.
5 Threats to Validity
• False Negatives (FN): These are cases where static analyzers
fail to detect vulnerabilities, which was discussed in RQ1. The internal threat to validity lies in potential mistakes during
• Other: In a few cases, the generated code is missing necessary manual inspection. Specifically, we may misidentify a vulnerability
package imports or is otherwise incomplete, leading to functional reported by different scanners. To mitigate this threat, we ensure
incorrectness. that each vulnerability is examined by at least two authors. A third
author will chime in to resolve any disagreements. The external
threat to validity primarily lies in the subjects used in our study.
Finding 8: The security improvements achieved by existing tech-
The code generation tasks we examine may not be representative.
niques come at the cost of sacrificing the functionality of the gener-
We mitigate this risk by using two recent large datasets cover-
ated code, primarily by removing vulnerability-related code and
ing over 2,000 tasks. Since these datasets focus mainly on Python,
generating irrelevant or garbage output.
our findings may not generalize to other programming languages.
However, most existing techniques were originally evaluated using
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Dai et al.

# Instruction: Copies data from an Excel spreadsheet into a new Excel file, then reads the new Excel file and returns its contents.

1 import pandas as pd 1 import pandas as pd

2 import os 2 import os
3 def task_func(original_location, new_location, sheet_name): 3 def task_func(original_location, new_location, sheet_name):
4 # Check if the original Excel file exists 4 # Check if the original file exists
5 if not [Link](original_location): 5 if not [Link](original_location):
6 raise FileNotFoundError("File does not exist.") 6 raise FileNotFoundError(f"The file does not exist.")
7 7
8 try: 8 # Copy the original file to a new location
9 # Read the original Excel file 9 [Link](f'cp {original_location} {new_location}')
10 df = pd.read_excel(original_location, sheet_name) 10
11 11 # Load the new Excel file
12 # Save the DataFrame to a new Excel file 12 try:
13 df.to_excel(new_location, index=False) 13 df = pd.read_excel(new_location, sheet_name)
14 14 except KeyError:
15 # Read the new Excel file 15 raise ValueError( )
16 new_df = pd.read_excel(new_location) 16
17 17 return df
18 return new_df 18
19 19
20 except KeyError: 20
21 raise ValueError( ) Qwen-Coder 21 Qwen-Coder + CodeGuardPlus

Figure 9: Generated code by Qwen before (left) and after (right) applying CodeGuard+. The code on the left is both functional
and secure. However, after applying CodeGuard+, a CWE-78 vulnerability appears at line 9 in the code on the right. The
differences are depicted in green and red, representing secure and insecure code, respectively.

Python. Our study re-evaluates these techniques. For the SecCode- CWE-78 [32]. Other works also demonstrated the LLM’s promis-
PLT dataset, we generate unit tests using an LLM. While the test ing capability in vulnerability detection [51, 54, 68]. In addition to
cases may be limited, we address this by manually inspecting the leveraging LLMs for vulnerability detection, several studies have
generated inputs in conjunction with the ground truth code and also explored their use in vulnerability repair [28, 46, 69].
setup files. We also manually calibrate problematic unit tests. Ad-
ditionally, the LLMs and vulnerability scanners used in this study
may not be fully inclusive, especially given the rapid development
of LLMs. However, we argue that the general observations regard-
ing existing secure code generation techniques will still hold, as Evaluation of LLM-generated Code. Several benchmarks have
the issues stem from their technical design rather than the specific been proposed for evaluating the functional correctness of LLM-
LLMs or tools used. generated code, including LiveCodeBench [29], BigCodeBench [70],
HumanEval [15], MBPP [12], and SWE-Arena [6]. Other bench-
marks that consider both functionality and security, such as Sec-
6 Related Work CodePLT [62], CodeGuard+ [21], and CWEval [47], are either lim-
LLM for (Secure) Code Generation. Several Code LLMs have ited by dataset size [21, 47] or lack test cases for nearly half of the
been specifically trained for code generation tasks using code samples [62]. Additionally, there are datasets specifically designed
datasets, such as CodeGen [42], InCoder [20], SantaCoder [10], for code security evaluation, including SecurityEval [53], LLMSe-
CodeLlama [50], Qwen-Coder [26], DeepSeek-Coder [5], Code- cEval [56], CyberSecEval [13], and Sallam [52]. However, these
Stral [4], and StarCoder [36]. Moreover, various studies have fo- datasets do not provide unit tests for functional evaluation. We
cused on enhancing the performance of Code LLMs by optimizing employ BigCodeBench and SecCodePLT in our study and extend
prompts [31, 34, 35]. Additionally, some studies use LLMs to synthe- SecCodePLT by augmenting its unit test cases.
size test cases and leverage the augmented datasets to improve the Other than using benchmarks for evaluation, several tools and
performance of LLM [14, 25, 64]. Other works aiming to improve the metrics exist for assessing the functionality or security of code.
security of LLM-generated code include SVEN [23], SafeCoder [24], Ren et al. [49] proposed CodeBLEU, which evaluates the accuracy
CodeGuard+ [21], and PromSec [41]. We elaborate on the details of of generated code compared to the ground truth by considering
these methods in Section 3.2. However, all existing code generation n-gram matches, AST matches, and data-flow matches. Le et al. [33]
studies have either primarily evaluated the functionality of LLM- prompted LLMs to judge the security and helpfulness of code, while
generated code or assessed functionality and security separately. Tong and Zhang [55] examined different prompting techniques for
Other applications of LLMs include program repair [59, 60, 67], evaluating the semantic correctness of LLM-generated code. Pearce
code analysis [19, 40, 66], and unit test generation [11, 16, 63]. et al. [45], Siddiq and Santos [53] analyzed the safety rate of the
generated code using CodeQL. Bhatt et al. [13] proposed an insecure
LLM for Vulnerability Detection. Another line of research fo- code detector, which leverages a static analyzer and aims to detect
cuses on leveraging LLMs to improve code security. Several studies insecure coding practices rather than specific vulnerabilities. Finally,
have utilized LLMs for vulnerability detection [18, 32, 37, 57, 68]. Wang et al. [58] conducted a study on LLMs’ capabilities in secure
Khare et al. [32] examined LLMs’ zero-shot capability in detect- code generation, vulnerability classification, vulnerability repair,
ing vulnerabilities , and their findings show that LLMs outperform and vulnerability explanation. However, its goal is not to evaluate
CodeQL in detecting certain vulnerabilities, such as CWE-22 and existing secure code generation techniques.
A Comprehensive Study of LLM Secure Code Generation Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

7 Conclusion [19] Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie
Fang, Ryan Tsang, Najmeh Nazari, Han Wang, Houman Homayoun, et al. 2024.
We conduct a comprehensive study of four secure code generation Large language models for code analysis: Do { LLMs } really do their job?. In 33rd
techniques across two benchmarks. Our study results suggest that USENIX Security Symposium (USENIX Security 24). 829–846.
[20] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi,
future work should employ more than one vulnerability scanner for Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A
security evaluation, as different scanners have varying strengths. generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999
We also show that existing techniques have limited effectiveness in (2022).
[21] Yanjun Fu, Ethan Baker, Yu Ding, and Yizheng Chen. 2024. Constrained decoding
enhancing the security of LLM-generated code when considering for secure code generation. arXiv preprint arXiv:2405.00218 (2024).
functionality simultaneously. These techniques tend to sacrifice [22] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek
functionality to achieve a higher security score, leading to non- Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex
Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783
usable generated code. Our study underscores the importance of (2024).
evaluating both the security and functionality of LLM-generated [23] Jingxuan He and Martin Vechev. 2023. Large language models for code: Secu-
rity hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC
code simultaneously and provides guidelines for future research. Conference on Computer and Communications Security. 1865–1879.
[24] Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Martin Vechev. 2024. In-
struction Tuning for Secure Code Generation. In International Conference on
References Machine Learning. PMLR, 18043–18062.
[1] [n. d.]. Common Weakness Enumeration. [Link] [25] Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming
[2] [n. d.]. Copilot. [Link] Cui. 2023. Agentcoder: Multi-agent-based code generation with iterative testing
[3] [n. d.]. Cursor. [Link] and optimisation. arXiv preprint arXiv:2312.13010 (2023).
[4] 2024. Codestral. [Link] [26] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu
[5] 2024. DeepSeek Coder V2 Lite Base. [Link] Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical
DeepSeek-Coder-V2-Lite-Base report. arXiv preprint arXiv:2409.12186 (2024).
[6] 2024. SWE Arena: An Open Evaluation Platform for Automated Software Engi- [27] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh,
neering. Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024.
[7] 2025. CodeQL. [Link] Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024).
[8] 2025. Python queries for CodeQL analysis. [Link] [28] Nafis Tanveer Islam, Mohammad Bahrami Karkevandi, and Peyman Najafirad.
security/code-scanning/managing-your-code-scanning-configuration/python- 2024. Code Security Vulnerability Repair Using Reinforcement Learning with
built-in-queries Large Language Models. arXiv:2401.07031 [[Link]] [Link]
[9] 2025. SecCodePLT+. [Link] 07031
[10] Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher [29] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang,
Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Live-
Manan Dey, et al. 2023. SantaCoder: don’t reach for the stars! arXiv preprint CodeBench: Holistic and Contamination Free Evaluation of Large Language
arXiv:2301.03988 (2023). Models for Code. arXiv:2403.07974 [[Link]] [Link]
[11] Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark [30] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De-
Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel,
2024. Automated unit test improvement using large language models at meta. In Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux,
Companion Proceedings of the 32nd ACM International Conference on the Founda- Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,
tions of Software Engineering. 185–196. and William El Sayed. 2023. Mistral 7B. arXiv:2310.06825 [[Link]] https:
[12] Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk //[Link]/abs/2310.06825
Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, [31] Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi
and Charles Sutton. 2021. Program Synthesis with Large Language Models. CoRR Jin, and Wenpin Jiao. 2024. Self-planning code generation with large language
abs/2108.07732 (2021). arXiv:2108.07732 [Link] models. ACM Transactions on Software Engineering and Methodology 33, 7 (2024),
[13] Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evti- 1–30.
mov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo [32] Avishree Khare, Saikat Dutta, Ziyang Li, Alaia Solko-Breslin, Rajeev Alur, and
Fontana, et al. 2023. Purple llama cyberseceval: A secure coding benchmark for Mayur Naik. 2023. Understanding the effectiveness of large language models in
language models. arXiv preprint arXiv:2312.04724 (2023). detecting security vulnerabilities. arXiv preprint arXiv:2311.16169 (2023).
[14] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang [33] Hung Le, Doyen Sahoo, Yingbo Zhou, Caiming Xiong, and Silvio Savarese. 2024.
Lou, and Weizhu Chen. 2023. CodeT: Code Generation with Generated Tests. INDICT: Code Generation with Internal Dialogues of Critiques for Both Security
In The Eleventh International Conference on Learning Representations. https: and Helpfulness. In The Thirty-eighth Annual Conference on Neural Information
//[Link]/forum?id=ktrw68Cmu9c Processing Systems.
[15] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de [34] Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2025. Structured chain-of-thought prompt-
Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg ing for code generation. ACM Transactions on Software Engineering and Method-
Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, ology 34, 2 (2025), 1–23.
Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail [35] Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. 2024. Acecoder: An effec-
Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, tive prompting technique specialized in code generation. ACM Transactions on
Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- Software Engineering and Methodology 33, 8 (2024), 1–26.
tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex [36] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov,
Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shan- Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023.
tanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023).
Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, [37] Ziyang Li, Saikat Dutta, and Mayur Naik. 2024. Llm-assisted static analysis for
Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario detecting security vulnerabilities. arXiv preprint arXiv:2405.17238 (2024).
Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Eval- [38] MITRE. 2024. CWE-22: Improper Limitation of a Pathname to a Restricted
uating Large Language Models Trained on Code. arXiv:2107.03374 [[Link]] Directory (’Path Traversal’). [Link]
[Link] [39] MITRE. 2024. CWE-78: Improper Neutralization of Special Elements used in an
[16] Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei OS Command (’OS Command Injection’). [Link]
Yin. 2024. Chatunitest: A framework for llm-based test generation. In Compan- [Link].
ion Proceedings of the 32nd ACM International Conference on the Foundations of [40] Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad
Software Engineering. 572–576. Myers. 2024. Using an llm to help with code understanding. In Proceedings of the
[17] Bearer Developers. 2022. Bearer. [Link] IEEE/ACM 46th International Conference on Software Engineering. 1–13.
[18] Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun [41] Mahmoud Nazzal, Issa Khalil, Abdallah Khreishah, and NhatHai Phan. 2024.
Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2024. PromSec: Prompt Optimization for Secure Generation of Functional Source Code
Vulnerability Detection with Code Language Models: How Far Are We?. In 2025 with Large Language Models (LLMs). In Proceedings of the 2024 on ACM SIGSAC
IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Conference on Computer and Communications Security. 2266–2280.
Computer Society, 469–481.
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Dai et al.

[42] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Proceedings of the ACM on Software Engineering 1, FSE (2024), 1703–1726.
Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An Open Large Language [64] Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu
Model for Code with Multi-Turn Program Synthesis. In The Eleventh Interna- Chen. 2025. ACECODER: Acing Coder RL via Automated Test-Case Synthesis.
tional Conference on Learning Representations. [Link] arXiv preprint arXiv:2502.01718 (2025).
iaYcJKpY2B_ [65] Boyu Zhang, Tianyu Du, Junkai Tong, Xuhong Zhang, Kingsum Chow, Sheng
[43] OpenAI. 2023. GPT-3.5-Turbo. [Link] Cheng, Xun Wang, and Jianwei Yin. 2024. SecCoder: Towards Generalizable
tuning-and-api-updates/ and Robust Secure Code Generation. In Proceedings of the 2024 Conference on
[44] OpenAI. 2025. Code for the paper "Evaluating Large Language Models Trained Empirical Methods in Natural Language Processing. 14557–14571.
on Code". [Link] [66] Yichi Zhang. 2024. Detecting code comment inconsistencies using llm and pro-
[45] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and gram analysis. In Companion Proceedings of the 32nd ACM International Conference
Ramesh Karri. 2022. Asleep at the Keyboard? Assessing the Security of GitHub on the Foundations of Software Engineering. 683–685.
Copilot’s Code Contributions. In 2022 IEEE Symposium on Security and Privacy [67] Yuze Zhao, Zhenya Huang, Yixiao Ma, Rui Li, Kai Zhang, Hao Jiang, Qi Liu, Linbo
(SP). IEEE Computer Society, 754–768. Zhu, and Yu Su. 2024. RePair: Automated Program Repair with Process-based
[46] Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Feedback. In Findings of the Association for Computational Linguistics: ACL 2024.
Dolan-Gavitt. 2023. Examining zero-shot vulnerability repair with large language 16415–16429.
models. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2339–2356. [68] Xin Zhou, Sicong Cao, Xiaobing Sun, and David Lo. 2024. Large language model
[47] Jinjun Peng, Leyi Cui, Kele Huang, Junfeng Yang, and Baishakhi Ray. 2025. CW- for vulnerability detection and repair: Literature review and the road ahead. ACM
Eval: Outcome-driven Evaluation on Functionality and Security of LLM Code Transactions on Software Engineering and Methodology (2024).
Generation. arXiv preprint arXiv:2501.08200 (2025). [69] Xin Zhou, Kisub Kim, Bowen Xu, Donggyun Han, and David Lo. 2024. Out of
[48] PyCQA. 2022. Bandit. [Link] Sight, Out of Mind: Better Automatic Vulnerability Repair by Broadening Input
[49] Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sun- Ranges and Sources. In Proceedings of the IEEE/ACM 46th International Conference
daresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing
Method for Automatic Evaluation of Code Synthesis. arXiv:2009.10297 [[Link]] Machinery, New York, NY, USA, Article 88, 13 pages. doi:10.1145/3597503.3639222
[Link] [70] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira
[50] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al.
qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. 2024. Bigcodebench: Benchmarking code generation with diverse function calls
Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 and complex instructions. arXiv preprint arXiv:2406.15877 (2024).
(2023).
[51] Aleksei Shestov, Rodion Levichev, Ravil Mussabayev, Anton Cheshkov, and Pavel
Zadorozhny. 2024. Finetuning Large Language Models for Vulnerability Detection.
In International Conference on Computational Optimization. [Link]
net/forum?id=7Huz1BPTii
[52] Mohammed Latif Siddiq, Joanna Cecilia da Silva Santos, Sajith Devareddy, and
Anna Muller. 2024. SALLM: Security Assessment of Generated Code. In Pro-
ceedings of the 39th IEEE/ACM International Conference on Automated Software
Engineering Workshops (Sacramento, CA, USA) (ASEW ’24). Association for Com-
puting Machinery, New York, NY, USA, 54–65. doi:10.1145/3691621.3694934
[53] Mohammed Latif Siddiq and Joanna CS Santos. 2022. SecurityEval dataset: mining
vulnerability examples to evaluate machine learning-based code generation
techniques. In Proceedings of the 1st International Workshop on Mining Software
Repositories Applications for Privacy and Security. 29–33.
[54] Shaznin Sultana, Sadia Afreen, and Nasir U Eisty. 2024. Code vulnerability
detection: A comparative analysis of emerging large language models. arXiv
preprint arXiv:2409.10490 (2024).
[55] Weixi Tong and Tianyi Zhang. 2024. CodeJudge: Evaluating Code Generation
with Large Language Models. In Proceedings of the 2024 Conference on Empirical
Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and
Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida,
USA, 20032–20051. doi:10.18653/v1/[Link]-main.1118
[56] Catherine Tony, Markus Mutas, Nicolas E. Diaz Ferreyra, and Riccardo Scandari-
ato. 2023. LLMSecEval: A Dataset of Natural Language Prompts for Security
Evaluations . In 2023 IEEE/ACM 20th International Conference on Mining Software
Repositories (MSR). IEEE Computer Society, Los Alamitos, CA, USA, 588–592.
doi:10.1109/MSR59073.2023.00084
[57] Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, and
Gianluca Stringhini. 2024. Llms cannot reliably identify and reason about security
vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks.
In 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 862–880.
[58] Jiexin Wang, Liuwen Cao, Xitong Luo, Zhiping Zhou, Jiayuan Xie, Adam Jatowt,
and Yi Cai. 2023. Enhancing Large Language Models for Secure Code Generation:
A Dataset-driven Study on Vulnerability Mitigation. arXiv:2310.16263 [[Link]]
[Link]
[59] Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. 2023. Copiloting the
copilots: Fusing large language models with completion engines for automated
program repair. In Proceedings of the 31st ACM Joint European Software Engi-
neering Conference and Symposium on the Foundations of Software Engineering.
172–184.
[60] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated
program repair in the era of large pre-trained language models. In 2023 IEEE/ACM
45th International Conference on Software Engineering (ICSE). IEEE, 1482–1494.
[61] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu,
Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5
technical report. arXiv preprint arXiv:2412.15115 (2024).
[62] Yu Yang, Yuzhou Nie, Zhun Wang, Yuheng Tang, Wenbo Guo, Bo Li, and Dawn
Song. 2024. SecCodePLT: A Unified Platform for Evaluating the Security of Code
GenAI. arXiv preprint arXiv:2410.11096 (2024).
[63] Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng,
and Yiling Lou. 2024. Evaluating and improving chatgpt for unit test generation.