Human-Written vs.

AI-Generated Code: A Large-Scale

Study of Defects, Vulnerabilities, and Complexity
Domenico Cotroneo, Cristina Improta, Pietro Liguori
University of Naples Federico II, Naples, Italy
{cotroneo, [Link], [Link]}@[Link]

Abstract—As AI code assistants become increasingly integrated class-based wrappers and utility code. These artifacts are then
into software development workflows, understanding how their integrated into production systems, open-source repositories,
code compares to human-written programs is critical for ensuring and developer workflows, often without rigorous inspection.
reliability, maintainability, and security. In this paper, we present
As a consequence, AI-generated code is reshaping the stylistic,
arXiv:2508.21634v1 [[Link]] 29 Aug 2025

a large-scale comparison of code authored by human developers

and three state-of-the-art LLMs, i.e., ChatGPT, DeepSeek-Coder, structural, and quality norms that have traditionally character-
and Qwen-Coder, on multiple dimensions of software quality: ized software engineering.
code defects, security vulnerabilities, and structural complexity. Unfortunately, current research provides an incomplete pic-
Our evaluation spans over 500k code samples in two widely ture of the impact of AI code generators on software quality.
used languages, Python and Java, classifying defects via Or-
thogonal Defect Classification and security vulnerabilities using Most existing studies evaluate AI-generated code in isolation,
the Common Weakness Enumeration. We find that AI-generated emphasizing correctness or performance on synthetic bench-
code is generally simpler and more repetitive, yet more prone marks [7], [8], or considering some aspects of quality [9],
to unused constructs and hardcoded debugging, while human- [10] and security [11], [12]. Few works compare AI-generated
written code exhibits greater structural complexity and a higher code directly with human-authored programs, limiting our un-
concentration of maintainability issues. Notably, AI-generated
code also contains more high-risk security vulnerabilities. These derstanding of how AI code differs across diverse dimensions
findings highlight the distinct defect profiles of AI- and human- such as structural complexity, defect profiles, or vulnerability
authored code and underscore the need for specialized quality classes. Even fewer studies adopt a standardized framework
assurance practices in AI-assisted programming. for evaluating code quality, leading to fragmented and often
Index Terms—AI Code Generation, Orthogonal Defect Classi- incomparable findings due to the wide variety of programming
fication, Software Defects, Software Security, Code Complexity,
Large Language Models languages, tools, and evaluation metrics considered.
These limitations raise a fundamental concern: the frame-
works and taxonomies traditionally used to assess software
I. I NTRODUCTION
quality are human-centered, designed around assumptions
The emergence of AI-assisted programming has triggered about human cognition, error modes, and review processes.
a paradigm shift in software engineering, redefining not only Yet, AI-generated code introduces new characteristics, includ-
how code is written but also who, or rather what, is writing ing structural redundancy, model-specific vulnerability pat-
it [1]. Large Language Models (LLMs) such as ChatGPT [2], terns, or even hallucinations (e.g., non-existing libraries or
GitHub Copilot [3], and DeepSeek-Coder [4] are no longer APIs) [13]. It is unclear whether existing defect classification
limited to generating code snippets or suggesting completions, frameworks can meaningfully characterize these patterns or
they now produce entire functions, classes, or even modules whether they must be revised and extended to reflect the
with minimal human intervention, transitioning programming properties and limitations of AI-generated software.
from a human-centric design activity into a model-driven In this work, we present a large-scale, cross-language com-
process [5]. According to forecasts by leading AI research parison of human-written and AI-generated code, focusing
labs, by the end of 2025, AI may be responsible for producing on multiple dimensions of software quality: (i) code defects,
up to 90% of all new code [6]. While this trend brings clear (ii) security vulnerabilities, and (iii) structural complexity. We
advantages in terms of development speed and accessibility, it focus on Python and Java, two of the most widely used
also raises critical concerns regarding the quality, correctness, programming languages in both industry and academia [14],
and reliability of automatically generated code. and analyze a large-scale dataset (>500k code samples) from
It is worth noting that this transformation is not merely an human developers (>17k GitHub projects) alongside outputs
evolution of tools and solutions, but a structural redefinition from three popular LLMs: ChatGPT, DeepSeek-Coder, and
of software development, with the role of developers evolving Qwen-Coder [15]. We rely on three state-of-the-art static
from designing and implementing code to supervising and analysis tools to detect issues in the code: Pylint [16] for
adapting model outputs, repositioning them from authors to finding defects in Python functions, PMD [17] for Java, and
curators. As AI-generated code proliferates, new patterns Semgrep [18] as a security-oriented solution for discovering
emerge: code tends to be more concise, heavily pattern-based, vulnerabilities in both.
and uniform, frequently reusing architectural motifs such as To normalize the analysis across tools and languages, we

1
adopt the Orthogonal Defect Classification (ODC) framework static analysis tools like SonarQube, revealing wide variability
by Chillarege et al. [19], a well-established method for in maintainability and correctness. Siddiq et al. [27] analyzed
categorizing defects along independent dimensions such as real-world developer interactions with ChatGPT, finding that
assignment, algorithm, and interface. ODC allows us to map generated code is often significantly modified prior to use due
language-specific defects into a common taxonomy, enabling to issues like variable misuse and documentation gaps. Liu
meaningful cross-language and cross-author (i.e., human vs. et al. [10] provide an empirical study on ChatGPT-generated
LLMs) comparisons. Python and Java code, evaluating 4,000 samples for correct-
Our study yields the following key findings. First, we reveal ness and insights into error types and failure rates stratified
systematic differences in defects between AI-generated and by task difficulty. Improta et al. [28] assessed the quality
human-written code. LLMs tend to produce code that is less of a large-scale Python dataset (>4M) mined from open-
structurally complex but more repetitive and prone to spe- source GitHub repositories using Semgrep and showed that
cific defect categories, such as variable assignment errors. In filtering out low-quality functions from training data improves
contrast, human-authored code exhibits more intricate control generation. Esfahani et al. [29] provide a well-structured defect
flow, algorithmic flaws, and improper exception handling, classification of Python code by using two small AI code
symptoms often associated with the maintainability and design generators, i.e., CodeT5+ and CodeGen, to generate code for
debt of real-world, mature codebases. Moreover, compared to the well-known open-source benchmark, HumanEval. Then,
software developers, AI models are better at generating Python they explore different prompting techniques to automatically
code yet much worse with Java code (−4k and +22k defective fix the problematic code.
samples, respectively). On the security side, we observed These studies suffer from restricted dataset sizes, single-
that AI-generated code more frequently triggers vulnerabilities language or single-model focus, and limited generalizabil-
associated with high-risk CWE categories, including command ity. A key reason for this limitation is that, when dealing
injection and hardcoded secrets. On average, compared to with AI-generated code, there is no established taxonomy
human code, AI models generate more vulnerable code both in for systematically classifying software defects. Unlike tradi-
Python (+5k samples) and Java (+18k samples), fewer lines tional software engineering, where code evolves within well-
of code (−6.75) and use significantly fewer tokens (−63.74), defined development lifecycles, AI-generated code is often
indicating reduced structural and logical complexity and lower produced as isolated, context-free snippets, making it difficult
lexical diversity. to apply process-aware classification frameworks directly. In
All code, datasets and results presented in our study are conventional systems, one of the most widely adopted models
available for transparency and replication purposes on Zen- for defect classification is the Orthogonal Defect Classifica-
odo [20] and GitHub [21]. tion framework, introduced by Chillarege et al. [19]. ODC
In the following, Section II discusses related work; Sec- was explicitly designed to support in-process measurement
tion III describes our research study; Section IV details the by categorizing software defects based on their observable
dataset construction process; Section V presents the exper- impact, independent of language, implementation details, or
imental evaluation; Section VI discusses threats to validity; development stage.
Section VII concludes the paper. We apply the ODC framework in our study to assess
whether it remains a suitable and informative abstraction in
II. R ELATED W ORK this new generation of code development. A recent study by
Recent studies on AI-generated code have primarily focused Bogaerts et al. [30] attempted to apply ODC to real-world
on correctness and security [11], [22], often evaluating just one Python vulnerabilities drawn from CVEs. While their work
or two popular models, typically ChatGPT or Copilot [23], showed that ODC can bring structure to vulnerability analysis,
on isolated tasks or benchmarks. For instance, Pearce et it also highlighted limitations in using it alone to capture the
al. [24] evaluated Copilot’s security by testing it on 89 coding specificity of modern security issues.
scenarios and found that around 40% of its outputs contained The most related to our work is the preliminary study
vulnerabilities. Fu et al. [12] evaluated code generated by three by Patel et al. [31], which compares Copilot-generated and
AI code assistants (i.e., GitHub Copilot, CodeWhisperer and human-written Java code. However, the study is limited in
Codeium) and performed extensive CWE-based security anal- several ways: it focuses exclusively on a single programming
ysis, including attempts to repair insecure snippets. Kharma language and two specific software bugs, considers only one
et al. [25] further highlighted that AI-generated code often AI model, and evaluates a small and narrow dataset of just 90
fails to adopt modern security best practices, particularly in LeetCode solutions, which lack diversity in scope and style.
languages like Java and C++. While these studies provide Our study addresses these limitations through a comprehen-
crucial insights into the risks of AI-assisted programming, sive and controlled comparison between human-written and
they are often language-specific, focused solely on security, LLM-generated code. We analyze human-written code mined
and lack broader quality metrics. from over 17k real-world GitHub projects and code generated
Beyond security, only a few studies addressed other aspects by three popular LLMs (i.e., ChatGPT, DeepSeek-Coder, and
of code quality. Yetiştiren et al. [9] benchmarked Copilot, Qwen-Coder) into two widely used languages (i.e., Python
CodeWhisperer, and ChatGPT using HumanEval [26] and and Java). Our evaluation spans more than 500k code samples

2
and multiple dimensions of code quality, including defect de- detect potential issues, we apply two widely-used static analy-
tection, security vulnerabilities, and structural complexity. To sis tools: Pylint [16] for Python and PMD [17] for Java. These
enable consistent and interpretable comparisons across tools, tools were selected due to their maturity, extensive adoption
code authors, and languages, we adopt the ODC framework in industrial and academic software analysis workflows, and
and MITRE’s Common Weakness Enumeration (CWE) [32], their ability to produce fine-grained, rule-specific outputs.
allowing us to normalize and categorize software issues across Pylint is a deep static analysis tool for Python programs
languages. that performs code linting, style checking, bug detection, and
design enforcement. It analyzes Python Abstract Syntax Trees
III. R ESEARCH S TUDY
(ASTs) to detect a wide range of issues, including syntax
We designed this research study with the aim of answering errors, type inconsistencies, uninitialized variables, and poor
the following research questions (RQs): programming practices. Pylint’s output is rule-driven, with
▷ RQ1 : Do defect types and frequencies differ between each rule assigned a unique symbolic name and severity level
human-written and AI-generated code across programming (e.g., convention, warning, error), for a total of 417 detection
languages? rules. In our setup, we exclude all style-related issues, such as
To answer this research question, we systematically compare formatting, naming conventions, and documentation warnings,
the distribution of defect types in human-written and AI- as they do not affect functional correctness, resulting in a final
generated code across two programming languages. To this set of 350 rules.
aim, we apply two state-of-the-art static analysis tools (i.e., PMD is an extensible static source code analyzer that parses
Pylint for Python, PMD for Java) and map the resulting defects Java code into an AST and applies a wide set of rule-
to standardized Orthogonal Defect Classification categories. based patterns to detect programming flaws. PMD checks
This approach allows for a structured comparison of defect for issues such as unnecessary object creation, performance
types and frequencies among code authors, enabling us to bottlenecks (e.g., avoid instantiating objects inside loops),
quantify differences in code quality between human and AI- possible bugs (e.g., empty catch blocks, unused variables), and
generated functions across Python and Java datasets. design problems (e.g., excessive method complexity). It sup-
▷ RQ2 : Do security vulnerabilities differ between human- ports multiple built-in rule sets categorized into best practices,
written and AI-generated code across languages, in terms of code style, design, documentation, error-prone, multithreading,
type and severity? performance, and security, for a total of 290 rules.
The goal of this RQ is to investigate and compare security vul- As before, we excluded from the analysis all rules concern-
nerabilities present in human-written and AI-generated code ing code style and documentation, as docstrings are stripped
across Python and Java code. Using Semgrep for static security from code functions and used as generation prompts (as
analysis, we map the identified vulnerabilities to the Common detailed in § IV). Additionally, we omitted the security ruleset
Weakness Enumeration taxonomy and categorize them by due to its limited coverage (i.e., only two rules), opting
severity level. This analysis enables us to assess differences instead to incorporate a more comprehensive, security-focused
in the types and criticality of security issues introduced by solution later on. In the end, our analysis encompasses 226
human developers versus AI code assistants, providing insight detection rules.
into potential real-world risks and author-specific security To systematically categorize and compare software defects
behavior. across different code authors, i.e., human developers or AI
▷ RQ3 : Do structural complexity metrics vary between code assistants, and across different programming languages,
human-written and AI-generated code across programming i.e., Python and Java, we rely on the Orthogonal Defect
languages? Classification [19]. ODC is a widely recognized method for
The third RQ assesses the structural characteristics of human- classifying software defects based on their nature and origin,
written and AI-generated code by analyzing complexity met- independent of the specific software development process or
rics. We evaluate code samples based on quantitative measures application domain. It is based on the definition of a set of
such as the number of lines of code, cyclomatic complexity, attributes, which are non-overlapping dimensions that measure
and token counts, to examine whether AI-generated code different aspects of defects, including their type, their impact,
shares similar surface-level and deeper structural traits with their trigger (i.e., how the defect was found), and their source.
code written by human programmers. This orthogonality facilitates standardized analysis and defect
pattern detection.
A. Code Defects Analysis In the paradigm of AI-assisted software development, where
In this study, we systematically investigate software defects developers query LLMs with a short description of the required
introduced by human developers and AI models when writing implementation, the generated code is usually composed of
code. A defect refers to any violation of correctness, logic, snippets, single functions or short classes, which are then
performance or design. These include issues like improper integrated into a larger codebase. For this reason, not all ODC
variable initialization, control flow anomalies, or faulty error attributes are applicable to this fast-prototyping scenario.
handling, problems that may lead to program misbehavior but For this reason, we applied the Defect Type attribute from
do not necessarily expose the system to external threats. To ODC to categorize software issues identified in the code. ODC

3
TABLE I: Summary of the mapping between Orthogonal Defect Classification (ODC) defect types and code smells in Python
and Java code. Normal text represents Python rules, while bold text represents Java rules.

ODC Defect Type Defect Characteristics Python Java Example Rules

Rules Rules
Errors in assignment, initialization, or used-before-assignment,
Assignment 46 13
variable binding UnusedAssignment
Logical flaws in computation or data ma- too-many-nested-blocks,
Algorithm 118 88
nipulation CyclomaticComplexity
Issues with interaction between modules, no-value-for-parameter,
Interface 113 12
functions, or APIs UseProperClassLoader
Faulty validation or error handling mech-
Checking 23 26 missing-timeout, EmptyCatchBlock
anisms
Concurrency, event ordering, or multi- useless-with-lock,
Timing/Serialization 1 11
threading defects AvoidSynchronizedStatement
Structural or design errors in function, redefined-outer-name,
Function/Class/Object 49 76
class, or object organization MissingOverride

Defect Type describes the nature of the fault independently of structured resolution of conflicts ensured conceptual alignment
specific code details or language constructs, covering eight dis- with ODC taxonomy. The complete rule-to-category mapping
tinct categories, including Assignment, Algorithm, Interface, is made available in the replication package for transparency
Checking, and Timing, among others. and future extensions [21].
We deliberately excluded other ODC attributes from our Using ODC allowed us to abstract away from tool-specific
analysis, such as the Trigger attribute. This dimension captures rule naming conventions and focus on the broader patterns
how a defect is discovered, typically during code inspections, of defects across datasets. It also enabled cross-comparison
unit testing, integration testing, or system-level testing. How- between human-written and AI-generated code, highlighting
ever, in our setting, defect detection is performed entirely differences in error types rather than being limited to tool
through static analysis tools applied uniformly across all code output frequencies.
samples. This approach does not simulate dynamic software Table I provides an overview of the categorization. Each
development workflows or staged testing processes in which ODC Defect Type is described by its defect characteristics,
trigger information would be meaningful. Moreover, LLM- indicating the nature of defects it is intended to capture. For
generated code is not compiled, tested, or executed in its each category, the table reports the number of mapped rules
original context, making it infeasible to reliably assign trig- originating from Pylint for Python and PMD for Java. Example
ger labels without speculative or artificial assumptions. By rules are provided to show typical violations of each defect
focusing solely on the defect type, we preserve the rigor and type, with Python rules displayed in regular text and Java rules
comparability of our findings across languages and authors. shown in bold.
For both Python and Java, we manually mapped all the To make the mapping process more concrete, consider
provided detection rules to the most appropriate ODC cat- two representative examples. The Pylint rule used-before-
egory. This mapping process was necessary to provide a assignment, which detects the use of variables before initializa-
standardized view of the analysis by normalizing results across tion in Python, was mapped to the Assignment category. This is
different languages and tools, given that static analysis rules because such violations reflect fundamental issues in variable
are tool-specific and not directly generalizable. Each rule was binding and initialization, which are key to the definition of
individually reviewed and assigned an ODC category based assignment-related defects in the ODC framework. On the
on its functional nature and the type of defect it targeted, Java side, the PMD rule CyclomaticComplexity, which flags
aiming to minimize ambiguity and maximize consistency methods with excessive branching logic, was classified under
across languages. the Algorithm category, as it reflects control-flow complexity
Given that Pylint and PMD define extensive rule sets with and potential logical flaws in computation. These examples
heterogeneous scopes and naming conventions, we performed illustrate how each rule was assigned based on the functional
a detailed manual classification to interpret the intent behind semantics of the issue it detects, rather than tool-specific
each rule. To ensure the reliability of this process, we adopted naming, ensuring consistency in our defect categorization
a two-stage validation approach. First, two authors indepen- across languages and tools.
dently categorized all rules based on their diagnostic seman- Finally, we excluded the Documentation and Build/Pack-
tics. Disagreements were then resolved through discussion, age/Merge defect types from our analysis. Documentation-
resulting in a consensus-based mapping. This procedure served related issues are omitted because docstrings are intentionally
as a form of inter-rater agreement, mitigating subjectivity removed from the code samples and repurposed as prompts for
and reinforcing the reproducibility of the classification. This AI code generation in the dataset construction (e.g., missing-

4
docstring). Similarly, Build/Package/Merge defects pertain to we aggregate security rules addressing both general application
broader integration concerns such as library dependencies, vulnerabilities and language-specific threats. The complete
version control, and packaging errors (e.g., import-error, configuration, including the list of security rules, is available
NoPackage), that are not applicable in our context, as we focus in our replication package [21].
exclusively on isolated code snippets rather than complete For each identified security issue, Semgrep provides rich
software systems. As such, these categories fall outside the metadata, including: (i) the rule identifier that triggered the
scope of our study. detection; (ii) the severity level of the issue (categorized as
Additionally, to ensure compatibility with PMD, which info, warning, error, or critical); (iii) the specific lines of code
requires valid Java class structures, all Java functions were where the issue occurs; (iv) a human-readable explanation of
wrapped in minimal dummy classes prior to analysis. This the detected vulnerability; and (v) the corresponding mapping
transformation preserved the original logic while satisfying to a Common Weakness Enumeration entry. This detailed
PMD’s parsing requirements. However, it also introduced output enables a systematic and standardized analysis of
artificial class contexts that could inadvertently activate class- vulnerabilities across both human-written and AI-generated
level rules unrelated to the function under evaluation. To code samples.
maintain fairness and analytical consistency, we filtered out
all PMD rules targeting class design or object structure (e.g., C. Code Complexity Analysis
number of fields, method modifiers). This ensured that our To evaluate structural differences between human-written
evaluation remained focused strictly on function-level logic, and AI-generated code, we measured several key code com-
the unit of generation for both humans and LLMs, and plexity metrics that are widely used in software engineering
not on artifacts introduced solely for tool compatibility. To to assess maintainability, readability, and logical depth. Each
this end, we excluded 12 additional class-related rules (e.g., metric captures a distinct dimension of code structure and
UseUtilityClass, ClassWithOnlyPrivateConstructorsShouldBe- computational intricacy, providing a multifaceted view of how
Final). Again, all code is provided in the replication package code is organized and how difficult it may be to understand,
for reproducibility [21]. test, or extend. By analyzing these metrics, we aim to identify
both quantitative differences between human and AI code, and
B. Code Security Analysis also stylistic or structural biases in how LLMs approach code
A security vulnerability is a specific type of defect that generation.
poses a risk of exploitation, potentially allowing unauthorized ■ Number of Lines of Code (NLOC). It reflects the overall
access, data leakage, or system compromise. These are clas- size of a block of code and can indicate code verbosity or
sified using the CWE framework and include issues such as density of implementation. It is computed by counting the total
command injection, hardcoded secrets, or use of unsafe APIs. number of lines excluding comments.
To detect security vulnerabilities in both Python and Java ■ Cyclomatic Complexity Number (CCN). It measures the
code, we employ Semgrep OSS [18], a modern, lightweight number of independent control paths through a function, thus
static analysis tool designed for finding security vulnerabilities, quantifying logical complexity and the degree of branching or
correctness bugs, and code quality issues across more than 30 decision-making within the code. It is computed by counting
programming languages. the number of linearly independent paths through the code.
Unlike traditional static analyzers that often require full ■ Token Count (TC). It captures the number of lexical units
code compilation (e.g., CodeQL [33], Bandit [34]), Sem- (e.g., constants, identifiers, operators, keywords) contained in
grep operates based on syntactic pattern-matching directly on code, providing a finer-grained measure of syntactic richness.
source code, making it particularly well-suited for scanning ■ Function Name Length (FNL). It acts as a proxy for
large-scale, heterogeneous datasets without requiring to build naming descriptiveness, with longer names often indicating
artifacts or full type inference. Semgrep provides an extensive higher semantic specificity. It is computed as the character
registry of curated rulesets, each targeting specific families count of the extracted function names.
of issues, such as detection of vulnerabilities from MITRE’s ■ Unique Tokens (UT). It is a proxy metric for the vocabulary
CWE Top 25 [35], unencrypted communication patterns, se- size. It aggregates the total number of distinct tokens used
crets management flaws, or injection vulnerabilities. across the corpus (i.e., for a single code author), serving
In this analysis, we focus exclusively on detecting security as an indicator of lexical diversity and potentially richer
vulnerabilities using Semgrep rules specifically designed for programming expression.
security analysis. We include all security-focused rules avail- We calculated all complexity metrics using a combination
able for Python and Java (as of April 2025). The selected of lizard [37] and tiktoken [38] libraries. We used the for-
rules cover a broad range of vulnerability categories, including mer, a lightweight static analysis tool capable of extracting
injection attacks, insecure communication, improper authenti- basic structural information from 23 different languages, to
cation, and sensitive data exposure. These rules are sourced extract code snippets. Then, we tokenized the source code
from a diverse set of curated Semgrep rulesets targeting using tiktoken, an efficient tokenizer aligned with GPT-4’s
well-established security standards such as the OWASP Top encoding schema, to track the average token count and unique
Ten [36] and the CWE Top 25. To ensure comprehensiveness, tokens per function. We used lizard to automatically calculate

5
 TABLE II: Dataset composition. this process, 3,259 Python and 539 Java samples could not
Avg.
be matched and were discarded, yielding a total of 285,249
Avg. Code Python and 221,796 Java instances.
Lang. # Samples # Repos Docstring
Len.
Len. Then, to further enrich the dataset with more diverse AI-
Python 285,249 12,632 25.73 87.21 generated implementations, we employed two additional state-
Java 221,795 4,296 24.86 79.30 of-the-art AI code assistants: DeepSeek-Coder-Instruct (DSC)
and Qwen2.5-Coder-Instruct (Qwen). For each sample, we
used the original docstring and corresponding function signa-
NLOC and CCN, while function name lengths were directly ture as prompts to generate alternative implementations with
measured as the character count of the extracted function both models.
names. The metrics were collected separately for each code ■ DeepSeek-Coder-Instruct (33B) [4] is part of a series of
author (human-written code, ChatGPT, DeepSeek-Coder, and code language models, each trained from scratch on 2T tokens,
Qwen-Coder) to preserve the granularity of the analysis per with a composition of 87% code and 13% natural language.
source. Full details of the computation pipeline, along with The base model is pre-trained on project-level code corpus by
the scripts used, are available in the replication package [21]. employing a window size of 16K and an extra fill-in-the-blank
task. We utilize a quantized, instruction-tuned model with 33
IV. DATASET
billion parameters, which was further fine-tuned on 2B tokens
To investigate whether there exist any differences in the of instruction data.
defect distribution of human-written and LLM-generated code, ■ Qwen2.5-Coder-Instruct (32B) [15] is the latest series
we curated a large-scale dataset consisting of both Python of Qwen large language models specifically designed for
and Java code. We selected these programming languages as code-related tasks. These models have been pre-trained on
the target for our study due to their widespread adoption, an extensive dataset exceeding 5.5 trillion tokens, achieving
differing programming paradigms, and relevance in software state-of-the-art performance in code generation, completion,
development [14]. Python, a dynamically-typed, high-level reasoning, and repair tasks. We employ a quantized model
scripting language, is heavily used in AI/ML development. with 32 billion parameters, which was further trained for
Java, on the other hand, is a statically-typed, object-oriented instruction-tuning through a multi-stage fine-tuning process.
language widely deployed in enterprise-grade applications, We followed the default generation settings recommended
where software quality and security requirements are stringent. by each model’s authors and used their preferred prompting
By analyzing both, our aim is to ensure that our findings formats. Example prompts for Python and Java code genera-
regarding code quality, security vulnerabilities, and defect tion are shown in the following listing.
profiles generalize across different language ecosystems and
programming practices. You are an AI programming assistant, utilizing the
We began by adopting and extending the HMCorp DeepSeek-Coder model, developed by DeepSeek Company,
dataset [39], which contains 288,508 ⟨human-code, ChatGPT- and you only answer questions related to computer science.
code⟩ pairs in Python and 222,335 in Java. HMCorp was con- ### Instruction:
structed by filtering the Python and Java subsets of the Code-
Java
SearchNet (CSN) dataset [40] to extract functions authored by “““{docstring}”””
human developers. Then, for each function, the corresponding {signature}
docstring was used to prompt ChatGPT (“gpt-3.5-turbo”, April
2023) to generate a matching AI implementation. During this ### Response:
filtering process, noisy or malformed samples, such as those
containing HTML tags, URLs, or empty function bodies, were You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
systematically removed to ensure dataset quality. The original
Python
CodeSearchNet dataset is a widely-used benchmark compris- {signature}
ing ⟨documentation, code⟩ pairs in six different languages “““{docstring}”””
mined from public, non-forked GitHub repositories sorted by
popularity (stars and forks), and is often used to train and Following generation, a comprehensive code cleanup phase
evaluate LLMs on code-related tasks [41], [42]. was applied to ensure consistency and validity across all sam-
However, the released HMCorp dataset excluded the origi- ples. The cleanup process included normalization of indenta-
nal docstrings and GitHub provenance. To restore this informa- tion, removal of leading or trailing whitespace inconsistencies,
tion, we performed a pre-processing step to match each HM- and lightweight syntactic parsing checks to detect incomplete
Corp sample back to its source in CSN using a combination of or invalid code fragments. In addition, special care was taken
function signature and body similarity. This step allowed us to remove any non-code text artifacts often produced by AI
to recover both the documentation and repository metadata, models, such as explanations of the code, usage examples,
resulting in the creation of complete ⟨repository, docstring, comments unrelated to the function’s purpose, or generic
human-code, ChatGPT-code⟩ tuples for each instance. During model disclaimers. To ensure the preservation of valid code,

6
TABLE III: Defect statistics by language and code author. Blue A sample is considered defective if it contains at least one
are best scores, red are the worst. detected issue; a sample is classified as incorrect if it is either
Defective Incorrect Total
syntactically invalid or entirely missing (e.g., empty response
Language Author due to incorrect interpretation of the input by the model). The
Samples Samples Defects
total number of defects accounts for multiple issues within
Human 158,221 3,712 429,247
Python ChatGPT 110,528 2,549 185,110 the same sample, reflecting the cumulative defect count rather
(285,249) DeepSeek 173,899 20,783 319,643 than unique instances.
Qwen 178,225 7,190 574,916 In Python, Qwen-generated code exhibits the highest total
Human 76,049 245 202,129 number of defects, surpassing both other LLMs and human-
Java ChatGPT 78,241 2,056 161,056 written code. This is driven by its high number of defective
(221,795) DeepSeek 155,077 34,382 242,437 samples (62.48%) and a considerable number of incorrect ones
Qwen 60,389 3,833 123,180 (2.52%). DSC shows a lower defects rate (60.96%) but suffers
from the highest number of incorrect samples (7.29%) reflect-
ing more frequent generation failures such as not parseable or
the cleaning procedure was refined iteratively: several rounds incomplete outputs (19,612 empty predictions) and, therefore,
of manual inspection were conducted on random subsets of potentially less surface for defects. In contrast, ChatGPT
the data to verify that correct code instances were not inadver- produces the lowest number of defects and defective samples
tently discarded or damaged. Cleaning rules were adjusted as (38.75%), reflecting a distinct generation style that tends to
necessary to minimize the loss of properly generated samples. produce cleaner yet potentially oversimplified code. Also no-
Our final dataset consists of 507,044 ⟨docstring, human- table, it yields the least number of incorrect samples (0.89%).
code, ChatGPT-code, DSC-code, Qwen-code⟩ instances, Human-written Python code results in 429,247 total defects,
285,249 for Python and 221,795 for Java, respectively. The which is higher than both ChatGPT and DSC, but lower
human-written functions span 16,928 unique open-source than Qwen. This is primarily due to the high number of
GitHub repositories, ensuring a representative mix of styles, defective samples (55.47%) and a relatively low number
domains, and development practices. Table II provides a full of incorrect samples (1.30%). However, as detailed later, a
breakdown of the dataset. significant portion of these defects (∼47%) stems from a
single code issue detected by Pylint, i.e., protected-access,
V. E XPERIMENTAL E VALUATION
which accounts for 202,259 violations. This issue refers to
This section illustrates the results of our investigation. First, accessing class members prefixed with an underscore (e.g.,
we dive into the defect categorization and distribution between _internal_method()), which are considered protected
human-written and AI-generated code across programming by convention. In open-source repositories, especially those
languages. We show which are the most occurring ODC involving complex inheritance hierarchies or performance-
defect types and the frequency with which each specific code sensitive designs, such practices are common and often inten-
issue appears in both Python and Java datasets. Then, we tional, even if discouraged. If we exclude these violations, the
focus on the assessment of the security vulnerabilities that total number of defects in human-written Python code drops
are introduced by human developers and AI code generators, to ∼227k, making it comparable to ChatGPT.
respectively. We classify security issues according to MITRE’s In Java, DSC exhibits the worst performance across all
CWE and analyze their distribution and severity. Finally, we dimensions. It produces the highest number of total defects
examine whether there are any significant differences in the (242,437), the most defective samples (69.92%), and the most
structure and complexity of Python and Java programs written incorrect outputs (15.50%, 19,071 of which are empty outputs)
by developers and generated by LLMs. among all sources, reaching more than 85% of problematic
samples. This suggests that DSC struggles significantly with
A. Code Defects Assessment Java’s stricter syntax and structural constraints, even though
To investigate differences in the nature and frequency of the most frequent issues refer to best practices and design
code defects across human-written and AI-generated code defects and not critical errors. In contrast, human-written Java
(RQ1 ), we analyze Python and Java code generated by LLMs code results in 202,129 defects across 34.29% of defective
(i.e., ChatGPT, DeepSeek-Coder, and Qwen-Coder) and com- samples, with only 245 incorrect cases, demonstrating higher
pare it to a baseline of human-authored functions. All code syntactic and semantic consistency. ChatGPT ranks between
samples are analyzed using Pylint (for Python) and PMD (for these extremes, generating 35.28% of defective samples. Sur-
Java), and the resulting violations are categorized using the prisingly, although being the worst with Python, Qwen shows
Orthogonal Defect Classification framework to ensure consis- great capabilities in the generation of Java code, achieving
tency across languages and tools. Due to space constraints, the most robust performance among all authors, with 123,180
in this section we provide an overview of our findings, but total defects, 27.23% defective samples, and 1.73% incorrect
complete results can be found in the replication package [21]. outputs (3,209 of which are empty).
Table III summarizes the distribution of code issues in terms To better understand the nature of the issues detected by
of defective samples, incorrect samples, and total defects. the static analysis tools, we categorize all reported violations

7
 300k within local scopes.
ODC Defect Type
250k Algorithm No timing-related problems are detected apart from 16
Assignment useless-with-lock issues in ChatGPT code, which fails to
Checking
200k
Issues per type

Function/Class/Object establish real locking behavior, potentially leading to thread-

Interface safety problems. This is mostly due to Pylint not being well-
150k Timing
suited to detect synchronization defects.
100k These differences in defects distributions indicate that AI-
generated code is often simpler and more repetitive, leading
50k
to a higher concentration of variable reuse, shallow argument
propagation, and underdeveloped class structures. The recur-
Human ChatGPT DeepSeek Qwen
rence of unused arguments across all AI models in extremely
(a) Python
large numbers suggests that they frequently replicate parameter
300k patterns from docstrings or function signatures without cor-
ODC Defect Type
250k Algorithm rectly using them in the generated body. Meanwhile, structural
Assignment
Checking issues like too-few-public-methods and redefined-outer-name
200k
Issues per type

Function/Class/Object reflect difficulties in generating coherent and contextually

Interface
150k Timing integrated class hierarchies, issue not commonly found in
human-written code.
100k As for Java (1b), defect distributions reveal additional
50k differences between human-written and AI-generated code,
particularly in how structural and complexity-related issues
Human ChatGPT DeepSeek Qwen manifest. Human code exhibits the highest number of al-
gorithmic defects, totaling over 96k, and also shows signif-
(b) Java
icant presence in the assignment and checking categories.
Fig. 1: Distribution of ODC defect types by code author. Indeed, the most occurring defects are CyclomaticComplexity
(15,435), AvoidInstantiatingObjectsInLoops (14,351), Cogni-
tiveComplexity (13,065), and AvoidCatchingGenericException
according to the ODC. Figure 1 plots the distribution of (9,749). These reflect code quality concerns that typically arise
ODC defect types across by code author for Python and Java, in real-world, mature codebases, such as overly complex con-
respectively. Several notable trends emerge. trol flow, algorithmic flaws, and improper exception handling.
In Python (1a), human-written code is overwhelmingly Such defects often relate to maintainability or design debt,
dominated by interface-related issues, accounting for over rather than outright correctness, and are consistent with the
250k violations, more than all other categories combined. This broader, more semantically aware development practices of
skew is almost entirely due to a protected member being human programmers.
accessed outside a class (protected-access). The second most In contrast, AI-generated Java code, particularly from DSC
frequent type is algorithm due to no-else-return (22,619) and and Qwen, exhibits very different error profiles. DSC stands
raise-missing-from (10,108), suggesting unnecessary code and out sharply, with high counts for ImmutableField (41,669
lack of exception traceability, which are typical of hand- of function/class/object type) and SystemPrintln (37,986),
written codebases. followed by violations such as UnusedPrivateField, Avoid-
In contrast, the top defects in AI-generated code show PrintStackTrace, and UnusedFormalParameter. This indicates
substantially different patterns. All three models, ChatGPT, that DSC-generated code frequently defines fields that could
DSC, and Qwen, are dominated by assignment issues and be marked final, overuses low-level debugging constructs like
the latter two by function/class/object issues, rather than [Link]() and printStackTrace(),
interface and algorithm violations. For instance, ChatGPT’s and includes unused or unreferenced fields and parameters,
most frequent violations are assignment-related (with 32,992 suggesting syntactically plausible but structurally shallow code
unused-argument and 14,408 unused-variable), along with with limited architectural awareness.
unspecified-encoding (14,312), indicating failure to declare Qwen’s Java outputs exhibit a similar behavior with Un-
encoding in file I/O, and protected-access (7,108), which is usedFormalParameter (24,660) and SystemPrintln (13,329) as
far less frequent than in human-written code. DSC displays its most frequent violations. These point again to common
a unique profile with extremely high counts for unused- problems in low-complexity utility methods and boilerplate
argument (111,703) and too-few-public-methods (64,279), the code, where unused inputs and basic performance inefficien-
latter reflecting structural problems in class design, such as cies occur frequently. ChatGPT exhibits a more even spread
the generation of classes with trivial or placeholder content. of defects. Its top violations, SystemPrintln (10,428) and Use-
Qwen, similarly, is dominated by unused-argument (213,264), Varargs (9,124), point to best-practice violations rather than
followed by protected-access (105,770), and redefined-outer- complex architectural flaws. Compared to DSC, ChatGPT’s
name (57,252), a warning related to overwriting global names Java output is better structured but still presents common

8
TABLE IV: Security vulnerabilities statistics by language and
code author. Blue are best scores, red are the worst. CWE-78 2,243 13,419 10,908 7,318
CWE-754 13,157 1,539 0 8,214
Vulnerable Unique Total
Language Author
Samples CWEs CWEs CWE-400 1,631 6,887 7,987 4,857
Human 15,835 45 25,678
CWE-611 0 4,587 6,372 3,521
Python ChatGPT 22,864 45 40,035 CWE-330 1,272 2,045 1,853 1,889
(285,249) DeepSeek 22,878 43 39,982
CWE-502 1,349 1,638 1,692 1,762
Qwen 16,316 45 34,250
CWE-327 1,097 1,626 1,668 1,148
Human 6,660 42 11,677
Java ChatGPT 18,144 49 29,807 CWE-89 1,152 1,294 1,305 1,179
(221,795) DeepSeek 43,386 51 76,678 CWE-95 1,065 1,462 0 1,076
Qwen 13,620 41 21,301
CWE-319 0 0 1,928 0
Human ChatGPT DeepSeek Qwen
coding anti-patterns. (a) Python
▶ Key Finding 1: Overall, the qualitative difference between
human and AI-generated Java code is clear: human developers CWE-532 3,736 6,915 30,031 9,271
tend to write complex, expressive logic that introduces main- CWE-489 1,948 5,659 17,533 4,680
tainability issues (e.g., cognitive and cyclomatic complexity), CWE-798 784 1,737 5,199 901
while AI code assistants reproduce syntactically valid but CWE-209 582 1,238 4,549 1,001
semantically shallow code that overuses hardcoded debugging
and unused constructs. ChatGPT exhibits a somewhat more CWE-89 1,208 1,874 2,731 903
stable profile but still deviates from human practices, par- CWE-22 0 2,886 3,738 0
ticularly in areas like method scoping and parameter usage. CWE-611 297 1,480 1,960 659
On average, compared to human code, AI models are better CWE-321 541 989 1,757 367
at generating Python code yet much worse with Java code
CWE-470 741 1,228 0 914
(−4k and +22k defective samples). These findings further
reinforce the need to evaluate AI-generated code not only for CWE-502 247 1,064 0 458
correctness but also for design-level quality. Human ChatGPT DeepSeek Qwen
(b) Java
B. Code Security Assessment
Although AI code assistants can reliably generate syntac- Fig. 2: Heatmap of top-10 CWEs distribution across code
tically correct and functionally valid code, recent evidence authors.
suggests that they may also propagate insecure coding patterns,
often in subtle and systematic ways. Understanding whether
the nature of these vulnerabilities diverges from defects tra- code. DSC produces the most insecure code, with 22,878
ditionally introduced by human developers is essential to vulnerable samples and 39,982 total CWE-triggered issues,
adapting software security tools and mitigation frameworks followed closely by ChatGPT (22,864 vulnerable samples and
accordingly. 40,035 issues), reaching ∼8% of the dataset. Both models
To address RQ2 , we used Semgrep, configured with an substantially exceed the human baseline of 5.55% vulnerable
extensive set of security detection rules mapped to the CWE, samples and 25,678 total issues. Qwen shows a somewhat
to extract issue counts, severity levels, CWE types, and vul- more conservative behavior (5.72% vulnerable samples and
nerability distributions for each code sample. This analysis 34,250 issues), though still higher. Interestingly, despite these
enables a structured comparison of both the frequency and large disparities in total issue count, the number of distinct
nature of security issues, with the aim of determining whether CWEs triggered by each AI model in Python is nearly identical
AI-generated vulnerabilities form a distinct risk profile. The to human code (45 for humans, ChatGPT and Qwen, 43 for
complete results are available in the replication package [21]. DSC), suggesting that AI-generated code exhibits not only
Table IV provides a summary of the frequency and density more vulnerabilities but also repeated instantiations of the
(i.e., nr. of issues per sample) of security vulnerabilities same types of flaws. Moreover, while human-written code
detected. We report the number of vulnerable samples (i.e., contains 1.62 issues on average, this density reaches 1.75 for
unique samples containing at least one issue), of unique CWE ChatGPT and DSC, and 2.10 for Qwen.
identifiers triggered, and the total number of issues detected In Java, the gap is even more pronounced. Code generated
across all samples (i.e., a single function may contain several by DSC stands out with an extremely high number of vul-
different vulnerabilities). nerable samples (19.56%) and total issues (76,678), triggering
In Python, AI-generated code exhibits a higher volume 51 distinct CWE types, the highest among all authors and
but equal coverage of vulnerabilities than human-written languages. This indicates both a broader vulnerability surface

9
and more frequent violations. ChatGPT and Qwen also exhibit ChatGPT and DSC, but rare in human-written code.
numerous vulnerable samples (8.18% and 6.14%, respectively) ▶ Key Finding 2: In summary, these findings confirm that
compared to the human baseline of 3%, with a similar or AI-generated code differs from human-written code not only in
higher CWE variety (49 and 41 vs. 42 for humans). Overall, the number of security vulnerabilities but also in their nature
DSC consistently demonstrates the worst security profile in and distribution. AI models are particularly prone to recurring
both languages, while human-written code is consistently the instances of high-severity issues such as injection flaws, inse-
most secure across all metrics, followed by Qwen being the cure configurations, and information exposure. These trends
least prone to generate insecure code. hold across both Python and Java, although they are more
Figure 2 presents detailed heatmaps illustrating the distri- extreme in the latter. On average, compared to human code,
bution of the ten most frequent CWEs in Python and Java, AI models generate more vulnerable code both in Python (+5k
respectively, across different code authors. CWEs belonging samples) and Java (+18k samples). The combination of higher
to MITRE’s Top 25 most dangerous software weaknesses [35] defect volume, greater CWE diversity, and a skew toward high-
are shown in bold. In Python (2a), AI-generated code dis- severity categories underscores the importance of properly
proportionately triggers CWE-78 (OS Command Injection), assessing the security risks of AI-generated code.
with DSC and ChatGPT showing 10,908 and 13,419 instances
respectively, compared to only 2,243 in human code. A sim- C. Code Complexity Assessment
ilar trend is observed for CWE-400 (Uncontrolled Resource
Consumption) and CWE-611 (Improper Restriction of XML TABLE V: Comparison of structural complexity metrics. Bold
External Entity Reference), which are almost absent in human- values indicate the highest score per metric.
written code but occur in thousands of cases in DSC and
Avg. Avg. Avg. Avg.
ChatGPT outputs. Notably, these two models exhibit highly Language Author UT
NLOC CCN TC FNL
similar insecure behavior in Python, both in terms of volume
Human 12.72 3.97 142.45 14.16 70,542
and distribution, indicating that despite model differences, they ChatGPT 6.89 2.47 79.33 17.56 62,717
may internalize and reproduce comparable insecure generation Python
DeepSeek 5.16 2.00 68.37 12.51 60,199
patterns. Human-authored Python code, by contrast, is heavily Qwen 4.47 1.84 58.68 13.95 63,265
skewed toward CWE-754 (Improper Check for Unusual or Human 13.38 3.48 119.66 13.98 57,356
Exceptional Conditions), with 13,157 instances, indicating a ChatGPT 8.42 2.35 76.51 18.92 52,540
Java
different defect profile and confirming the frequent lack of DeepSeek 6.70 1.83 59.60 19.32 47,737
exception handling in projects written by human programmers. Qwen 6.14 1.90 61.43 13.85 50,698
DSC further distinguishes itself by generating code vul-
nerable to CWE-319 (Cleartext Transmission of Sensitive To answer RQ3 , we analyzed structural complexity metrics,
Information), which does not appear in any other author’s including the number of code lines (NLOC), the cyclomatic
code. These gaps emphasize that while AI models broadly complexity (CCN), average and unique token counts (TC and
follow similar trends, certain failure modes are model-specific UT), and function name length (FNL), across human-written
and need to be addressed differently from traditional human- and AI-generated code samples. The goal is to both quantify
written insecure code. differences in surface-level complexity and also to understand
Java results (2b) reveal a similar divergence in defect whether AI-generated code mirrors the structural richness
distributions. DSC again produces the highest frequencies and diversity observed in human-written software. Table V
across several high-impact CWE types. For instance, its code summarizes the results.
accounts for 30,031 instances of CWE-532 (Information Ex- In Python, the structural gap between human-written and
posure Through Log Files), compared to 3,736 for humans. It AI-generated code is particularly pronounced. Human-written
also dominates in CWE-489 (Active Debug Code), CWE-798 Python functions average 12.72 NLOC and exhibit a CCN of
(Use of Hardcoded Credentials), and CWE-209 (Information 3.97 on average, reflecting deeper logical branching and con-
Exposure Through an Error Message), all of which are sig- trol flow compared to AI-generated counterparts. In contrast,
nificantly underrepresented in human-written code. ChatGPT code snippets generated by ChatGPT, DSC, and Qwen are
and Qwen show similar but less extreme trends, while human- substantially shorter and simpler, with average NLOC values
authored Java code displays a flatter distribution with fewer of 6.89, 5.16, and 4.47 respectively, and CCN values of 2.47,
high-impact CWEs overall. 2.00, and 1.84. The average length of human Python code
The increased number of unique CWEs in AI-generated (142.45 tokens per function) is markedly higher than in AI-
code reflects a broader attack surface and a tendency to generated functions, with the most concise code produced by
reproduce known insecure patterns more frequently. Notably, Qwen (58.68 tokens). Finally, AI-generated Python functions
many of the most prevalent vulnerabilities align with cate- tend to have comparable or slightly longer function names
gories listed in MITRE’s Top 25, underscoring their severity. relative to human code, with ChatGPT being the most prolix.
CWE-78 (OS Command Injection), CWE-400 (Uncontrolled In Java, a similar but slightly less extreme trend emerged.
Resource Consumption), and CWE-798 (Use of Hardcoded Human-written Java functions demonstrate significantly higher
Credentials) are particularly frequent in code produced by scores across all metrics: the average NLOC for human

10
functions is 13.38, compared to 8.42 for ChatGPT, 6.70 for lacking executable context, making dynamic approaches such
DSC, and 6.14 for Qwen. Cyclomatic complexity is also as runtime verification and testing impractical. Nevertheless,
higher in human Java code (3.48) relative to AI-generated the use of mature and widely adopted static analysis tools like
output, which ranged from 1.83 to 2.35. Token counts follow Pylint and PMD ensures consistent, meaningful defect detec-
the same pattern, with human-written Java averaging 119.66 tion, effectively addressing our research goal of comparative
tokens per function, while AI code ranges between 59.60 quality assessment between human and AI-generated code.
and 76.51 tokens. Interestingly, AI-generated Java functions Moreover, given the heterogeneous nature of rule definitions
exhibit notably longer average function names, with DSC across static analyzers like Pylint and PMD, mapping them
(19.32 characters) and ChatGPT (18.92) exceeding the average to ODC categories inevitably involves interpretive judgment.
function name length chosen by human developers of 13.98, To mitigate potential subjectivity, we employed a structured
suggesting a pattern of more verbose or explicitly descriptive two-stage validation protocol where two authors independently
naming conventions in AI outputs. classified each rule, followed by a consensus resolution of
In addition to conventional structural metrics, we also any disagreements. While this process enhances reliability and
analyzed the unique token count for each author, representing reproducibility, some degree of ambiguity remains, particularly
the number of distinct lexical units (e.g., keywords, identifiers, for rules that straddle multiple defect types. Future work could
operators) used across the code samples. Vocabulary diversity explore automated or learning-based classification approaches
serves as a proxy for syntactic richness and expressiveness: a to further improve consistency.
higher number of unique tokens may indicate a broader use of Internal Validity: Our dataset builds upon the publicly avail-
language features, more varied code patterns, and potentially able and peer-reviewed HMCorp dataset [39], which includes
more nuanced or sophisticated program logic. Conversely, a over 500k human-written and ChatGPT-generated code sam-
low unique token count suggests more repetitive, template- ples in Python and Java. While this provides a strong base-
like code structures. In both Python and Java, human-written line, potential biases could be introduced during our dataset
code exhibited significantly higher unique token counts than expansion phase. Specifically, we generated additional AI
AI-generated code, with 70k unique tokens in human Python code using two state-of-the-art models, i.e., DeepSeek-Coder-
samples compared to 62k, 60k and 63k for ChatGPT, DSC Instruct and Qwen2.5-Coder-Instruct, based on the original
and Qwen, and 57k unique tokens in human Java compared to docstrings and function signatures. Although the generation
52k, 47 and 50k for AI counterparts. These findings suggest process was standardized to ensure consistency across models,
that, especially when writing Python code, human developers discrepancies in model behavior and prompt interpretation
employ a richer and more varied programming style, while could introduce subtle variability. Nonetheless, we mitigate
AI models, despite generating syntactically valid code, tend this threat by (i) incorporating outputs from multiple LLMs to
to reuse a narrower subset of language constructs. increase diversity, (ii) applying the same prompting templates
▶ Key Finding 3: Overall, the results consistently demon- across models to reduce bias, and (iii) anchoring our study on
strate that human-written code tends to be structurally more a high-quality and publicly available dataset that meets modern
complex, both in terms of size and logical structure, across standards for scale, diversity, and cross-language support.
both programming languages. In contrast, AI-generated code, External Validity: Our findings’ external validity may be con-
particularly that produced by DSC and Qwen, favors brevity strained by the selection of programming languages (Python
and reduced complexity. On average, it contains fewer lines of and Java), the specific LLMs used (ChatGPT, DeepSeek-
code (−6.75), exhibits lower cyclomatic complexity (−1.66), Coder, Qwen-Coder), and our prompt-based generation strat-
and uses significantly fewer tokens (−63.74), indicating re- egy relying primarily on docstrings. While these selections
duced structural and logical complexity. Additionally, AI- represent practical and widely used configurations in both
generated code tends to contain fewer unique tokens (−7,756), academia and industry, results could differ when employing
suggesting lower lexical diversity. These findings suggest that other programming languages, LLMs with varying model
AI models prioritize generating compact code over producing sizes, or alternative prompting approaches. To mitigate these
highly structured or richly branched functions. While reduced limitations, we employed a robust experimental design with
complexity and lexical variety may offer benefits such as a large-scale dataset comprising over 500k code samples,
improved readability, it may also introduce risks of under- including human-written code sourced from approximately
specification, limited error and edge cases handling, or shal- 17k real-world GitHub repositories, ensuring broad coverage
low logic coverage. and representativeness. Nonetheless, investigating additional
configurations remains an important direction for future work
VI. T HREATS TO VALIDITY to further validate and expand our conclusions.
Construct Validity: Our study exclusively relies on static
analysis tools for detecting code defects, which inherently lim- VII. C ONCLUSION
its our evaluation to syntactic and structural aspects, excluding This study offers a large-scale comparison of human-written
runtime behaviors, context-specific errors, and deeper semantic and AI-generated code on multiple dimensions of software
correctness. This limitation arises because the code generated quality, including defects, vulnerabilities, and structural com-
by LLMs often comprises standalone functions or snippets plexity. By applying standardized taxonomies, i.e., ODC and

11
CWE, we uncover fundamental differences in how code is [14] TIOBE, “TIOBE Index,” [Link]
produced by humans versus AI code assistants. Human-written [15] B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang,
B. Yu, K. Dang et al., “Qwen2.5-coder technical report,” arXiv preprint
code tends to be more structurally complex but prone to arXiv:2409.12186, 2024.
maintainability and design issues, whereas AI-generated code [16] Logilab, “PyLint - Python linter,” [Link]
is simpler but more repetitive and semantically shallow, with [17] PMD, “Source Code Analyzer,” [Link]
[18] Semgrep, “Static Code Analyzer,” [Link]
frequent unused constructs and hardcoded debugging. Criti- [19] R. Chillarege, I. S. Bhandari, J. K. Chaar, M. J. Halliday, D. S.
cally, AI-generated code also exhibits a higher prevalence of Moebus, B. K. Ray, and M.-Y. Wong, “Orthogonal defect classification-
high-severity security vulnerabilities, underscoring the need a concept for in-process measurements,” IEEE Transactions on software
Engineering, vol. 18, no. 11, pp. 943–956, 1992.
for stricter safeguards in AI-assisted development workflows. [20] D. Cotroneo, C. Improta, and P. Liguori, “Artifact,” [Link]
By systematically exposing these differences and structuring 5281/zenodo.16853222.
them within a consistent evaluative framework, we believe that [21] ——, “Replication Package,” [Link] vs
AI Code Quality.
this study lays the groundwork for developing a standardized [22] D. Cotroneo, C. Improta, P. Liguori, and R. Natella, “Vulnerabilities
methodology that better reflects the quality and risk profiles in AI Code Generators: Exploring Targeted Data Poisoning Attacks,”
of AI-generated code when compared to human-authored in Proceedings of the 32nd IEEE/ACM International Conference on
Program Comprehension, 2024, pp. 280–292.
programs. [23] A. Clark, D. Igbokwe, S. Ross, and M. F. Zibran, “A quantitative
analysis of quality and consistency in ai-generated code,” in 2024
ACKNOWLEDGMENT 7th International Conference on Software and System Engineering
This work has been partially supported by the (ICoSSE). IEEE, 2024, pp. 37–41.
[24] H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep
IDA—Information Disorder Awareness Project funded at the keyboard? assessing the security of github copilot’s code contri-
by the European Union-Next Generation EU within the butions,” Communications of the ACM, vol. 68, no. 2, 2025.
SERICS Program through the MUR National Recovery and [25] M. Kharma, S. Choi, M. AlKhanafseh, and D. Mohaisen, “Security and
quality in llm-generated code: A multi-language, multi-model analysis,”
Resilience Plan under Grant PE00000014, and the SERENA- arXiv preprint arXiv:2502.01853, 2025.
IIoT project funded by MUR (Ministero dell’Università e [26] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan,
della Ricerca) and European Union (Next Generation EU) H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large
language models trained on code,” arXiv preprint arXiv:2107.03374,
under the PRIN 2022 program (project code 2022CN4EBH). 2021.
[27] M. L. Siddiq, L. Roney, J. Zhang, and J. C. D. S. Santos, “Quality
R EFERENCES assessment of chatgpt generated code and their use by developers,” in
[1] N. A. Ernst and G. Bavota, “Ai-driven development is here: Should you Proceedings of the 21st International Conference on Mining Software
worry?” IEEE Software, vol. 39, no. 2, pp. 106–110, 2022. Repositories, 2024, pp. 152–156.
[2] Open AI, “Open AI ChatGPT,” [Link] [28] C. Improta, R. Tufano, P. Liguori, D. Cotroneo, and G. Bavota, “Quality
[3] GitHub, “GitHub Copilot,” [Link] in, quality out: Investigating training data’s role in ai code generation,”
[4] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, in 2025 IEEE/ACM 33rd International Conference on Program Com-
X. Bi, Y. Wu, Y. Li et al., “Deepseek-coder: When the large language prehension (ICPC). IEEE Computer Society, 2025, pp. 454–465.
model meets programming–the rise of code intelligence,” arXiv preprint [29] A. M. Esfahani, N. Kahani, and S. A. Ajila, “Understanding defects
arXiv:2401.14196, 2024. in generated codes by language models,” in 2024 34th International
[5] M. R. Lyu, B. Ray, A. Roychoudhury, S. H. Tan, and P. Thongtanunam, Conference on Collaborative Advances in Software and COmputiNg
“Automatic programming: Large language models and beyond,” ACM (CASCON). IEEE, 2024, pp. 1–10.
Transactions on Software Engineering and Methodology, 2024. [30] F. C. G. Bogaerts, N. Ivaki, and J. Fonseca, “A taxonomy for python
[6] Anthropic CEO Dario Amodei, [Link] vulnerabilities,” IEEE Open Journal of the Computer Society, 2024.
anthropics-ceo-said-all-code-will-be-ai-generated-in-a-year/91163367. [31] A. Patel, K. Z. Sultana, and B. K. Samanthula, “A comparative analysis
[7] H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, between ai generated code and human written code: A preliminary
Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code study,” in 2024 IEEE International Conference on Big Data (BigData).
generation with generative pre-trained models,” in Proceedings of the IEEE, 2024, pp. 7521–7529.
46th IEEE/ACM International Conference on Software Engineering, [32] The MITRE Corporation (MITRE), “Common Weakness Enumeration,”
2024, pp. 1–12. [Link]
[8] N. S. Mathews and M. Nagappan, “Test-driven development and llm- [33] GitHub, “CodeQL,” [Link]
based code generation,” in Proceedings of the 39th IEEE/ACM Interna- [34] PyCQA, “Bandit,” [Link]
tional Conference on Automated Software Engineering, 2024. [35] MITRE, “Top 25 Most Dangerous Software Weaknesses,” [Link]
[9] B. Yetiştiren, I. Özsoy, M. Ayerdem, and E. Tüzün, “Evaluating the [Link]/top25/.
code quality of ai-assisted code generation tools: An empirical study [36] OWASP, “2021 OWASP Top 10,” [Link]
on github copilot, amazon codewhisperer, and chatgpt,” arXiv preprint [37] T. Yin, “Lizard,” [Link]
arXiv:2304.10778, 2023. [38] Hauntsaninja, “Tiktoken, fast BPE tokenizer,” [Link]
[10] Y. Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X.- tiktoken/.
B. D. Le, and D. Lo, “Refining chatgpt-generated code: Characterizing [39] X. Xu, C. Ni, X. Guo, S. Liu, X. Wang, K. Liu, and X. Yang,
and mitigating code quality issues,” ACM Transactions on Software “Distinguishing llm-generated from human-written code by contrastive
Engineering and Methodology, vol. 33, no. 5, pp. 1–26, 2024. learning,” ACM Transactions on Software Engineering and Methodol-
[11] N. Tihanyi, T. Bisztray, M. A. Ferrag, R. Jain, and L. C. Cordeiro, “How ogy, vol. 34, no. 4, pp. 1–31, 2025.
secure is ai-generated code: a large-scale comparison of large language [40] H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt,
models,” Empirical Software Engineering, vol. 30, no. 2, pp. 1–42, 2025. “CodeSearchNet challenge: Evaluating the state of semantic code
[12] Y. Fu, P. Liang, A. Tahir, Z. Li, M. Shahin, J. Yu, and J. Chen, “Se- search,” arXiv preprint arXiv:1909.09436, 2019.
curity weaknesses of copilot generated code in github,” arXiv preprint [41] Z. Zheng, K. Ning, Y. Wang, J. Zhang, D. Zheng, M. Ye, and J. Chen,
arXiv:2310.02059, 2023. “A survey of large language models for code: Evolution, benchmarking,
[13] S. Kabir, D. N. Udo-Imeh, B. Kou, and T. Zhang, “Is stack overflow and future trends,” arXiv preprint arXiv:2311.10372, 2023.
obsolete? an empirical study of the characteristics of chatgpt answers to [42] Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware
stack overflow questions,” in Proceedings of the 2024 CHI Conference unified pre-trained encoder-decoder models for code understanding and
on Human Factors in Computing Systems, 2024, pp. 1–17. generation,” arXiv preprint arXiv:2109.00859, 2021.

12