A Large-Scale Empirical Study of AI-Generated Code in

Real-World Repositories
Tianhao Mao Dongfang Zhao Haixu Tang
Indiana University Bloomington Indiana University Bloomington Indiana University Bloomington
tianmao@[Link] zhaodo@[Link] hatang@[Link]

Xiaofeng Wang Hang Zhang

Nanyang Technological University Indiana University Bloomington
[Link]@[Link] hz64@[Link]

Abstract Copilot and Cursor, have enabled developers to offload a wide spec-
arXiv:2603.27130v2 [[Link]] 3 Apr 2026

Large language models (LLMs) are increasingly used in software trum of programming tasks to AI. These tools are now routinely
development, generating code that ranges from short snippets to used not only for generating small code snippets or completing
substantial project components. As AI-generated code becomes functions, but increasingly for synthesizing entire files, modules,
more common in real-world repositories, it is important to under- and even full-fledged projects. As a result, AI-assisted programming
stand how it differs from human-written code and how AI assistance is transitioning from an experimental capability to a mainstream
may influence development practices. However, existing studies development paradigm, fundamentally altering how software is
have largely relied on small-scale or controlled settings, leaving a written and maintained.
limited understanding of AI-generated code in the wild. This paradigm shift raises critical questions about the charac-
In this work, we present a large-scale empirical study of AI- teristics and implications of AI-generated code in real-world soft-
generated code collected from real-world repositories. We examine ware systems. While early studies have examined LLM-generated
both code-level properties, including complexity, structural char- code in controlled environments—typically focusing on small, self-
acteristics, and defect-related indicators, and commit-level charac- contained programming tasks or benchmark datasets—such settings
teristics, such as commit size, activity patterns, and post-commit fail to capture the complexity and dynamics of real-world devel-
evolution. To support this study, we develop a detection pipeline opment. In practice, AI-generated code is often intertwined with
that combines heuristic filtering with LLM-based classification to human-written code, iteratively refined, and deployed within large,
identify AI-generated code and construct a large-scale dataset for evolving codebases. Understanding the properties of such code
analysis. therefore requires moving beyond lab-based evaluations toward
Our study provides a comprehensive view of the characteristics comprehensive empirical studies in real-world repositories.
of AI-generated code in practice and highlights how AI-assisted A key aspect of this understanding lies in analyzing code-level
development differs from conventional human-driven development. characteristics of AI-generated code. These include traditional soft-
These findings contribute to a better understanding of the real- ware quality and complexity metrics, such as cyclomatic complex-
world impact of AI-assisted programming and offer an empirical ity, control-flow and data-flow complexity, loop nesting depth,
basis for future research on AI-generated software. and language-specific constructs (e.g., object-oriented metrics for
Java or pointer-related complexity in C/C++). Additionally, defect-
ACM Reference Format:
related indicators, such as bugs per KLOC, are essential for assessing
Tianhao Mao, Dongfang Zhao, Haixu Tang, Xiaofeng Wang, and Hang
reliability and maintainability. Comparing these metrics between
Zhang. 2026. A Large-Scale Empirical Study of AI-Generated Code in Real-
World Repositories. In . ACM, New York, NY, USA, 12 pages. [Link] AI-generated and human-written code can provide insights into
10.1145/[Link] whether AI assistance leads to simpler, more complex, or potentially
more error-prone implementations.
1 Introduction Beyond code-level properties, it is equally important to inves-
tigate commit-level characteristics associated with AI-generated
Recent advances in artificial intelligence, particularly large lan-
code. Modern software development is inherently collaborative
guage models (LLMs), are profoundly reshaping modern software
and version-controlled, making commits a natural unit of analysis.
engineering practices. Systems such as OpenAI Codex, GPT-series
Questions arise as to whether AI-assisted development changes
models, and Claude, along with IDE-integrated assistants like GitHub
developer behavior and workflow dynamics: Do AI-generated com-
Permission to make digital or hard copies of all or part of this work for personal or mits modify larger portions of code? Do they accelerate develop-
classroom use is granted without fee provided that copies are not made or distributed ment by increasing commit frequency or reducing development
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than the
time? How stable are such commits—do they require more frequent
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or subsequent fixes, revisions, or reverts? Addressing these questions
republish, to post on servers or to redistribute to lists, requires prior specific permission is crucial for understanding the broader impact of AI on software
and/or a fee. Request permissions from permissions@[Link].
Conference’17, Washington, DC, USA
engineering processes.
© 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. Despite the growing interest in AI-assisted programming, there
ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM remains a significant research gap. Existing work primarily focuses
[Link]
Conference’17, July 2017, Washington, DC, USA Tianhao Mao, Dongfang Zhao, Haixu Tang, Xiaofeng Wang, and Hang Zhang

on limited sets of code-level metrics and evaluates LLM-generated classification or detection purposes, rather than for a systematic
outputs in controlled, small-scale settings. To the best of our knowl- characterization of how LLM-generated code differs from human-
edge, no prior study has conducted a comprehensive, large-scale em- written code across multiple structural dimensions.
pirical analysis that jointly examines both code-level and commit- Security is another major focus in prompt-based evaluations
level characteristics of AI-generated code in real-world repositories. of LLM-generated code. Early work conducted exploratory assess-
This gap limits our ability to draw reliable conclusions about the ments of the security of ChatGPT-generated code [16]. Subsequent
practical implications of adopting AI in software development. studies expanded the evaluation to multiple models and further
In this paper, we aim to bridge this gap by conducting a compre- examined how model parameters and prompting strategies affect
hensive empirical study of AI-generated code in real-world projects. the security of generated code [3]. To enable more systematic as-
A central challenge in this endeavor is identifying and collecting sessment, [33] proposed a unified framework integrating code gen-
AI-generated code from large-scale repositories, where such in- eration and security analysis, while [22] provided a more detailed
formation is not explicitly labeled. To address this, we develop study of the impact of prompt engineering on code security. [29]
multi-stage rule-based message filter and LLM-assisted classifier studied LLM-based security patching, which also sheds light on
for Github commit messages to construct a large dataset of AI- the security implications of LLM-generated code. In addition, [38]
generated code instances. Based on this dataset, we perform, to the and [42] developed datasets and benchmarks for more systematic
best of our knowledge, the most extensive measurement study to evaluation of the quality and security of LLM-generated code. Some
date, covering a wide range of code-level and commit-level metrics. work has further explored mitigation strategies, such as prompt
Our analysis yields a set of actionable insights into the current engineering for safer code generation [30] and model fine-tuning
state of AI-assisted software development. These insights shed to improve the security of generated code [13]. Relatedly, several
light on how AI-generated code differs from human-written code studies have investigated the use of LLMs themselves for vulner-
in terms of complexity, quality, and evolution, as well as how it ability and bug discovery in code [4, 19]. Overall, these studies
influences development practices at the commit level. Such findings provide important foundations for understanding the security of
have important implications for researchers, practitioners, and tool LLM-generated code, but they still predominantly evaluate model
designers seeking to better understand and improve the integration outputs under controlled settings.
of AI into software engineering workflows. Beyond controlled prompting, a growing body of work has be-
In summary, this paper makes the following contributions: gun to examine LLM-generated code in real-world open-source
• We build and release a large-scale (by far largest to our knowledge) development. [25] and [8] studied GitHub Copilot-generated code
dataset of AI-generated code extracted from real-world repositories, in real GitHub repositories, focusing on security issues and the
enabling various analytical study. ability of LLMs to repair vulnerabilities. [40] analyzed self-admitted
• Based on the dataset, we perform comprehensive measurement LLM usage in GitHub repositories, examining how generative AI
study of both code-level and commit-level characteristics, covering is used in practice and surveying developers about usage policies.
an extensive set of metrics, comparing AI-generated and human- Using the AIDev dataset [17] of LLM-agent pull requests, [34] inves-
written code. tigated security-related issues, review interactions, and ecosystem
• We distill multiple insights from our measurement findings re- characteristics, while [12] examined cyclomatic complexity and
garding AI-assisted programming in practice and discuss potential code quality in the same setting. Other studies focused on specific
implications for skateholders including programmars and AI ven- aspects of AI-generated code in repositories, such as code smells
dors. in GitHub Copilot-generated code [43] and adoption patterns re-
flected in modified lines and touched files at both commit and file
levels for AI-agent-generated code [28]. These studies move beyond
2 Related Works synthetic prompting scenarios and provide valuable evidence about
Existing research on LLM-generated code can be broadly grouped how LLMs are actually used in practice.
into three lines: controlled prompt-based evaluation of generated Another related line of work studies the detection and measure-
outputs, empirical studies of real-world AI-assisted development, ment of LLM-generated code in the wild. [7] measured the usage
and detection or measurement of LLM-generated code in the wild. of LLMs for coding by training a classifier for LLM-generated code,
Most prior work evaluates LLM-generated code in prompt-based while [36], [24], and [2] also explored methods for detecting LLM-
or benchmark-style settings, where researchers design program- generated code. These studies are closely related because they help
ming tasks, ask LLMs to generate code, and then analyze the re- identify AI-generated code in large-scale repositories, but their
sulting outputs. For example, [23] studied differences in maintain- main goal is detection or usage estimation rather than a system-
ability and reliability between LLM-generated and human-written atic comparison of the characteristics of real-world LLM-generated
code, while [18] examined performance and efficiency, and [35] code and human-written code.
focused on call calibration and correctness of LLM-generated code. Despite this growing body of work, an important gap remains.
[41] compared code style differences between human-written and Prompt-based studies mainly evaluate generated outputs on de-
LLM-generated code and further tracked style trends in GitHub signed tasks, which do not fully reflect how LLMs are used in real
open-source projects. Other studies investigated code-level features software development. Real-world repository studies, in contrast,
such as line counts, cyclomatic complexity, nesting depth, and class usually focus on isolated aspects such as security issues, code smells,
counts [27], while [10] examined lexical properties such as lex- review interactions, or adoption patterns. As a result, there is still
ical density, however these features were often used mainly for
A Large-Scale Empirical Study of AI-Generated Code in Real-World Repositories Conference’17, July 2017, Washington, DC, USA

Phase 1. Initial Data Collection. Following prior work [31], we

leverage the GitHub REST API to retrieve code files containing
explicit mentions of popular LLM tools. We extend the scope to
include recent models (e.g., Claude, Gemini, DeepSeek) and repos-
itories up to 2025. After filtering for comments containing tool
mentions, we obtain an initial dataset of 31,726 commits and 44,616
unique code files.
Phase 2. Multi-Stage Filtering. To improve data quality, we de-
velop a multi-stage filtering pipeline:
(1) Keyword normalization. We eliminate false matches caused by
substring-based search by ensuring that matched prefixes (e.g.,
“use”, “by”) appear as standalone words in comments. This step
reduces noise introduced by the GitHub search mechanism.
(2) Semantic attribution filtering. We identify explicit attribution
patterns (e.g., “generated by”, “written by”, “enhanced by”) and
analyze their surrounding context to ensure that the LLM is used
for code generation. Mentions referring to non-code artifacts (e.g.,
Figure 1: Measurement Pipeline.
text, media, or data generation) are excluded.
(3) Usage-based exclusion. We remove cases where LLMs are men-
tioned only as tools for auxiliary tasks (e.g., API calls, analysis, or
limited understanding of how real-world LLM-generated code sys- communication), rather than directly contributing to code genera-
tematically differs from human-written code, not only in code-level tion.
characteristics but also in development activity patterns. After rule-based filtering, we obtain 4,882 commits and 8,141
Our work addresses this gap by conducting a systematic compar- files. Manual validation on randomly sampled data shows high
ative study of real-world LLM-generated code and human-written precision (TP=88/100, TN=95/100), indicating the effectiveness of
code in open-source repositories. Rather than relying on crafted our rules.
prompts or benchmark tasks, we analyze code that has actually Phase 3. LLM-assisted Classification. For samples that cannot
been produced and integrated into real GitHub projects. Moreover, be confidently classified by rules, we further employ an LLM-based
instead of focusing on a single aspect such as security, code smell, classifier (ChatGPT 5-mini) to infer whether code is AI-generated.
or adoption, we jointly examine both code-level characteristics and This step yields an additional 7,867 commits (10,713 files). Remain-
development-level characteristics, providing a broader empirical ing uncertain cases are manually labeled, resulting in 686 additional
view of how LLM assistance affects real-world software develop- positive samples. A secondary validation confirms the reliability of
ment. LLM-based classification (TP=85/100, TN=93/100).
Final Dataset. After deduplication using repository, file path, and
3 Measurement Pipeline commit hash, our final dataset contains 12,749 commits and
19,816 code files with confirmed AI involvement. To our knowl-
In this section we describe our measurement pipeline, from real-
edge, this is the largest dataset of real-world AI-generated code by
world dataset collection to the various metrics we consider and
far, which we will make public. Fig. 2a and Fig. 2b illustrate its dis-
the methodology to measure them. The process of our pipeline is
tribution across LLM models/tools and programming languages, re-
described in Figure 1.
spectively. Consistent with prior observations [31], general-purpose
models (e.g., ChatGPT) dominate real-world usage, while code spans
3.1 Dataset Collection and Alignment a diverse set of languages with Python and JavaScript being the
As we target real-world code repositories, the first challenge is most prevalent.
how to collect AI-generated code within them. To that end, we got Human Code Alignment. To enable fair comparison between
inspiration from recent work [31] that demonstrates the feasibility AI-generated and human-written code, we construct a matched
of identifying AI-generated code from GitHub by searching for human code dataset. Direct comparison without control may in-
explicit hints in code comments (e.g., “generated by ChatGPT”). troduce confounding factors, such as repository-specific coding
However, such approaches suffer from several limitations. First, practices, project complexity, or code scale differences. For exam-
keyword search often introduces substantial noises due to sub- ple, variations in defect density or complexity metrics may stem
string matching (e.g., matching “use” in unrelated words), leading from repository characteristics rather than the effect of AI assis-
to false positives. Second, attribution comments are frequently am- tance.
biguous, as mentions of LLMs may refer to non-code tasks (e.g., To mitigate such biases, we adopt an alignment strategy: for
documentation or data generation). Third, prior filtering strategies each AI-generated file, we randomly select two human-written files
remain relatively coarse-grained and cannot reliably distinguish from the same repository and same programming language, with
true AI-generated code from incidental references. To address these similar size (within 20% lines of code when possible). This design
challenges, we design a refined data collection pipeline that im- controls for repository-level and language-level factors, ensuring
proves both the precision and reliability:
Conference’17, July 2017, Washington, DC, USA Tianhao Mao, Dongfang Zhao, Haixu Tang, Xiaofeng Wang, and Hang Zhang

ChatGPT (83.3%) Copilot (10.8%) DeepSeek (2.2%) (e.g., bugs per KLOC), while structural metrics are measured at the
Gemini (1.7%) Cursor (1.3%) Others (0.7%) function or file level (e.g., average number of loops per function).
Statistical Significance Testing. To draw meaningful and reliable
0% 20% 40% 60% 80% 100% conclusions in code characteristic differences betwene AI and hu-
man, we perform statistical hypothesis testing to determine whether
(a) Distribution of AI-generated code by tools observed AI/human differences are statistically significant.
Implementation. We employ various tools to compute the afore-
Python (41.7%) JavaScript (15.4%) Java (8.6%)
mentioned metrics (e.g., cloc for basic statistics, CK and Understand
C++ (7.8%) TypeScript (7.8%) C# (4.8%)
for OOP metrics, CodeQL for security analysis, etc.). More details
Swift (3.8%) C (3.7%) Others (6.4%)
can be found in Table 1, which summarizes all measured metrics,
along with their analysis tools and normalization scopes.
0% 20% 40% 60% 80% 100%
(b) Distribution of AI-generated code by languages
3.3 Study Commit-Level Patterns
Beyond code-level characteristics (§3.2), it is equally important
Figure 2: Distribution of AI-generated code records by tools
to understand how AI-generated code is produced, integrated, and
and programming languages.
evolved within real-world development workflows. Commit-level
analysis provides a natural lens to study such dynamics, as commits
capture both developer activity and project evolution. By examin-
that observed differences more accurately reflect the impact of AI ing commit histories associated with AI-generated code, we aim
assistance rather than external confounders. to uncover how developers collaborate with AI tools in practice
After these preprocessing, our final dataset contains 36,467 matched and how AI-assisted code influences development processes and
human written code files and 19,816 AI-generated files, enabling outcomes.
robust comparative analysis under controlled conditions. Research Questions. We specifically aim at answering the fol-
lowing research questions during our commit-level study, which
collectively characterize the real-world usage patterns and lifecycle
3.2 Study Code-Level Characteristics
dynamics of AI-assisted code.
We aim at comprehensively understanding the differences between
• RQ1: Temporal patterns. Do AI-assisted commits exhibit different
AI-generated and human-written code in real-world settings, to
temporal characteristics (e.g., working hours, release proximity)
that end, we measaure an extensive and diverse set of code-level
compared to human-written code?
characteristics, surpassing prior works and providing a holistic
• RQ2: Change characteristics. Do AI-generated commits involve dif-
comparison.
ferent magnitudes or types of changes (e.g., number of modified
Metric Taxonomy. Specifically, our scope includes the following
files, lines of code, churn)?
categories of metrics:
• RQ3: Post-commit evolution. Are AI-generated commits more
(1) Code Size and Basic Statistics. Basic properties such as lines of
likely to be revised, reverted, or modified after submission?
code (LOC), number of files, and token-level statistics, providing a
• RQ4: Collaboration and workflow. How does AI-assisted code
baseline understanding of code scale and granularity.
affect developer collaboration patterns, such as handoffs and
(2) Structural Complexity. We analyze fine-grained structural com-
multi-author contributions?
ponents extracted from abstract syntax trees (ASTs), including
• RQ5: Stability and maintenance. Does AI-generated code require
functions, parameters, classes, statements, conditionals, loops, and
more frequent fixes or exhibit different stabilization patterns
recursion. We further derive complexity-related metrics such as
compared to human-written code?
cyclomatic complexity, nesting depth, and decision counts.
(3) Code Style and Duplication. Stylistic properties such as code Methodology. To answer these questions, we extract commit-level
duplication, reflecting maintainability and reuse patterns. information from real-world repositories corresponding to both
(4) Object-Oriented Design Metrics. For object-oriented languages AI-generated and human-written code. Specifically, information is
(e.g., Java, C++, Python), we measure classical OOP metrics (e.g., collected from the following dimensions:
class-level complexity and coupling) to understand design quality. • Temporal Characteristics (for RQ1). Commit timestamps, including
(5) Program Graphs. We analyze program characteristics from their time of day, day of week, and distance to the nearest release.
control-flow, data-flow, and call graphs, capturing higher-level prop- • Change Characteristics (for RQ2). The scale and scope of code
erties regarding both syntax and semantic. changes, including the number of modified files, lines, and overall
(6) Code Defects and Security. We measure security-related proper- churn.
ties such as vulnerability patterns and counts of potential defects, • Post-Commit Evolution (for RQ3, RQ4, and RQ5). The frequency
gaining insights of comparative code quality. of post-commit modification and revert within fixed time win-
Metric Normalization. As the absolute volume of AI-generated dows (e.g., 7, 30, and 90 days), the number of subsequent edits,
and human-written code is different in our dataset (§3.1), direct participating authors, and time to first modification.
metric comparison by raw counts is misleading. To address this, • Workflow and Maintenance Patterns (for RQ4, and RQ5). These in-
we normalize each metric using an appropriate measurement unit. clude fix likelihood, determined by whether subsequent commits
For example, defect-related metrics are normalized by lines of code contain fix-related keywords (e.g., “fix”, “revert”), stabilization
A Large-Scale Empirical Study of AI-Generated Code in Real-World Repositories Conference’17, July 2017, Washington, DC, USA

Table 1: Summary of code-level metrics, analysis tools, and measurement scopes.

Category Metrics Tool Scope
Code Size & Statistics LOC, token count, file size cloc [6], pygments [5] file / function
Structural Complexity #functions, parameters, statements, conditions, loops, recursion tree-sitter [37] per function / per file
Cyclomatic complexity, nesting depth, decision count AST-based analysis function
Code Style Code duplication ratio jscpd [15] file / project
OOP Design Class/method metrics (e.g., complexity, coupling) CK (Java) [1], Understand (C++, Python) [32] class
Program Graphs CFG/DFG properties, call graph size, dependencies joern [14] + networkx [11] function / file
Security Vulnerability count, CWE patterns CodeQL [9] per KLOC
Hard-coded secrets (passwords, API keys) String analysis file

time, measured as the time until the last modification of a file, Table 2: Basic code statistics per file.
and handoff latency, defined as the time until another developer Metric AI Human
modifies the code. We consider related issues in the interval of
90 days after the commit. PLOC 256.57 192.68
SLOC 171.47 132.14
Statistical Hypothesis Testing. Similar to §3.2, we ensure that our BLOC 38.87 25.93
comparative conclusions are statistically significant. Specifically, CLOC 46.22 34.61
beyond hypothesis testing between AI-generated code and human Blank ratio 15.15% 13.46%
written code, we also implemented the pairwise comparisons, that Comment ratio 18.01% 17.96%
compare each AI-generated with one of the human written code in PLOC: physical lines; SLOC: source lines;
the same repository. we use Chi-squared tests [26] for categorical BLOC: blank lines; CLOC: comment lines
variables and Mann-Whitney U tests [20] for numerical variables
in group-level analysis, and McNemar’s tests [21] and Wilcoxon
signed-rank tests [39] for paired comparisons. Table 3: Code Token Statistics Distribution per Function
Implementation. To collect the aforemntioned information and Items AI mean±STD Human mean±STD p-value cohen-d
answer our research questions, we mainly use the git toolset with Token 122.49±333.59 94.75±3681.62 0.000 0.009
various corresponding commands (e.g., git diff for change char- Unique token 32.66±41.16 23.11±591.71 0.000 0.019
acteristics and git log for some temporal metadata, etc.). Unique Operator 8.79±4.72 6.09±5.80 0.000 0.489
Unique Operand 20.14±37.47 14.29±591.45 0.000 0.012
Content ratio(%) 35.91±15.20 23.51±20.05 0.000 0.658
4 Code-Level Measurement Results Lexical Density 0.531±0.260 0.692±0.301 0.000 -0.555
We present our measurement results and distilled findings regarding Keyword count 7.75±20.87 5.93±171.11 0.000 0.012
code-level metrics in this section, as outlined previously in §3.2. Identifier count 35.67±91.46 24.45±447.21 0.000 0.029
Literal count 17.46±87.41 16.04±1895.41 0.554 0.001
identifier count: amount of identifier tokens (e.g. variable, function, class names)
4.1 Basic Code Statistics literal count: amount of literal tokens (e.g. numeric, string, boolean)
Table 2 summarizes file-level basic statistics. Overall, AI-generated content ratio: (identifier count+literal count)/total token amounts
code exhibits consistently larger size across all LOC types. At lexical density: ratio of unique tokens/total tokens

the function level, token-based analysis (Table 3) reveals that AI-

generated code uses more (unique) tokens, significantly more op-
erators (Cohen’s 𝑑 = 0.489), operands, keywords, and identifiers. Insight 2: Explicitness over Compactness. AI-generated code
Furthermore, Table 4 shows that AI code also involves more per- favors explicit and expanded representations of logic (higher
function syntax constructs in general (except the switch-case content ratio), while human-written code tends to compress se-
constructs). These results suggest that rather than merely being mantics into denser expressions (higher lexical density).
larger, AI-generated code appears systematically more verbose across
structural and non-structural elements, while the nearly identical
comment ratio (Table 2) suggests that verbosity is not driven by 4.2 Structural Complexity
documentation but by expanded code bodies. We next examine structural and control-flow complexity at both
Insight 1: Verbosity vs. Compression. AI-generated code is function and file levels.
consistently more verbose across files and functions, not due to Function-Level Structure. Table 4 shows that AI-generated func-
additional comments but due to expanded structural and token- tions contain more statements (+2.61), conditions, loops, and slightly
level representations. In contrast, human-written code achieves more parameters. However, effect sizes are generally small (𝑑 < 0.2),
similar functionality with more compact representations. suggesting consistent but moderate differences.
Another notable pattern is the contrast between content ratio and Insight 3: Expanded Local Structure. AI-generated functions
lexical density(Table 3). AI-generated code shows higher content tend to expand logic within functions by introducing more state-
ratio but lower lexical density, suggesting that it expresses more ments and control structures, leading to more explicit but not
logic through repeated or explicit constructs rather than compact drastically more complex implementations.
abstractions.
Conference’17, July 2017, Washington, DC, USA Tianhao Mao, Dongfang Zhao, Haixu Tang, Xiaofeng Wang, and Hang Zhang

Table 4: Code structure components statistics per Function Table 7: Code structure components statistics per File
Items AI mean±STD Human mean±STD p-value cohen-d Items AI codes Human codes
Statement count 8.20±17.46 5.59±16.53 0.000 0.155 Function count 6.62 11.67
Return count 0.761±1.56 0.746±1.96 0.001 0.008 Function size(AST nodes) 138.97 201.82
Loop count 0.385±1.98 0.287±1.08 0.000 0.070
Method count 4.95 4.59
Condition count 1.06±3.18 0.759±3.21 0.000 0.094
Switch count 0.013±0.181 0.016±0.154 0.000 -0.015 Method size(AST nodes) 103.82 97.27
Case count 0.084±1.55 0.116±2.51 0.000 -0.014 Class count 0.988 0.885
Parameter count 1.50±1.88 1.39±1.70 0.000 0.062 Class size(AST nodes) 584.91 574.59
Enumeration count 0.080 0.078
Recursive functions count 0.476 0.525
Table 5: Control Flow Metrics per Function Recursive call count 0.587 0.672
Items AI mean±STD Human mean±STD p-value cohen-d
Cyclomatic Complexity 2.62±4.73 2.47±5.44 0.000 0.030
Modified Cyclomatic Complexity 2.55±4.32 2.37±4.66 0.000 0.041
Stricted Cyclomatic Complexity 2.86±5.55 2.76±6.47 0.000 0.016
Max nesting depth(All) 0.787±1.42 0.582±1.09 0.000 0.172 more methods and comparable class-level structure, suggesting a
Average nesting depth(All) 0.280±0.634 0.187±0.440 0.000 0.184
Max nesting depth(Loop) 0.265±0.586 0.227±0.552 0.000 0.068
shift toward method-based organization.
Average nesting depth(Loop) 0.138±0.307 0.118±0.288 0.000 0.068 Insight 6: Function-Oriented vs. Method-Oriented Design.
Max nesting depth(Condition) 0.256±1.10 0.301±0.843 0.000 -0.048
Average nesting depth(Condition) 0.137±0.563 0.162±0.452 0.000 -0.051 Human-written code tends to organize logic into more and larger
Max nesting depth(Function) 0.092±0.379 0.087±0.397 0.000 0.013 standalone functions, while AI-generated code favors method-
Average nesting depth(Function) 0.051±0.210 0.049±0.219 0.006 0.008
Boolean expression complexity 0.451±1.97 0.465±2.34 0.007 -0.006 based structuring within classes, indicating different decomposi-
Decision points 1.62±4.73 1.47±5.44 0.000 0.030 tion strategies.

Table 6: Halstead Metrics per Function Additionally, recursive patterns are slightly less frequent in AI-
generated code, suggesting a preference for iterative or explicitly
Items AI mean±STD Human mean±STD p-value cohen-d expanded logic.
Vocabulary 28.93±39.70 20.38±591.62 0.000 0.017 Insight 7: Reduced Use of Recursion. AI-generated code is less
Length 114.61±316.35 88.72±3628.00 0.000 0.008
Volume 765.59±2961.67 1096.37±77227.68 0.008 -0.005
likely to use recursion, favoring iterative and explicitly expanded
Difficulty 10.88±9.50 10.98±12.40 0.000 -0.009 control flow.
Effort 21766.91±410243.36 73577.37±17204915.43 0.062 -0.004

4.3 Code Style

Control-Flow Complexity. Table 5 shows that classical complexity Code style reflects how developers organize, format, and reuse code.
metrics (cyclomatic complexity, decision points) are only slightly We examine two complementary aspects: duplication patterns and
higher in AI-generated code, with very small effect sizes. How- formatting structure.
ever, a more meaningful difference emerges in nesting behavior. AI-
generated code has higher overall and loop nesting depth, whereas Code Duplication. Table 8 summarizes duplication metrics. Over-
human-written code exhibits slightly higher condition nesting and all, human-written code exhibits substantially higher duplication
boolean complexity. ratios than AI-generated code across all major measures (e.g., du-
Insight 4: Layered vs. Compressed Complexity. AI-generated plicated lines: 25.89% vs. 18.69%, duplicated intervals: 39.14% vs.
code tends to build complexity through deeper structural nest- 29.23%). This difference is primarily driven by cross-file duplica-
ing (especially loops), whereas human-written code concentrates tion, where human-written code shows significantly higher reuse
complexity within conditional logic and boolean expressions. (24.52% vs. 17.20%). In contrast, within-file duplication remains low
and similar across both groups.
Halstead Complexity. Table 6 reveals an interesting distributional Insight 8: Cross-File Reuse vs. One-Off Generation. Human-
pattern. AI-generated code has higher median vocabulary and written code contains substantially more cross-file duplication,
length, indicating that typical AI functions are larger and more reflecting systematic reuse (e.g., shared utilities and templates),
diverse. In contrast, human-written code shows significantly higher whereas AI-generated code is more often produced independently
variance, with much larger maximum values. for each task, resulting in lower cross-file duplication.
Insight 5: Heavy-Tailed Human Complexity. AI-generated Interestingly, AI-generated code exhibits a slightly higher num-
code is consistently moderate in size and complexity, whereas ber of clone instances per file (0.679 vs. 0.534), suggesting a different
human-written code exhibits a heavy-tailed distribution with a duplication pattern.
small number of extremely large and complex functions.
Insight 9: Fragmented vs. Structured Duplication. AI-
generated code tends to produce more localized and fragmented
File-Level Structure. Table 7 shows that human-written code con- duplication patterns, while human-written code exhibits fewer
tains substantially more functions per file (11.67 vs. 6.62), and these but larger and more structured duplicated regions.
functions are also larger. In contrast, AI-generated code has slightly
A Large-Scale Empirical Study of AI-Generated Code in Real-World Repositories Conference’17, July 2017, Washington, DC, USA

Table 8: Code Duplication Results Table 9: OOP Metrics per Class

Items AI codes Human codes Items WMC DIT NOC CBO RFC LCOM
Duplicated lines ratio 18.69% 25.89% AI code C++ 15.79 0.209 0.226 1.22 9.76 41.34
Duplicated tokens ratio 19.71% 26.76% Human code C++ 31.52 0.507 0.263 4.60 12.98 40.54
Duplicated interval ratio 29.23% 39.14% AI code Java 15.99 1.29 0.013 4.29 14.51 159.16
Clone instances(per file) 13455(0.679) 19477(0.534) Human code Java 15.14 1.30 0.023 5.00 14.14 155.58
Within file duplicate lines ratio 1.86% 1.72% AI code Python 10.06 1.27 0.339 1.94 8.95 N/A
Cross file duplicate lines ratio 17.20% 24.52% Human code Python 10.71 1.14 0.320 2.10 7.62 N/A
Within file duplicate interval ratio 3.57% 3.23% WMC: weighted methods per class; DIT: depth of inheritance tree;
Cross file duplicate interval ratio 26.20% 36.54% NOC: number of children; CBO: coupling between objects;
RFC: response for class; LCOM: lack of cohesion.

Formatting and Structural Style. Beyond duplication, formatting-

related metrics reveal additional stylistic differences. As shown in
Table 2, AI-generated code has a higher blank-line ratio (15.15% vs.
13.46%), while comment ratios are nearly identical. groups, with only a slight reduction in CBO for AI-generated code,
Insight 10: Spacing-Oriented Formatting. AI-generated code suggesting that AI-generated Java code largely follows established
uses slightly more blank lines, suggesting a tendency toward more object-oriented design conventions. Python falls between these
spaced or segmented formatting, while human-written code is two cases: AI-generated classes exhibit slightly lower internal com-
comparatively denser in layout. plexity (WMC, CBO) but slightly higher RFC, indicating somewhat
simpler internal structures while preserving, or slightly expanding,
The similar comment ratios (18.01% vs. 17.96%) indicate that externally reachable behavior.
AI-generated code does not rely more heavily on comments despite
Insight 13: Object-Oriented Differences Are Language-
being larger in size.
Dependent. The gap between AI-generated and human-written
Insight 11: Verbosity Without Additional Documentation. object-oriented design is not uniform across languages. It is most
The increased size of AI-generated code is not driven by addi- pronounced in C++, minimal in Java, and moderate in Python,
tional comments, but by expanded code structures, reinforcing the suggesting that AI aligns more closely with mature and standard-
observation that AI favors explicit implementation over compact ized design conventions, but diverges more in languages where
abstraction. deeper structural design and coupling patterns are common.

4.4 Object-Oriented Design Metrics

We analyze object-oriented design characteristics using standard 4.5 Graph-Level Structural Analysis
CK-style metrics as listed in Table 9. Results are reported per class We analyze program structure using graph-based representations,
across C++, Java, and Python. including control-flow graphs (CFG), data-flow graphs (DFG), and
call graphs. These graphs capture how execution, data dependen-
Language-dependent differences. A key observation is that differ-
cies, and inter-procedural interactions are organized beyond surface-
ences between AI-generated and human-written code are strongly
level syntax.
language-dependent. AI-generated object-oriented design is not
uniformly simpler or more complex, but instead reflects different
trade-offs across languages. The gap is largest in C++, minimal Overall observations. Across CFG and DFG metrics (Table 10 and
in Java, and moderate in Python, highlighting the importance of Table 11), AI-generated code consistently exhibits slightly larger
language-specific context when evaluating AI-generated code. graphs (more nodes and edges) but lower density and centralization.
These differences are statistically significant but small in magnitude.
Insight 12: Language Sensitivity of OOP Differences. Dif-
ferences in object-oriented design between AI-generated and Insight 14: Expanded but Less Centralized Graphs. AI-
human-written code vary significantly across languages, indicat- generated code tends to produce larger but less densely connected
ing that AI behavior is shaped by language-specific paradigms control-flow and data-flow graphs, suggesting a more distributed
rather than a single consistent pattern. and stepwise representation of logic rather than tightly optimized
structures.
Cross-language differences in object-oriented design. The object- This pattern aligns with earlier findings: AI-generated code ex-
oriented design differences between AI-generated and human-written pands logic into more explicit intermediate steps, resulting in ad-
code vary substantially across languages. In C++, human-written ditional nodes and edges while reducing structural concentration
classes are much more complex and interconnected: WMC and CBO (e.g., lower PageRank and density).
are both more than twice as large, and DIT is also notably higher,
indicating deeper inheritance hierarchies and stronger inter-class Control-flow vs. data-flow consistency. The same structural trend
dependencies. By contrast, Java shows near parity across almost all appears in both CFG and DFG: AI-generated code has slightly higher
metrics. WMC, DIT, and RFC are nearly identical between the two node/edge counts and diameter, but lower density and centrality.
Conference’17, July 2017, Washington, DC, USA Tianhao Mao, Dongfang Zhao, Haixu Tang, Xiaofeng Wang, and Hang Zhang

Table 10: Control Flow Graph per Function Table 12: Call Graph per File
Items AI mean±STD Human mean±STD p-value cohen-d Items AI mean±STD Human mean±STD p-value cohen-d
Node count 27.76±99.91 26.23±131.51 0.000 0.013 Node count 14.22±25.92 13.52±49.56 0.117 0.017
Edge count 27.95±102.45 26.38±133.77 0.000 0.013 Edge count 16.13±53.52 15.76±158.16 0.777 0.003
Density 0.205±0.186 0.213±0.186 0.000 -0.041 Density 0.206±0.170 0.223±0.175 0.000 -0.095
Max Degree 2.34±1.97 2.29±1.80 0.000 0.030 Max Degree 4.28±5.10 3.90±5.61 0.000 0.070
Largest SCC ratio(%) 25.46±20.47 25.36±20.06 0.009 0.005 Largest SCC ratio(%) 21.70±17.42 23.43±17.85 0.000 -0.098
Self Loop count 0.024±0.267 0.019±0.217 0.000 0.023 Self Loop count 0.198±0.844 0.201±1.30 0.812 -0.003
Simple Cycle count 1.68±122.43 2.55±245.85 0.017 -0.004 Simple Cycle count 3.41±216.60 0.710±54.44 0.207 0.019
Betweenness max 0.225±0.145 0.222±0.145 0.000 0.023 Call Graph depth 2.03±1.35 1.89±1.30 0.000 0.108
Pagerank max 0.285±0.237 0.295±0.237 0.000 -0.042 Betweenness max 0.054±0.115 0.050±0.112 0.003 0.036
Diameter in LWCC 16.97±42.91 15.78±42.53 0.000 0.028 Pagerank max 0.333±0.215 0.355±0.218 0.000 -0.102
SCC: Strongly connected component; LWCC: largest weakly connected component Diameter in LWCC 3.06±2.23 2.80±2.04 0.000 0.123
SCC and LWCC same with Table 10

Table 11: Data Flow Graph per Function Table 13: CodeQL Alerts per k-Lines
Items AI mean±STD Human mean±STD p-value cohen-d Items AI codes Human codes
Node count 20.82±47.20 19.39±45.86 0.000 0.031 Total Alerts 12.81 11.58
Edge count 42.82±129.41 39.73±132.80 0.000 0.024
Errors 0.840 0.846
Density 0.203±0.108 0.210±0.107 0.000 -0.064
Max Degree 14.84±36.55 13.85±36.26 0.000 0.027 Warnings 2.41 2.23
Largest SCC ratio(%) 14.88±10.02 15.39±10.04 0.000 -0.051 Recommendations 8.29 7.80
Self Loop count 0±0 0±0 / / Critical Security Alerts 0.042 0.052
Simple Cycle count 12.30±678.04 11.67±654.53 0.641 0.001 High Security Alerts 0.934 0.464
Betweenness max 0.139±0.146 0.142±0.149 0.000 -0.020 Medium Security Alerts 0.282 0.182
Pagerank max 0.349±0.101 0.355±0.100 0.000 -0.062
Diameter in LWCC 2.84±0.978 2.79±0.956 0.000 0.057
SCC and LWCC same with Table 10
4.6 Code Security and Defects
We analyze security and defect characteristics using static analysis
(CodeQL) and manual inspection of sensitive information leakage.
Insight 15: Consistent Expansion Across Abstraction Lev- Metrics are normalized per k-lines to ensure fair comparison.
els. The structural differences between AI and human code are
Overall security and defect density. As shown in Table 13, AI-
consistent across both control-flow and data-flow representa-
generated code triggers slightly more total alerts than human-
tions, indicating that AI’s tendency toward explicit expansion is
written code (12.81 vs. 11.58 per KLOC),which mainly driven by
a general property rather than a metric-specific artifact.
warnings and recommendations, whereas error rates are nearly
identical between the two groups. From the risk perspective, AI-
Call graph structure. At the inter-procedural level (Table 12), the generated code contains fewer critical alerts, but noticeably more
overall size of call graphs (nodes and edges) is comparable between high-risk (0.934 vs. 0.464) and medium-risk alerts (0.282 vs. 0.182).
AI and human code. However, AI-generated code shows lower Insight 18: Higher Density of Security-Relevant Issues.
density but greater depth and diameter. AI-generated code does not show more critical or correctness-
Insight 16: Deeper but Not Broader Call Structures. AI- related failures overall, but it contains substantially more high-
generated code does not introduce more functions or calls overall, and medium-risk alerts. This suggests a denser concentration of
but organizes them into deeper and more hierarchical call struc- security-relevant issues, even if the most severe category does
tures, suggesting a tendency toward layered decomposition rather not increase.
than flat interaction patterns.
Vulnerability composition. Table 14 shows that both groups share
In contrast, human-written code shows slightly higher density the same dominant CWE categories (CWE-563, CWE-561, CWE-
and centrality, indicating more concentrated call interactions. 396), suggesting similar primary sources of issues. Differences
emerge in secondary categories: AI-generated code shows relatively
Interpretation. Taken together, these results reveal a consistent
more input validation (CWE-20), resource consumption (CWE-400),
structural pattern: AI-generated code is not fundamentally more
and logging-related issues (CWE-117), while human-written code
complex in terms of connectivity or centrality, but is more expanded
shows more concurrency, resource lifecycle, and logical condition
and layered. Human-written code, in contrast, tends to concentrate
issues (e.g., CWE-208, CWE-772, CWE-570).
logic into fewer, more interconnected structures.
Insight 19: Shifted Vulnerability Composition. AI-generated
Insight 17: Layered vs. Concentrated Structure. AI-generated
and human-written code share similar dominant defect types, but
code builds program structure through layered and distributed
differ in secondary distributions, suggesting that AI shifts the
graph organization, while human-written code tends to concen-
composition of common issues rather than introducing entirely
trate complexity into more tightly connected regions.
new vulnerability patterns.
A Large-Scale Empirical Study of AI-Generated Code in Real-World Repositories Conference’17, July 2017, Washington, DC, USA

Table 14: CodeQL high frequency CWEs Table 16: Private Message in codes per 10k lines
Rank AI code CWE ratio(count) Human code CWE ratio(count) Items AI code count Human code count
1 CWE-563 32.3%(8379) CWE-563 34.4%(9976) Hardcode Password 0.703 1.02
2 CWE-561 9.08%(2358) CWE-561 13.4%(3904) Simple Password 0.512 0.811
3 CWE-396 6.28%(1630) CWE-396 6.42%(1863) API key or token 0.194 0.098
4 CWE-20 2.88%(747) CWE-208 3.63%(1053) Private key 0.021 0.006
5 CWE-390 2.86%(742) CWE-772 3.33%(968)
Simple password: Pure number or letters, short password,
6 CWE-400 2.63%(683) CWE-390 3.00%(870)
contains ’password’ itself or varieties
7 CWE-772 2.38%(619) CWE-248 2.26%(656)
8 CWE-208 2.36%(614) CWE-570 2.16%(626)
9 CWE-117 2.34%(608) CWE-571 2.16%(626)
10 CWE-248 2.19%(568) CWE-20 2.06%(597) Table 17: Results of Github Commit analysis
Items AI values Human values p-value q-value 1/2
Table 15: CodeQL Alerts Language Distribution per k-lines
Non-work Hours 0.372 0.358 0.001 0.000 / 0.000
Weekend Commit 0.232 0.237 0.190 0.753 / 1.000
Language AI Alerts Human Alerts AI high risk Human high risk
Merge Commit 0.126 0.000 0.000 0.000 / 0.000
C 4.95 4.28 0.904 0.420 Release in 30 days 0.622 0.694 0.000 0.000 / 0.000
C++ 5.64 3.88 0.053 0.049 Files changed in commit 92.85±2014.33 344.64±3104.45 0.000 0.000 / 0.000
C# 13.25 12.20 0 0 Lines churned in commit 24.70±147.40 152.02±1833.92 0.000 0.000 / 0.000
Files touched 7 days 0.004 0.002 0.001 0.243 / 1.000
Java 15.12 16.32 0.357 0.238
Files touched 30 days 0.013 0.003 0.000 0.000 / 0.000
Javascript 13.98 6.25 4.02 1.14 Files touched 90 days 0.020 0.003 0.000 0.000 / 0.000
Python 18.63 21.71 0.551 0.453 Touch times 7 days 0.006±0.116 0.002±0.058 0.001 0.040 / 0.450
Ruby 0.241 1.29 0.241 1.29 Touch times 30 days 0.031±0.359 0.003±0.069 0.000 0.000 / 0.000
Typescript 4.92 5.68 0.078 0.794 Touch times 90 days 0.050±0.541 0.004±0.075 0.000 0.000 / 0.000
Authors touched 7 days 0.004±0.065 0.002±0.048 0.001 0.203 / 1.000
Authors touched 30 days 0.016±0.156 0.003±0.054 0.000 0.000 / 0.000
Authors touched 90 days 0.024±0.194 0.003±0.057 0.000 0.000 / 0.000
Revert in 90 days 0.000 0.000 0.040 0.286 / 0.588
Line churned 7 days 0.293±14.79 0.222±8.55 0.001 0.272 / 0.353
Language-dependent security patterns. Table 15 reveals strong Line churned 30 days 1.69±43.69 0.239±8.62 0.000 0.000 / 0.000
language-specific effects. AI-generated code shows higher alert Line churned 90 days 3.19±59.08 0.275±9.73 0.000 0.000 / 0.000
Fixing commit in 7 days 0.001±0.033 0.000±0.025 0.737 0.338 / 0.450
density in C, C++, C#, and especially JavaScript (13.98 vs. 6.25),
Fixing commit in 14 days 0.003±0.064 0.001±0.025 0.000 0.000 / 0.001
while human-written code is higher in Java, Python, Ruby, and First edit days 26.16±22.41 11.10±19.96 0.000 0.441 / 0.085
TypeScript. The difference is particularly pronounced for high-risk Stabilization days 0.658±5.55 0.035±1.27 0.000 0.000 / 0.000
Time to revert days 29.28±18.38 32.00±27.68 0.819 1.000 / 1.000
alerts: JavaScript shows a large increase in AI-generated code (4.02 Handoff latency days 28.28±23.27 11.23±21.64 0.000 1.000 / 0.333
vs. 1.14), whereas TypeScript shows the opposite trend (0.078 vs.
0.794).
Insight 20: Language-Dependent Security Impact. The se-
curity impact of AI-generated code varies substantially across between each AI-associated sample and its matched human-control
languages. In particular, JavaScript shows a clear increase in high- commit; q-value 1 and q-value 2 denote the first and second matched
risk vulnerabilities, while TypeScript exhibits the opposite pattern, controls, respectively. Throughout this paper, values below 0.05 are
highlighting ecosystem-specific effects. considered statistically significant. Therefore, significance in both 𝑝
and 𝑞 indicates a robust difference; significance only in 𝑝 indicates
Sensitive information leakage. Table 16 shows comparable over- a group-level difference that weakens after matching; significance
all leakage levels but different distributions. Human-written code only in 𝑞 indicates a difference that is clearer after matching; and
contains more password-related leaks, while AI-generated code significance in neither indicates no reliable difference. Results are
shows higher rates of API keys, tokens, and private keys. organized by the research questions introduced in §3.3.
Insight 21: Different Secret Leakage Patterns. Human-
written code is more prone to password-like leakage, whereas RQ1: Temporal patterns. Commits contains AI-associated con-
AI-generated code more frequently exposes token- and key-based tents are slightly more likely to occur during non-working hours
credentials, suggesting different leakage mechanisms and mitiga- (0.372 vs. 0.358), while weekend activity shows no consistent dif-
tion needs. ference under paired comparisons. In addition, commits with AI-
associated codes are less likely to occur within 30 days before a
release (0.622 vs. 0.694).
5 Commit-Level Measurement Results
Insight 22: Flexible but Risk-Aware Usage. Commits with
Table 17 summarizes commit-level comparisons between AI-associated AI-assisted codes are more likely to occur outside regular working
and human-written code, and we report both pooled group-level hours, but are less frequent near release deadlines, suggesting that
and matched pair-wise results. The 𝑝-value compares all AI-associated developers use AI flexibly while avoiding it in stability-critical
samples with all human-control samples as two overall groups, stages.
while the 𝑞-value is derived from matched pair-wise comparisons
Conference’17, July 2017, Washington, DC, USA Tianhao Mao, Dongfang Zhao, Haixu Tang, Xiaofeng Wang, and Hang Zhang

RQ2: Change characteristics. AI-associated commits are substan- • Statistical conclusion validity is influenced by the large sample
tially smaller in scope. They involve fewer files and significantly size, which can make small effects statistically significant. We inter-
lower line churn per commit compared to human-written code. pret results using both statistical significance and effect size, and
Insight 23: Localized and Fine-Grained Edits. AI-assisted complement group-level analysis with paired comparisons where
coding is primarily used for small, localized modifications rather applicable.
than large-scale, multi-file changes, indicating a role as a fine- • External validity is limited to public GitHub repositories with
grained development aid. explicit AI-related signals, and is dominated by certain tools and
languages (e.g., ChatGPT, Python, JavaScript). While this may limit
generalization, it reflects real-world open-source usage and pro-
RQ3 & RQ5: Post-commit evolution and maintenance. Differences vides an in-vivo perspective on AI-assisted development.
are limited in the short term (7 days), but become pronounced over Scope and Interpretation of the Results. Unlike prior work that
longer horizons. AI-generated code is more likely to be revisited evaluates LLM-generated code under controlled prompts, we study
within 30 and 90 days, with higher touch frequency, more participat- the observable outcome of real-world AI-assisted programming: code
ing authors, and greater accumulated churn. It also shows slightly that has been selected, edited, and integrated into real repositories.
higher likelihood of follow-up fixing commits and significantly Our goal is therefore not to characterize all possible model out-
longer stabilization time. However, most median values remain puts—which are inherently prompt-dependent—but to understand
zero, indicating that these effects are driven by a subset of commits the typical properties of AI-involved code in practice.
with repeated follow-up activity. Under this scope, a consistent pattern emerges: AI-involved code
Insight 24: Delayed but Concentrated Maintenance. AI- tends to be more explicit, more verbose, and more layered. It is gen-
generated code does not exhibit substantially higher immediate erally larger, uses more structural constructs, and exhibits deeper
instability, but it incurs more follow-up modifications and longer nesting, while AI-associated commits are smaller, more localized,
stabilization over time, suggesting increased downstream adjust- and more likely to incur follow-up modification over time. These
ment costs concentrated in a subset of cases. findings reflect the combined effect of model behavior and human
integration, rather than raw model outputs alone.
Implications for Developers and AI Vendors. Our results sug-
RQ4: Collaboration and workflow. AI-associated commits involve gest that AI assistance reshapes development effort rather than sim-
longer handoff latency (28.28 vs. 11.23 days) and more follow-up ply reducing it. While AI-generated code is effective for localized
contributors over time, indicating a different collaboration pattern. edits and rapid drafting, it is associated with increased follow-up
At the same time, merge commits appear more frequently in AI- modification and longer stabilization over time. Developers should
associated changes. therefore anticipate a shift of effort from initial coding to downstream
Insight 25: Slower Handoffs and Distributed Follow-Up. AI- validation and maintenance, and allocate review and testing effort
assisted code tends to remain longer with the original author accordingly beyond the immediate post-commit window.
before being handed off, but eventually involves more distributed More importantly, our findings indicate that AI-generated code
follow-up activity, suggesting a shift toward delayed but broader tends to be more explicit, less compact, and more layered. This sug-
collaboration. gests that the primary risk is not correctness failure, but structural
inefficiency—including unnecessary verbosity, reduced reuse, and
fragmented logic. Developers may benefit from treating AI outputs
6 Discussion as drafts to be consolidated, focusing on abstraction, deduplication,
and structural simplification after generation.
Threats to Validity. We discuss some potential threats to validity
For AI tool vendors, these results highlight a gap between gener-
of our work as follows:
ation quality and long-term code quality. Current systems optimize
• Construct validity is challenged by the difficulty of identifying
for immediate correctness and completion, but less for maintain-
AI-generated code in the wild, as AI involvement is rarely explicitly
ability. Our findings suggest that future tools should better support
labeled and there is no large-scale real-world dataset. We miti-
post-generation refinement, such as encouraging reuse across files,
gate this using a multi-stage filtering pipeline as described in §3.1,
reducing unnecessary expansion, and warning about overly deep or
though false positive and negatives can still occur regardlessly, our
fragmented structures. In addition, the strong language-dependent
filtering achieves high accuracy in sampled verification and ensures
differences observed in our study indicate that AI assistants should
high-fidelity statistical study.
adopt more language-aware optimization strategies rather than
• Internal validity may be affected by repository-level confounders.
relying on uniform generation behavior.
We reduce this risk by aligning AI and human samples (§3.1) within
the same repository, language, and similar code size, though residual
differences may remain. Conclusion
• Measurement validity depends on the accuracy and coverage of In this paper, we presented a large-scale empirical study of AI-
analysis tools. While we employ well-established tools and normal- generated code in real-world repositories. Unlike prior work based
ized metrics, tool limitations and language-specific behaviors may on controlled prompts, we analyze code that has been selected,
introduce noise. Therefore, we emphasize consistent cross-metric edited, and integrated into real projects, and compare it with matched
trends rather than conclusions from any single metric. human-written code. Our results show that AI-generated code is
A Large-Scale Empirical Study of AI-Generated Code in Real-World Repositories Conference’17, July 2017, Washington, DC, USA

generally more verbose, more explicit, and more structurally lay- In 32nd Annual Network and Distributed System Security Symposium, NDSS
ered, while human-written code is more compact and relies more 2025, San Diego, California, USA, February 24-28, 2025. The Internet Soci-
ety. [Link]
on reuse. At the process level, AI-associated commits are smaller comparative-evaluation-of-large-language-models-in-vulnerability-detection/
and more localized, yet incur more follow-up modification and [20] H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random
Variables is Stochastically Larger than the Other. The Annals of Mathematical
longer-term maintenance. These findings suggest that AI assistance Statistics 18, 1 (1947), 50 – 60. doi:10.1214/aoms/1177730491
does not simply change who writes code, but reshapes how code [21] Quinn McNemar. 1947. Note on the Sampling Error of the Difference Between
is expressed and evolved. It acts as a local productivity aid while Correlated Proportions or Percentages. Psychometrika 12, 2 (1947), 153–157.
doi:10.1007/BF02295996
shifting effort toward downstream refinement. By releasing our [22] Ahmad Mohsin, Helge Janicke, Adrian Wood, Iqbal H. Sarker, Leandros Maglaras,
measurement pipeline, we provide a foundation for future studies to and Naeem Janjua. 2024. Can We Trust Large Language Models Generated Code?
track the evolving impact of AI on software engineering in practice. A Framework for In-Context Learning, Security Patterns, and Code Evaluations
Across Diverse LLMs. arXiv:2406.12513 [[Link]] [Link]
[23] Alfred Santa Molison, Marcia Moraes, Glaucia Melo, Fabio Santos, and Wesley
K. G. Assunção. 2025. Is LLM-Generated Code More Maintainable & Reliable
References Than Human-Written Code?. In 2025 ACM/IEEE International Symposium on
[1] Maurício Aniche. 2026. CK. [Link] Accessed: Empirical Software Engineering and Measurement (ESEM). 151–162. doi:10.1109/
2026-03-26. ESEM64174.2025.00036
[2] Tamas Bisztray, Bilel Cherif, Richard A. Dubniczky, Nils Gruschka, Bertalan [24] Daniil Orel, Dilshod Azizov, and Preslav Nakov. 2025. CoDet-M4: Detecting
Borsos, Mohamed Amine Ferrag, Attila Kovacs, Vasileios Mavroeidis, and Nor- Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain
bert Tihanyi. 2026. I Know Which LLM Wrote Your Code Last Summer: LLM Settings. In Findings of the Association for Computational Linguistics: ACL 2025,
generated Code Stylometry for Authorship Attribution. In Proceedings of the 18th Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pile-
ACM Workshop on Artificial Intelligence and Security (AISec ’25). Association for hvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 10570–
Computing Machinery, New York, NY, USA, 28–39. doi:10.1145/3733799.3762964 10593. doi:10.18653/v1/[Link]-acl.550
[3] Gavin S. Black, Bhaskar P. Rimal, and Varghese Mathew Vaidyan. 2025. Balancing [25] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and
Security and Correctness in Code Generation: An Empirical Study on Commercial Ramesh Karri. 2022. Asleep at the Keyboard? Assessing the Security of GitHub
Large Language Models. IEEE Transactions on Emerging Topics in Computational Copilot’s Code Contributions. In 2022 IEEE Symposium on Security and Privacy
Intelligence 9, 1 (2025), 419–430. doi:10.1109/TETCI.2024.3446695 (SP). IEEE Computer Society, 754–768.
[4] Hongbo Chen, Yifan Zhang, Xing Han, Tianhao Mao, Huanyao Rong, Yuheng [26] Karl Pearson. 1900. X. On the criterion that a given system of deviations from
Zhang, XiaoFeng Wang, Luyi Xing, Xun Chen, and Hang Zhang. 2025. Line- the probable in the case of a correlated system of variables is such that it can
Breaker: Finding Token-Inconsistency Bugs with Large Language Models. In be reasonably supposed to have arisen from random sampling. The London,
2025 40th IEEE/ACM International Conference on Automated Software Engineering Edinburgh, and Dublin Philosophical Magazine and Journal of Science 50, 302
(ASE). 893–905. doi:10.1109/ASE63991.2025.00079 (1900), 157–175. doi:10.1080/14786440009463897
[5] Pygments contributors. 2026. Pygments. [Link] Accessed: 2026- [27] Musfiqur Rahman, SayedHassan Khatoonabadi, Ahmad Abdellatif, and Emad
03-26. Shihab. 2025. Automatic Detection of LLM-Generated Code: A Comparative
[6] Albert Danial. 2026. cloc: v2.08. doi:10.5281/zenodo.5760077 Case Study of Contemporary Models Across Function and Class Granularities.
[7] Simone Daniotti, Johannes Wachs, Xiangnan Feng, and Frank Neffke. 2026. Who arXiv:2409.01382 [[Link]] [Link]
is using AI to code? Global diffusion and impact of generative AI. Science 391, [28] Romain Robbes, Théo Matricon, Thomas Degueule, Andre Hora, and Stefano
6787 (2026), 831–835. doi:10.1126/science.adz9311 Zacchiroli. 2026. Agentic Much? Adoption of Coding Agents on GitHub.
[8] Yujia Fu, Peng Liang, Amjed Tahir, Zengyang Li, Mojtaba Shahin, Jiaxin Yu, and arXiv:2601.18341 [[Link]] [Link]
Jinfu Chen. 2025. Security Weaknesses of Copilot-Generated Code in GitHub [29] Amirali Sajadi, Kostadin Damevski, and Preetha Chatterjee. 2025. How Safe
Projects: An Empirical Study. ACM Trans. Softw. Eng. Methodol. 34, 8, Article 218 Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and
(Oct. 2025), 34 pages. doi:10.1145/3716848 Agentic Automated Program Repair on SWE-bench. arXiv:2507.02976 [[Link]]
[9] GitHub. 2025. CodeQL. [Link] [Link]
[10] Jiaxun Guo, Ziyuan Yang, Mengyu Sun, Hui Wang, Jingfeng Lu, and Yi Zhang. [30] Andreas Schaad, Stefan Götz, and Dominik Binder. 2025. You Still have to Study
2026. Code Fingerprints: Disentangled Attribution of LLM-Generated Code. On the Security of LLM Generated Code. In ICT Systems Security and Privacy
arXiv:2603.04212 [[Link]] [Link] Protection, Lili Nemec Zlatolas, Kai Rannenberg, Tatjana Welzer, and Joaquin
[11] Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. 2008. Exploring Network Garcia-Alfaro (Eds.). Springer Nature Switzerland, Cham, 111–124.
Structure, Dynamics, and Function using NetworkX. In Proceedings of the 7th [31] Maximilian Schreiber and Pascal Tippe. 2025. Security Vulnerabilities in AI-
Python in Science Conference. 11–15. doi:10.25080/TCWV9851 Generated Code: A Large-Scale Analysis of Public GitHub Repositories. Springer
[12] S M Mahedy Hasan, Md Fazle Rabbi, and Minhaz Zibran. 2026. The Quiet Contri- Nature Singapore, 153–172. doi:10.1007/978-981-95-3537-8_9
butions: Insights into AI-Generated Silent Pull Requests. arXiv:2601.21102 [[Link]] [32] SciTools. 2026. Understand. [Link] Accessed: 2026-03-26.
[Link] Mining Challenge track of the 23rd International [33] Mohammed Latif Siddiq, Joanna Cecilia da Silva Santos, Sajith Devareddy, and
Conference on Mining Software Repositories (MSR 2026). Anna Muller. 2024. SALLM: Security Assessment of Generated Code. In Pro-
[13] Jingxuan He and Martin Vechev. 2023. Large Language Models for Code: Security ceedings of the 39th IEEE/ACM International Conference on Automated Software
Hardening and Adversarial Testing. In Proceedings of the 2023 ACM SIGSAC Engineering Workshops (ASEW ’24). ACM, 54–65. doi:10.1145/3691621.3694934
Conference on Computer and Communications Security (Copenhagen, Denmark) [34] Mohammed Latif Siddiq, Xinye Zhao, Vinicius Carvalho Lopes, Beatrice Casey,
(CCS ’23). Association for Computing Machinery, New York, NY, USA, 1865–1879. and Joanna C. S. Santos. 2026. Security in the Age of AI Teammates: An Empirical
doi:10.1145/3576915.3623175 Study of Agentic Pull Requests on GitHub. arXiv:2601.00477 [[Link]] https:
[14] [Link]. 2026. Joern: The Bug Hunter’s Workbench. [Link] //[Link]/abs/2601.00477
joern [35] Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam
[15] jscpd contributors. 2026. jscpd. [Link] Accessed: Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, and Toufique Ahmed. 2025.
2026-03-26. Calibration and Correctness of Language Models for Code. In Proceedings of the
[16] Raphaël Khoury, Anderson R. Avila, Jacob Brunelle, and Baba Mamadou Camara. IEEE/ACM 47th International Conference on Software Engineering (Ottawa, Ontario,
2023. How Secure is Code Generated by ChatGPT?. In 2023 IEEE International Canada) (ICSE ’25). IEEE Press, 540–552. doi:10.1109/ICSE55347.2025.00040
Conference on Systems, Man, and Cybernetics (SMC). 2445–2451. doi:10.1109/ [36] Hyunjae Suh, Mahan Tafreshipour, Jiawei Li, Adithya Bhattiprolu, and Iftekhar
SMC53992.2023.10394237 Ahmed. 2025. An Empirical Study on Automatically Detecting AI-Generated
[17] Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. 2025. The Rise of AI Teammates Source Code: How Far Are We?. In Proceedings of the IEEE/ACM 47th International
in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshap- Conference on Software Engineering (Ottawa, Ontario, Canada) (ICSE ’25). IEEE
ing Software Engineering. arXiv:2507.15003 [[Link]] [Link] Press, 859–871. doi:10.1109/ICSE55347.2025.00064
15003 [37] tree-sitter contributors. 2025. tree-sitter. [Link]
[18] Shuang Li, Yuntao Cheng, Jinfu Chen, Jifeng Xuan, Sen He, and Weiyi Shang. [38] Jiexin Wang, Xitong Luo, Liuwen Cao, Hongkui He, Hailin Huang, Jiayuan
2026. Performance analysis of AI-generated code: A case study of Copilot, Copilot Xie, Adam Jatowt, and Yi Cai. 2024. Is Your AI-Generated Code Really Safe?
Chat, CodeLlaMa, and DeepSeek-Coder models. Empirical Softw. Engg. 31, 3 (Jan. Evaluating Large Language Models on Secure Code Generation with CodeSecEval.
2026), 52 pages. doi:10.1007/s10664-025-10776-1 arXiv:2407.02395 [[Link]] [Link]
[19] Jie Lin and David Mohaisen. 2025. From Large to Mammoth: A Com- [39] Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods. Biometrics
parative Evaluation of Large Language Models in Vulnerability Detection. Bulletin 1, 6 (1945), 80–83. [Link]
Conference’17, July 2017, Washington, DC, USA Tianhao Mao, Dongfang Zhao, Haixu Tang, Xiaofeng Wang, and Hang Zhang

[40] Tao Xiao, Youmei Fan, Fabio Calefato, Christoph Treude, Raula Gaikovina Kula, [42] Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang,
Hideaki Hata, and Sebastian Baltes. 2026. Self-Admitted GenAI Usage in Open- Ying Li, Qianxiang Wang, and Tao Xie. 2024. CoderEval: A Benchmark of Prag-
Source Software. arXiv:2507.10422 [[Link]] [Link] matic Code Generation with Generative Pretrained Models . In 2024 IEEE/ACM
[41] Yuliang Xu, Siming Huang, Mingmeng Geng, Yao Wan, Xuanhua Shi, and Dong- 46th International Conference on Software Engineering (ICSE). IEEE Computer
ping Chen. 2026. code-transformed: The Influence of Large Language Models on Society, Los Alamitos, CA, USA, 428–439. doi:10.1145/3597503.3623322
Code. In Findings of the Association for Computational Linguistics: EACL 2026, Vera [43] Beiqi Zhang, Peng Liang, Qiong Feng, Yujia Fu, and Zengyang Li. 2024. Copilot-
Demberg, Kentaro Inui, and Lluís Marquez (Eds.). Association for Computational in-the-Loop: Fixing Code Smells in Copilot-Generated Python Code using Copilot.
Linguistics, Rabat, Morocco, 5462–5490. doi:10.18653/v1/[Link]-eacl.290 In Proceedings of the 39th IEEE/ACM International Conference on Automated Soft-
ware Engineering (Sacramento, CA, USA) (ASE ’24). Association for Computing
Machinery, New York, NY, USA, 2230–2234. doi:10.1145/3691620.3695290