EvoCodeBench: An Evolving Code Generation Benchmark Aligned with

Real-World Code Repositories

Jia Li ♂1 , Ge Li1 , Xuanming Zhang1 , Yihong Dong1 , Zhi Jin1

1
School of Computer Science, Peking University
lijia@[Link], lige@[Link]

def has_close_elements(numbers, threshold):

Abstract for idx, elem in enumerate(numbers):
for idx2, elem2 in enumerate(numbers):
How to evaluate Large Language Models if idx != idx2:
distance = abs(elem - elem2)
(LLMs) in code generation is an open question. if distance < threshold:
arXiv:2404.00599v1 [[Link]] 31 Mar 2024

Existing benchmarks demonstrate poor align- return True

ment with real-world code repositories and are return False

insufficient to evaluate the coding abilities of (a) A standalone function in HumanEval

LLMs. This paper proposes a new benchmark - # [Link]
EvoCodeBench to address the preceding prob- def namespace(self):
lems, which has three primary advances. ❶ data = self._command_and_check(“namespace”)
EvoCodeBench aligns with real-world reposi- parts = []
for item in parse_response(data):
tories in multiple dimensions, e.g., code dis- (more lines . . )
tributions and dependency distributions. ❷ for prefix, separator in item:
EvoCodeBench offers comprehensive annota- if self.folder_encode:
tions (e.g., requirements, reference code, and prefix = decode_utf7(prefix)
[Link]((prefix, to_unicode)
reference dependencies), and robust evalua- [Link](tuple(converted))
tion metrics (e.g., Pass@k and Recall@k). ❸ return Namespace(*parts)
EvoCodeBench is an evolving benchmark to
(b) A non-standalone function in a real-world project
avoid data leakage. We build an automatic
pipeline to update EvoCodeBench from the lat- Figure 1: Examples of standalone and non-standalone
est repositories. We release the first version functions. Dependencies are highlighted, i.e., yellow:
- EvoCodeBench-2403, containing 275 sam- intra-class dependencies, green: intra-file dependencies,
ples from 25 real-world repositories. Based and blue: cross-file dependencies.
on EvoCodeBench, we propose repository-
level code generation and evaluate 10 popular
LLMs (e.g., gpt-4, gpt-3.5, DeepSeek Coder, 2023). In practice, human developers typically
StarCoder 2, CodeLLaMa, Gemma, and Qwen write the code for code repositories. Thus, evalu-
1.5). Our experiments reveal the coding abili-
ating the coding abilities of LLMs in real-world
ties of these LLMs in real-world repositories.
For example, the highest Pass@1 of gpt-4 repositories is necessary. We analyze over 1 mil-
only is 20.73% in our experiments. We also lion functions from 500 real-world repositories (see
analyze failed cases and summarize the short- Section 3) and think a good benchmark should sat-
comings of existing LLMs in EvoCodeBench. isfy the following features.
We release EvoCodeBench, all prompts, and
LLMs’ completions for further community • Real-world Repository. The benchmark should
analysis1 . be collected from real-world code repositories
(Yu et al., 2023).
1 Introduction
• Real Code Distribution. Real-world reposito-
Code generation with Large Language Models ries comprise two types of code, i.e., standalone
(LLMs) has attracted lots of researchers’ attention and non-standalone code. As shown in Figure 1,
(Guo et al., 2024; Rozière et al., 2023; Lozhkov a standalone function solely uses built-in or pub-
et al., 2024), and some commercial products have lic libraries, while a non-standalone one contains
been produced, e.g., GitHub Copilot (GitHub, context-aware dependencies (i.e., invocations of
1
[Link] code elements defined in current repositories).
EvoCodeBench The benchmark should cover both types of code
 Table 1: The comparison between existing benchmarks and EvoCodeBench.

Benchmark Real Repo. Real Code Distribution Comprehensive Annota. Robust Metric Avoiding Data Leak.
CoNaLA (Yin et al., 2018) é é é é é
Concode (Iyer et al., 2018) Ë é é é é
HumanEval (Chen et al., 2021) é é é é é
MBPP (Austin et al., 2021) é é é é é
APPS (Hendrycks et al., 2021) é é é é é
PandasEval (Zan et al., 2022) é é é é é
NumpyEval (Zan et al., 2022) é é é é é
AixBench (Li et al., 2023b) Ë é é é é
ClassEval (Du et al., 2023) é é é é é
CoderEval (Yu et al., 2023) Ë é é Ë é
EvoCodeBench (Ours) Ë Ë Ë Ë Ë

and ensure their ratios are realistic. The number and report Pass@k. It also proposes Recall@k
of dependencies should also be consistent with to evaluate the dependencies in predictions. ❺
real-world repositories. EvoCodeBench is an evolving benchmark and
will be dynamically updated every period (e.g.,
• Comprehensive Annotations. The benchmark 6 months) to avoid data leakage. In this paper,
can offer comprehensive annotations, including we release the first version - EvoCodeBench-2403,
natural language requirements, original reposito- which consists of 275 samples from 25 real-world
ries, and ground truths (code and dependencies). repositories. Details of the collection process can
be found in Section 3.
• Robust Evaluation Metrics. The benchmark
should contain test cases to evaluate models’ pre- Based on EvoCodeBench, we propose
dictions and report Pass@k. Metrics are also repository-level code generation, which simu-
required to assess the abilities of LLMs to gener- lates the developers’ coding process in a working
ate dependencies. repository. The task asks models to write the code
based on requirements and a complete repository.
• Avoidng Data Leaking. With more LLMs We evaluate 10 popular LLMs (i.e., gpt-4 (Ope-
emerging, the benchmark should avoid potential nAI, 2023b), gpt-3.5 (OpenAI, 2023a), DeepSeek
data leakage (Huang et al., 2023). Coder (Guo et al., 2024), StarCoder 2 (Lozhkov
et al., 2024), CodeLLaMa (Rozière et al., 2023),
However, as shown in Table 1, none of the existing Gemma (GemmaTeam, 2024), and Qwen 1.5 (Bai
benchmarks satisfies all aforementioned features. et al., 2023)). These LLMs exhibit low perfor-
The problem hinders the evaluation and develop- mance on EvoCodeBench, especially compared to
ment of LLMs in the real development process. their performance on previous benchmarks. For
To address the above problem, we pro- example, gpt-4-turbo-1106 achieves a Pass@1
pose a new code generation benchmark named score of 80% on HumanEval, while its highest
EvoCodeBench, which aligns with real-world Pass@1 on EvoCodeBench is only 20.73%. Our
code repositories. As shown in Table 1, results reveal the coding abilities of these LLMs in
EvoCodeBench satisfies the above features. ❶ real-world repositories. We further analyze failed
EvoCodeBench is collected from high-quality cases and summarize the shortcomings of existing
open-source repositories in the real world. ❷ LLMs in EvoCodeBench.
EvoCodeBench is constructed through a rigorous In summary, our contributions are as follows:
pipeline and aligns with real-world repositories.
Specifically, the distributions of code and depen- • We summarize five features (see Table 1) that a
dencies in EvoCodeBench are consistent with the code generation benchmark for real-world repos-
ones in 500 real-world repositories. Detailed statis- itories should satisfy.
tics are in Section 2.4. ❸ EvoCodeBench of-
fers comprehensive annotations, e.g., detailed re- • We propose a new code generation benchmark
quirements, original repositories, reference code, - EvoCodeBench, satisfying the above features.
and reference dependencies. ❹ EvoCodeBench We released the first version and will continually
leverages test cases to check models’ predictions update it.
 EvoCodeBench
Stats: A evolutionary code generation benchmark
Evaluation Task: Repository-level code generation: ① ② ③ → ④
Evaluation Metrics: Pass@k (functional correctness, label: ⑥), Recall@k (recall of reference dependencies, label: ⑤)

① Signature ③ Repository
def create(self, model: str,
path: Optional[Union[str, PathLike]] = None, import httpx
modelfile: Optional[str] = None, import platform
stream: bool = False, ...
) -> Union[Mapping[str, Any],Iterator[Mapping[str, Any]]]: class BaseClient:
...
② Requirement class Client(BaseClient):
Initiate a request to create a model based on the provided ...
path. Handle the request either as a single response or as class AsyncClient(…):
a stream of responses, depending on the `stream` parameter. ...
def _encode_image(…):
:param self: Client. An instance of Client class. ...
:param model: str, The model to be created. def _as_path(…):
:param path: Optional[…], . . . ...
:param modelfile: Optional[str], . . . (more 700 lines . . .)
:param steam, bool, . . .
:return Union[…], . . .
Intra-class Dependency:
④ Reference Code ollama._client.Client._parse_modelfile
ollama._client.Client._request_stream
if (realpath := _as_path(path)) and [Link]():
modelfile = self._parse_modelfile(realpath.read_text())
elif modelfile: Intra-file Dependency:
modelfile = self._parse_modelfile(modelfile) ollama._client._as_path
else: ⑤ Reference
raise RequestError('must provide path or modelfile‘) Cross-file Dependency: Dependency
return self._request_stream(…) ollama._types.RequestError

def test_client_create_path_relative(httpserver: HTTPServer): . . .

def test_client_create_modelfile(httpserver: HTTPServer): . . . ⑥ Test cases
def test_client_create_from_library(httpserver: HTTPServer): . . .

Figure 2: An overview of EvoCodeBench. Each sample consists of six components.

• We propose repository-level code generation, code. ❸ Repository: The current repository con-
which provides a challenging and realistic evalu- tains hundreds of code files. ❹ Reference Code:
ation scenario. A developer-written implementation of the target
code. This code may invoke dependencies defined
• We evaluate 10 popular LLMs on in the current repository. ❺ Reference Depen-
EvoCodeBench, analyzing their strengths dency: The dependencies invoked in the reference
and shortcomings in repository-level code code include intra-class, intra-file, and cross-file
generation. dependencies. ❻ Test Cases: Test cases are used
We hope EvoCodeBench can align with the ac- to check the functional correctness of the code.
tual experiences of developers during the practical
development process. By EvoCodeBench, practi- 2.2 Task Definition
tioners can pick up superior LLMs and facilitate Traditional benchmarks fall into a sim-
the application of code generation techniques in ple requirement-to-code task. In contrast,
real-world repositories. EvoCodeBench proposes a more realistic task
- repository-level code generation. This task
2 EvoCodeBench simulates the developers’ coding process in a
In this section, we first show an overview of working repository. Given a requirement and a
EvoCodeBench and then describe its tasks and repository, LLMs are tasked to generate the code
metrics. Finally, we present the first version - for the repository.
EvoCodeBench-2403 and its statistics.
2.3 Evaluation Metrics
2.1 Overview Pass@k (Functional Correctness). Following pre-
Figure 2 shows a sample in EvoCodeBench. Each vious studies (Chen et al., 2021; Austin et al., 2021;
sample consists of six components. Yu et al., 2023), we assess the functional correct-
❶ Function Signature: The signature of the ness of programs by executing test cases and com-
target code. ❷ Requirement: An English de- pute the unbiased Pass@k. Specifically, we gen-
scription detailing the functionality of the target erate n ≥ k programs per requirement, count the
Table 2: The comparison between existing code generation benchmarks and EvoCodeBench-2403. SA and Depend
are the abbreviations of “standalone” and “dependency”, respectively.

Code Distribution Dependency Distribution

Benchmark Annotation Release Data
#Repo. #Sample SA Non-SA #Type #Avg.
CoNaLa – 500 100% 0% 0 0 NL, Code 2018-5
HumanEval – 164 100% 0% 0 0 NL, Code 2021-7
MBPP – 974 100% 0% 0 0 NL, Code 2021-8
APPS – 5,000 100% 0% 0 0 NL, Code 2021-11
PandasEval – 101 100% 0% 0 0 NL, Code 2022-6
NumpyEval – 101 100% 0% 0 0 NL, Code 2022-6
AixBench N/A 175 100% 0% 0 0 NL, Code 2023-2
ClassEval – 100 100% 0% 0 0 NL, Code, Depend. Name 2023-8
Concode N/A 2,000 20% 80% 1 1.23 NL, Code 2018-8
CoderEval 43 230 36% 64% 3 1.73 NL, Code, Depend. Name 2023-2
NL, Code, Depend. Path,
EvoCodeBench-2403 25 275 27% 73% 3 3.46 2024-3
Repository
500 Real Repositories 500 1M+ 27% 73% 3 3.22 – –

number of correct programs c ≤ n that pass test EvoCodeBench-2403 is consistent with that of 500
cases, and calculate the Pass@k: real-world repositories. The average number of
  dependencies per program in EvoCodeBench-2403
n−c

is also close to that of 500 real-world repositories.
 k  ❷ Comprehensive Annotations. We provide
Pass@k := 1 − 
E
Requirements  n
  (1) requirements, reference code, reference depen-
k dency and the complete repository for each sam-
ple. Previous works (i.e., CoderEval, ClassEval)
Recall@k (Recall of Reference Dependency). only provide dependencies’ names (e.g., close).
Besides the functional correctness, we expect Because many functions have the same name
LLMs to invoke relevant dependencies defined in in practice, it is hard to identify whether gen-
contexts. Hence, we propose Recall@k, which erated dependencies are correct by relying on
gauges the recall of reference dependencies in gen- names. EvoCodeBench annotates dependencies
erated programs. with paths (e.g., [Link]::ClassB::close), ad-
Specifically, LLMs generate k programs per re- dressing ambiguity and biases. These annotations
quirement. For the i-th program, we employ a offer a broad arena to explore repository-level code
parser2 to extract its dependencies as Pi . Subse- generation and evaluation.
quently, we compare Pi with reference dependen- ❸ Latest repositories to avoid data leak-
cies R and compute the Recall@k: age. Considering the latest LLM’s (Lozhkov
  et al., 2024) training data is up to 2023-9,
|R ∩ Pi |
Recall@k := E max (2) EvoCodeBench-2403 is collected from real-world
Requirements i∈[1,k] |R|
repositories that were created from 2023-10 to
where | · | means the number of elements of a set. 2024-2. In future versions, we will continually
update EvoCodeBench using the latest repositories.
2.4 EvoCodeBench-2403
3 Benchmark Collection Pipeline
This paper releases the first version of
EvoCodeBench named EvoCodeBench-2403. The We build an automatic pipeline for collecting
statistics of EvoCodeBench-2403 are shown in EvoCodeBench from the latest repositories. The
Table 2. We discuss its features as follows. pipeline consists of four stages as follows.
❶ Alignment with real-world code reposito- ❶ Repository Selection. We crawl high-quality
ries. EvoCodeBench-2403 consists of 275 sam- repositories from GitHub satisfying the following
ples collected from 25 real-world repositories. criteria: open-source Python projects; created in re-
As shown in Table 2, the code distribution of cent months; non-fork and non-malicious projects;
2
We develop the parser based on an open-source static more than 50 stars; and having explicit unit tests.
analysis tool - Pyan (Pyan, 2023). ❷ Function Parse. We extract functions from
repositories and exclude trivial functions (e.g., Table 3: Studied LLMs in this paper. Context L.: Con-
empty or initialization functions). We extract each text Window.
function’s signature and function body (i.e., refer- Type Name Version Context W.
ence code). We developed a static analysis-based gpt-4 gpt-4-turbo-1106 128,000
Closed-source
parser to extract reference dependencies with the gpt-3.5 gpt-3.5-turbo-1106 16,385
reference code. StarCoder 2 15B 16,384
StarCoder 2 7B 16,384
❸ Tests Construction. For each function, we ex- DeepSeek Coder 33B 16,384
tract test cases invoking it from current repositories. Open-source
DeepSeek Coder 6.7B 16,384
We use pip3 to automatically install required pack- CodeLLaMa 13B 16,384
CodeLLaMa 7B 16,384
ages for each repository and leverage Pytest4 to Gemma 7B 8,192
run test cases. Functions without executable test Qwen 1.5 7B 32,768

cases are excluded.

❹ Deduplication. To improve the diversity of
4.2 Experimental Setting
EvoCodeBench, we perform repository-level dedu-
plication based on the Jaccard similarities in code Repository-level code generation takes a require-
files and imports. The former removes duplicate ment and a repository as inputs. Typically, a repos-
repositories in text surfaces, and the latter removes itory consists of hundreds of code files and is very
repositories in domains that are too similar. long. For example, the average length of 500 real-
❺ Requirement Annotation. Manually writing world repositories is 1.1 million tokens, surpassing
requirements is time-consuming and laborious. In- the context windows of existing LLMs (e.g., gpt-4:
spired by the powerful abilities of LLMs in code 128k tokens). Inspired by related work (Shrivas-
comment generation (Geng et al., 2024), we lever- tava et al., 2023), we try to extract parts of code
age LLMs to generate natural language require- contexts from the repository as inputs and design
ments. Specifically, we manually craft a few-shot the following experimental settings.
prompt, which teaches LLMs to write requirements ❶ Without context. In this setting, we ignore
in a specific format (i.e., functional descriptions contexts and directly generate the code based on
and input-output parameters). Appendix A.1 the requirements and signatures.
details of prompts. ❷ Local File (Completion). The local file de-
❻ Benchmark Construction. Finally, we select notes the code file where the reference code is in.
samples from the outputs of step ❺ to construct This setting simulates the scenario where devel-
EvoCodeBench. We strive to make EvoCodeBench opers continue to write code at the end of a file.
satisfy the following goals: consistent with the Thus, we consider code snippets above the refer-
code distribution observed in 500 real-world reposi- ence code in the local file as contexts. Then, LLMs
tories; close to the average number of dependencies generate code in an autoregressive manner based
in 500 real-world repositories; including as many on requirements, signatures, and contexts.
samples as possible. ❸ Local File (Infilling). Different from the Lo-
cal File (Completion) setting, this setting simulates
4 Experiments the scenario where developers infill code in the
middle of a file. Thus, we use the code snippets
4.1 Studied LLMs
above and below the reference code in the local
Table 3 shows 10 studied LLMs in our experi- file as contexts. We evaluate LLMs that support
ments. They cover closed-source LLMs (i.e., gpt-4 code infilling and construct input sequences using
(OpenAI, 2023b), gpt-3.5 (OpenAI, 2023a)) and official formats.
open-source LLMs (i.e., DeepSeek Coder (Guo The prompt templates used in the above settings
et al., 2024), StarCoder 2 (Lozhkov et al., 2024), are shown in Appendix B.2. We note that there are
CodeLLaMa (Rozière et al., 2023), Gemma (Gem- other approaches to extracting relevant contexts.
maTeam, 2024), and Qwen 1.5 (Bai et al., 2023)). We consider this to be beyond the scope of this
We use official interfaces or implementations to paper and leave the exploration to future work.
reproduce these LLMs. The details of LLMs can
be found in Appendix B.1. 4.3 Evaluation
3
[Link] We use Pass@k and Recall@k (see Section 2.3)
4
[Link] to assess generated programs. In this paper, k ∈
Table 4: Pass@k and Recall@k of LLMs on EvoCodeBench-2403. Bold and underlined data indicate top-1 and
top-2 results, respectively.

LLMs Size Pass@1 Pass@3 Pass@5 Pass@10 Recall@1 Recall@3 Recall@5 Recall@10
Local File (Infilling)
gpt-4 N/A 20.73 23.03 24.11 25.34 68.24 70.63 72.05 73.52
gpt-3.5 N/A 17.82 21.78 23.06 24.46 61.94 68.13 69.69 70.85
DeepSeek Coder 33B 19.64 22.78 24.29 26.01 71.46 79.93 82.11 86.25
DeepSeek Coder 6.7B 17.82 21.02 22.40 23.97 69.58 74.04 78.00 83.22
StarCoder 2 15B 14.91 17.54 18.63 19.86 50.90 53.29 55.89 61.76
StarCoder 2 7B 15.27 17.29 18.63 20.09 56.35 60.59 63.74 74.20
Local File (Completion)
gpt-4 N/A 17.45 19.65 20.80 22.41 63.49 68.67 70.00 72.07
gpt-3.5 N/A 15.64 17.29 18.21 19.36 61.44 66.25 66.82 69.89
DeepSeek Coder 33B 14.18 17.57 18.66 19.95 66.90 72.83 74.40 80.02
DeepSeek Coder 6.7B 13.45 17.10 18.81 21.07 65.76 72.32 75.61 78.45
StarCoder 2 15B 13.45 15.44 17.84 19.59 68.55 71.37 74.76 77.70
StarCoder 2 7B 13.82 15.15 16.18 17.65 62.93 69.85 73.54 78.40
CodeLLaMa 13B 12.73 15.78 16.86 18.19 63.34 71.26 76.43 80.11
CodeLLaMa 7B 12.73 15.33 16.00 16.93 63.33 69.79 71.91 76.50
Gemma 7B 10.55 13.25 14.31 15.48 58.02 70.57 74.44 80.76
Qwen 1.5 7B 5.45 7.04 7.91 9.07 39.21 44.02 50.17 58.42
Without Contexts
gpt-4 N/A 7.27 10.05 10.70 11.49 21.58 23.93 25.69 26.23
gpt-3.5 N/A 6.55 7.85 8.28 8.73 21.66 24.31 24.77 25.40
DeepSeek Coder 33B 6.91 8.92 9.79 11.03 27.67 32.73 34.92 37.76
DeepSeek Coder 6.7B 5.82 8.56 9.67 11.26 25.89 32.06 35.59 38.33
StarCoder 2 15B 6.18 8.77 9.95 11.53 24.03 29.86 33.62 36.91
StarCoder 2 7B 5.82 6.72 7.43 8.62 27.39 32.60 34.88 36.81
CodeLLaMa 13B 5.45 7.38 8.37 9.95 25.52 31.28 33.66 36.36
CodeLLaMa 7B 5.45 6.94 7.75 9.03 26.97 31.17 34.08 36.82
Gemma 7B 6.18 6.86 7.64 8.66 21.84 29.98 33.61 35.23
Qwen 1.5 7B 4.00 4.72 5.38 6.18 16.33 13.56 16.34 21.06

[1, 3, 5, 10]. When k = 1, we use the greedy search HumanEval, while it scores 7.27 on Pass@1 upon
and generate a single program per requirement. EvoCodeBench-2403. On the one hand, the de-
When k > 1, we use the nucleus sampling with creases validate our motivation that existing bench-
a temperature 0.4 and sample 20 programs per re- marks can not comprehensively assess the coding
quirement. We set the top-p to 0.95 and the max abilities of LLMs in practical projects. On the other
generation length to 500. hand, the results emphasize the importance of con-
Because EvoCodeBench is an evolving texts in repositories. Without the necessary context,
benchmark, this paper evaluates LLMs upon LLMs lack the domain knowledge of current repos-
EvoCodeBench-2403. Note that the Pass@k itories and generate the wrong programs.
and Recall@k between different versions of Local File (Completion) and (Infilling). After
EvoCodeBench are not comparable. introducing the contexts within local files, the
Pass@k and Recall@k of all LLMs obviously in-
4.4 Main Results crease. For example, the Pass@1 of gpt-4 is im-
The Pass@k and Recall@k in three experimental proved by 104% and 152% in two settings, respec-
settings are shown in Table 7. tively. We attribute the improvements to the do-
Without Context. gpt-4 achieves the highest main knowledge contained in contexts. Figure 3
Pass@k among all LLMs. However, compared shows a uniquely successful case in the Local File
to previous benchmarks, these LLMs’ performance (Completion) setting. The key to writing this func-
in EvoCodeBench-2403 drops dramatically. For tion is to know cache directories. Without context,
example, gpt-4 achieves a Pass@1 score of 88.4 on gpt-4 fabricated a non-existent field as cache direc-
Contexts (above):
import concurrent
import json
. . .
def _get_num_nodes() -> int:
. . .
def _get_cache_dir(name: Optional[str] = None):
. . .
def _get_cache_data_dir(name: Optional[str] = None):
. . .

Input Code:
def _cleanup_cache(self) -> None:
""" Clean up cache directories by removing them if
they exist to prevent issues from corrupted files
from previous runs, and then recreates these Figure 4: Pass@1 of gpt-4 on different program types.
directories to ensure they are available for use.
:param self: DataProcessor. ...
:return: No return values.
"""
gpt-4-turbo-1106 (without context):
find that LLMs can generate some dependencies
for cache_dir in self.cache_directories:
without context. We manually inspect successful
# Check if the cache directory exists cases and summarize two reasons. First, LLMs
if [Link](cache_dir):
. . . can reason about some easy dependencies from re-
gpt-4-turbo-1106 (local completion): quirements, e.g., initialization functions of returned
# Define the cache directories objects. Second, LLMs can “guess” dependencies
cache_data_dir = _get_cache_data_dir()
cache_chunks_dir = _get_cache_dir() from their functionalities. In practice, dependen-
# Remove the cache data directory if it exists
cies’ names are relevant to their functional descrip-
if [Link](cache_data_dir): tions, e.g., send_request() means send a re-
. . .
quest to the server. LLMs are trained with a large
code corpus and can learn the naming conventions.
Figure 3: A uniquely successful case in Local File (Com- Thus, LLMs may successfully guess some depen-
pletion) setting.
dencies from their functionalities.
❷ More contexts benefit code generation.
Based on Table 7, we compare the performance
tories, generating the incorrect code. In fact, two
of an LLM (e.g., gpt-4) under different settings.
functions for returning the cache directories are
Obviously, the more input contexts, the better the
available in the local file. After introducing the lo-
performance of the LLM. It inspires practitioners
cal file, gpt-4 successfully gets cache directories by
to extend the context windows of LLMs and input
invoking these functions and generates the correct
more contexts.
code.
❸ The gpt family models have higher Pass@k
Error Analyses. Although promising, the Pass@k and lower Recall@k, while other models are the
of existing LLMs is still low and far from practical opposite. We speculate the reason is that gpt fam-
applications. To determine LLMs’ shortcomings, ily models are instruction-tuned models and focus
we manually analyze 50 error cases of gpt-4 in the on performing tasks based on given instructions.
Local File (Infilling) setting. We found that most of With limited contexts, gpt family models are con-
the cases (29 cases) failed due to implementation servative and tend to generate code independently.
logic errors. 20 cases failed since the necessary con- Other LLMs are standard language models trained
texts were missing, e.g., APIs defined in other files. with real code files containing dependencies. They
Besides, one case failed because of the vague re- are aggressive and generate dependencies that may
quirement. It shows that existing LLMs’ reasoning exist. The comparisons show the importance of
and coding abilities need to be improved. Mean- instruction tuning in practical applications.
while, how to utilize more contexts is necessary to
explore. 4.5 Empirical Leassons
We also obtain some interesting findings from Based on the above experiments, we summarize
Table 7. the empirical lessons we learned as
❶ LLMs successfully generate some depen- ❶ EvoCodeBench pose new challenges, i.e.,
dencies without context. Theoretically, LLMs do repository-level code generation. The performance
not see the contexts and cannot generate dependen- of existing LLMs on EvoCodeBench-2403 drops
cies. According to Table 7, we are surprised to dramatically compared to their performance on pre-
Table 5: The comparison between auto-generated re- Table 6: Pass@1 and Recall@1 with retrieval-
quirements and human-written requirements. augmented generation.

Annotator Win / Tie / Lose Cost (Time) Cost (Money) LLMs Setting Pass@1 Recall@1
gpt-4 5 / 41 / 4 12m30s $0.54 Without Context 8.31 21.08
Human 4 / 41 / 5 4h10m $31.25 gpt-4
Similar Functions 12.29 45.14
Without Context 6.64 21.16
gpt-3.5
vious benchmarks. Similar Functions 11.62 41.93
❷ LLMs benefit from code contexts in current
repositories. With limited context windows, the
for us to use gpt-4 to annotate requirements for
contexts from local files can improve gpt-4 by
EvoCodeBench.
152% in Pass@1.
❸ The main reasons why programs generated by Retrieval-Augmented Generation (RAG). RAG
LLMs fail are logic errors and incomplete contexts. is to enhance generative models with retrieved in-
How to enhance reasoning abilities and context formation and has achieved promising results in
windows of LLMs is important. code generation (Li et al., 2023b,c). We try to
apply RAG to repository-level code generation
5 Discussion and consider the repository to be a retrieval cor-
pus. Because most programs in repositories are not
Evaluation of auto-generated requirements. We equipped with documentation, we retrieve top-k
leverage an LLM (gpt-4 in this paper) to generate (i.e., k = 5 in this paper) functions with similar
natural language requirements for functions auto- names to the target function. Specifically, we split
matically. To assess the quality of auto-generated names into tokens based on underscore or camel-
requirements, we randomly select 50 functions case formatting and then match the tokens of names.
from EvoCodeBench-2403 and compare require- Finally, we use similar functions as contexts in
ments from gpt-4 and human developers. We prompts and further generate code. The experimen-
hire two developers to write the requirements tal results are shown in Table 6. The performance
and two others to evaluate auto-generated and of both LLMs is improved after introducing similar
human-written requirements. The evaluation met- functions. LLMs can know relevant algorithms and
rics include completeness (whether the require- dependencies from similar functions, which benefit
ments cover the intent of code), clarity (whether writing new programs. In the future, we will ex-
the requirements are clear and user-friendly). All plore more advanced RAG techniques to improve
developers are paid according to the relevant poli- repository-level code generation.
cies5 ($7.5 per hour). Results on different program types. Figure 4
The evaluation results are shown in Table 5. The shows Pass@1 of gpt-4 on different program types
Cohen’s Kappa coefficient between the two eval- (i.e., standalone and non-standalone). The results
uators is 0.92. On 41 functions, gpt-4 and devel- are consistent with the above Table 7. Code con-
opers tie. In the remaining functions, gpt-4 wins texts significantly improve the Pass@1 on non-
by 5 functions, and developers win by 4 functions. standalone functions. Meanwhile, the Pass@1 on
These results show that gpt-4 can produce high- standalone functions also slightly increases. We
quality requirements comparable to human-written speculate that the domain knowledge (e.g., private
requirements in most cases (92% = 46/50). We objects) within contexts helps LLMs understand
also inspect the four functions lost by gpt-4 and find requirements. However, the Pass@1 on both types
that some necessary details (e.g., hyper-parameters) of programs is still low. The coding abilities of
are missing in its requirements. In the future, we existing LLMs in real-world repositories need to
will explore new techniques to solve this problem, be further improved.
e.g., controllable text generation (Dekoninck et al.,
Results on different dependency types. Figure 5
2023). Besides the high-quality requirements, gpt-
shows the Recall@1 of gpt-4 on different depen-
4 shows advantages in costs. As shown in Table 5,
dency types (i.e., intra-class, intra-file, and cross-
got-4 costs less time and money to annotate require-
file). The results yield two insights. ❶ Without
ments. Thus, it is a feasible and efficient approach
context, LLMs can reason about some simple de-
5
[Link] pendencies from requirements (e.g., initialization
 achieving significant advancements in code gen-
eration, such as Codex (Chen et al., 2021), Chat-
GPT (OpenAI, 2023a), CodeLlama (Rozière et al.,
2023), DeepSeek Coder (Guo et al., 2024), and
StarCoder2 (Lozhkov et al., 2024).

To effectively steer LLMs in various code gen-

eration scenarios, some works focus on improving
the prompt technologies by introducing specific pat-
Figure 5: Recall@1 of gpt-4 on different dependency
types.
terns, e.g., Structured Chain-of-Thought (Li et al.,
2023a), Self-planning (Jiang et al., 2023), Self-
debug (Chen et al., 2023), Self-collaboration (Dong
functions of returned objects), but still exhibit low
et al., 2023), and AceCoder (Li et al., 2023c).
Recall@1 values across three dependency types. ②
Local files contain code implementations of intra-
Code Generation Benchmarks. Early code gen-
class and intra-file dependencies and thus improve
eration benchmarks (Yin et al., 2018; Chen et al.,
the Recall@1 on both types of dependencies. It is
2021; Austin et al., 2021; Zan et al., 2022) eval-
surprising that Recall@1 on cross-file dependen-
uate code generation on relatively Python func-
cies also increases. We inspect successful cases and
tions, such as HumanEval (Chen et al., 2021) and
find that LLMs copy cross-file dependencies from
MBPP (Austin et al., 2021). APPS (Hendrycks
relevant programs in local files. However, copied
et al., 2021) evaluates code generation on more dif-
cross-dependencies may be inconsistent with the
ficult competition-style problems. ClassEval (Du
current code, e.g., inconsistent arguments. It results
et al., 2023) evaluates LLMs on class-level code
in gpt-4 with higher Recall@1 but lower Pass@1.
generation and contains 100 human-crafted self-
Thus, it is necessary for LLMs to see more contexts
contained Python classes. Concode (Iyer et al.,
about cross-file dependencies.
2018) and CoderEval (Yu et al., 2023) further in-
The bias of Recall@k. As stated in Section 2.3,
troduce non-standalone programs.
we develop a static analysis-based parser to extract
dependencies in generated programs automatically. We release EvoCodeBench to extend code gen-
Because Python is a dynamically typed language, eration benchmarks. Compared to existing bench-
certain dependencies are only determined at run- marks, EvoCodeBench aligns with real-world code
time and may elude our parser. It may lead to lower repositories (e.g., the distributions of code and de-
Recall@k than actual values. pendency) and contains more comprehensive an-
To gauge the above bias, we randomly select notations (e.g., reference dependencies). Besides,
50 programs generated by gpt-4 and annotate de- EvoCodeBench is an evolving benchmark and is
pendencies with them by our parser and two hu- dynamically updated to address data leakage.
man developers, respectively. Based on the human-
annotated and auto-extracted dependencies, we We have also noticed that some benchmarks have
compute two Recall@1 values. The bias of two recently been proposed for repository-level tasks.
Recall@1 values is 0.16. Compared to the average CrossCodeEval (Ding et al., 2023), RepoBench
variations between LLMs (7.77 in Table 7), 0.16 is (Liu et al., 2023), and RepoEval (Zhang et al.,
slight. Consequently, we believe that Recall@k can 2023) are code completion benchmarks. They lack
effectively rank different LLMs, notwithstanding the necessary annotations (e.g., natural language
its slight bias. requirements) for code generation. SWE-bench
(Jimenez et al., 2023) focuses on repairing repos-
6 Related Work
itories’ issues by revising existing programs. In
Large Language Models for Code Generation. contrast, EvoCodeBench is collected for code gen-
The rise of pre-training technology has brought eration and aims to generate new programs based
new impetus to the field of code generation, both in on requirements for a repository. EvoCodeBench
academia and industry (Li et al., 2022; Shen et al., offers comprehensive annotations (e.g., natural lan-
2022; Nijkamp et al., 2023; Fried et al., 2023). In guage requirements, original repositories, reference
this context, more and more LLMs have emerged, code, and reference dependencies).
7 Conclusion and Future Work human-annotated and auto-extracted dependencies,
we compute two Recall@1 values. The bias of two
In this paper, we propose a new code genera-
Recall@1 is 0.16. Compared to the average varia-
tion benchmark named EvoCodeBench. Collected
tions between LLMs (7.77 in Table 7), 0.16 is slight.
through a meticulous pipeline, EvoCodeBench
Consequently, the Recall@k can effectively rank
aligns with real-world code repositories in mul-
different LLMs, notwithstanding its slight bias.
tiple dimensions, e.g., code distributions and de-
Besides, our evaluation experiments can be fur-
pendency distributions. Besides, EvoCodeBench
ther improved in three aspects. ❹ More LLMs. Due
is an evolving benchmark and will be dynamically
to the limited computing budgets, we mainly eval-
updated every period (e.g., 6 months). Based on
uate 10 mainstream LLMs. It is worth evaluating
EvoCodeBench, we propose repository-level code
LLMs with different sizes and fine-tuned models.
generation and evaluate 10 popular LLMs. The re-
❺ More investigations of contexts. As shown in
sults reveal the strengths and weaknesses of LLMs
Table 7, we design two straightforward approaches
in real-world repositories. Compared to previous
to extracting code contexts from repositories. In
benchmarks, EvoCodeBench offers a more chal-
the future, we will introduce cross-file contexts
lenging and realistic evaluation scenario. We hope
and further explore how to utilize contexts effec-
EvoCodeBench can facilitate the applications of
tively. ❻ Tuning hyper-parameters. It is known that
LLMs in real-world repositories.
LLMs are sensitive to sampling hyper-parameters
In the future, we will continue to update
and prompts. We ensure all LLMs are evaluated
EvoCodeBench, e.g., multilingual samples. Be-
under the same experimental settings. Due to the
sides, we will explore how to improve the perfor-
limited computing budgets, we do not carefully
mance of LLMs in repository-level code genera-
tune hyper-parameters and prompts. Thus, there
tion, e.g., retrieval-augmented and tool-augmented
may be better hyper-parameters and prompts to
generation.
improve the performance of LLMs further.
8 Limitations
9 Ethics Consideration
We believe that EvoCodeBench itself has four limi-
EvoCodeBench is collected from open-source
tations. ❶ EvoCodeBench is a monolingual bench-
repositories from the real world. We manually
mark (i.e., requirements in English and code in
check all samples in EvoCodeBench. We ensure
Python) and ignores other languages. In practice,
all samples do not contain private information
LLMs require understanding requirements in dif-
or offensive content. We ensure all programs in
ferent natural languages (e.g., Chinese, Spanish)
EvoCodeBench are behaving normally and exclude
and generating programs in various programming
any malicious programs.
languages (e.g., Java, C). Thus, we plan to build
a multilingual EvoCodeBench in future work. ❷
Auto-generated requirements can be improved. As References
stated in Section 5, auto-generated requirements
Jacob Austin, Augustus Odena, Maxwell I. Nye,
are comparable to human-written requirements but Maarten Bosma, Henryk Michalewski, David Dohan,
may lack necessary details. In the future, we will Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le,
leverage more advanced generation strategies (e.g., and Charles Sutton. 2021. Program synthesis with
controlled text generation (Dekoninck et al., 2023)) large language models. CoRR, abs/2108.07732.
to generate requirements. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang,
❸ As stated in Section 5, Recall@k values in Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei
EvoCodeBench may have slight biases, i.e., they Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin,
may be slightly less than actual values. Because Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu,
Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren,
Python is a dynamically typed language, certain Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong
dependencies can only be identified at runtime and Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang
may elude our parser. To gauge the bias introduced Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian
by our parser, we manually annotate dependencies Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi
Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang,
within 50 programs generated by gpt-4. Simulta- Yichang Zhang, Zhenru Zhang, Chang Zhou, Jin-
neously, we employ the parser to extract depen- gren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.
dencies in the same 50 programs. Based on the Qwen technical report. CoRR, abs/2309.16609.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds,
Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Tom Henighan, Danny Hernandez, Tristan Hume,
Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Josh Jacobson, Scott Johnston, Shauna Kravec,
Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, Catherine Olsson, Sam Ringer, Eli Tran-Johnson,
and Yi Zhang. 2023. Sparks of artificial general Dario Amodei, Tom Brown, Nicholas Joseph, Sam
intelligence: Early experiments with GPT-4. CoRR, McCandlish, Chris Olah, Jared Kaplan, and Jack
abs/2303.12712. Clark. 2022. Red teaming language models to re-
duce harms: Methods, scaling behaviors, and lessons
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, learned. CoRR, abs/2209.07858.
Henrique Pondé de Oliveira Pinto, Jared Kaplan,
Harrison Edwards, Yuri Burda, Nicholas Joseph, GemmaTeam. 2024. Gemma: Open models based
Greg Brockman, Alex Ray, Raul Puri, Gretchen on gemini research and technology. CoRR,
Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- abs/2403.08295.
try, Pamela Mishkin, Brooke Chan, Scott Gray,
Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Mingyang Geng, Shangwen Wang, Dezun Dong, Hao-
Kaiser, Mohammad Bavarian, Clemens Winter, tian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xi-
Philippe Tillet, Felipe Petroski Such, Dave Cum- angke Liao. 2024. Large language models are few-
mings, Matthias Plappert, Fotios Chantzis, Eliza- shot summarizers: Multi-intent comment generation
beth Barnes, Ariel Herbert-Voss, William Hebgen via in-context learning. In Proceedings of the 46th
Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie IEEE/ACM International Conference on Software En-
Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, gineering, ICSE 2024, Lisbon, Portugal, April 14-20,
William Saunders, Christopher Hesse, Andrew N. 2024, pages 39:1–39:13. ACM.
Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan
Morikawa, Alec Radford, Matthew Knight, Miles GitHub. 2023. Github copilot. [Link]
Brundage, Mira Murati, Katie Mayer, Peter Welinder, com/features/copilot.
Bob McGrew, Dario Amodei, Sam McCandlish, Ilya
Sutskever, and Wojciech Zaremba. 2021. Evaluating Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai
large language models trained on code. CoRR. Dong, Wentao Zhang, Guanting Chen, Xiao Bi,
Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wen-
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and feng Liang. 2024. Deepseek-coder: When the large
Denny Zhou. 2023. Teaching large language models language model meets programming - the rise of code
to self-debug. CoRR, abs/2304.05128. intelligence. CoRR, abs/2401.14196.
Jasper Dekoninck, Marc Fischer, Luca Beurer-Kellner, Dan Hendrycks, Steven Basart, Saurav Kadavath, Man-
and Martin T. Vechev. 2023. Controlled text gen- tas Mazeika, Akul Arora, Ethan Guo, Collin Burns,
eration via language model arithmetic. CoRR, Samir Puranik, Horace He, Dawn Song, and Jacob
abs/2311.14479. Steinhardt. 2021. Measuring coding challenge com-
petence with APPS. In Proceedings of the Neural
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Han- Information Processing Systems Track on Datasets
tian Ding, Ming Tan, Nihal Jain, Murali Krishna Ra- and Benchmarks 1, NeurIPS Datasets and Bench-
manathan, Ramesh Nallapati, Parminder Bhatia, Dan marks 2021, December 2021, virtual.
Roth, and Bing Xiang. 2023. Crosscodeeval: A di-
verse and multilingual benchmark for cross-file code Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong,
completion. CoRR, abs/2310.11248. Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen,
Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023. Self- Chen Lin, Nan Duan, et al. 2023. Competition-level
collaboration code generation via chatgpt. CoRR, problems are effective llm evaluators. arXiv preprint
abs/2304.07590. arXiv:2312.02143.

Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and
Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Luke Zettlemoyer. 2018. Mapping language to code
Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A in programmatic context. In Proceedings of the 2018
manually-crafted benchmark for evaluating llms on Conference on Empirical Methods in Natural Lan-
class-level code generation. CoRR, abs/2308.01861. guage Processing, Brussels, Belgium, October 31 -
November 4, 2018, pages 1643–1652. Association
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, for Computational Linguistics.
Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih,
Luke Zettlemoyer, and Mike Lewis. 2023. Incoder: Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang,
A generative model for code infilling and synthesis. and Ge Li. 2023. Self-planning code generation with
In ICLR. [Link]. large language model. CoRR, abs/2303.06689.

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Carlos E. Jimenez, John Yang, Alexander Wettig,
Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik
Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Narasimhan. 2023. Swe-bench: Can language
Andy Jones, Sam Bowman, Anna Chen, Tom Con- models resolve real-world github issues? CoRR,
erly, Nova DasSarma, Dawn Drain, Nelson Elhage, abs/2310.06770.
Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2023a. Struc- Disha Shrivastava, Hugo Larochelle, and Daniel Tar-
tured chain-of-thought prompting for code genera- low. 2023. Repository-level prompt generation for
tion. arXiv preprint arXiv:2305.06599. large language models of code. In International Con-
ference on Machine Learning, ICML 2023, 23-29
Jia Li, Yongmin Li, Ge Li, Zhi Jin, Yiyang Hao, and July 2023, Honolulu, Hawaii, USA, volume 202 of
Xing Hu. 2023b. Skcoder: A sketch-based approach Proceedings of Machine Learning Research, pages
for automatic code generation. In 45th IEEE/ACM 31693–31715. PMLR.
International Conference on Software Engineering,
ICSE 2023, Melbourne, Australia, May 14-20, 2023, Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
pages 2124–2135. IEEE. Martinet, Marie-Anne Lachaux, Timothée Lacroix,
Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal
Jia Li, Yunfei Zhao, Li Yongmin, Ge Li, and Zhi Jin. Azhar, Aurélien Rodriguez, Armand Joulin, Edouard
2023c. Acecoder: Utilizing existing code to enhance Grave, and Guillaume Lample. 2023. Llama: Open
code generation. arXiv preprint arXiv:2303.17780. and efficient foundation language models. CoRR,
abs/2302.13971.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan
Julian Schrittwieser, Rémi Leblond, Tom Eccles, Vasilescu, and Graham Neubig. 2018. Learning to
James Keeling, Felix Gimeno, Agustin Dal Lago, mine aligned code and natural language pairs from
et al. 2022. Competition-level code generation with stack overflow. In Proceedings of the 15th Interna-
alphacode. Science, 378(6624):1092–1097. tional Conference on Mining Software Repositories,
MSR 2018, Gothenburg, Sweden, May 28-29, 2018,
Tianyang Liu, Canwen Xu, and Julian J. McAuley. 2023. pages 476–486. ACM.
Repobench: Benchmarking repository-level code
auto-completion systems. CoRR, abs/2306.03091. Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang,
Yuchi Ma, Guangtai Liang, Ying Li, Tao Xie, and
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Fed- Qianxiang Wang. 2023. Codereval: A benchmark
erico Cassano, Joel Lamy-Poirier, Nouamane Tazi, of pragmatic code generation with generative pre-
Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, trained models. CoRR, abs/2302.00288.
et al. 2024. Starcoder 2 and the stack v2: The next
generation. arXiv preprint arXiv:2402.19173. Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu
Kim, Bei Guan, Yongji Wang, Weizhu Chen, and
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Jian-Guang Lou. 2022. CERT: continual pre-training
Wang, Yingbo Zhou, Silvio Savarese, and Caiming on sketches for library-oriented code generation. In
Xiong. 2023. Codegen: An open large language Proceedings of the Thirty-First International Joint
model for code with multi-turn program synthesis. Conference on Artificial Intelligence, IJCAI 2022,
In ICLR. [Link]. Vienna, Austria, 23-29 July 2022, pages 2369–2375.
[Link].
OpenAI. 2023a. gpt-3.5-turbo. https:
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin
//[Link]/docs/models/
Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and
gpt-3-5.
Weizhu Chen. 2023. Repocoder: Repository-level
code completion through iterative retrieval and gen-
OpenAI. 2023b. GPT-4 technical report. CoRR, eration. In Proceedings of the 2023 Conference on
abs/2303.08774. Empirical Methods in Natural Language Process-
ing, EMNLP 2023, Singapore, December 6-10, 2023,
Pyan. 2023. Pyan. [Link] pages 2471–2484. Association for Computational
davidfraser/pyan. Linguistics.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten
Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi,
Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom
Kozhevnikov, Ivan Evtimov, Joanna Bitton, Man-
ish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori,
Wenhan Xiong, Alexandre Défossez, Jade Copet,
Faisal Azhar, Hugo Touvron, Louis Martin, Nico-
las Usunier, Thomas Scialom, and Gabriel Synnaeve.
2023. Code llama: Open foundation models for code.
CoRR, abs/2308.12950.

Sijie Shen, Xiang Zhu, Yihong Dong, Qizhi Guo,

Yankun Zhen, and Ge Li. 2022. Incorporating do-
main knowledge through task augmentation for front-
end javascript code generation. In ESEC/SIGSOFT
FSE, pages 1533–1543. ACM.
 Please write natural language comments for the given Python function. et al., 2022) techniques. However, the pre-
The template of natural language comments is shown as follows: training data scope and scale, model size, and
1. [Required] What does this function do.
2. [Required] Input-Output parameters parameters remain closed-source at present.
:param arg1: data type [optional], what it is [required], how it is used [optional].
:param arg2: ....
:return: data type [optional], what it is [required]. Or No return values. • gpt-3.5-turbo (OpenAI, 2023a) is an im-
Here is an example: proved gpt-3 model enhanced by a three-stage
Input Code:
```Python reinforcement learning with human feedback
{example_code} (RLHF) algorithm. Apart from improving
```
Requirement: instruction-following capabilities, the RLHF
```
{example_requirement} algorithm proves highly effective in mitigat-
```
ing the generation of harmful or toxic content,
Input Code:
```Python
which is crucial for the practical deployment
{input_code} of LLMs in security-sensitive contexts. we uti-
```
Requirement: lized the released versions of gpt-3.5, namely
gpt-3.5-turbo-1106, with training data up to
Figure 6: The prompt template for generating require- September 2021. However, similar to gpt-4,
ments with gpt-4. the training details, training data, and model
weights are currently closed-source.

A Benchmark Collection Details • CodeLLaMa (Rozière et al., 2023), based on

A.1 Requirement Annotation the LLama2 architecture by Meta-AI6 , was
fine-tuned and open-sourced by the company
Figure 6 shows the prompt template for gen- on August 25, 2023, with versions of 7B,
erating requirements. {example_code}, 13B, and 34B. A 70B version was released
{example_requirement}, and on January 30, 2024 (Rozière et al., 2023).
{input_code} are placeholders. We man- CodeLLama is primarily trained on nearly
ually write a function with requirements as deduplicated publicly available code datasets.
the demonstration example. Then, we fill The first three models were trained on 500
{input_code} with functions and leverage gpt- billion tokenized code, while the latest 70B
4 to generate the corresponding requirements. We model was trained on 1T tokens. Similar
use the greedy search and generate a requirement to the LLaMa series, CodeLLaMa also fol-
for each function. lows a decoder-only architecture. We eval-
A.2 Projects in EvoCodeBench-2403 uated CodeLLaMa-Python-{7B, 13B} upon
our EvoCodeBench.
B Experimental Details
B.1 Base LLMs • DeepSeek Coder (Guo et al., 2024) is a large
language model for programming tasks re-
In this paper, we select 6 popular LLMs as base
leased by DeepSeek-AI7 in November 2, 2023.
LLMs and evaluate them on EvoCodeBench-2403.
DeepSeek Coder consists of a series of code
The details of these LLMs are described as follows.
language models, each trained from scratch
• gpt-4 (OpenAI, 2023b), released by Ope- on 2T tokens, containing 87% code and 13%
nAI on March 14, 2023, marks another mile- natural language. DeepSeek Coder provides
stone in the field of natural language process- code models with 1.3B, 6.7B and 33B parame-
ing. gpt-4 demonstrates superior performance ter sizes. In terms of model architecture, each
compared to previous gpt models (Bubeck model integrates a decoder-only Transformer,
et al., 2023). In our experiments, we use the incorporating Rotary Position Embedding and
version - gpt-4-1106. Its training data up to FlashAttention v2. We evaluated DeepSeek
April 2023. It continues the auto-regressive Coder-{6.7B, 33B} on our EvoCodeBench.
prediction of the next token training objective
inherited from the GPT series models. It in- • StarCoder 2 (Lozhkov et al., 2024) was re-
corporates reinforcement learning with human 6
[Link]
7
feedback (RLHF) and red-teaming (Ganguli [Link]
 Table 7: Projects detail on EvoCodeBench-2403.

Repository Created Stars Py Files Py Lines Samples Domin

Test-Agent 2023-10-20 440 85 15278 1 Deep Learning
skfolio 2023-12-14 813 158 33852 13 Statistical Learning
camp_zipnerf 2024-01-19 523 53 18973 54 Image Processing
microagents 2023-12-11 674 45 2918 18 Deep Learning
open-iris 2023-12-09 161 140 13933 14 Image Processing
litdata 2024-02-15 114 56 11713 59 Data Science
nlm-ingestor 2024-01-17 643 56 16674 4 Internet
AutoRAG 2024-01-10 259 115 7735 13 Data Science
XAgent 2023-10-16 7054 148 17623 3 Deep Learning
tanuki_py 2023-10-16 606 108 10146 9 Text Processing
UHGEval 2023-11-06 148 34 2938 3 Data Science
Generalizable-BEV 2023-10-30 136 570 132407 8 Image Processing
EasyVolcap 2023-12-07 442 308 51723 20 Image Processing
UniRef 2023-12-22 208 382 70042 23 Image Processing
contrastors 2024-01-30 346 62 13774 1 Deep Learning
gaussian-splatting-lightning 2023-10-06 168 76 9935 1 Image Processing
scepter 2023-12-21 190 244 41519 1 Deep Learning
microsearch 2024-02-05 336 5 231 2 Internet
ollama-python 2023-12-09 898 13 2089 12 Deep Learning
Python-Type-Challenges 2023-10-23 343 121 3208 1 Education
stable-fast 2023-10-17 871 82 11948 2 Deep Learning
stable-diffusion-webui-forge 2024-01-14 2537 1112 210946 3 Deep Learning
openlogprobs 2023-11-22 174 6 524 1 Deep Learning
searcharray 2023-11-03 133 25 4217 6 Data Science
deluder 2023-12-01 115 34 1894 3 Internet

leased by BigCode8 on December 8, 2023 is released by Google DeepMind9 on Febru-

with 3 different parameters, 3B, 7B and 15B. ary 21, 2024, based on the transformer de-
StarCoder2 is trained on The Stack v2, a new coder. Models are trained on a context length
large-scale, high-quality code dataset. All of 8192 tokens. We evaluated Gemma-7b in
models were trained using Grouped Query our EvoCodeBench performance.
Attention, a contextual window of 16,384 to-
kens with a sliding window attention of 4,096 • Qwen (Bai et al., 2023) is a comprehensive
tokens, using the Fill-in-the-Middle objec- language model series containing different
tive. Following DeepseekCoder (Guo et al., models with different parameters, including
2024) and Code LLaMA (Rozière et al., 2023), basic pre-trained language models and chat
StarCoder2 use Rotary Positional Encodings. models fine-tuned through human alignment
We evaluated StarCoder2-{7B, 15B} on our technology. The Qwen series is open-sourced,
EvoCodeBench, which was trained on over including Qwen-1.8B, Qwen-7B, Qwen-14B,
3.5 trillion tokens in 17 programming lan- and Qwen-72B, and the corresponding Qwen-
guages from Stack v2. Chat series, which the Alibaba Group released
on August 3, 2023. Qwen has conducted
• Gemma (GemmaTeam, 2024) is a lightweight stable pre-training on multi-lingual data of
open model family built from the research up to 3 trillion tokens, covering fields, lan-
and technology used to create Gemini mod- guages (focusing on Chinese and English),
els. Gemma models demonstrate strong per- etc. QWEN is designed using a modified ver-
formance on academic benchmarks in lan- sion of the Transformer architecture, specif-
guage understanding, reasoning, and security. ically adopting the open-source approach of
Gemma releases models in two sizes (2 billion LLaMA (Touvron et al., 2023). We evalu-
and 7 billion parameters) and provides pre- ated the performance of Qwen1.5-7b on our
training and fine-tuning checkpoints. Gemma EvoCodeBench.
8 9
[Link] [Link]
 Please complete the {function_name} function in the given Python
code.

Input Code:
```Python
{signature}
{requirement}
```
Completed Code:
Please complete the {function_name} function in the middle of a file.

Figure 7: The prompt template in the without context The contexts above the function are:
setting. ```Python
{contexts_above}
```
Please complete the {function_name} function based on the The contexts below the function are:
contexts above the function. ```Python
{contexts_below}
The contexts above the function are: ```
```Python
{contexts_above} The code to be completed is:
``` ```Python
{signature}
The code to be completed is: {requirement}
```Python ```
{signature} Completed code:
{requirement}
```
Completed code: Figure 9: The prompt template in the local file (infilling)
setting.
Figure 8: The prompt template in the local file (comple-
tion) setting.

B.2 Prompt Templates

The prompt templates used for instruction-tuning
models (i.e., gpt-4 and gpt-3.5) are shown in
Figure 7, 8, 9, and 10. {function_name},
{contexts_above}, {contexts_below},
{signature}, and {requirement} are
placeholders.
For other standard language models, the prompt
templates are shown as follows: ① Without
context: [signature; requirement]; ② Please complete the {function_name} function based on some
functions with similar names.
Local file (completion): [context_above;
signature; requirement]; ③ Local file The functions with similar names are:
```Python
(infilling): [prefix_id; context_above; {similar_functions}
signature; requirement; ```

suffix_id; context_below; The code to be completed is:

middle_id]. Where [;] denotes the con- ```Python
{signature}
catenation operation of strings. {prefix_id}, {requirement}
{suffix_id}, {middle_id} are special ```
Completed code:
tokens used in code infilling. For different LLMs,
we reuse their official special tokens to make Figure 10: The prompt template in the similar function
prompts. setting.