LLM for Coding
Wei Lu (mailwlu@[Link])
History of program synthesis
开发者地域分布
In logical programming languages On End-to-End Codex: The first pre-trained
Agents: GPT-
(Prolog), requirements are “no- Program code generation model of
Pilot, MetaGPT,
expressed as logical expressions code” /”low- Generation from 10B+ scale was trained
ChatDev,
for computers to perform code“ User Intention by through unsupervised learning
Devin, SWE-
automated reasoning to derive movement Deep Neural techniques on an extensive
Agent
solutions. Networks (RNNs). corpus of code.
1957 1972 200?2004/2011 2015 2020 2021 ～ 2022 2023 2024
GraphCodeBERT ChatGPT:
CugLM: a pre-trained
FORTRAN compilers are Some of the UML UniXCoder interactive code
language model for
recognized as diagrams (Class, State) GPT-C generation
code understanding
automatic code can be converted into Grammformer CodeLlama,
and generation, and
generation systems implementation code, ReACC DeepSeek-coder,
then fine-tune it on
that transform high- with Rule and Template CodeReviewer WizardCoder
code completion.
level programming based approaches. … CodeQwen
CodeBERT: a bimodal
language statements StarCoder
pre-trained model for
into machine code. programming language
(PL) and natural Common sense,
language (NL). Reasoning,
Flipped
Higher level of abstraction
Interaction
SE: Needs & Expectations
The most time-consuming activities 开发者地域分布
广东、北京、上海是开发者聚集较多的地域，占全国总数的 38% 。江苏、浙江、四川、山东处于第二梯
队，占全国总数的 24% 。
Source: [Link]
Consistent in all
SE: Needs & Expectations experience levels of
the survey
respondents?
How frequently do you use the How likely is it开发者地域分布
that you would delegate
following features of the existing AI the following activities to an AI
assistants for coding? assistant?
广东、北京、上海是开发者聚集较多的地域，占全国总数的 38% 。江苏、浙江、四川、山东处于第二梯
队，占全国总数的 24% 。
Source: [Link]
SE: Needs & Expectations
Higher acceptance rate by less experienced GitHub Copilot user
开发者地域分布
广东、北京、上海是开发者聚集较多的地域，占全国总数的 38% 。江苏、浙江、四川、山东处于第二梯
队，占全国总数的 24% 。
Source: [Link]
AI: SOTA
Human/ Environment/
Execution feedback
Requirement
开发者地域分布-
GPT-4
/CodeLlama software
Repo
Completion
Text - Code Tool / API
GPT-3/Codex Generation use
Func/Class
Completion
Slower
Repo Edit/
NL Code Issue Fix
Search GPT-2
Code
Review
Line
Completion
Retrieval- BERT/GPT Customized
augmented Model
Code-Completion Effectiveness
Acceptance Rate Used by 开发者地域分布 Used by
Developer Organization
GitHub Copilot 28%(1st m)- 1M 20,000
34%(6th m)
Baidu Comate 46%(internal)- 80%(internal) 10,000+
49%(External)
Alibaba Tongyi 30-40%(Online)-
Lingma 40-50%(Offline)
广东、北京、上海是开发者聚集较多的地域，占全国总数的 38% 。江苏、浙江、四川、山东处于第二梯
队，占全国总数的 24% 。
Implementation
Code Completion,
FIM Test Generation
Curation , Cleaning
GitHub Format
Private Repo Curriculum Pre-training Context
Data
Doc Deduplication
Fine-tuning RAG
Weighting/ Mix
Synthetic data LLM
ratio IDE, FS,
KB
Agent Chunking
Reasoning Context
Loop Embedding
Tools Vector
Conversation Model
DB
memory
Planning
Question
Complex Tasks External
APIs
Data engineering - NL vs. Code
Nature Language model Code segment is dependent on prefix,
predicts the next token as suffix, other files in the project or external
one of a sequence of tokens library.
Data engineering – Fill In the Middle (FIM) task
• AST subtree
span
• Context
• Call-graph
• File
dependency →
file re-ranking
• Long context
helps.
• RAG helps in
inference.
Data engineering – Synthetic data
• CodeAlpaca (Chaudhary, 2023) comprises 20K code instructions automatically generated by applying self-instruct
(Wang et al., 2023b) to ChatGPT, utilizing 21 seed tasks as the foundation.
• WizarCoder (Luo et al., 2023) describes a method, Evol-Instruct-Code, that applies a variety of heuristics to enrich
the complexity of initial code instructions, building upon the dataset provided by CodeAlpaca.
• MagicCoder (Wei et al., 2023)/Magicoder-OSS-Instruct employs a robust LLM to generate novel coding
challenges, sourcing inspiration from a diverse range of open-source code snippets.
• WaveCoder (Yu et al., 2023) implements an LLM
• Seeds, Human framework for creating code instruction
generator-discriminator
data, offering customization and control overKnow-How,
the data generationSOPprocess. They introduce CodeOcean, a dataset
comprising 20,000 instruction instances across • 4Engineering
universal code-related tasks, which is aimed at augmenting the
effectiveness of instruction tuning and improving the generalization ability of fine-tuned model.
• Code-Feedback (Zheng et al., 2024) comprises 68KArtifacts
multi-turn interactions that include both user instructions and
compiler feedback, enabling the model to not• only Intelligence
generate but also refine code based on execution outputs and
human guidance. flywheel
• VGX: Large-Scale Sample Generation for Boosting Learning-Based Software Vulnerability Analyses (Nong et al.,
2024))
• TACO: The training set (25,443 problems) and the test set (1,000 problems) consist of more challenging
programming competition problems that more closely simulate real-world programming scenarios.
• StarCoder2-Instruct (Wei et al., 2024) data generation consists of three steps: Extract high-quality and diverse
seed functions from The Stack v1, Self-OSS-Instruct & NL response self-validation with StarCoder2-15B.
Data engineering: Evaluation
HumanEval(+/X) Needs
• Algorithm or competition • Tasks closer to development
questions scenarios
• Generation of independent • Dependencies and context
functions or methods without • More accurate evaluation
dependency indicators (Compile, UnitTest)
• Easily contaminated during • Specialized in specific
training programming language,
library/package.
SWE-bench • Comparing to human experience
• To resolve Real-World GitHub levels
Issues
• Too difficult now
Data engineering: Evaluation
HumanEval(+/X) Needs
• Algorithm or competition • Tasks closer to development
questions scenarios
• Generation of independent • Dependencies and context
functions or methods without • More accurate evaluation
dependency indicators (Compilation, UnitTest)
• Easily contaminated during • Comparing to human experience
training level
SWE-bench
• To resolve Real-World GitHub
Issues
• Too difficult now
Source:[Link]
Copiloting: Online code completion
Quality & Speed (latency time < 600ms)
• Rule vs. AI
Wh • Cache
en
• Before cursor
Co • After cursor
nte • Dependencies
xt
• Prompt engineering, SFT, RAG
Ge • Speculative Decoding
ner
ate
• Verify
Ap • Merge
ply
• Multiple solutions
Bet • Stronger models
ter
RAG
Chunking based on AST: function/method/class → KB Node/Document
…
block_comment [173, 4] - [183, 7]
method_declaration [184, 4] - [213, 5]
modifiers [184, 4] - [185, 10]
marker_annotation [184, 4] - [184, 13]
name: identifier [184, 5] - [184, 13]
…
Vector Index/Database
• FAISS
• LanceDB, Chroma, Milvus, pgvector, … [Link]
Code line# Path Vector Summary Class name
[123,134] src/… [0.12,-0.34,…] Project management … Manager
RAG
Summary convolving → Hierarchical KB
• AST-File-Dir/Package, Dependency
• Call-graph
Summary 1 Summary 2
File Summary 3
Sub-dir Summary 4
File main delete push
Dir File
Sub-dir File
Sub-dir File add close call C
Dir File int
int File main i
Sub-dir Program call
function D callforE call F
main i (argv) int ++
function
(argv)
for File
++
body main a i
body a Program function for
(argv) > ++
Program
> if
int if body b a
b
main >
function {} {}
RAG
Retrieval
• Vector-based retrieval + keyword-based (TF-IDF, BM25)
⎯⎯re-rank with cross-encoder/decoder→ top-k
• Recall@k: 0.6~0.8
Long context (e.g. 1M in Gemini)
• Like a larger cross-encoder + decoder
• Neural KB
🧠
Source: [Link]
Agent
Source: [Link]
Agent
Source: [Link]
Agent
GPT-Pilot with more agents/roles of a software project
• Spec Writer Agent — writes project description from user input
• Architect Agent — creates architecture
• Tech Lead Agent — breaks down the project and creates tasks
• Developer Agent — Implements Tasks
• Reviewer Agent — Checks CodeBest Practice
Agent
SWE-Agent
• Tools/commands
• linter that ensures syntactical correctness
before code edits are applied
Tools
• file viewer and editor with scrolling and SOP
search capabilities Simulation of
• full directory string searching causal
command for codebase reasoning
navigation and match
• Prompt Templates
• mimic the workflow of a human
developer
📣 What Open Source can do?
"word-of-mouth" is essentially a small amount of test
data used by real users. Such data is highly valuable and
Big Data → the new “source code” would become an important role of the open-source
• Seeds movement, much like open-source code.
• SOP
• Engineering Artifacts (Requirement, Design, …)
Evaluation metrics are
• Proprietary/Corporate Code & Doc one aspect, word-of-
mouth is another
• De-identification/Anonymization aspect, and the latter
will become
• Differential privacy I personally have very few fictional use cases, increasingly important.
but such use cases are the most effective for
• LLM testing a model's capabilities. Models that
cannot pass my personal use cases aren't
worth spending more time on.
Appropriate model, at the Appropriate Time, Appropriate
Place for the Appropriate Task and Appropriate People
Thanks!
mailwlu@[Link]
@storyfinland