1
CS59200 AI-Assisted Software Engineering
Instructor: Tianyi Zhang, Assistant Professor of Computer Science
Email: tianyi@[Link]
Lecture: 12:00pm-1:15pm TTh @ LWSN B134
Office Hours: 1:30pm-2:30pm TTh @ LWSN 3154H
Instructional Modality: Face-to-Face
Course Credits: 3.0
Prerequisites: Python programming skills and basic understanding of machine learning
are required. Knowing how to use scikit-learn and PyTorch (or Keras) is recommended.
Course Description
Have you ever wondered about these:
- Will programming jobs no longer exist because of large language models like
ChatGPT?1
- Will software engineering become prompt engineering in the next decade?
- How far are we from the “black art” of natural language programming as Dijkstra
called it 40 years ago?2
- What essential software engineering skills are needed in the age of AI?
This course will help you answer those questions. In this course, you will learn concepts
and research topics about (1) the role of human programmers in the age of AI, (2) how AI
and ML technologies have been applied in different software engineering tasks such as
code generation and software testing, (3) how AI-based software development tools work
with or clash against the existing workflow of programmers, and (4) how to improve the
design of AI-based programming tools to support human-AI partnership.
Activities in this course include lectures, presentations of research papers, and discussion
of relevant literature in Software Engineering, Machine Learning, and Human-Computer
Interaction. You should expect to present two research papers
during the semester. You also need to write a short paper review (2 or 3 paragraphs) in
the form of comments and questions and post it on Piazza before each paper
discussion. There will be a course project, in which you will work in groups to design
1
Matt Welsh, The end of programming. Communications of the ACM, January 2023.
2
E. W. Dijkstra. On the foolishness of “natural language programming”. In Program Construction, pages
51–53. Springer, 1979.
2
and carry out research projects related to AI-assisted programming. During the semester,
we will have two guest speakers from other universities and industry to present their
current research in human-AI interaction. This is a good opportunity to connect with
faculty from other universities and learn about their research.
Learning Resources, Technology, Texts
Since this is a relatively new research topic, there is currently no textbook available for
this course. I recommend you reading some surveys to get a basic understanding about
the literature.
• Yang, Yanming, Xin Xia, David Lo, and John Grundy. "A survey on deep learning
for software engineering." ACM Computing Surveys (CSUR) 54, no. 10s (2022):
1-73. [Link]
• Watson, Cody, Nathan Cooper, David Nader Palacio, Kevin Moran, and Denys
Poshyvanyk. "A systematic literature review on the use of deep learning in software
engineering research." ACM Transactions on Software Engineering and
Methodology (TOSEM) 31, no. 2 (2022): 1-58.
[Link]
• Zan, Daoguang, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan,
Wang Yongji, and Jian-Guang Lou. "Large language models meet NL2Code: A
survey." In Proceedings of the 61st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pp. 7443-7464. 2023.
[Link]
• Wang, Junjie, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing
Wang. "Software testing with large language model: Survey, landscape, and
vision." arXiv preprint arXiv:2307.07221 (2023). [Link]
To develop useful AI-assisted programming tools, it is very important to understand user
interface design and Human-AI Interaction. I recommend read the following articles:
● Jakob Nielson, 10 Usability Heuristics for User Interface Design
● Google PAIR, People + AI Guidebook
● Apple, Human Interface Guidelines for Machine Learning
● Microsoft, HAX Toolkit
If you are not very familiar with Machine Learning. I recommend reading the following
textbooks.
● Christopher M. Bishop (2006), Pattern Recognition and Machine Learning
● Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep Learning
3
We will use Piazza to make announcements, ask & answer questions, look for teammates,
etc. Please join this course on Piazza. The signup link will be posted soon.
All lecture slides, assignment instructions, and other materials will be posted on Purdue
BrightSpace.
Learning Outcomes
At the end of this course, students should be able to:
• design and implement AI-based software engineering tools
• develop interaction mechanisms to improve the usefulness, usability, and
transparency of your systems
• evaluate an AI-based software engineering tools through quantitative experiments
and user studies
• write a research paper and give research presentations
Grading
Reading assignments [20%]
Paper presentation [20%]
Course project [50%]
Quizzes [5%]
Class participation and discussion [5%]
Course Schedule
Week 1. Introduction to AI-assisted Software Engineering
Sample course project ideas released by the instructor on 1/19
Week 2: Probability Theory and Linear Algebra Review
Week 3: Classic Machine Learning Algorithms
Week 4. Deep Learning Models
Week 5. Transformers and Large Language Models
Project proposal due on 2/15 midnight
4
Week 6. Machine Learning Models for Code Generation
Week 7. No-Code/Low-Code Platforms
Week 8. Machine Learning Models for Requirement Engineering and Software Design
Week 9. Machine Learning Models for Software Testing and Verification
Week 10 No Class (Spring Vacation)
Week 11. Machine Learning Models for Software Defect Prediction
Mid-point project summary due on 3/20 midnight
Week 12. Machine Learning Models for Program Repair
Week 13. Machine Learning Models for Software Evolution and Maintenance
Week 14. Usability Challenges in AI-based Software Engineering Tools
Week 15. User Interface Design for AI-based Software Engineering Tools
Week 16. Final Project Presentation
Final project report due on 5/3 midnight
Paper Reading Assignment
For each lecture, you should expect to read one research paper on a specific topic in
Human-AI Interaction. I will provide some optional readings related to the topic but you
are not required to read them. The optional readings are mostly for students who are
particularly interested in the topic or who are doing a course project in the topic.
For each required paper reading, you need to submit a short paper review (one or two
paragraphs) in the form of questions and comments on Piazza before the class. The
grading of your paper review will depend on the overall quantity and quality of your
questions and comments. As you read a paper or write your review, focus on the following
perspectives.
5
- Motivation of the work. If the paper presents a new tool, who are the target users?
Do they really need such a tool? What pain points does this tool address for those
users? If the paper presents an empirical study, what are the research questions
this study aims to answer? How important are these studies? Who will care about
the findings and why should they care?
- Novelty and significance of the work. What is new here? What are the main
contributions of the paper? What did you find most interesting?
- Limitations, flaws, and blind spots. Are there any unrealistic or false
assumptions about the target users or the approach? Are there flaws or mistakes
in the tool design, technical approach, or the study design?
- Future work. How would you improve on this work? Does this paper inspire any
new ideas in your own research?
You should expect to present two research papers during the course. The instructor will
ask students to sign up papers to present by the end of the second week. The instructor
will present the rest of the unselected papers during the course.
Each paper presentation should be no more than 40 minutes, so we can have enough
time for discussion. The presentation should focus on elaborating the motivation, related
work, tool/study design, research questions, findings, limitations, and future work of the
assigned paper. To make your presentation more insightful, try to center your
presentation based on the literature and tell the audience why this work is proposed in
the first place, how it advances people’s understanding about a topic, and how it is
different from other related work in the past. You are also encouraged to connect the
assigned paper to your own research. You should prepare for a set of questions (either
came up by yourself or based on questions other students post on Piazza) and co-lead
an in-class discussion with the instructor based on these questions after the presentation.
The in-class discussion will follow the think-pair-share format.
- 1) Think. The presenter or the instructor will provoke students' thinking with a question.
The students should take one or two minutes just to THINK about the question.
- 2) Pair. Using designated partners (such as with Clock Buddies), nearby neighbors, or
a deskmate, students PAIR up to talk about their answers with each other. They compare
their mental or written notes and identify the answers they think are best, most convincing,
or most unique.
- 3) Share. After students talk in pairs for a few minutes, the presenter or instructor will
call for pairs to SHARE their thinking with the rest of the class
Course Project Instructions
6
You are expected to work on a course project either alone or in groups (no more than 3
students in a group). You can pick any topics related to AI-based software engineering,
e.g., a new LLM-based code generation pipeline, a new code summarization model, a
new ML-based bug detection tool, etc. At the end of Week 1, I will release a list of sample
project ideas with paper references, but feel free to work on your own ideas. Between
Week 2 and Week 5, please stop by during office hours to discuss your project ideas with
the instructor to get early feedback on the relevance, novelty, feasibility, and significance
of your ideas.
A short project proposal is due on Feb 15 midnight (Week 5). This proposal should
describe the project idea, the motivation of this idea, and (optional) a usage scenario if
you propose to build a new tool. The proposal could be any length but no longer than 4
pages. It will be evaluated based on the quality of the idea and writing, not the length of
your writing.
A mid-point project summary is due on Mar 20 midnight (Week 11). This summary should
describe the envisioned approach/methodology/design as well as which parts have been
done so far. The summary could be any length but no longer than 4 pages.
In Week 16, each team will deliver a presentation of their project. The presentation will
be about 20 minutes. You will get another 5 minutes for Q&A.
A final project report is due on May 3 midnight (max 10 pages plus references). Your final
project report should be built upon your proposal and project summary. Feel free to reuse
sections from those two reports in your final report. You may include an appendix beyond
10 pages, but your paper must be understandable without it. Submissions should be in
the ACM format.
Your final report should be structured like a conference paper. It should contain:
● Abstract
● A well-motivated introduction
● Related work with proper citations
● Description of your methodology
● Evaluation results
● Discussion of your approach, threats to validity, and additional experiments
● Conclusions and future work
If you are doing a project that involves implementation, please include a link to your Github
repository in your final report. Please also add a README file in your repository to
describe how to run and test your code.
7
Important dates:
● Project proposal due on 2/15 midnight
● Mid-point project summary due on 3/20 midnight
● Final project presentation on Week 16
● Final project report due on 5/3 midnight
Course Policies and Expectations
Late submissions
Late submissions are accepted with 20% decaying credit per day.
Attendance
This course follows Purdue’s academic regulations regarding attendance, which states
that students are expected to be present for every meeting of the classes in which they
are enrolled. Your final grade will depend on your participation in the class. Please come
to the class continuously, read the assigned papers, and participate in discussions.
If you feel sick, have any symptoms associated with COVID-19, or suspect you have been
exposed to the virus, you should stay home and contact the Protect Purdue Health Center.
Please also notify the instructor so that the instructor can arrange remote participation for
you. If you miss classes because of COVID-related reasons, your final grade will not be
affected by your absence of classes. For more guidance on class attendance related to
COVID-19 are outlined in the Protect Purdue Pledge for Fall 2021 on the Protect Purdue
website.
For other conflicts or absences, such as for many University-sponsored activities and
religious observations, the student should inform the instructor of the situation as far in
advance as possible. When the student is unable to make direct contact with the instructor
and is unable to leave word with the instructor’s department because of circumstances
beyond the student’s control, and in cases falling under excused absence regulations, the
student or the student’s representative should contact or go to the Office of the Dean of
Students website to complete appropriate forms for instructor notification. Under
academic regulations, excused absences may be granted for cases of grief/bereavement,
military service, jury duty, and parenting leave. For details, see the Academic Regulations
& Student Conduct section of the University Catalog website.
Academic Guidance in Event of Quarantine or Isolation
If you must miss class at any point in time during the semester, please reach out to me
via email so that we can communicate about how you can maintain your academic
8
progress. If you find yourself too sick to progress in the course, notify your adviser and
notify me via email or Brightspace. We will make arrangements based on your particular
situation. Please note that, according to Details for Students on Normal Operations for
Fall 2021 announced on the Protect Purdue website, “individuals who test positive for
COVID-19 are not guaranteed remote access to all course activities, materials, and
assignments.”
Feedback to the instructor
During this course, I will be asking you to give me feedback on your learning in both
informal and formal ways. Occasionally, at the end of a lecture, I will hand out index cards
to collect anonymous comments and questions about this class and your learning
experience. In the middle of the semester, I will send out an anonymous midpoint survey
about how my teaching strategies are helping or hindering your learning. It is very
important for me to know your reaction to what we are doing in the class, so I encourage
you to respond to these surveys, ensuring that we can create an environment effective
for teaching and learning.
Classroom Guidance Regarding Protect Purdue
Any student who has substantial reason to believe that another person is threatening the
safety of others by not complying with Protect Purdue protocols is encouraged to report
the behavior to and discuss the next steps with their instructor. Students also have the
option of reporting the behavior to the Office of the Student Rights and Responsibilities.
See also Purdue University Bill of Student Rights and the Violent Behavior Policy under
University Resources in Brightspace.
Academic Integrity
Please read and follow the academic integrity policy from the CS department (link) and
from Purdue University (link). For paper reading assignments and the course project, you
are encouraged to discuss with your classmates and course instructor. However, you
should ensure that any written work you submit for evaluation is the result of your own
research and writing and that it reflects your own approach to the topic. You must also
adhere to standard citation practices in this discipline and properly cite any books, articles,
websites, lectures, etc. that have helped you with your work.
Nondiscrimination Statement
Purdue University is committed to maintaining a community which recognizes and values
the inherent worth and dignity of every person; fosters tolerance, sensitivity,
9
understanding, and mutual respect among its members; and encourages each individual
to strive to reach his or her potential. In pursuit of its goal of academic excellence, the
University seeks to develop and nurture diversity. The University believes that diversity
among its many members strengthens the institution, stimulates creativity, promotes the
exchange of ideas, and enriches campus life. A hyperlink to Purdue’s full
Nondiscrimination Policy Statement can be found here.
Accessibility
Purdue University is committed to making learning experiences accessible. If you
anticipate or experience physical or academic barriers based on disability, you are
welcome to let me know so that we can discuss options. You are also encouraged to
contact the Disability Resource Center at: drc@[Link] or by phone: 765-494-1247.
Mental Health/Wellness Statement
If you find yourself beginning to feel some stress, anxiety and/or feeling slightly
overwhelmed, try WellTrack. Sign in and find information and tools at your fingertips,
available to you at any time.
If you need support and information about options and resources, please contact or
see the Office of the Dean of Students. Call 765-494-1747. Hours of operation are M-F,
8 am- 5 pm.
If you find yourself struggling to find a healthy balance between academics, social
life, stress, etc. sign up for free one-on-one virtual or in-person sessions with a Purdue
Wellness Coach at RecWell. Student coaches can help you navigate through barriers and
challenges toward your goals throughout the semester. Sign up is completely free and
can be done on BoilerConnect. If you have any questions, please contact Purdue
Wellness at evans240@[Link].
If you’re struggling and need mental health services: Purdue University is committed
to advancing the mental health and well-being of its students. If you or someone you know
is feeling overwhelmed, depressed, and/or in need of mental health support, services are
available. For help, such individuals should contact Counseling and Psychological
Services (CAPS) at 765-494-6995 during and after hours, on weekends and holidays, or
by going to the CAPS office on the second floor of the Purdue University Student Health
Center (PUSH) during business hours.
Basic Needs Security
10
Any student who faces challenges securing their food or housing and believes this may
affect their performance in the course is urged to contact the Dean of Students for support.
There is no appointment needed and Student Support Services is available to serve
students 8 a.m.-5 p.m. Monday through Friday. Considering the significant disruptions
caused by the current global crisis as it is related to COVID-19, students may submit
requests for emergency assistance from the Critical Needs Fund.
Emergency Preparation
In the event of a major campus emergency, course requirements, deadlines and grading
percentages are subject to changes that may be necessitated by a revised semester
calendar or other circumstances beyond the instructor’s control. Relevant changes to this
course will be posted onto the course website or can be obtained by contacting the
instructors or TAs via email or phone. You are expected to read your @[Link] email
on a frequent basis.