// Large language models
// and the end of programming
// Matt Welsh <matt@[Link]>
//
// May 2023
*** COMPUTER SCIENCE IS DOOMED ***
Computer Science has always been about one thing:
Translating ideas into programs.
CS is the study of how to take a problem and map it onto
instructions that can be executed by a Von Neumann machine.
2
*** COMPUTER SCIENCE IS DOOMED ***
Critically, the goal of CS has always been that programs
are implemented, maintained, and understood by humans.
But -- spoiler alert! -- humans suck at all of these things.
3
Let’s just make programming easier!
Fifty years of programming language research has done
nothing to improve the state of affairs.
No amount of improvement to type systems, debugging, static
analysis, linters, or documentation is going to magically
solve this problem.
4
Let’s just make programming easier!
FORTRAN (1957)
5
Let’s just make programming easier!
BASIC (1964)
6
Let’s just make programming easier!
APL (1966)
7
Let’s just make programming easier!
Malbolge (1998)
8
Let’s just make programming easier!
Whitespace (2003)
9
Let’s just make programming easier!
Rust (2010)
10
Let’s just make programming easier!
11
Let’s just make programming easier!
Rust (2010)
12
Bro, do you even CoPilot?
Programmer without CoPilot,
ca 2020 (colorized)
13
Bro, do you even CoPilot?
Programmer with CoPilot,
ca 2021
14
CoPilot is a tectonic shift
CoPilot will READ YOUR MIND
Just start typing, it finishes the job
CoPilot KNOWS ALL
Not sure how to do something? CoPilot it!
15
CoPilot is just the beginning
The only thing stopping CoPilot from
getting really, really good is more data
and more compute.
Both of which are in abundance.
16
17
How much does it cost to replace one human with AI?
Typical SWE salary: $220,000
Benefits, taxes, free breakfast, lunch, dinner, snacks,
masseuse, shuttle bus, on-site doctor, bowling alley, ...
$92,000
Total: $312,000
Number of working days per year: 260
Total cost for one-human-SWE-day: $1200
18
How much does it cost to replace one human with AI?
Let’s do the math!
Average lines of code checked in per day ~= 100
Average number of GPT-3 tokens per line ~= 10
Price for GPT-3 = $0.02 / 1K tokens
19
How much does it cost to replace one human with AI?
Let’s do the math!
Average lines of code checked in per day ~= 100
Average number of GPT-3 tokens per line ~= 10
Price for GPT-3 = $0.02 / 1K tokens
Total cost for one-human-SWE-day equivalent work: $0.12
20
How much does it cost to replace one human with AI?
$0.12 / day
21
How much does it cost to replace one human with AI?
$0.12 / day $1200 / day
22
How much does it cost to replace one human with AI?
The robot does not take breaks.
The robot does not require catered
lunches or on-site massage.
The robot takes the same length of
time whether it’s a prototype or
final production code.
The robot makes plenty of
mistakes, but makes them
incredibly quickly.
23
Cutting humans out of the loop
The age old question
24
Presenting ... The software team of the future
Red-blooded PM
AI robot
Human code (still human)
code monkey
reviewer
25
AI-generated code
Why is everyone freaking out about ChatGPT?
Pong
(1972)
Why is everyone freaking out about ChatGPT?
Pong Red Dead Redemption 2
(1972) (2018)
Why is everyone freaking out about ChatGPT?
What Computers Can’t Do Superintelligence
(1972) (2014)
The Evolution of Programming
The Dawn of Time
----------------
Humans directly write
machine instructions
29
The Evolution of Programming
Early Prehistory
----------------
Humans write in high level
languages, computers compile
to machine instructions
Bjarne Stroustrup, looking cool
30
The Evolution of Programming
Modern Times
------------
Humans write in high-level
languages, assisted by AI
CoPilot doing my job for me
31
The Evolution of Programming
The Future
----------
Humans teach AI models how to
solve problems directly
32
Chain of thought reasoning in LLMs
LLMs like GPT-4 have been shown to perform reasoning:
- Take a complex problem statement
- Break it into steps
- “Execute” the steps, one at a time, manipulating a
world model as execution proceeds
Implication: LLMs are general purpose problem solvers
33
34
35
Teaching, not programming
Gradually, programming gets replaced by teaching models new
skills:
- How to interface to an API
- How to pull data from a database
- How to transform data
- How to use software meant for humans
36
The Natural Language Computer
A new kind of computational substrate!
External tools
("peripherals")
Natural
language
Large Language
"program" Model
{ API }
Short-term Long-term
memory memory
GPT Task
at
Ch
Task
Task
Vector DB 37
AND NOW, A WORD
FROM OUR SPONSORS
THE FIXIE LLM APPLICATION PLATFORM
Customer Enterprise
BI Dashboard
Service App Automation
Database Salesforce G Drive Github Custom
agent agent agent agent agent
{ API }
39
CUSTOMER SERVICE EXAMPLE
Customer support ticket (or Slack “Fetch the order history for customer xxx”
message, email, API call, etc.)
Order history agent
“Check stock for SKU yyy”
Stock lookup agent
“Issue a return label to <address>”
Return label agent
“Draft an email reply with...”
Email reply agent
Large Language
Model
Fixie Platform
40
FIXIE AGENTS
FEW-SHOT LEARNING EXAMPLES Q: What is the current price for Apple?
Example query, iteration, and Ask Func[quote]: AAPL
Func[quote] says: $155.33
responses, all driven by the LLM A: The current share price for Apple is $155.33.
Q: How is Tesla doing?
EMBEDDED FUNCTIONS Ask Func[quote]: TSLA
Minimal code to invoke external APIs Func[quote] says: $214.24
A: The current share price for Tesla is $214.24.
Cloud-hosted and serverless
Any programming language
def quote(symbol: str):
url = f"[Link]
f"function=GLOBAL_QUOTE&apikey={API_KEY}&symbol={symbol}"
response = [Link](url)
raw_price = [Link]()["Global Quote"]["05. price"]
dot = raw_price.index(".")
return raw_price[:dot+3]
41
MAKING AGENTS AUTOMATICALLY
import fixieai
URLS = [
"[Link]
"[Link] SPECIFY DOCUMENT SOURCES
"[Link] Fixie crawls and indexes every
"[Link]
document in the provided list
"[Link]
"[Link]
"[Link]
] AUTOMATICALLY GENERATE
AGENT FROM DOC CONTENTS
CORPORA = [[Link](urls=URLS)]
No need for any custom code
agent = [Link](CORPORA, conversational=True)
42
WE NOW RETURN YOU TO YOUR
REGULARLY SCHEDULED TECH TALK
Evolving Computer Science
Slide rule
1859-1975
44
Evolving Computer Science
Slide rule Computer science
1859-1975 1959-2030
45
Evolving Computer Science
Over time, CS looks more like EE: A more technical skill set
necessary in some very specialized occupations.
The vast majority of people building “software” will not be
programming: they will be interacting with an AI.
AI greatly expands access to computing to anyone who can
express themselves in natural language.
46
Evolving Computer Science
The network is the computer.
-- John Gage, 1984
47
Evolving Computer Science
The network is the computer.
-- John Gage, 1984
The model is the computer.
-- Matt Welsh, 2023
48
Challenges ahead...
Dirty secret: Nobody understands how large AI models work.
“Chain of thought” reasoning in LLMs was discovered
empirically, not designed into the model architecture!
49
The silver lining
Writing code sucks anyway!
Let the robots do it.
Go and have a good life.
50
matt@[Link]