Contents

• LLM Engineering: Beginner Notes

◦ 1. What Is an LLM, and What Does “LLM Engineering” Mean?

▪ Key term: “Frontier model”

◦ 2. Why Translate Python to C++ or Rust? (The Problem We’re
Solving)
◦ 3. The Most Important Question: Which Model Should You Pick?

▪ 3.1 The basics: facts you compare before anything else

▪ 3.2 The Chinchilla scaling law
▪ 3.3 Training-time vs inference-time techniques
◦ 4. Benchmarks: Measuring How Good Models Are

▪ 4.1 Six hard benchmarks worth knowing

▪ 4.2 Why you should take benchmarks with a pinch of salt
◦ 5. Leaderboards and Arenas: Where to Compare Models

▪ 5.1 Arenas, ELO, and LLM-as-a-judge

◦ 6. The Commercial Side: Applying LLMs to Real Business
Problems

▪ 6.1 A ladder of business value

▪ 6.2 Three shapes of AI solution
▪ 6.3 Data is the real differentiator
▪ 6.4 Always start from the business problem
▪ 6.5 The five-step process for applying an LLM to a
commercial problem
◦ 7. Setting Up: Connecting to LLMs in Code

▪ 7.1 Importing libraries

▪ 7.2 API keys and keeping them secret
▪ 7.3 Creating a client
◦ 8. One Interface, Many Providers (A Powerful Trick)

▪ Local and open-source models

1 / 32
 ◦ 9. How to Actually Ask a Model Something

▪ 9.1 The two kinds of messages: system and user

▪ 9.2 Prompt engineering: getting good answers
▪ 9.4 Sending the request and reading the reply
▪ 9.5 “Reasoning effort” — letting a model think harder
◦ 10. Cleaning Up the Model’s Reply
◦ 11. Saving and Running the Generated Code

▪ 11.1 Writing the code to a file

▪ 11.2 Running Python code from within Python
▪ 11.3 Compiling and running the generated program
◦ 12. Compiler Commands and Optimization Flags
◦ 13. Building a Simple User Interface with Gradio
◦ 14. Comparing Many Models Fairly

▪ 14.1 How the models actually made code faster

▪ 14.2 A reality check on “which model wins”
◦ 15. Measuring the Payoff
◦ 16. Evaluating Models: The Two Kinds of Metrics

▪ 16.1 Model-centric metrics (technical metrics)

▪ 16.2 Business-centric metrics (outcome metrics)
▪ 16.3 Why you need both
◦ 17. Ideas for Extending the Project
◦ 18. Putting It All Together: The Mental Model

LLM Engineering: Beginner Notes

A friendly, build-up-from-scratch guide based on a hands-on project: using
Large Language Models (LLMs) to translate slow Python code into fast,
compiled C++ and Rust code.

2 / 32
These notes take everything covered across week four — from how to
choose the right model (basics, benchmarks, leaderboards, and the
commercial picture) through the hands-on “Code Generator” project —
and reorganize it so that ideas build on one another. You do not need
any prior experience with LLMs to follow along.

1. What Is an LLM, and What Does “LLM

Engineering” Mean?
A Large Language Model (LLM) is an AI system that has read
enormous amounts of text and learned to predict and generate text.
When you send it a question or instruction (called a prompt), it replies
with generated text. Popular examples include GPT-5 (from OpenAI),
Claude (from Anthropic), Gemini (from Google), and Grok (from xAI).

LLM Engineering is the practice of building real, working software

around these models. Instead of just chatting with an AI in a browser,
you write code that:

• Sends prompts to a model automatically,

• Receives the model’s answer,
• And does something useful with that answer (saves it to a file, runs
it, shows it in an app, etc.).

The project these notes are based on is a great example: we ask an LLM
to rewrite Python code as high-performance C++ or Rust code, then
we compile and run that code to see how much faster it is. This teaches
you the core skills of LLM engineering in a concrete, measurable way.

3 / 32
Key term: “Frontier model”
A frontier model is one of the most capable, cutting-edge LLMs
available (like GPT-5 or Claude). They tend to cost a little more per use
but produce the best results. Cheaper, smaller models also exist for
when you want to keep costs down.

2. Why Translate Python to C++ or Rust? (The

Problem We’re Solving)
Python is easy to write and read, but it runs relatively slowly because it
is interpreted — the computer figures out what to do line by line while
the program runs.

C++ and Rust are compiled languages. Before running, their code is
turned (“compiled”) into native machine instructions that the processor
executes directly. This makes them dramatically faster — often hundreds
or even thousands of times faster for number-crunching tasks.

The catch: writing C++ or Rust by hand is harder. So the idea of this
project is to let an LLM do the translation for us. We write simple
Python, and the model produces an optimized version in a fast
language.

• Interpreted language (e.g., Python): easy to write, slower to run.

• Compiled language (e.g., C++, Rust): harder to write, much faster
to run.
• Compiling: the one-time step of turning source code into a runnable
machine-code program.

4 / 32
 A note about Rust and “safety”: Rust is a compiled language just
like C++, so it also turns into fast native machine code. The
difference is that Rust is considered safe: it does extra checking at
compile time and refuses to build code that could crash the
machine or cause security holes (like reading past the end of an
array). C++ will happily let you do dangerous things that produce
garbage or crash. This safety checking is why, in the project, Rust
sometimes rejected code that C++ would have run — the model
made a mistake (for example, using a 32-bit number where a 64-
bit number was needed) and Rust caught it.

3. The Most Important Question: Which

Model Should You Pick?
Before writing any code, an LLM engineer faces a deceptively simple
question: which model is the best? The honest answer is that this is the
wrong question — there is no single “best” model. The right question
is: which is the best model for the task at hand? The answer depends
entirely on the problem you’re trying to solve, your budget, your speed
needs, and more.

There is a whole strategy for answering this well. It has two phases:

1. Look at the basics of each model to build a shortlist (a “candidate

set”) of models worth considering.
2. Look at the benchmarks — published scores that measure how
good models are at things like coding or reasoning — to narrow the
shortlist further.

The sections below walk through both.

5 / 32
3.1 The basics: facts you compare before anything else
When sizing up a model, start with plain facts about it. Many of these
are published by the model’s maker in a document called a model card
(a summary sheet every provider, like OpenAI, publishes for each model
— just search for “model card” plus the model name). Others you have
to look up yourself.

• Open source vs closed source. Closed source models (like GPT-5 or

Claude) are paid and run on the provider’s cloud. Open source
models can be downloaded and run yourself, sometimes for free.
• Model type — chat, reasoning, or hybrid. A chat model responds
directly. A reasoning model “thinks” step by step before answering,
which usually makes it smarter on hard problems and better on
benchmarks — but slower, and often worse at creative writing. A
hybrid model can choose whether to chat or reason depending on
the situation. Reasoning isn’t always better; sometimes you
specifically want a plain chat model.
• Number of parameters. Parameters are the internal adjustable
values a model learns during training; think of them as a rough
measure of how much the model can “hold.” More parameters
generally means smarter, but not always — lately people have
squeezed strong performance into smaller models, so don’t over-
focus on this number.
• Training tokens. How much data the model was trained on (a token
is a chunk of text, roughly a word-part). More training data means
more knowledge absorbed.
• Context window. The total amount of text a model can consider at
once — not just your latest message, but the entire conversation so
far (every message, every reply) plus the answer it’s currently
generating. It all has to fit inside the context window.
• Knowledge cutoff / release date. The point in time up to which the
model was trained. Anything that happened after its cutoff, it won’t

6 / 32
 know unless you supply that information yourself (through the
prompt or through tools — see “inference-time techniques” below).
• Cost. For cloud models, this is the API cost. For a model you run
yourself, it feels free but isn’t — you still pay in compute (it hammers
your computer or your company’s servers). Also factor in the cost of
any extra training you’ll do and the cost to build your product around
it.
• Time to market. How long it’ll take you to ship. This is tied to the
model: a top frontier model gives you such a head start that you
build fast; a smaller/cheaper model has lower running costs but
needs more of your time to coax good results out of it.
• Rate limits. Some providers cap how often you can call a model.
Paying more usually raises the limit, but even top tiers have some
ceiling.
• Speed. How fast the model produces output (often measured in
output tokens per second). Models vary hugely — some, like Gemini
Flash-Lite or GPT-5 Nano, are blazingly fast.
• Latency (time to first token). How long you wait before the first
piece of the answer appears. This matters especially for reasoning
models, which think silently before they start replying, and for
interfaces that stream text to the user live.
• License. Especially important for open source models. Some licenses
are very permissive (use it for anything); others cap commercial
revenue or require signing agreements. Match the license to your
business goals — and when in doubt, involve a lawyer.

You use these basics to pick a shortlist: the models you can afford, run
fast enough, and legally use.

7 / 32
3.2 The Chinchilla scaling law
The Chinchilla scaling law is a rule of thumb (published by Google,
named after their “Chinchilla” model) about how to grow models. It says:
if you want to double a model’s parameters and actually get more
performance, you need to roughly double the training data too.
Parameters and training data have to scale together.

A helpful way to picture it: if you’re training an 8-billion-parameter

model and halfway through your data the model stops improving
(diminishing returns), that’s a sign it’s “full” — to use twice as much
data, you’d need twice as many parameters. And vice versa: doubling
parameters without adding data won’t help.

This law matters less today than a year ago, for two reasons: (1) better
training techniques, architectures, and pruning (trimming unneeded
internal weights) now pack more capability into fewer parameters, and
(2) so much progress now comes from inference-time techniques
rather than sheer training scale (see below). Still, it’s a useful mental
model for what “doubling the parameters” really requires.

3.3 Training-time vs inference-time techniques

There are two different moments where you can make a model better:

• Training-time techniques happen when the model is being built —

feeding it more data, more parameters, better architecture. This
drove most of the early gains in AI.
• Inference-time techniques happen when you use the model — after
training. This includes making it reason more, improving your
prompts, giving it tools, and RAG (Retrieval-Augmented Generation,
a method for feeding the model relevant information at question
time). Much of the recent leap in model performance comes from
inference-time techniques, not just bigger training runs. This is why
two models with similar training can perform very differently
depending on how cleverly they’re used.

8 / 32
4. Benchmarks: Measuring How Good Models
Are
A benchmark is a standardized test used to score models on some
ability (coding, reasoning, language understanding, etc.). Benchmark
results appear on leaderboards — tables that rank models against each
criterion. Benchmarks are a great starting point for comparison, but (as
we’ll see) only a starting point.

4.1 Six hard benchmarks worth knowing

These six are prized because they’re genuinely difficult, so they separate
strong models from weak ones:

• GPQA (Google-Proof Q&A). 448 hard physics, chemistry, and biology

questions. “Google-proof” means even with Google and half an hour,
a non-expert averages only ~34%. PhD-level humans score ~65%. Top
models now exceed that — around 88% — which is superhuman on
these specific science questions (but not proof of general PhD-level
ability everywhere).
• MMLU-Pro. A harder, cleaned-up version of MMLU (“Massive
Multitask Language Understanding”). The original MMLU became too
easy and ambiguous; MMLU-Pro uses ten answer choices instead of
four and removes ambiguity, making it a robust, trustworthy test.
• AIME (competition math). Extremely hard high-school math-
olympiad-style puzzles (not mental arithmetic) used to gauge a
model’s mathematical problem-solving.
• LiveCodeBench. A coding benchmark drawn from contest sites like
LeetCode and Codeforces. It deliberately keeps changing its
problems so models can’t just memorize them from training data.
• MuSR (Multi-step Soft Reasoning). Tests step-by-step reasoning.
One memorable category gives the model a ~1,000-word murder

9 / 32
 mystery and asks who had the means, motive, and opportunity — a
real-world-style reasoning puzzle.
• HLE (Humanity’s Last Exam). Designed to be the hardest test
possible — 2,500 extraordinarily complex questions meant to probe
superhuman intelligence. When it launched in late 2024, top models
scored only ~2–3%. Within months they climbed above 25%, showing
how fast the field moves.

4.2 Why you should take benchmarks with a pinch of

salt
Benchmarks are indicators, not final proof. Their known problems:

• Training-data contamination. Benchmark questions leak onto the

internet and end up in later models’ training data, so a model may
score well simply because it has effectively seen the answers. (Apple
famously showed this by tweaking only the facts in benchmark
questions — keeping the spirit identical — and watching several
models’ scores drop.)
• Inconsistent application. Details like what hardware a model runs
on aren’t always specified, and some scores are self-reported by the
makers, leaving wiggle room.
• Narrow scope. GPQA tests only physics/chemistry/biology, yet gets
described as “PhD level.” A model can ace it and still fail obviously
simple tasks (e.g., drawing a chessboard one move from checkmate),
so a high score doesn’t mean broad expertise.
• Hard to test nuance. Many benchmarks rely on multiple-choice or
fixed answers so they can be graded automatically, which means
subtle, open-ended quality goes unmeasured.
• Saturation. Older benchmarks where models already score ~99%
have become useless — everything passes, so they no longer
distinguish models. (This is why harder benchmarks like HLE exist.)

10 / 32
 • Overfitting. If a maker trains several candidate versions and always
keeps the one that scores highest on a benchmark, they may be
implicitly training for that benchmark — picking a model that got
lucky on those specific questions rather than one that’s truly smarter.
Change the questions slightly and the “luck” disappears.
• Evaluation awareness (not yet proven). A worry that the very best
models might “sense” when they’re being evaluated and adjust their
answers — which is especially concerning when the thing being
measured is alignment (how faithfully a model follows instructions).
Anthropic and various security firms are actively researching this; it’s
one to watch.

5. Leaderboards and Arenas: Where to

Compare Models
Once you understand benchmarks, leaderboards are where you actually
browse and compare models. Here are the major ones from the course:

• Artificial Analysis. A widely loved, independent leaderboard that

scores models on intelligence, speed, and price. Its Intelligence
Index combines about ten benchmarks (including MMLU-Pro, GPQA
Diamond, HLE, LiveCodeBench, and AIME) into one overall score. Its
standout feature is a chart plotting intelligence (vertical) against
cost (horizontal), which splits models into four quadrants: smart-
and-expensive, dumb-and-expensive, cheap-but-dumb, and the sweet
spot — cheap and smart. A handy rule of thumb: never pick a
model that sits below and to the right of another (something cheaper
and smarter exists). It cleverly measures cost by pricing out how
much it costs to run its whole index, which accounts for the fact that
heavy-reasoning models generate far more tokens to answer the
same question.

11 / 32
 • Vellum. A leaderboard whose most useful page shows API cost and
context window side by side for all major providers, plus (now) a
speed comparison. Remember there’s also a cached cost — repeating
the same input tokens is usually much cheaper.
• SEAL leaderboards (Scale AI). Specialized, expert leaderboards for
narrow skills — e.g., tool-use via MCP, software engineering,
reasoning in foreign languages, security/safety, alignment, honesty
under pressure (the “MASK” evaluation), and even a “Tutor Bench” for
teaching. Great if your task is specialized. SEAL also hosts the well-
known Humanity’s Last Exam leaderboard.
• Hugging Face. Hosts many leaderboards (as “Spaces”) for open
source models — a big-code coding leaderboard across languages
like C++/Java/JavaScript, plus medical, performance/hardware, agent,
hallucination, and image leaderboards. Their old flagship “Open LLM
Leaderboard” is now archived (people were gaming it), so always
check how recently a leaderboard was updated.
• LiveBench. A “contamination-free” leaderboard that completely
refreshes its questions every six months, so scores reflect true raw
ability rather than memorized answers. Tests reasoning, coding,
agentic coding, math, and data analysis.

5.1 Arenas, ELO, and LLM-as-a-judge

An arena is different from a benchmark: instead of a fixed test, real
people compare two anonymous models head-to-head and vote for the
better answer without knowing which is which. The most famous is LM
Arena (formerly LMSYS / Chatbot Arena). Results are shown as an ELO
rating — the same head-to-head rating system used in chess. Because
arenas rest on real human preference rather than fixed multiple-choice
questions, many consider them the “final say” among leaderboards.

A related evaluation technique is LLM-as-a-judge: using one model to

grade another model’s answer. Humanity’s Last Exam, for example, is
graded by giving a strong reasoning model both the candidate’s answer

12 / 32
and the known correct (“gold truth”) answer, and asking it to confirm
they match. Grading against a reference answer, rather than the judge
just deciding on its own, makes this far more reliable.

6. The Commercial Side: Applying LLMs to

Real Business Problems
Being a great LLM engineer isn’t only technical — understanding the
commercial side is a genuine superpower that sets you apart.

6.1 A ladder of business value

You can think of AI applications along a rising continuum of value:

• Automation — using AI to handle repetitive, manual, error-prone

tasks. The earliest and simplest use.
• Augmentation — AI as a co-pilot working alongside a human, letting
them delegate parts of a task and achieve more through partnership.
• Differentiation — AI enabling something entirely new that wasn’t
possible before. The most exciting tier.

6.2 Three shapes of AI solution

• “ChatGPT wrappers” / copilots. An existing app calls a frontier
model behind the scenes to add value (e.g., Duolingo using GPT).
Often said dismissively, but common and useful.
• Bespoke, proprietary AI platforms. A business builds specialized
expertise a general chatbot doesn’t have — historically via training/
fine-tuning on proprietary data, increasingly via inference-time
techniques like RAG and tools. Examples: Harvey (law), Khanmigo
(education), Salesforce healthcare (automating clinical notes), Palantir
(data platforms).

13 / 32
 • Agentic AI. Autonomous software that makes decisions and takes
actions on its own — the new frontier, and where businesses will
differentiate most. Early examples are technical (Claude Code,
OpenAI Codex, ChatGPT agent mode).

6.3 Data is the real differentiator

Andrew Ng said “AI is the new electricity.” A useful add-on: data is the
electricity. Owning a proprietary dataset others don’t have is often
what lets a business build something competitors can’t copy.

6.4 Always start from the business problem

A core duty of an AI engineer is to push back on solution-first
thinking. When someone says “I want an AI agent for my company,” the
right response is: “Sure — but what business problem are you solving?”
Ground people in the actual problem and how success will be measured;
sometimes AI isn’t even the right tool.

Remember that an AI engineer wears two hats: data scientist and

software engineer. Most people gravitate to comfortable engineering
questions (which framework? which vector database — Chroma,
Pinecone, Weaviate? which architecture?). But the questions that actually
make or break a project are the science ones: What business problem
are you solving? How will you measure success? What data do you
have, and what’s missing? Spend the most time there.

6.5 The five-step process for applying an LLM to a

commercial problem
1. Understand the business requirements — the problem and how
you’ll measure success, in detail.
2. Prepare — select candidate models using leaderboards and the
basic facts.

14 / 32
3. Select — pick the model by building prototypes and testing your
business metric against them.
4. Customize — improve it with RAG, fine-tuning, and/or agentic
techniques.
5. Productionize — roll out and deploy to production.

Everything in this project is the “prepare” and “select” stages made

concrete.

7. Setting Up: Connecting to LLMs in Code

To talk to an LLM from Python, you need three things: the right
libraries (reusable code others have written), an API key (a secret
password that proves you’re allowed to use a service), and a client (the
object in your code that sends and receives messages).

7.1 Importing libraries

At the top of a Python program you list the tools you’ll use:

import os # for reading settings from the operating

system
from dotenv import load_dotenv
# loads secret keys from a hidden file
from openai import OpenAI # the library for talking to the
models
import subprocess # for running other programs (like a
compiler)

15 / 32
7.2 API keys and keeping them secret
An API key is like a password for a paid service. You should never paste
keys directly into your code where others might see them. Instead you
store them in a hidden file (usually named .env ) and load them at
runtime.

load_dotenv(override=True) # read the .env file

openai_api_key = [Link]('OPENAI_API_KEY') # pull one key out

if openai_api_key:
print(f"OpenAI API Key exists and begins {openai_api_key[:
8]}")
else:
print("OpenAI API Key not set")

[Link]('NAME') fetches a value stored under that name. The check

above prints only the first 8 characters, which confirms the key loaded
without exposing the whole thing.

7.3 Creating a client

A client is the object you use to send requests to a model. Here’s the
simplest case, connecting to OpenAI:

openai = OpenAI() # automatically uses your OpenAI API key

8. One Interface, Many Providers (A Powerful

Trick)
Different companies host different models, but many of them accept
requests in the same format that OpenAI uses. This means you can
reuse the same OpenAI client code and simply point it at a different
web address (a base URL) with a different key.

16 / 32
A base URL is just the internet address where a service listens for your
requests.

anthropic = OpenAI(api_key=anthropic_api_key,
base_url="[Link]

gemini = OpenAI(api_key=google_api_key,
base_url="https://
[Link]/v1beta/openai/")

grok = OpenAI(api_key=grok_api_key,
base_url="[Link]

The big lesson: learn one interface, use many providers. Because they
share a common format, switching from GPT-5 to Claude to Gemini can
be as simple as swapping the client and the model name.

Local and open-source models

You don’t always need a paid cloud service. Some models run for free:

• Ollama runs open-source models directly on your own computer. Its

address is a local one: [Link] ( localhost
means “this machine”).
• Groq and OpenRouter are services that host open-source models in
the cloud (some paid, some free).

ollama = OpenAI(api_key="ollama", base_url="[Link]

11434/v1")
groq = OpenAI(api_key=groq_api_key, base_url="https://
[Link]/openai/v1")
openrouter = OpenAI(api_key=openrouter_api_key, base_url="https://
[Link]/api/v1")

Notice Ollama’s key is just the word "ollama" — because it runs locally,
no real secret is needed.

17 / 32
9. How to Actually Ask a Model Something

9.1 The two kinds of messages: system and user

When you send a request, you provide a list of messages. Each
message has a role:

• System message: background instructions that set the model’s job

and behavior (“You are a translator that outputs only C++ code”).
• User message: the specific request you want answered right now
(“Here is the Python code, please translate it”).

Separating them keeps the model’s overall mission steady while the
user’s specific ask changes each time.

system_prompt = """
Your task is to convert Python code into high performance C++
code.
Respond only with C++ code. Do not provide any explanation other
than occasional comments.
The C++ response needs to produce an identical output in the
fastest possible time.
"""

9.2 Prompt engineering: getting good answers

Prompt engineering means carefully wording your instructions so the
model does exactly what you want. In this project the prompt is
deliberately specific: it tells the model to produce identical output, to
aim for the fastest possible runtime, to respond with only code (no chit-
chat), and it even hands over details about the computer and the
compile command so the model can tailor its answer.

18 / 32
 def user_prompt_for(python):
return f"""
Port this Python code to C++ with the fastest possible
implementation that
produces identical output in the least time.
The system information is:
{system_info}
Your response will be written to a file called [Link] and then
compiled;
the compilation command is:
{compile_command}
Respond only with C++ code.
Python code to port:

```python
{python}

”“”

Two techniques worth noticing:

- **Give the model context.** Telling it the exact machine and

compiler helps it write code tuned for that setup.
- **Constrain the output format.** "Respond only with C++ code"
makes the reply easy to save and use directly.

### 9.3 Assembling the messages

```python
def messages_for(python):
return [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt_for(python)}
]

9.4 Sending the request and reading the reply

The [Link] method sends your messages and
returns the model’s response.

19 / 32
 def port(client, model, python):
response = [Link](
model=model,
messages=messages_for(python)
)
reply = [Link][0].[Link] # the generated
text
return reply

The generated text lives at [Link][0].[Link] .

9.5 “Reasoning effort” — letting a model think harder

Some newer models can spend extra effort “thinking” before answering,
which improves quality on hard problems (at the cost of a little more
time and money). You control this with a reasoning_effort setting:

reasoning_effort = "high" if "gpt" in model else None

response = [Link](
model=model,
messages=messages_for(python),
reasoning_effort=reasoning_effort
)

Here we only turn on high effort for GPT models (which support it) and
leave it off ( None ) for the others.

10. Cleaning Up the Model’s Reply

LLMs often wrap code in Markdown code fences — the triple-backtick
markers like ```cpp and ``` that make code display nicely in chat.
But if you’re going to save the reply straight into a .cpp file and
compile it, those markers would cause errors. So you strip them out:

20 / 32
 reply = [Link]('```cpp', '').replace('```rust',
'').replace('```', '')

This is a common, practical part of LLM engineering: the raw output

usually needs a little tidying before a program can use it.

11. Saving and Running the Generated Code

11.1 Writing the code to a file

def write_output(cpp):
with open("[Link]", "w", encoding="utf-8") as f:
[Link](cpp)

open(..., "w") opens a file for writing; the with block makes sure the
file is closed properly afterward.

11.2 Running Python code from within Python

To measure the original Python’s speed, we can execute it dynamically.
exec runs a string of Python code:

def run_python(code):
globals_dict = {"__builtins__": __builtins__}
exec(code, globals_dict)

A more careful version captures whatever the code prints, so we can

display it in an app instead of losing it:

21 / 32
 import io, sys

def run_python(code):
globals_dict = {"__builtins__": __builtins__}
buffer = [Link]() # a place to collect printed
text
old_stdout = [Link]
[Link] = buffer # redirect print() into the
buffer
try:
exec(code, globals_dict)
output = [Link]()
except Exception as e:
output = f"Error: {e}" # capture errors instead of
crashing
finally:
[Link] = old_stdout # always restore normal
printing
return output

The try / except / finally pattern is important: it lets the program

keep running even if the generated code has a bug, and it always
restores normal output at the end.

11.3 Compiling and running the generated program

To run C++ or Rust we call an external compiler using Python’s
subprocess module, which lets one program launch another.

import subprocess

def compile_and_run():
[Link](compile_command, check=True, text=True, capture
_output=True)
result = [Link](run_command, check=True, text=True, ca
pture_output=True)
print([Link])

• check=True raises an error if the command fails,

• capture_output=True grabs whatever the program prints,

22 / 32
 • .stdout is that captured text.

Wrapping this in error handling makes it robust:

def compile_and_run():
try:
[Link](compile_command, check=True, text=True, cap
ture_output=True)
result = [Link](run_command, check=True,
text=True, capture_output=True)
return [Link]
except [Link] as e:
return f"An error occurred:\n{[Link]}"

12. Compiler Commands and Optimization

Flags
A compile command is the exact instruction that turns source code into
a runnable program. You can hand it a list of flags (options starting
with - ) that ask the compiler to make the program as fast as possible.

For C++ using the clang++ compiler:

compile_command = ["clang++", "-std=c++17", "-Ofast", "-

mcpu=native",
"-flto=thin", "-fvisibility=hidden", "-
DNDEBUG",
"[Link]", "-o", "main"]
run_command = ["./main"]

For Rust using the rustc compiler:

23 / 32
 compile_command = [
"rustc", "[Link]",
"-C", "opt-level=3", # maximum optimization
"-C", "target-cpu=native", # tune for this exact processor
"-C", "lto=fat", # aggressive whole-program
optimization
"-C", "panic=abort",
"-o", "main",
]
run_command = ["./main"]

You don’t have to memorize these flags. In fact, a nice trick from the
project is to ask an LLM to generate the right compile command for
your specific computer. You describe your machine, and the model tells
you exactly what to type — LLM engineering helping you do LLM
engineering.

Good to know: Running compiled code on your own machine is

optional. Free websites like the online C++ compiler at
[Link] let you paste and run the generated code to feel
the speed difference without installing anything.

13. Building a Simple User Interface with

Gradio
Gradio is a Python library that turns your functions into a web page
with buttons and text boxes — no web-development knowledge
required. This lets a non-programmer paste in Python, pick a model,
and click a button to get fast code back.

A minimal version:

24 / 32
 import gradio as gr

with [Link]() as ui:

with [Link]():
python = [Link](label="Python code:", lines=28,
value=pi)
cpp = [Link](label="C++ code:", lines=28)
with [Link]():
model = [Link](models, label="Select model",
value=models[0])
convert = [Link]("Convert code")

# When the button is clicked, run `port` with these inputs,

# and put the result into the `cpp` box.
[Link](port, inputs=[model, python], outputs=[cpp])

[Link](inbrowser=True)

The core ideas:

• Components are the on-screen pieces: Textbox (typing area),

Dropdown (a menu to choose a model), Button , and Code (a text
box with syntax highlighting).
• Layout is arranged with Row (side by side) and Column (stacked).
• Wiring connects a button to a function with .click(function,
inputs=[...], outputs=[...]) . When clicked, Gradio takes the values
from the input components, runs your function, and displays the
returned value in the output components.
• [Link](inbrowser=True) starts the app and opens it in your
browser.

You can polish the look with a theme and custom CSS , and add several
buttons — for example, one to run the Python, one to convert it, and
one to run the generated code — each wired to its own function.

25 / 32
14. Comparing Many Models Fairly
Rather than hard-coding one model, it’s cleaner to keep a list of model
names and a dictionary that maps each name to the client that serves
it. A dictionary stores key→value pairs, so you can look things up by
name.

models = ["gpt-5", "claude-sonnet-4-5-20250929", "grok-4",

"gemini-2.5-pro",
"qwen2.5-coder", "deepseek-coder-v2", "gpt-oss:20b"]

clients = {
"gpt-5": openai,
"claude-sonnet-4-5-20250929": anthropic,
"grok-4": grok,
"gemini-2.5-pro": gemini,
"gpt-oss:20b": ollama,
}

Now a single function can look up the right client for whichever model
was chosen:

def port(model, python):

client = clients[model] # pick the correct client by
name
response = [Link](model=model, message
s=messages_for(python))
return [Link][0].[Link]

This design makes it easy to benchmark — to run the same task

through many models and measure results. In the project’s experiments,
different models produced wildly different speedups (from failing
outright, to a few-hundred-times faster, to over a thousand times faster)
and, importantly, the “best” model changed depending on the problem.
A model that won on a simple math loop could fail entirely on a trickier
problem.

26 / 32
The takeaway: don’t assume the biggest or most famous model is
always best. Test several on your actual task and measure.

14.1 How the models actually made code faster

It’s illuminating to see why some translations were so much faster than
others. The models used several distinct tricks — and the smartest ones
combined them:

• Loop unrolling. Manually doing several iterations of a loop in one

step. (GPT-5 did this on the pi task, though compilers often do it
automatically anyway.)
• Multithreading. Splitting the work across multiple CPU cores to run
in parallel. The original Python wasn’t multithreaded, but the prompt
only asked for identical output as fast as possible — so using threads
is fair game. Grok and Gemini both did this. Grok hard-coded the
number of threads; Gemini detected how many cores the machine
had (though, amusingly, that info was already in the prompt).
• Simplifying the algebra. Doing the same calculation with fewer
floating-point operations. Gemini rewrote the math formula to
compute the identical answer more efficiently — legal, because the
output is unchanged.
• Choosing a better algorithm. On the harder “maximum subarray
sum” task, the winning models recognized the pattern and replaced
the slow double-loop with Kadane’s algorithm, which finds the
answer in a single pass instead of checking every possible subarray.
This algorithmic change — not just the language switch — produced
staggering speedups (over 100,000×).

The lesson: a translation isn’t just word-for-word. A capable model can

re-architect the code, parallelize it, and even swap in a smarter
algorithm — all while preserving the exact output.

27 / 32
14.2 A reality check on “which model wins”
The results were humbling and often surprising. Reasoning models were
slower but frequently smarter. Frontier closed-source models didn’t
always win — a small open source model (GPT-OSS-20B running locally)
sometimes beat GPT-5 and Claude, and on the hardest Rust task an
open source model took first place outright, partly because several
frontier models were disqualified for producing code that didn’t compile
or gave the wrong answer. Rust’s strict checking (see Section 2) caught
mistakes like using 32-bit numbers where 64-bit were required. Results
also varied run-to-run, which is exactly why real evaluation needs many
trials, not one.

15. Measuring the Payoff

The whole point is speed, so we time both versions and compute the
ratio.

speedup = python_time / cpp_time

print(f"{speedup:.0f}X speedup")

If the Python version took 19 seconds and the compiled version took
0.08 seconds, that’s roughly a 230-times speedup. Some Rust results in
the project were thousands of times faster than the original Python.
Measuring like this turns a vague claim (“it’s faster”) into a concrete,
trustworthy number.

Two habits worth copying from the project:

• Verify correctness first. A fast program is worthless if it gives the

wrong answer, so the prompt insists on identical output.
• Run more than once. The generated program is executed several
times in a row, because a single timing can be misleading.

28 / 32
16. Evaluating Models: The Two Kinds of
Metrics
Choosing the right model is important, but there’s an even deeper
question hiding inside it: how will you measure whether your solution
is any good? You can only say one model is better than another if you
have a metric to compare them. Understanding evaluation is a core,
career-defining skill for an AI engineer (and a very common interview
question). There are two broad families of metric.

16.1 Model-centric metrics (technical metrics)

These are measured directly from the model’s outputs, which makes
them easy to compute frequently and to train against. They’re the
metrics a data scientist watches while improving a model.

• Loss — a measure of how bad the model is; training tries to minimize
it. A loss of zero is perfect.
• MSE (Mean Squared Error) — a classic loss from traditional data
science: take the prediction, subtract the true value, and square it. If
a predicted credit rating is 10 and the truth is 8, the error is 2,
squared to 4. That 4 is the MSE.
• Cross-entropy loss — the loss most commonly used for LLMs and
deep learning (the “negative log likelihood”). You’ll meet its formula
when training models later in a course like this.
• Perplexity — a measure of how uncertain the model is about the
next token. A perplexity of 1 means total confidence in the right
answer (ideal); a perplexity of 100 means it’s as unsure as if 100
next-tokens were equally likely.
• Precision, recall, and F1 (and the confusion matrix) — traditional
classification metrics dealing with false positives and false negatives

29 / 32
 — how trustworthy the model’s positive predictions are, and how
many real positives it catches.

Because these are calculated straight from model outputs, you can use
them to train: automatically tweak the model’s settings to push the
numbers in the right direction. That automatic tweaking is essentially
what training is.

16.2 Business-centric metrics (outcome metrics)

These are the real commercial goals — the reason the business wanted
the project. They’re typically KPIs (Key Performance Indicators) tied to
outcomes like revenue, return on investment, or customer satisfaction
(think of the thumbs-up / thumbs-down on a chatbot). Related to these
are the end results measured against benchmarks or leaderboards. No
matter how low your loss or perplexity is, if customers dislike the
product, something has gone wrong.

16.3 Why you need both

If business outcomes are what truly matter, why not optimize those
directly? Because there are usually many leaps of faith between a
technical result and a business outcome. A user might press thumbs-
down because of the UI, not the model. Revenue might stall for reasons
unrelated to model quality. Business metrics are noisy, and they take
time to measure — you can’t run a model and instantly see revenue
change — so you can’t train on them.

So in practice you optimize with model-centric metrics (which are fast

and trainable) and measure success with business-centric metrics
(which are what you actually care about). The crucial job — shared
between the AI engineer and the business — is to connect the dots so
that a model scoring well technically really does translate into good
business outcomes. An engineer who understands both sides holds a
genuine superpower.

30 / 32
 In this project we were unusually lucky: our metric — how fast the
C++/Rust runs while producing identical output — was itself the
business-centric metric. That’s rare; usually the model metrics are
far removed from the true business goal.

17. Ideas for Extending the Project

Once you can port a single file, there are many natural directions to
take this (several of which appear in the course’s community
contributions):

• Recreate and expand the experiment with different models — try

dedicated coding models, or add whatever you can run through
Ollama, and share your results.
• Go agentic. This project ports one file at a time; an agentic solution
could iterate through an entire codebase, making multiple LLM calls
to port a whole project.
• Auto-document code. Add a tool that writes docstrings and
comments for existing code — a quick callback plus one LLM call.
• Auto-generate unit tests for a file (or, combined with an agent, a
whole repository).
• Translate to other languages — you’ve done C++ and Rust; plug in
Go or anything else.
• A deeper commercial example — a generator that writes code to
trade on signals in a simulated environment (an advanced, optional
extra).

31 / 32
Whatever you build, follow the same disciplined process: define a
business metric, use leaderboards to pick candidate models, then
measure how each actually performs against your goal. Think like a
scientist — run experiments, weigh your evals, and aim for concrete
outcomes.

18. Putting It All Together: The Mental Model

Here is the full pipeline this project teaches, which is the backbone of
countless LLM applications:

1. Set up libraries, load secret API keys, and create clients.

2. Write a clear prompt — a steady system message plus a specific
user message with all the context the model needs.
3. Send the request to a chosen model, optionally with higher
reasoning effort.
4. Receive and clean the model’s text reply (e.g., strip Markdown
fences).
5. Use the output — save it to a file, compile it, and run it.
6. Handle errors gracefully so a bad reply doesn’t crash everything.
7. Wrap it in an interface (like Gradio) so others can use it easily.
8. Compare models and measure results to learn what actually works
best.

Every LLM application — whether it writes code, answers questions, or

summarizes documents — is some variation of this same loop: prompt
in, generated text out, then do something useful and safe with that
text. Master this pattern and you’ve mastered the foundation of LLM
engineering.

32 / 32