Contents

• LLM Engineering: Beginner’s Notes

◦ 1. What Is an LLM (and What Does “Engineering” One

Mean)?

▪ Two ways to reach a model

▪ What’s actually inside an LLM: the Transformer
▪ The context window
◦ 2. API Keys and Environment Variables

▪ Keeping keys secret with a .env file

◦ 3. Your First API Call: The Messages Format

▪ Common settings you can pass with a request

▪ Every model has its own “character”
▪ A note on the “training cutoff”
◦ 4. Talking to Many Providers with One Style of Code

▪ Running models locally with Ollama

▪ Routers and abstraction layers
◦ 5. Understanding Tokens, Cost, and Caching

▪ Tokens
▪ Prompt caching
◦ 6. Reasoning Models and “Thinking” Effort
◦ 6b. Hallucinations and the Importance of Context
◦ 7. Multi-Turn Conversations: Giving a Model Memory

▪ Fun demonstration: two chatbots talking to each other

▪ Scaling to three or more participants: the “whole history
in the prompt” pattern
◦ 8. Prompting Techniques: Steering the Model with Words
◦ 9. Streaming Responses
◦ 10. Building a User Interface with Gradio

▪ The core idea

▪ What Gradio is actually doing behind the scenes

1 / 37
 ▪ Handy options for .launch()
▪ A note on dark mode
▪ Customizing inputs and outputs
▪ The three kinds of Gradio interface
◦ 11. Building a Chatbot
◦ 12. Tools: Letting the Model Take Actions

▪ Step 1: Write an ordinary function

▪ Step 2: Describe the function so the model understands
it
▪ Step 3: Offer the tools and handle the model’s request
▪ Two practical notes for real projects
◦ 13. Connecting a Tool to a Real Database
◦ 14. Going Multi-Modal: Images and Audio

▪ Generating images
▪ Generating speech (text-to-speech)
◦ 15. Bringing It All Together: A Multi-Modal Assistant
◦ 16. A First Look at Agents and Agentic AI

▪ What is an “agent”?
▪ Common hallmarks of agents
▪ Tools and structured outputs are two sides of the same
coin
▪ Frameworks do the heavy lifting
◦ Where This Fits in the Bigger Picture
◦ Quick Glossary

2 / 37
LLM Engineering: Beginner’s
Notes
A friendly, self-contained guide to building applications with Large
Language Models (LLMs). These notes start with the absolute basics
and gradually build up to a complete, multi-modal AI assistant. You
do not need prior experience—every term is explained the first
time it appears.

1. What Is an LLM (and What Does

“Engineering” One Mean)?
A Large Language Model (LLM) is a computer program trained
on huge amounts of text. Its core skill is simple to describe: given
some text, it predicts what text should come next. From that single
ability comes everything else—answering questions, writing code,
holding conversations, and more.

A frontier model is one of the most capable, cutting-edge LLMs

available. Examples you’ll meet in these notes include OpenAI’s
GPT models, Anthropic’s Claude models, and Google’s Gemini
models.

LLM engineering means writing code that talks to these models

and wraps them in useful applications. You typically don’t build or
train the model yourself. Instead, you send it text and receive text
back, then combine that with your own logic, data, and user
interface to create something valuable.

Two ways to reach a model

• Chat UI: the website or app where you type to a model directly
(like the ChatGPT website). Good for humans.

3 / 37
 • API (Application Programming Interface): a way for your
code to talk to the model over the internet. This is what LLM
engineering is built on. An API lets your program send a request
and get a structured response back automatically.

What’s actually inside an LLM: the Transformer

Almost every modern LLM is built on an architecture called the
Transformer, introduced in a famous 2017 research paper titled
“Attention Is All You Need.” You don’t need to build one, but a little
intuition helps you talk to these models well.

• Before Transformers, language models used designs like

LSTMs (Long Short-Term Memory networks) that read text
strictly word by word and struggled to connect words that were
far apart.
• The Transformer’s big idea is “attention.” Instead of
reading in a rigid left-to-right march, the model can look across
the whole sentence or paragraph at once and figure out which
words matter most for understanding each other word. Self-
attention is the mechanism that lets every word “pay
attention” to every other word to work out meaning in context.
• A helpful mental picture: a Transformer is like a very smart e-
reader that can glance over an entire passage, decide which
earlier words are important, and use them to understand what
comes next.

You may hear a few component names when people describe

Transformers:

• Input embeddings — turning words/tokens into lists of

numbers the model can process.
• Positional encoding — adding information about where each
word sits, since attention looks at everything at once and would
otherwise lose word order.

4 / 37
 • Encoder and decoder stacks — the original design had two
halves. Most of today’s text-generating models are decoder-
only.
• Output layer — produces the final prediction: the next token.

At the end of all this machinery, the model still does just one thing:
predict the most likely next token. Everything impressive it
does emerges from doing that extremely well.

The context window

The context window is the maximum amount of text (measured
in tokens) a model can consider at once—everything you send plus
everything it generates has to fit inside it. It’s effectively the
model’s short-term working memory for a single request. Bigger
context windows let you feed in more background material (long
documents, more conversation history) in one go.

2. API Keys and Environment Variables

To use a model’s API, you need an API key—a secret password-like
string that identifies you and lets the provider bill you for usage.
Most providers require you to add a small amount of money to your
account before the key works, though some (like Google Gemini or
Groq) offer free tiers.

Keeping keys secret with a .env file

You should never paste API keys directly into your code, because
you might accidentally share them. Instead, you store them in a
hidden file called .env :

5 / 37
 OPENAI_API_KEY=xxxx
ANTHROPIC_API_KEY=xxxx
GOOGLE_API_KEY=xxxx

An environment variable is a named value that lives outside

your code, in your computer’s environment. The .env file holds
these values. A small Python library called python-dotenv loads
them into your program:

import os
from dotenv import load_dotenv

# Read the .env file and make its values available

load_dotenv(override=True)

# Pull one value out by name

openai_api_key = [Link]('OPENAI_API_KEY')

if openai_api_key:
print(f"Key found, begins with {openai_api_key[:8]}")
else:
print("Key not set")

Important: every time you edit your .env file, save it and re-run
load_dotenv(override=True) so your program picks up the changes.

3. Your First API Call: The Messages

Format
Almost every modern LLM API uses the same idea: you send a list
of messages, and the model replies with a new message.

Each message is a small dictionary with two parts:

• role — who is “speaking.” The three roles are:

6 / 37
 • system — background instructions that set the model’s behavior
and personality (the “rules of the game”).
• user — what the human says.
• assistant — what the model says back.
• content — the actual text.

Here is a complete first call using OpenAI’s Python library:

from openai import OpenAI

openai = OpenAI() # automatically finds OPENAI_API_KEY from your

environment

messages = [
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "Tell me a joke about
programming"}
]

response = [Link](
model="gpt-4.1-mini",
messages=messages
)

print([Link][0].[Link])

A few things to notice:

• model="gpt-4.1-mini" picks which model to use. Smaller/”mini”

or “nano” models are cheaper and faster; larger ones are more
capable but cost more.
• The reply is buried inside the response object at
[Link][0].[Link] . This exact path shows up
constantly, so it’s worth memorizing.

7 / 37
Common settings you can pass with a request
Besides model and messages , most APIs accept extra options that
shape the response. A few you’ll meet often:

• temperature — controls how random or “creative” the output is,

usually on a scale from 0 to about 2. A low temperature (near
0) makes the model more focused, predictable, and repeatable—
good for factual or structured tasks. A higher temperature (e.g.
1.0–1.5) makes it more varied and surprising—good for
brainstorming or creative writing, but more prone to going off
track.
• stream=True — return the answer piece by piece as it’s
generated (covered in the Streaming section).
• response_format (JSON mode) — force the model to reply with
valid JSON instead of free text. This is very handy when your
program needs to parse the answer reliably rather than read it
as prose. (For example, response_format={"type": "json_object"}
with OpenAI.)
• reasoning_effort — on newer reasoning models, dial how hard
the model “thinks” (covered in the Reasoning section).

Every model has its own “character”

Different models don’t just differ in raw ability—they differ in
personality and style, shaped by how each company trained them.
A fun way to see this is to pose the same dilemma to several
models. For instance, ask each to play the classic prisoner’s
dilemma (cooperate for a shared reward, or betray for a bigger
solo reward). In practice, models trained with a strong emphasis on
alignment and safety (the practice of making a model behave
helpfully and in line with human values) often lean toward
cooperation, while others lean toward the “steal” strategy. Knowing

8 / 37
a model’s tendencies helps you pick the right one for a task—and
reminds you that model choice is a real engineering decision, not
just a cost question.

A note on the “training cutoff”

An LLM only knows about things that existed when its training data
was collected. If you ask “What is today’s date?” it will often guess
wrong or reveal how recent its knowledge is. This moment—the
most recent date it reliably knows about—is called its training
cutoff. Anything after that, the model simply hasn’t seen.

4. Talking to Many Providers with One

Style of Code
There are many model providers: OpenAI (GPT), Anthropic (Claude),
Google (Gemini), DeepSeek, Groq, xAI (Grok), and more. Learning a
brand-new library for each would be tedious.

Happily, many providers offer an OpenAI-compatible endpoint—

they accept requests in the same format OpenAI uses. The OpenAI
Python library lets you change the base_url (the internet address
requests are sent to) so the same code can talk to different
providers:

9 / 37
 from openai import OpenAI

# Each provider just needs its own key and its own base_url
gemini = OpenAI(
api_key=google_api_key,
base_url="[Link]
openai/"
)
deepseek = OpenAI(
api_key=deepseek_api_key,
base_url="[Link]
)

# Now call them the same way as before

response = [Link](
model="gemini-2.5-flash-lite",
messages=messages
)

Providers also publish their own native libraries if you prefer them:

# Anthropic's own library (note: it uses max_tokens and a slightly

different response path)
from anthropic import Anthropic

client = Anthropic()
response = [Link](
model="claude-sonnet-4-5-20250929",
messages=[{"role": "user", "content":
"Describe the color blue in one sentence"}],
max_tokens=100
)
print([Link][0].text)

10 / 37
Running models locally with Ollama
You can also run smaller open-source models on your own
computer—no internet, no per-use cost, and your data stays
private. A tool called Ollama makes this easy. Once Ollama is
running, you download a model and talk to it through the same
OpenAI-style code, just pointed at your own machine:

# Ollama exposes an OpenAI-compatible endpoint on your local

machine
ollama = OpenAI(api_key="ollama", base_url="[Link]
11434/v1")

response = [Link](
model="llama3.2",
messages=messages
)

Routers and abstraction layers

As your projects grow, you may want to switch between dozens of
models without rewriting code. Two kinds of helpers make this
easier:

• Routers like OpenRouter are a remote service you connect

out to. You send your request to the router, and it forwards
(“routes”) it to whichever provider you named. The big
convenience: one account and one bill instead of separate
keys and top-ups for every provider. You pick a model by writing
provider/model , e.g. z-ai/glm-4.5 .

• Abstraction layers (also called frameworks) like LiteLLM or

LangChain are code that runs on your own machine. You call
one consistent function, and the library figures out how to talk
to each provider. Nothing extra sits between you and the
provider—you still use your own keys.

11 / 37
So the difference is where the switching happens: a router does it
remotely as a service; an abstraction layer does it locally in your
code. Both aim at the same goal—use many models without
rewriting code.

• LiteLLM is deliberately lightweight: one completion() function,

the same [Link][0].[Link] shape as OpenAI,
and easy switching between models. It even reaches managed
cloud services—you can call AWS Bedrock, Azure, or Google
Vertex models just by changing the model string (e.g.
bedrock///. ). It also makes it easy to track tokens and cost per
call, which is useful for monitoring the “unit economics” of a
production system.
• LangChain is powerful and widely used but much heavier, with
many abstractions to learn. It’s worth knowing it exists; you’ll
likely meet it again when building more complex applications
(especially retrieval systems).

# LiteLLM: one function works across providers by naming

"provider/model"
from litellm import completion

response = completion(model="openai/gpt-4.1", messages=messages)

print([Link][0].[Link])

# The same call reaches managed services just by changing the

prefix:
# completion(model="bedrock/[Link]-...",
messages=messages)

# LangChain: a heavier framework, shown here making the same

simple call
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-5-mini")
response = [Link](messages)
print([Link])

12 / 37
5. Understanding Tokens, Cost, and
Caching

Tokens
Models don’t read letters or whole words—they read tokens, which
are chunks of text (a token is roughly ¾ of a word on average).
Everything is measured in tokens: how much you send, how much
you get back, and how much you pay.

Most APIs report token usage in the response so you can track
cost:

print(f"Input tokens: {[Link].prompt_tokens}")

print(f"Output tokens: {[Link].completion_tokens}")
print(f"Total tokens: {[Link].total_tokens}")

You are billed separately for input tokens (what you send) and
output tokens (what the model generates). Output tokens usually
cost more.

Prompt caching
If you send the same large chunk of text over and over (for
example, an entire book as background context), you can save
money with prompt caching. The provider stores that repeated
portion so it doesn’t have to reprocess it every time.

Key ideas that apply across providers:

• Caching works on exact prefix matches—the repeated content

must be at the beginning of your prompt and identical each
time. So put fixed content (instructions, examples, big

13 / 37
 documents) first, and variable content (the user’s specific
question) last.
• Different providers price it differently. With OpenAI, cached input
is much cheaper (around 4× cheaper). With Anthropic, you pay
a bit more the first time to “prime” the cache, then far less on
reuse. Gemini supports both automatic (“implicit”) and manual
(“explicit”) caching.

The practical lesson: structure your prompts with the big,

unchanging parts up front.

6. Reasoning Models and “Thinking”

Effort
Newer models can be told to reason before answering—to
internally work through a problem step by step, much like showing
your work on a math test. This is sometimes called inference-time
scaling: spending more computation while answering to get better
results on hard problems.

Some models let you dial how much effort they put in:

# "minimal" is fast and cheap; higher effort tackles harder

problems more reliably
response = [Link](
model="gpt-5-nano",
messages=[{"role": "user", "content": "A tricky logic
puzzle//."}],
reasoning_effort="minimal"
)

14 / 37
Simple questions do fine with minimal effort. Genuinely hard
puzzles benefit from more reasoning effort (at higher cost and
slower speed). Part of the engineer’s job is matching the model and
effort level to the difficulty of the task.

There are two independent ways to get better answers on hard

problems, and both work:

• Training-time scaling — use a bigger, more capable model.

• Inference-time scaling — give the same model more
reasoning effort so it thinks longer before answering.

A tiny model with minimal reasoning might get a tricky puzzle

wrong; dialing up either the model size or the reasoning effort can
fix it.

6b. Hallucinations and the Importance

of Context
A hallucination is when a model states something false as if it
were true. This happens because an LLM’s core drive is to produce
the most likely next tokens—not to look up verified facts. When a
model is backed into a corner and doesn’t actually know the
answer, it will often still respond, and it tends to do so confidently
and incorrectly, because a confident-sounding answer looks like
plausible text. Watch for this: fluent and sure-sounding does not
mean correct.

The single most powerful fix is to give the model the relevant
information inside the prompt. If you ask a model a detailed
question about a specific book and it guesses wrong, then re-ask

15 / 37
the same question but paste the book’s text in as context, it will
typically answer correctly. The model wasn’t “smarter” the second
time—it simply had the facts in front of it.

# Without context: the model may confidently invent an answer

question = [{"role": "user", "content": "In Hamlet, what is the
reply to 'Where is my father?'"}]

# With context: paste the source text in, and accuracy jumps
question[0]["content"] += "\n\nFor context, here is the full text:
\n\n" + full_play_text

This idea—retrieving relevant text and feeding it to the model so it

can answer accurately—is the seed of a major technique called
RAG (Retrieval-Augmented Generation), which you’ll encounter
in more advanced work. For now, the takeaway is simple: when
accuracy matters, supply the facts in the prompt rather than
hoping the model remembers them.

7. Multi-Turn Conversations: Giving a

Model Memory
An LLM has no memory of its own—each API call starts fresh. To
hold a conversation, you must send the entire history back every
time. You do this by growing the messages list: after each
exchange you append the user’s message and the assistant’s reply,
then send the whole list again.

16 / 37
 messages = [
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "My name is Sam."},
{"role": "assistant", "content": "Nice to meet you, Sam!"},
{"role": "user", "content": "What's my name?"} # model can
answer because history is included
]

Because the history travels with every call, the model appears to
“remember” the conversation. This message-list structure is the
foundation of every chatbot you’ll build.

Fun demonstration: two chatbots talking to each

other
A great way to internalize this is to make two models converse. You
give each its own system message (its personality)—for example
one argumentative and snarky, the other polite and eager to find
common ground—and keep two running lists of what each has said.
On each turn, you build a messages list from one model’s point of
view—labeling its own past lines as assistant and the other
model’s lines as user —and ask it for its next reply. Loop that back
and forth and you get an automatic dialogue. It’s a memorable way
to see how roles and history drive a conversation, and how
different system prompts produce different characters.

Scaling to three or more participants: the

“whole history in the prompt” pattern
The role-flipping trick above works nicely for two speakers, but it
gets tangled with three or more. A cleaner, more general approach
is to stop relying on the assistant/user role structure to represent
who-said-what, and instead put the entire conversation so far
into a single user prompt as plain text, telling the model which
participant it is:

17 / 37
 system_prompt = """
You are Alex, a snarky, argumentative chatbot.
You are in a conversation with Blake and Charlie.
"""

user_prompt = f"""
You are Alex, in conversation with Blake and Charlie.
The conversation so far is:
{conversation}
Now respond with what you would say next, as Alex.
"""

This “just hand the model the full context and ask for the next
line” pattern is far more coherent for multi-party chats—and,
importantly, it generalizes well beyond this toy example to many
real tasks where you assemble context and ask for a single next
step. (If you only have access to one model, you can still run a
multi-way conversation by giving the same model different system
prompts to play different personas.)

8. Prompting Techniques: Steering the

Model with Words
The prompt is the text you give the model. How you word it
dramatically changes the output. A few core techniques:

• System prompt for context and tone. Use the system

message to tell the model who it is, what it knows, and how it
should behave. Example: “You are a helpful assistant in a
clothes store. Gently encourage customers toward items on
sale.”
• Zero-shot prompting. You just ask, giving no examples. The
model relies entirely on its training.

18 / 37
 • One-shot / few-shot prompting. You include one or more
example answers inside the prompt to show the model the style
or format you want. Example: “If the customer says ‘I’m looking
for a hat’, you could reply, ‘Wonderful—we have lots of hats,
including several on sale.’” Showing an example is often far
more effective than describing it.
• Adding context dynamically. You can adjust the prompt
based on the user’s specific message before sending it:

relevant_system_message = system_message
if 'belt' in [Link]():
relevant_system_message += " We don't sell belts; point out
other items on sale instead."

The takeaway: prompting is a real skill. Clear instructions, useful

context, and concrete examples are your main levers for controlling
behavior.

9. Streaming Responses
By default, an API call waits until the whole answer is ready, then
returns it all at once. For long replies this feels slow. Streaming
instead sends the answer piece by piece as it’s generated—the
familiar effect of text appearing word by word.

In Python, streaming pairs naturally with a generator: a function

that uses yield to hand back partial results one at a time instead
of a single return at the end.

19 / 37
 def stream_gpt(prompt):
messages = [
{"role": "system", "content": system_message},
{"role": "user", "content": prompt}
]
stream = [Link](
model="gpt-4.1-mini",
messages=messages,
stream=True # ask for streamed chunks
)
result = ""
for chunk in stream:

# each chunk carries a small piece of text (or None, hence "or
''")
result += [Link][0].[Link] or ""
yield result # hand back the answer-so-far

Note that with streaming, the text arrives at

[Link][0].[Link] (a delta, meaning “the new bit”)
rather than the usual [Link] .

10. Building a User Interface with

Gradio
So far everything has lived in code. Gradio is a Python library that
turns your functions into a real web interface with almost no effort
—perfect for demos, prototypes, and internal tools.

The core idea

You write a normal Python function, and Gradio builds input and
output boxes around it. When the user submits, Gradio calls your
function and displays what it returns.

20 / 37
 import gradio as gr

def shout(text):
return [Link]()

# "textbox" in, "textbox" out; .launch() starts a local web server

[Link](fn=shout, inputs="textbox",
outputs="textbox").launch()

The key concept is the callback function: you hand Gradio a

function (here, shout ), and Gradio promises to call it back
whenever the user takes an action, feeding your inputs in and
displaying your outputs. Gradio doesn’t know or care what the
function does—whether it shouts text or calls an LLM, it’s just “the
function to call.” This is why you can swap shout for a function
that calls GPT and nothing else changes: to Gradio it’s still just a
callback.

What Gradio is actually doing behind the scenes

It feels magical, but it’s really three sensible steps:

1. Builds a front-end. From your Python description of the UI,

Gradio generates a web interface (using the Svelte front-end
framework, which compiles down to plain JavaScript).
2. Starts a web server. When you call .launch() , Gradio spins
up a small web server (built on the Starlette framework) that
listens on a free port—typically 7860 , ticking up to the next port
if that one’s busy—and serves your interface to anyone who
visits.
3. Wires up routes to your callbacks. It connects the “Submit”
button to a backend route that runs your Python function and
returns the result.

That’s the whole trick—three good ideas combined to give the

“instant app” feeling.

21 / 37
Handy options for .launch()
• inbrowser=True — automatically opens a browser window locally
as soon as you launch.
• share=True — creates a temporary public link (valid for about
a week) so others can try your app. Cleverly, the app is hosted
remotely but still calls the callback back on your own machine
using a technique called HTTP tunneling (similar to a tool
called ngrok). Because that’s a fairly advanced network
behavior, corporate firewalls and antivirus software may block it
—skip this step if you’re on a locked-down work network. For a
permanent, fully above-board deployment, Gradio offers gradio
deploy instead.

• auth=("username", "password") — adds a simple login screen.

You can also pass a list of username/password tuples for
multiple users. (Note: never keep real passwords in plain text
like this in production—at minimum use your .env , and ideally a
proper hashed password store. But it’s a very easy way to gate
a shared demo.)

A note on dark mode

Gradio shows in light or dark mode based on the user’s own
browser/system settings. You can force dark mode with a small
snippet of JavaScript passed via js=//. , but Gradio recommends
against it: respecting the user’s choice matters for accessibility.

Customizing inputs and outputs

You can label boxes, add example inputs, and render the output as
Markdown (formatted text with headings, bold, lists) instead of
plain text:

22 / 37
 message_input = [Link](label="Your message:", lines=7)
message_output = [Link](label="Response:")

[Link](
fn=message_gpt,
title="Chat with GPT",
inputs=[message_input],
outputs=[message_output],
examples=["Explain transformers to a beginner"]
).launch()

Gradio supports streaming too: if your function is a generator that

yield s, Gradio updates the screen live as the text arrives. You can
also add a dropdown to let users pick between models, and
Gradio will pass the selection into your function.

The three kinds of Gradio interface

• [Link] — the quickest option for a simple “inputs →
function → outputs” screen.
• [Link] — a ready-made chatbot layout (message box,
conversation view, history handled for you).
• [Link] — full manual control over layout and behavior, for
when you need a custom UI with many components wired
together.

11. Building a Chatbot

[Link] makes a full chatbot from a single callback
function. That function must have a specific shape:

23 / 37
 def chat(message, history):
//.
return reply

• message is the newest thing the user typed.

• history is the conversation so far, which Gradio manages for
you automatically.

The type="messages" flag tells Gradio to hand you the history in the
same role/content format that OpenAI uses, which is
convenient because you’re usually about to pass it straight to an
LLM. One small gotcha: Gradio’s history entries can carry extra
fields (like metadata ) that OpenAI ignores but some other providers
(e.g. Gemini) reject. Rebuilding each entry with just role and
content avoids that error—harmless for OpenAI, essential for
others.

Your job is to combine the system prompt, the history, and the new
message into a proper messages list, call the model, and return its
reply:

def chat(message, history):

# Gradio gives history as role/content dicts already—normalize

just in case
history = [{"role": h["role"], "content": h["content"]} for h
in history]
messages = (
[{"role": "system", "content": system_message}]
+ history
+ [{"role": "user", "content": message}]
)
response = [Link](model=MODEL, message
s=messages)
return [Link][0].[Link]

[Link](fn=chat, type="messages").launch()

24 / 37
To make replies stream in live, turn chat into a generator with
yield , exactly as in the streaming section. With just this, you have
a working conversational assistant with memory and a web UI.

12. Tools: Letting the Model Take

Actions
By default an LLM can only produce text. Tools (also called
function calling) let the model use functions you wrote—to look
something up, do a calculation, or take an action in the real world.

It’s less spooky than it sounds. The model can’t run your code by
itself. Instead, when it decides a tool would help, it asks you to run
it and tells you what arguments to use. Your code runs the
function, sends the result back, and the model incorporates it into
its answer.

Step 1: Write an ordinary function

ticket_prices = {"london": "$799", "paris": "$899", "tokyo":

"$1400"}

def get_ticket_price(destination_city):
print(f"Tool called for {destination_city}")
return ticket_prices.get(destination_city.lower(), "Unknown
price")

Step 2: Describe the function so the model

understands it
The model needs a structured description—its name, what it does,
and what arguments it takes—so it knows when and how to request
it:

25 / 37
 price_function = {
"name": "get_ticket_price",
"description": "Get the price of a return ticket to the
destination city.",
"parameters": {
"type": "object",
"properties": {
"destination_city": {
"type": "string",
"description": "The city the customer wants to
travel to",
},
},
"required": ["destination_city"],
"additionalProperties": False
}
}

tools = [{"type": "function", "function": price_function}]

Step 3: Offer the tools and handle the model’s

request
You pass tools=tools in your API call. If the model wants a tool, its
response comes back with finish_reason /= "tool_calls" . You then
run the requested function, package the result as a message with
role: "tool" , add it to the conversation, and call the model again
so it can finish its answer:

26 / 37
 def handle_tool_calls(message):
responses = []
for tool_call in message.tool_calls: # there may be
several at once
if tool_call.[Link] /= "get_ticket_price":
args = [Link](tool_call.[Link])
price = get_ticket_price([Link]("destination_city"))
[Link]({
"role": "tool",
"content": price,
"tool_call_id": tool_call.id # links the
result to the request
})
return responses

def chat(message, history):

history = [{"role": h["role"], "content": h["content"]} for h
in history]
messages = [{"role": "system", "content": system_message}] + h
istory + [{"role": "user", "content": message}]
response = [Link](model=MODEL, message
s=messages, tools=tools)

# A while-loop handles a model that needs several rounds of

tools before answering
while [Link][0].finish_reason /= "tool_calls":
message = [Link][0].message
results = handle_tool_calls(message)
[Link](message) # the model's tool request
[Link](results) # your tool results
response = [Link](model=MODEL, mes
sages=messages, tools=tools)

return [Link][0].[Link]

A few details that make this click:

• The tool role. Alongside system , user , and assistant , there’s

a fourth role: tool . You send your function’s result back as a
message with role: "tool" , and its tool_call_id must match
the id of the request the model made—that’s how the model
links a result to the specific call it asked for.

27 / 37
• Two round-trips. The first call to the model is where it says
“please run this tool.” After you run it and append the result,
the second call is where the model reads that result and writes
its final answer. (Printing out the messages list right before the
second call is a great way to see the whole system → user →
assistant(tool request) → tool(result) sequence.)
• Multiple tools in one response: loop over message.tool_calls
—the model may request several at once. Handling only the first
one ( tool_calls[0] ) is a common early bug.
• Multiple rounds: use a while loop (not a single if ) so the
model can call tools repeatedly, one after another, before it’s
ready to answer. LLMs tend to stop on their own, but if you’re
worried about an endless loop you can add a maximum-
iterations guard.

Two practical notes for real projects

• Avoid long if/elif chains. Dispatching tools with if name /=
"get_ticket_price": //. gets clunky as you add more tools. A
more “pythonic” approach is to look up the function by its name
automatically (e.g. from a dictionary of {name: function} ) and
call it, handling the case where the name isn’t found.
• Streaming plus tools is fiddly. Streaming the reply and
supporting tool calls at the same time is possible but messy
(you have to inspect the finish reason and piece together the
tool JSON as it streams). In practice, most real projects don’t
hand-write all this JSON and dispatch logic—they use an agent
framework (such as the OpenAI Agents SDK) that builds the
tool descriptions, selects tools, and handles streaming for you.
Writing it by hand once, as above, is valuable for understanding;
frameworks are the shortcut afterward.

28 / 37
With tools, your assistant graduates from answering questions to
doing things—checking databases, calling other services, even
making bookings.

13. Connecting a Tool to a Real

Database
Hard-coding data in a Python dictionary is fine for learning, but real
applications store data in a database. Python includes SQLite, a
simple database saved as a single file—no separate server needed.

import sqlite3

DB = "[Link]"

# Create a table once (if it doesn't already exist)

with [Link](DB) as conn:
[Link]('CREATE TABLE IF NOT EXISTS prices (city TEXT
PRIMARY KEY, price REAL)')
[Link]()

# A tool that reads from the database

def get_ticket_price(city):
with [Link](DB) as conn:
cursor =
[Link]('SELECT price FROM prices WHERE city = ?', ([Link]
r(),))
result = [Link]()
return f"Ticket price to {city} is ${result[0]}" if
result else "No price data available"

The ? placeholder safely inserts values into the query. Swapping

the dictionary-based tool for this database-backed version requires
no change to the rest of your chatbot—the model still just calls
get_ticket_price . This shows a powerful pattern: tools are the
bridge between an LLM and your real systems.

29 / 37
14. Going Multi-Modal: Images and
Audio
Multi-modal means working with more than just text—images,
audio, and more. Importantly, these use different APIs and
different models from the chat-completions API you’ve used so
far. The same providers offer them, but you call a separate
endpoint for each capability.

Generating images
Image generation uses a dedicated image endpoint. You pass a
model, a prompt describing the picture, a size, and a response
format. Two model choices you’ll see are OpenAI’s DALL·E 3
(older but quick, easy, and no special approval needed) and newer
image models built on GPT (higher quality but sometimes requiring
extra approval to use via the API).

import base64
from io import BytesIO
from PIL import Image # PIL / Pillow is a Python library for
handling images

def artist(city):
image_response = [Link](
model="dall-e-3", # or a newer
image model
prompt=f"A vibrant pop-art image of a vacation in {city}",
size="1024x1024",
n=1, # how many
images to make
response_format="b64_json"
)
# The image comes back encoded as text (base64); decode it
into a real image
image_data = base64.b64decode(image_response.data[0].b64_json)
return [Link](BytesIO(image_data))

30 / 37
A couple of things worth knowing:

• base64 is a way of representing binary data (like an image) as

plain text so it can travel inside a normal API response. You
decode that text back into real image bytes, then open it with
Pillow (PIL).
• How image generators work under the hood: most use
diffusion models—a family of models that start from random
noise and gradually “denoise” it into a coherent picture that
matches your prompt. This is a different kind of model from the
token-predicting LLMs; it’s specialized for images. (You’ll meet
diffusion models directly in more advanced, open-source work.)
• Cost: generating images costs real money per image (a few
cents each)—use them sparingly while experimenting.

Generating speech (text-to-speech)

Audio uses yet another endpoint. TTS stands for Text-To-Speech:
you give it text and a voice, and it returns spoken audio. Different
voices (e.g. “onyx”, “alloy”, “coral”) give different vocal styles.

def talker(message):
response = [Link](
model="gpt-4o-mini-tts",
voice="onyx", # try different voices for
different styles
input=message
)
return [Link] # audio bytes you can play back

31 / 37
15. Bringing It All Together: A Multi-
Modal Assistant
The final project combines everything: a chatbot with memory,
tools that read a database, spoken replies, and generated images—
wrapped in a custom UI.

For a UI this rich, [Link] gives you the control to place each
component (chat window, image panel, audio player, text box) and
wire events yourself:

with [Link]() as ui:

with [Link]():
chatbot = [Link](height=500, type="messages")
image_output = [Link](height=500)
with [Link]():
audio_output = [Link](autoplay=True)
with [Link]():
message = [Link](label="Chat with our AI Assistant:")

# When the user submits: first add their message, then run the
assistant
[Link](
put_message_in_chatbot, inputs=[message, chatbot],
outputs=[message, chatbot]
).then(
chat, inputs=chatbot, outputs=[chatbot, audio_output, imag
e_output]
)

[Link]()

The chat function here does the full pipeline: run the model with
tools, loop through any tool calls, get the final text reply, turn that
reply into speech with talker , and—if a city came up—generate a
picture with artist . It returns all three (updated chat, audio,
image) at once so the UI can display them together. Notice how

32 / 37
many LLM calls are stitched together here—one that decides to use
a tool, one that produces the final text, plus the image and audio
calls—all coordinated into a single experience.

This is a meaningful milestone: an assistant that converses,

remembers, looks things up in real data, speaks, and illustrates. It’s
also a first step toward agentic systems, which the next section
explains.

16. A First Look at Agents and Agentic

AI
The multi-modal assistant you just built—stitching together several
LLM calls and tools to accomplish one goal—points toward one of
the biggest ideas in the field: agents.

What is an “agent”?
There’s a running joke that everyone defines “agent” differently.
Two definitions have become the most common:

1. An LLM controls the workflow. Rather than you hard-coding

every step, the model itself decides what should happen and
when. This is tied to the idea of autonomy—the model making
decisions. (Stay grounded: an LLM only ever generates tokens.
“The LLM controls the workflow” really means the tokens it
generates are interpreted by your code to steer what happens
next.)
2. An LLM runs tools in a loop to achieve a goal. This is the
newer, “keep thinking and acting until done” sense—the feeling
you get from tools like coding agents that work through a task
over many steps.

33 / 37
An older, looser definition simply called anything with multiple
chained LLM calls “agentic.” That still lingers, but the two
definitions above are the ones that have really taken hold.

Common hallmarks of agents

The more of these a system has, the more “agentic” it is:

• Memory / persistence — it remembers things across steps or

sessions.
• Planning — it can lay out a sequence of actions in advance.
• Autonomy — it uses its generated tokens to decide what to do
next.
• Orchestration — it coordinates multiple steps, and can even
call other LLMs.
• Tool use — it’s equipped with tools so it can take real actions.

Tools and structured outputs are two sides of

the same coin
An agent can decide what to do next in two equivalent-feeling
ways: by calling a tool, or by producing structured output (like
JSON) that your code reads and acts on. Experienced practitioners
note these are essentially the same idea—both are the model
emitting something your program interprets to drive the next step.
This is why JSON mode (from earlier) matters so much for agent-
style systems.

Frameworks do the heavy lifting

Building full agents by hand is a lot of plumbing, so people use
agent frameworks (such as the OpenAI Agents SDK, and others
you’ll meet later) that handle tool descriptions, tool selection,
looping, streaming, memory, and planning for you. The assistant

34 / 37
you built is a hand-made “flavor” of agentic behavior—a great way
to understand what those frameworks are doing under the hood
before you lean on them.

Where This Fits in the Bigger Picture

The material in these notes—frontier model APIs, UIs with Gradio,
tools, and a multi-modal assistant—is one stage of a longer journey
in LLM engineering. Typical next topics build on this foundation:

• Open-source models you run yourself (e.g. via Hugging Face).

• Choosing the right LLM for a given job.
• RAG (Retrieval-Augmented Generation) — the “give the
model the facts it needs” idea, done systematically.
• Fine-tuning — adapting a model to your own data, for both
frontier and open-source models.
• Agentic AI — building the autonomous, tool-using systems
introduced above.

The recurring lesson throughout: the best way to learn is to build—

take each technique and apply it to a real problem you care about.

Quick Glossary
• LLM — a model that predicts and generates text; the engine of
everything here.
• Frontier model — a top-tier, cutting-edge LLM.
• Transformer — the neural-network architecture behind modern
LLMs (from the 2017 paper “Attention Is All You Need”).

35 / 37
• Attention / self-attention — the mechanism that lets a model
weigh how words relate to each other to understand context.
• Context window — the maximum amount of text (in tokens) a
model can consider at once.
• API — the interface your code uses to talk to a model.
• API key — your secret credential for using an API.
• Environment variable / .env — a safe place to store secrets
outside your code.
• Token — the chunk of text models read and are billed by.
• Prompt — the text you send the model.
• System / user / assistant / tool roles — labels marking who
(or what) produced each message; tool carries a tool’s result
back to the model.
• Zero-/one-/few-shot — prompting with zero, one, or several
examples.
• Temperature — a setting controlling how random vs. focused
the output is.
• JSON mode ( response_format ) — forcing the model to reply
with valid JSON your code can parse.
• Hallucination — a confident but false answer; often fixed by
supplying the facts in the prompt.
• RAG (Retrieval-Augmented Generation) — retrieving
relevant text and feeding it to the model so it answers
accurately.
• Streaming — receiving the answer piece by piece as it’s
generated.
• Delta — the small new piece of text in each streamed chunk.
• Generator / yield — a Python function that returns results one
at a time.
• Reasoning effort — how much a model “thinks” before
answering.

36 / 37
• Training-time vs. inference-time scaling — getting better
answers via a bigger model vs. more thinking on the same
model.
• Prompt caching — reusing repeated prompt content to save
cost.
• Router — a remote service (e.g. OpenRouter) that forwards
your request to many providers under one account.
• Abstraction layer / framework — local code (e.g. LiteLLM,
LangChain) giving one interface to many providers.
• Gradio — a library for building quick web UIs from Python
functions.
• Callback function — the function you hand Gradio (or a
framework) to be called when the user acts.
• HTTP tunneling — the technique behind Gradio’s share=True
that lets a public link call code on your machine.
• Tool / function calling — letting the model request that your
code run a function.
• SQLite — a simple file-based database built into Python.
• Multi-modal — working with images and audio, not just text.
• base64 — a way to represent binary data (like an image) as
plain text inside an API response.
• Diffusion model — the type of model behind most image
generators; builds pictures by removing noise step by step.
• TTS (Text-To-Speech) — turning text into spoken audio.
• Agent / Agentic AI — a system where an LLM decides the
workflow and/or runs tools in a loop to reach a goal.
• Alignment — training/prompting a model to behave helpfully
and in line with human values.

37 / 37