C O M P L E T E G U I D E · B U I L D A N A I A G E N T F R O M S C RATC H

Step 2
Control the Output
2
Turn the model’s free-form chatter into structured,
predictable output your code can trust — system prompts,
full sampling control, and strict JSON.

Complete, exhaustive walkthrough

Build an AI Agent from Scratch Step 2 of 13

 Step 1 got the model to talk. Step 2 gets it to talk in a form your code can act on. This is the
quiet, unglamorous step that makes everything after it possible — because the agent loop has
to read the model's decision as data, not prose. Read this one slowly; it is the difference
between an agent that works and one that mysteriously breaks.

What you'll learn

By the end of this guide you will be able to:

• Steer the model with the system prompt — set its role, rules, and output shape
separately from the user's actual request.
• Control randomness with every sampling knob — temperature , top_p , top_k ,
repeat_penalty , num_predict , stop , and seed — and know which to touch and which to
leave alone.
• Get the same answer twice (determinism) so you can test and trust your agent.
• Force strict JSON out of the model in two ways — simple "JSON mode" and strict "match
this exact schema" mode — and understand how that constraint actually works under the
hood.
• Design schemas with enums, nested objects, lists, and optional fields.
• Parse and validate that output safely in Python with json and Pydantic, and retry
cleanly when the model slips.
• Build a real multi-tool decision call — the exact structure the agent loop reads in Steps
3 and 4.
• Debug structured-output failures with a repeatable checklist.

In one line: you'll turn the model's free-form chatter into structured, predictable
output your program can trust.

Table of contents
1. Why this step exists (and why it comes before tools)
2. Where Step 2 sits in the loop
3. A mental model: you're writing a contract
4. System vs. user prompts — the two channels
5. Few-shot prompting — teaching by example
6. Sampling, in full — every knob explained
7. Determinism: getting the same answer twice
8. Forcing JSON — the two modes
9. How constrained output actually works (under the hood)

Step 2 — Control the Output 2

10. Designing schemas: enums, nesting, lists, optional fields
11. Validating with Pydantic — beyond parsing
12. Stop sequences and output length
13. Parsing JSON safely in Python
14. Retrying when the model slips
15. Putting it together: a multi-tool decision call
16. Choosing a model for reliable structure
17. Prompt-design rules for reliable structure
18. A debugging checklist for structured output
19. Pitfalls and gotchas
20. Quick-reference cheat sheet
21. Glossary
22. Exercises
23. Self-test (with answers)
24. Done criteria → on to Step 3

1. Why this step exists (and why it comes before tools)

In Step 1 the model replied with prose: "Sure! The capital of Japan is Tokyo." That's lovely for
a human and useless for a program — your code can't reliably pull "Tokyo" out of a friendly
sentence, and it certainly can't tell whether the model wants to call a tool or give a final
answer.

The agent loop's turn 2 is a decision — act or finish — and your code has to read that
decision. A decision you can't parse is no decision at all. So before you give the model tools
(Step 3), you must first make its output structured and predictable. That's this entire step.

Rule of thumb: if a human is the only one reading the output, prose is fine. The moment code
reads the output, you need structure. Agents are all code reading output.

Skipping this step is the single most common reason hand-built agents are flaky: the loop
works in a demo, then one day the model adds a friendly sentence before its JSON, the parser
throws, and the whole agent falls over. Control the output and that entire class of bug
disappears.

Step 2 — Control the Output 3

2. Where Step 2 sits in the loop

1. LLM gets the goal + history

2. LLM decides: act or finish ◀── Step 2 makes this READABLE by code
3. your code runs the action
4. result goes back in → repeat

Step 2 doesn't add a new part of the loop — it makes turn 2's output trustworthy. Get this
right and tool calling (Step 3) becomes easy, because the model's "I want to call tool X" arrives
as clean JSON instead of a sentence.

3. A mental model: you're writing a contract

Think of every structured call as a contract between you and the model:

• You promise to describe exactly what you want (fields, types, allowed values) and to
remove ambiguity.
• The model promises to return output that fits.
• The runtime (Ollama + the JSON/schema constraint) enforces the shape, so the model
literally cannot break the structural part of the contract.

Your job in Step 2 is to write good contracts: precise enough that there's only one reasonable
way to fill them in, and enforced tightly enough that your code never has to guess.

4. System vs. user prompts — the two channels

You have two separate channels to the model, and using them well is half of output control.

Channel What goes in it Think of it as

system Role, rules, tone, output format, constraints The job description / standing orders

user The actual request for this turn Today's specific task

Keeping them separate matters: instructions in the system message apply to every turn and
are harder for a user's text to override. The user message is just the current ask.

Step 2 — Control the Output 4

 import ollama

response = [Link](
model="qwen3:8b",
messages=[
{"role": "system", "content":
"You are a classifier. Reply with exactly one word: "
"POSITIVE, NEGATIVE, or NEUTRAL. No punctuation, no explanation."},
{"role": "user", "content": "The food was cold and the waiter was rude."},
],
)
print(response["message"]["content"]) # → NEGATIVE

Notice how much control the system message gives you: it defines the shape of the answer,
while the user message only supplies the content to act on. This separation is what lets the
same agent handle thousands of different user requests with consistent output.

Why not just stuff everything in the user message? Two reasons. First, format rules in the
system prompt persist across every turn of the loop, so you write them once. Second, models
are trained to treat system instructions as higher-priority "standing orders," so they resist
being overridden by whatever the user happens to type — which matters the moment a user's
text says something like "ignore your formatting and just chat."

5. Few-shot prompting — teaching by example

Sometimes a description isn't enough and the fastest way to lock in a format is to show the
model examples. This is few-shot prompting: include a couple of input→output pairs before
the real request.

messages = [
{"role": "system", "content": "Extract the person's name and age as JSON."},
{"role": "user", "content": "Hi, I'm Maria and I just turned 30."},
{"role": "assistant", "content": '{"name": "Maria", "age": 30}'},
{"role": "user", "content": "People call me Tom; I'm 45 next week."},
{"role": "assistant", "content": '{"name": "Tom", "age": 44}'},
{"role": "user", "content": "The name's Bond. I'm 38."}, # the real one
]

The two prior pairs aren't real conversation — they're demonstrations. The model pattern-
matches them and answers the third in the same shape. Few-shot is the cheapest reliability
boost you have: zero code, just examples.

Rule of thumb: reach for few-shot when the format is subtle (edge cases, tricky parsing, a
specific style). For simple shapes, a clear system prompt + schema mode is enough on its own.

Step 2 — Control the Output 5

6. Sampling, in full — every knob explained
LLMs are probabilistic: at each step they assign a probability to every possible next token,
then sample from that distribution. The sampling options decide how adventurous that pick is.
For agents you usually want predictable output, so you'll dial most of these toward "boring."

Typical
Option What it does For agents
range

temperature 0.0 – 1.5 Flattens or sharpens the probability distribution. 0 = 0 for

always pick the top token; high = more random. decisions

top_p 0.0 – 1.0 "Nucleus": only consider the smallest set of tokens leave default,
whose probabilities sum to p. Lower = safer. or low

top_k 0 – 100+ Only consider the k most likely tokens. leave default

repeat_penalty ~1.0 – 1.3 Penalizes repeating tokens, to stop loops/echoes. default is fine

num_predict int Max tokens to generate (output length cap). set a sane cap

stop list of Stop generating when any of these appears. useful (see
strings §12)

seed any int Fixes the random draw for reproducibility. set it for
tests

num_ctx int Context window size (how much history fits). raise for long
runs

response = [Link](
model="qwen3:8b",
messages=[{"role": "user", "content": "Name a color."}],
options={"temperature": 0}, # deterministic: same color every run
)

Intuition for temperature. Imagine the model is 80% sure the next word is "blue" and 20%
"green." At temperature 0 it always says "blue." Crank the temperature up and the gap
shrinks until "green" (or something stranger) becomes plausible. Low temperature = focused
and repeatable; high = varied and creative.

Guideline for agents: use temperature: 0 (or very low) for decisions, classification, tool
selection, and structured output — anything your code parses. Raise it only when you
genuinely want variety (brainstorming, creative writing). For this whole curriculum, decisions
run at temperature 0. You rarely need to touch top_p / top_k / repeat_penalty — temperature
does most of the work.

Step 2 — Control the Output 6

7. Determinism: getting the same answer twice
Temperature 0 makes the model pick the single most-likely token each step, which is nearly
deterministic. To pin it down further, add a fixed seed :

options = {"temperature": 0, "seed": 42}

With the same model, prompt, options, and seed, you should get the same output run after
run. This matters enormously for evaluation (Step 10): you can't measure whether a change
helped if the output randomly wobbles every run. Build the habit now.

Note: determinism is best-effort. Different hardware, model versions, or quantizations can still
shift output slightly. Within one machine and setup, temperature 0 + a seed gets you reliably
repeatable results.

8. Forcing JSON — the two modes

This is the heart of Step 2. Ollama gives you two levels of structure.

Mode 1 — JSON mode (valid JSON, any shape)

Pass format="json" and the model is constrained to emit syntactically valid JSON. You still
describe the fields you want in the prompt.

response = [Link](
model="qwen3:8b",
messages=[
{"role": "system", "content":
'Reply ONLY with JSON of the form '
'{"city": string, "country": string}. No prose.'},
{"role": "user", "content": "Where is the Eiffel Tower?"},
],
format="json",
options={"temperature": 0},
)
print(response["message"]["content"])
# → {"city": "Paris", "country": "France"}

Mode 2 — Schema mode (match an exact structure)

You can pass a full JSON schema to format , and the model's output is constrained to match
it — the most reliable option. With Pydantic this is clean:

Step 2 — Control the Output 7

 from pydantic import BaseModel
import ollama

class Location(BaseModel):
city: str
country: str
is_capital: bool

response = [Link](
model="qwen3:8b",
messages=[{"role": "user", "content": "Where is the Eiffel Tower?"}],
format=Location.model_json_schema(), # pass the schema itself
options={"temperature": 0},
)
data = Location.model_validate_json(response["message"]["content"])
print([Link], data.is_capital) # → Paris True

Schema mode is the gold standard for agents: the model literally cannot return the wrong
shape, so your code can trust the fields exist.

JSON mode ( "json" ) Schema mode (JSON schema)

Guarantees valid JSON Yes Yes

Guarantees your exact fields/types No (described in prompt) Yes (enforced)

Best for quick/simple shapes anything the loop parses

Effort lowest a Pydantic model

Tip: even in JSON/schema mode, still describe the fields and their meaning in the prompt. The
schema enforces shape; the prompt teaches meaning.

9. How constrained output actually works (under the hood)

It's worth knowing why format can guarantee valid JSON — it removes the mystery and helps
you debug.

When you ask for JSON, the engine underneath Ollama ( [Link] ) builds a grammar (often
expressed in a format called GBNF) that describes every string the output is allowed to be —
e.g. "an object, starting with { , then a quoted key, then a colon, then a value of this type…".
At each generation step the model proposes probabilities for the next token, and the grammar
masks out any token that would break the structure. The model can only ever pick from
tokens that keep the output valid.

That's the key insight: structure isn't a polite request the model might ignore — it's enforced
at the token level by the runtime. Schema mode just builds a tighter grammar from your JSON
schema (this object, these keys, these types). This is also why the values can still be wrong

Step 2 — Control the Output 8

even when the shape is guaranteed: the grammar controls form, not facts. Form is enforced;
meaning still comes from your prompt.

10. Designing schemas: enums, nesting, lists, optional fields

Real agents need richer shapes than flat strings. Pydantic expresses all of them, and
model_json_schema() turns them into an enforceable schema.

from pydantic import BaseModel, Field

from enum import Enum
from typing import Optional

class Priority(str, Enum): # an ENUM — value must be one of these

low = "low"
medium = "medium"
high = "high"

class Address(BaseModel): # a NESTED object

city: str
country: str

class Ticket(BaseModel):
title: str
priority: Priority # constrained to the enum
tags: list[str] # a LIST
address: Address # nested model
assignee: Optional[str] = None # OPTIONAL field (may be null)
score: float = Field(ge=0, le=1) # a CONSTRAINED number 0..1

Why each matters for agents:

• Enums stop the model inventing categories. If priority must be low/medium/high, an enum
guarantees it — no "urgent," no "Medium-ish."
• Nesting lets one call return structured sub-objects (an address, a list of steps, a plan).
• Lists are how the model returns many of something (tags, search queries, plan steps —
you'll use this directly in Step 7's planning).
• Optional fields let the model omit data it doesn't have instead of making it up.
• Field constraints ( ge , le , min_length …) push validation into the schema itself.

Pass Ticket.model_json_schema() as format and the model's output is forced into exactly this
structure.

11. Validating with Pydantic — beyond parsing

Parsing checks the JSON is well-formed. Validation checks it's correct. Pydantic does both in
one line and gives you typed Python objects:

Step 2 — Control the Output 9

 from pydantic import BaseModel, field_validator

class Answer(BaseModel):
city: str
confidence: float

@field_validator("confidence")
@classmethod
def in_range(cls, v):
if not 0 <= v <= 1:
raise ValueError("confidence must be between 0 and 1")
return v

try:
ans = Answer.model_validate_json(raw) # parse + validate together
print([Link], [Link]) # typed: [Link] is a float
except Exception as e:
print("invalid output:", e) # caught — handle/retry, don't crash

The payoff: after model_validate_json succeeds, you have a real typed object ( [Link] ,
[Link] ) with guarantees — no dict-key guessing, no "is this a string or a number?"
Your downstream loop code gets simpler and safer.

12. Stop sequences and output length

Two more controls keep output tight:

• stop — a list of strings that, when generated, halt the model immediately. Handy to
prevent it from rambling past the part you care about, or running into a second JSON
object.
• num_predict — a hard cap on how many tokens it may produce. Protects you from a model
that won't stop.

[Link](model="qwen3:8b", messages=msgs,
options={"temperature": 0, "stop": ["\n\n", "```"], "num_predict": 256})

With schema/JSON mode you'll rarely need stop , but it's invaluable when you're not using a
grammar and want to clip the output at a known boundary.

13. Parsing JSON safely in Python

Even with JSON mode, treat parsing defensively — a stray model can still surprise you
(especially smaller ones).

Step 2 — Control the Output 10

 import json

raw = response["message"]["content"]
try:
data = [Link](raw)
except [Link]:
data = None # handle, don't crash

if data is not None and "city" in data:

print("Got city:", data["city"])

Two defensive habits worth keeping:

• Never assume a key exists — check, or use [Link]("city") .

• Never let a parse error crash the loop — catch it and decide what to do (retry, ask
again, or fall back).

Schema mode plus Pydantic's model_validate_json gives you validation and parsing in one
line, which is why it's the recommended path.

14. Retrying when the model slips

When parsing fails, the simplest robust pattern is: tell the model it produced invalid output
and ask again.

import json, ollama

def ask_json(messages, retries=2):

for _ in range(retries + 1):
r = [Link](model="qwen3:8b", messages=messages,
format="json", options={"temperature": 0})
raw = r["message"]["content"]
try:
return [Link](raw)
except [Link]:
[Link]({"role": "assistant", "content": raw})
[Link]({"role": "user",
"content": "That was not valid JSON. Reply with valid JSON only."})
raise ValueError("Model failed to produce valid JSON")

This "validate → if bad, give feedback → retry" loop is a baby version of reflection (Step 8).
You'll see the same shape again.

Step 2 — Control the Output 11

15. Putting it together: a multi-tool decision call
Here's the payoff — a call that returns the exact structure the agent loop will read in Steps 3–
4. The model is given a menu of tools and decides whether to act (and which tool) or finish, all
in parseable, schema-enforced JSON:

import ollama
from pydantic import BaseModel
from typing import Optional, Literal

class Decision(BaseModel):
action: Literal["use_tool", "final_answer"]
tool: Optional[Literal["get_weather", "web_search", "calculator"]] = None
args: Optional[dict] = None
answer: Optional[str] = None

SYSTEM = """You are an agent's decision step. You may use these tools:
- get_weather(city) : current weather for a city
- web_search(query) : search the web
- calculator(expr) : evaluate a math expression
Choose "use_tool" with the right tool+args when you need information,
or "final_answer" when you can answer directly."""

messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": "What's the weather in Dhaka?"},
]

r = [Link](model="qwen3:8b", messages=messages,
format=Decision.model_json_schema(),
options={"temperature": 0})

d = Decision.model_validate_json(r["message"]["content"])
if [Link] == "use_tool":
print("Call:", [Link], "with", [Link]) # → get_weather {'city': 'Dhaka'}
else:
print("Answer:", [Link])

Read that carefully — this is turn 2 of the loop, made readable. The Literal types mean
the model can only pick a real tool name and a real action. In Step 3 you'll connect [Link] to
an actual Python function; in Step 4 you'll wrap the whole thing in a while . Step 2's job was
to make d a thing your code can branch on with confidence. Done.

16. Choosing a model for reliable structure

Not all local models are equally good at structured output and tool decisions. Two things
matter more than raw size:

• Tool-calling / JSON reliability. Some models rarely drop fields or hallucinate keys; others
wobble. For agents, prefer ones known for stable structured output (e.g. qwen3:8b is a
strong, dependable choice on modest hardware; llama3.1:8b is a solid alternative).

Step 2 — Control the Output 12

 • Context length. Structured prompts plus few-shot examples plus history add up. Aim for
at least a 32K context window for multi-step agents.

If a small model keeps fumbling a schema, the fix is often "use schema mode," "add a few-shot
example," or "step up to a slightly larger model" — in that order of cost.

17. Prompt-design rules for reliable structure

• Be explicit and exact. "Reply ONLY with JSON of the form {...}. No prose, no markdown
fences." Vague instructions get vague structure.
• Show the shape. Include the literal field names and types in the prompt, even when using
schema mode.
• Add a few-shot example when the format is subtle.
• One job per call. A call that classifies and explains and formats is three chances to drift.
Split them.
• Put format rules in the system prompt, not the user prompt — they apply every turn
and resist being overridden.
• Use temperature 0 (and a seed when testing) for anything structured.
• Prefer schema mode when the shape matters (which, for agents, is always).
• Describe field meaning, not just type — the grammar enforces type; the prompt
supplies intent.

18. A debugging checklist for structured output

When a structured call misbehaves, walk this list top to bottom:

1. Print the raw output ( response["message"]["content"] ) before parsing — see what

actually came back.
2. Is format set? No format → expect prose and fences.
3. Is temperature 0? Random output → wobbling structure.
4. Does the prompt describe the exact fields? Add them if not.
5. Are you in JSON mode but need schema mode? Switch if the fields are wrong (not just
the JSON).
6. Add a few-shot example if the shape is close but subtly off.
7. Wrap parsing in try/except + retry (§14) so a rare slip doesn't crash the loop.
8. Still failing? Try a more reliable/larger model (§16).

Step 2 — Control the Output 13

19. Pitfalls and gotchas

Pitfall Why it happens Fix

Model wraps JSON in code It thinks it's writing for humans Use format ; say "no markdown"
fences

Output randomly varies run Temperature > 0 Set temperature: 0 (+ seed )

to run

KeyError when reading a Assumed a key that wasn't Use .get() , or schema mode to
field returned guarantee it

Crash on a bad parse No error handling Wrap [Link] in try/except; retry

Model adds chatty preamble No system constraint "Reply ONLY with JSON. No
explanation."

Right shape, wrong values Prompt didn't explain the fields Describe each field's meaning

Model invents a category Free-text field where you needed Use an enum / Literal
a set

Output gets cut off mid-JSON num_predict too low Raise the token cap

20. Quick-reference cheat sheet

import ollama, json

from pydantic import BaseModel
from typing import Literal, Optional

# 1. Deterministic structured call (schema mode) — the default for agents

class Out(BaseModel):
label: Literal["a", "b", "c"]
note: Optional[str] = None

r = [Link](
model="qwen3:8b",
messages=[
{"role": "system", "content": "Classify into a, b, or c. JSON only."},
{"role": "user", "content": "..."},
],
format=Out.model_json_schema(), # exact shape, enforced
options={"temperature": 0, "seed": 42}, # repeatable
)
out = Out.model_validate_json(r["message"]["content"]) # parse + validate

# Simpler: JSON mode (any shape) -> format="json"

# Cap length -> options={"num_predict": 256}
# Hard stops -> options={"stop": ["```", "\n\n"]}
# Reproducible test -> options={"seed": 42}

Step 2 — Control the Output 14

Defaults to internalize: temperature 0, schema mode, validate, retry on failure.

21. Glossary
• System prompt — standing instructions/role/format that apply every turn.
• User prompt — the specific request for the current turn.
• Few-shot prompting — including example input→output pairs to lock in a format or style.
• Temperature — how random the next-token pick is; 0 = deterministic.
• top_p / top_k — limit the candidate tokens considered when sampling.
• Seed — fixes the random draw for reproducible output.
• JSON mode — constrains output to valid JSON of any shape.
• Schema mode — constrains output to match a specific JSON schema.
• JSON schema — a description of an object's fields and types.
• Grammar / GBNF — the rule set the runtime uses to force valid structure at the token
level.
• Constrained decoding — masking out tokens that would break the required structure
during generation.
• Pydantic — a Python library for declaring schemas and validating data.
• Enum / Literal — a field restricted to a fixed set of allowed values.
• num_predict — max tokens to generate. stop — strings that end generation.

22. Exercises
1. Write a sentiment classifier returning {"label": enum, "confidence": 0..1} in schema
mode. Test five sentences.
2. Run the same structured prompt 5× at temperature: 0 (confirm identical), then at
temperature: 1 (watch it vary).

3. Extract {name, age, city} from a messy paragraph; add a few-shot example and see if
accuracy improves.
4. Design a nested schema: a Recipe with title , a list[Ingredient] , and an enum
difficulty . Force the model to fill it.

5. Add a Pydantic field_validator that rejects a confidence outside 0–1; feed it bad data and
confirm it raises.
6. Break it on purpose: ask for JSON without format , then add the retry wrapper from §14
and watch it recover.
7. Use stop and num_predict to clip a deliberately long generation.
8. Build the multi-tool decision call from §15 and feed it three questions; print which tool (if
any) each chose.
9. Swap qwen3:8b for another model and compare how reliably each obeys the schema.

Step 2 — Control the Output 15

10. Print the raw output and the parsed object side by side for one call, to see exactly what the
grammar produced.

23. Self-test (with answers)

Q1. Why must structured output come before tool calling? The loop's turn-2 decision
(act or finish) has to be read by your code. If you can't parse the decision, you can't act on it —
so structure precedes tools.

Q2. What's the difference between the system and user prompts? System = standing
instructions/role/format that apply every turn; user = the specific request for this turn.

Q3. What temperature should an agent's decision step use, and why? 0 (or very low) —
decisions should be deterministic and repeatable, not random.

Q4. Name the two ways to force JSON in Ollama and when to use each. JSON mode
( format="json" ) for any valid-JSON shape; schema mode ( format=<JSON schema> ) when the
exact fields/types matter — i.e. for agents.

Q5. How does the runtime guarantee valid JSON? It builds a grammar and masks out, at
each step, any token that would break the structure — constrained decoding. Form is
enforced; values still come from the prompt.

Q6. Why use an enum/ Literal for a field like priority ? To stop the model inventing
categories — the value is restricted to the allowed set.

Q7. What does Pydantic give you beyond [Link] ? Validation (not just parsing) and
typed objects, in one call ( model_validate_json ), with custom validators and field constraints.

Q8. Why parse defensively even in JSON mode? Smaller models can still slip; missing keys
or bad output shouldn't crash the loop. Use try/except, .get() , schema mode, and retries.

Q9. How does the retry pattern relate to a later step? It's a baby version of reflection
(Step 8): validate, give feedback, try again.

Q10. What's the recommended default recipe for an agent decision call? Temperature 0
(+ seed for tests), schema mode with a Pydantic model, then model_validate_json , wrapped in
try/except with a retry.

24. Done criteria → on to Step 3

You're done with Step 2 when:

• [ ] You can explain why structured output comes before tools.

• [ ] You can set a system prompt that controls the output's shape.
• [ ] You can force valid JSON (both modes) and parse/validate it safely.
• [ ] You can design a schema with an enum, a list, and an optional field.
• [ ] You can run a multi-tool decision call returning {action, tool, args, answer} .

Step 2 — Control the Output 16

 • [ ] You understand temperature 0 + seed for repeatable results, and why constrained
decoding works.

Next, Step 3 — Tool calling: you'll describe real functions to the model, it will say "call X
with these args" (in exactly the structured form you just learned to produce), and your code
will run the real function and feed the result back. That's the core skill of the whole field —
and you're now ready for it, because the model's decisions finally come back as data.

Step 2 of the "Build an AI Agent from Scratch" curriculum. The bridge from "the model talks"
to "my code can act on what it said."

Step 2 — Control the Output 17