Introduction to Applied
Machine Learning
11: Chain of Thought, In-
Context Learning, and
Retrieval-Augmented
Generation
(or intuitively, what can we do without changing model parameters?)
Prompting a Model
Prompting a Model
For a causal autoregressive language
model, the “easiest” prompts end where
you want the model to begin generating.
Prompt Engineering is the process of designing an
effective prompt for a given task.
One instruction-tuned model can perform many different
tasks with different prompts. Rather than fine-tuning
parameters, you are searching in the space of contexts on
which to condition.
Examples of Different Prompt
Results
System Prompts Recall that causal masking only
allows attending to prior tokens.
Standard prompts end where the causal language
model should begin generating:
{input} Rewritten with improved clarity and grammar:
System prompts are prepended to the context,
allowing the representation of {input} to be
contextualized with attention to the system prompt.
You are a helpful writing assistant. Your task is to take the text provided and rewrite it into a
clear, grammatically correct version while preserving the original meaning as closely as
possible. Correct any spelling mistakes, punctuation errors, verb tense issues, word choice
problems, and other grammatical mistakes.
{input} Rewritten with improved clarity and grammar:
Prompting vs. Fine-Tuning
Prompting Fine-Tuning
No need for curated fine- Need to curate good fine-
tuning data tuning dataset
No need for gradient updates Expensive gradient updates to
to model parameters and can model parameters and need
run on closed-weight models access to model weights
Highly sensitive to wording No ad-hoc tuning of prompt
choices, ad-hoc exploration wording
May not yield the best
May yield better performance
performance
Long prompts add inference-
No inference-time overhead
time overhead
In-Context Learning
In-Context Learning: In which a model “learns”
new information, or to complete a new task, purely
by processing in-context – the input sequence to
the transformer.
Zero-Shot, One-Shot, Few-Shot
Zero Shot. No
demonstrations, just
instructions.
One Shot. 1 single
demonstration in
the prompt.
Gives 3 examples of Few Shot. 2 or more
reading comprehension demonstrations in
completions. the prompt.
Chain-of-Thought Prompting
For formal reasoning tasks, demonstrations that show the thought
process outperform those that simply give the answer.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought
prompting elicits reasoning in large language models. Advances in neural information processing systems.
Hallucinations
A hallucination is any LLM generated text that is
factually/objectively untrue.
Example Prompt: Who was the first person to walk on the moon?
Example Hallucination: Yuri Gagarin was the first person to walk on
the moon in 1961.
Fabricating false information (such as false citations) is
another common example.
In the real world: In 2023 a practicing attorney was
caught citing 6 fake legal cases that were suggested by
ChatGPT. AP News Article
Hallucinations Persist Despite
Alignment Efforts
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Lowe, R. (2022). Training language
models to follow instructions with human feedback. Advances in neural information processing systems
LLMs are Poorly Calibrated
Model confidence does not correspond to model accuracy
– LLMs may not “know” what they do not “know.”
“Across all generations, only 53% of
generations with expressions of certainty
are correct (random accuracy being 25%).”
Relying on the Unreliable: The Impact of Language Models’
Reluctance to Express Uncertainty Kaitlyn Zhou, Jena D. Hwang,
Xiang Ren, Maarten Sap
Why: Retrieval Augmented
Generation (RAG)
How can we incorporate:
• Domain-specific factual knowledge,
• In a way that a causal LM directly references,
• To improve task-specific performance,
• And reduce hallucinations?
Information retrieval provided in-context to
augment generation.
How: Retrieval Augmented
Generation (RAG)
Automatically search a database for relevant
information and provide to the model in-context.
Prepending
documents allows
query to be
contextualized with
attention to
documents.
Key Advantages of RAG
1. Reduce hallucination by providing curated
and material directly in-context.
2. Can provide appropriate and relevant
citations to improve trust and verifiability.
3. Dynamic information: Can incorporate new
information post model-training cutoff.
Information Retrieval
• Traditional approaches look for matches on
keywords using metrics like tf-idf (term frequency –
inverse document frequency).
• But these approaches do not handle synonyms,
word order, or contextual understanding.
RAG Implementation with
Dense Bi-Encoder Embeddings
In practice, the most common approach is to compute
dense embeddings (numerical vectors) of both the query
(prompt) and document (candidate for retrieval) using a
pretrained encoder Transformer like BERT.
Select the highest scoring
documents to add to the
the RAG prompt.
Complexity of RAG with Dense
Bi-Encoder Approach
𝒛𝑑 can be precomputed for all documents 𝑑 in your
database ahead of time.
At inference time with a query (prompt) 𝑞:
1. Compute 𝒛𝑞 (forward propagation in encoder)
2. Compute score(𝑞, 𝑑) for all documents 𝑑
3. Select highest scoring documents and pass to
RAG prompt
Linear scaling with
Only done once Can compute all
number of documents
before generation scores in parallel
in database
Less Common Alternative:
Unified Encoder
Better performance jointly encoding query and
document to predict a score.
But impractical for large document collections.
Requires a separate forward pass for each.
Web-Grounded Generation
Increasingly common to use existing internet search engines as a
document retrieval tool to augment (or “ground”) generation.
[Link]/gemini-api/docs/google-search (retrieved 9/30/2025)
“AI-Overview” in Search as Web-
Grounded Generation Systems