Prompt Engineering
Complete Training Course
LLM Evaluation & AI Annotation
Your Mission in This Evaluation
• Create challenging prompts from set categories
• Lead realistic, thoughtful multi-turn conversations
• Review responses for reasoning, accuracy, and clarity
• Push the model to reveal strengths and weaknesses
You are not expected to formally score or grade responses, but you are expected to
understand what a strong or weak response looks like and push the model accordingly.
Key Terminology
A type of AI system trained on large amounts of text to learn language
LLMs patterns, enabling them to answer questions, write, summarize, and explain
ideas naturally.
Prompt Your input that starts or continues the conversation with the model.
Hallucination A false or misleading response generated by the model.
Constraint A requirement for a prompt, such as word count, tone, structure, or format.
A back-and-forth exchange where each message builds on previous ones,
Multi-Turn
using context for clarifications, problem solving, or brainstorming.
Extreme A long, complex dialogue across multiple topics or goals, requiring the model
Multi-Turn to track details, adapt, and stay consistent.
What Makes a Good Prompt?
Require Multi-Step Reasoning or Synthesis
1 Ask questions that need layered thinking, not simple one-line answers.
Reveal Gaps in Logic, Factuality, or Coherence
2 Push the model to show where it might make mistakes or oversimplify.
Include at Least One Constraint
3 Guide the response's format, tone, or focus.
Types of Constraints
Length
Set any type of limiter for the response — word count, number of sentences, or paragraph limit.
Examples:
• "Respond in under 200 words"
• "Summarize in 3 paragraphs of 4 sentences each"
Format
Define the structure or layout (lists, tables, JSON, outlines) to ensure easy-to-read output.
Examples:
• "Present your answer as bullet points"
• "Display the data in a two-column table with headers"
Style
Define the tone, voice, or personality the model should use when writing the response.
Examples:
• "Use a formal academic tone"
• "Explain as if speaking to a beginner with no prior knowledge"
Content
Set requirements for the specific information the response should include — or exclude.
Examples:
• "Include at least two examples from Southeast Asia"
• "Focus only on events from the 20th century"
Writing Realistic & Challenging Prompts
Think Like an Expert: Ask the kind of question a specialist would spend time researching.
Avoid Easy Wins: Skip simple factual lookups — good prompts require reasoning, synthesis, or
structured responses.
Aim for Clarity: The answer you expect should be clear enough that experts could agree on it.
Example — Good vs Poor Prompt
■ Good Prompt ■ Poor Prompt
"Explain how renewable energy policies in Germany "Tell me about energy."
have evolved over the last 10 years. Present your
answer as a timeline in bullet points."
Requires synthesis of policy changes over time. Too vague, no clear scope or constraints.
Constraint: Timeline format + bullet points.
3 Steps to Build a Strong Prompt
Step 1
Write Realistic & Challenging Prompts
Think like an expert. Avoid easy wins — good prompts require reasoning, synthesis, or structured
responses. Aim for clarity so experts could agree on the expected answer.
Step 2
Define the Scope Clearly
Set boundaries so the model doesn't go off track. Stay focused: if your question could lead in
many directions, narrow it to one.
■ "Describe three environmental impacts of plastic waste in oceans, focusing only on Southeast
Asia."
■ "Tell me about plastic waste."
Step 3
Target Key Reasoning Skills
Test the model's ability in at least one of these areas:
• Complex Reasoning: combining multiple ideas or logical steps.
• Contextual Understanding: interpreting tone, nuance, or cultural context.
Prompt Structure
Once your prompt has strong content, it also needs a clear structure — easy to read and simple for
the model to follow.
• Aim for 80–100 words total
• Use 2–3 short paragraphs for clarity
• Keep context and instructions separate
• Say exactly what you want — no guesswork and avoid long, complicated sentences
Challenge Types
A good prompt should focus on at least one of these challenges and apply structure, detail, or
complexity to make it meaningful. Choose one as the main focus but allow others to emerge naturally.
Factual Recall
Tests if the model can retrieve and present accurate, up-to-date information.
Example: "List the top 5 renewable energy companies in the U.S. and their most recent annual revenue."
Multi-Step Reasoning
Checks if the model can solve problems that require layered logic or calculations.
Example: "If a train travels 400 km at 80 km/h and makes three 10-minute stops, how long is the journey?"
Summarization & Abstraction
Evaluates if the model can distill complex or lengthy information into clear takeaways.
Example: "Summarize the key arguments in this legal document regarding data privacy regulations."
Structured Generation
Measures the ability to produce responses in a specific format or tone.
Example: "Write a structured report analyzing the impact of AI on the job market."
Comparative Analysis
Tests if the model can identify key differences or similarities between concepts.
Example: "Compare the economic recovery strategies of the U.S. and the EU post-2020."
Context Awareness
Checks if the model understands and adapts to evolving context across multiple turns.
Example: "Interpret the impact of recent interest rate hikes on small businesses, considering historical
trends."
Combined Abilities
Evaluates the model's ability to handle multiple challenge types at the same time.
Example: "Using current inflation rates, calculate how much a $10,000 investment will grow in five years, and
explain the economic context."
Multi-Turn Conversation — Example
Turn 1 — Opening
"Summarize the causes of the French Revolution in under 100 words."
■ Good opening: clear, has a constraint, requires reasoning.
Turn 2 — Build Complexity
"Now compare those causes to the causes of the American Revolution, using a side-by-side table."
■ Keeps the topic, increases complexity, adds a format constraint.
Turn 3 — Deep Reasoning
"Explain in one paragraph which revolution had the bigger long-term global impact, and why."
■ Pushes the model into deeper reasoning and synthesis.
Evaluating Responses — What to Look For
Factuality & Completeness
1 Ensure the response fully addresses the question, providing accurate, verifiable, and
reliable information. Avoid vague, partial answers or hallucinated/incorrect details.
Logical Flow
2 Check that the response follows a clear, logical flow of thought, without contradictions or
confusing points.
Context Awareness
3 Responses should build on the ongoing conversation and align with prior context. The
response does not ignore or misinterpret previous information.
Relevance
4 The response should stay on-topic and focused on the prompt. Avoid responses that stray
off-topic or include irrelevant filler.
Spotting Hallucinations
Stay alert and keep an eye out for model errors, commonly known as hallucinations. These can be
tricky, but spotting them is key!
Look out for responses that:
✓ Mention facts, names, or events you can't confirm — if you can't find them in a trusted source, they
may be made up.
✓ Give outdated details — check if the information is old, especially for fast-changing topics like laws,
technology, or recent events.
✓ Sound convincing but conflict with reliable sources — even if the writing sounds confident, verify it.
✓ Contradict earlier statements — if the model says something different from what it said before without
explaining the change, it's suspicious.
Pro Tip: Search reputable sources to confirm unusual facts. Compare the response with earlier
conversation turns for consistency. For dates, names, or numbers — verify them directly.