Prompt Engineering
Agenda
• What is Prompt Engineering
• Basic Prompt Engineering Elements
• Prompt Engineering Techniques
Customize LLMs
Data Base LLM GenAI Applications
• Text (Foundation Model)
• Images
• Speech
• Structured Pre-training Model Adaptation
Data
Retrieval Agents with
Supervised
Prompt Augmented Tools and
Fine-tuning
Engineering Generation Workflow
Improve how we use the model Improve the model
Prompt Engineering
Task Description
Query
Examples
Format
User … LLM
Prompt
Response
Prompt Engineering
• Prompts do not require a specific format.
• With today's language model capabilities, you just need to clearly
describe the task you need
Online New Assistant
Do not know things Basic knowledge
that are specific to and cognitive skills
you
Basic Prompt Elements
Basic Prompt Elements
• Instruction - a specific task or instruction you want the model to perform
• Context - external information or additional context that can steer the
model to better responses
• Input Data - the input or question for that we are interested in finding a
response
• Output Indicator - the type or format of the output.
Source: [Link]
An Example: Image Annotation
• A research project: annotating Airbnb host
images
• What we need:
1) Is there a person in the image?
2) Are there other people or pets?
3) What proportion does the person's face occupy in
the image?
4) What is the gender of the person?
5) Is the person smiling?
• How do we do prompt engineering (PE)?
PE Input & Output
Prompt Output
Typical PE Elements
• Example
• Input
• Output
• Role play
• Task description
• Context
Prompt Source: [Link]
Typical PE Elements
• Example
• Input
• Output
• Role play
• Task description
• Context
Typical PE Elements
• Example
• Input
• Output
• Role play
• Task description
• Context
Where Do These Work?
Prompt Techniques
Zero-Shot Prompting
Definition: Zero-shot prompting asks a language model to perform a
task without any task-specific examples.
Relies on: Efficient, scalable, & baseline
➢ Pretraining on large, diverse corpora Sensitive to prompt wording;
low performance on domain-specific tasks
➢ Natural-language task instructions
Common tasks: Classification, summarization, translation, reasoning
Example:
• Classify the text into neutral, negative or positive. [INSTRUCTIONS/TASKS]
• Text: I think the food was okay. [INPUT DATA]
• Sentiment: [OUTPUT INDICATOR]
Source: [Link]
Few-Shot Prompting
Definition: Few-shot prompting provides a language model with
a small number of task-specific examples within the
prompt to guide its behavior.
Those examples typically include:
➢ Input-output pairs
Few-Shot Prompt
An example:
• Classify the text into neutral, negative or positive.
[INSTRUCTIONS/TASKS]
• Example: The food was awesome. [CONTEXT]
• Sentiment: positive [CONTEXT] In-context learning
• Text: I think the vacation is okay. [INPUT DATA]
• Sentiment: [OUTPUT INDICATOR]
Source: [Link]
Few-Shot Prompting
Definition: Few-shot prompting provides a language model with
a small number of task-specific examples within the
prompt to guide its behavior.
Those examples typically include:
➢ Input-output pairs
➢ Demonstrations of reasoning and fine-tuning
Higher accuracy, & serve users’ intent
Consume prompt tokens;
performance sensitive to examples
Few-Shot Prompt with Random Formats
Few-Shot Prompt with Random Labels
Do LLMs really
learn from the
examples?
Larger models show
stronger tendency to
imitate demonstrations!
Larger language models do in-context
learning differently
Source: [Link]
Do LLMs really
learn from the
examples?
Do LLMs really
learn from the
examples?
Zero-shot prompting asks the model to answer directly;
Few-shot prompting shows the model what to do
through examples.
Zero-shot = What to do;
Few-shot = What good answers look like.
LLMs Settings and Roles
Image Source: [Link]
Key Parameters for Controlling LLM Behavior
• Temperature: controls randomness (i.e., accuracy vs.
creativity)
• Top P
• Max Length
• Stop Sequences
• Frequency Penalty
• Presence Penalty
System and User Level Prompts
• You are a helpful assistant who reviews and classifies customer feedback
SYSTEM PROMPT
into consumer sentiment. [PERSONA]
• Classify the text into neutral, negative or positive.
[INSTRUCTIONS/TASKS]
• Example: The food was awesome. [CONTEXT]
• Sentiment: positive [CONTEXT]
• Text: I think the vacation is okay. [INPUT DATA]
USER
• Sentiment: [OUTPUT INDICATOR]
Recap
• What is Prompt Engineering
• Prompt Engineering Elements
• Prompt Engineering Techniques